Anyone familiar with Data Analytics/Big Data world may be aware of the 3 V’s of Big Data. They are Volume, Velocity and Variety (there is a 4th V called Veracity). With these principles, Big Data Analytics forms the bedrock of modern Information Systems. They form the first principles of the discipline of Data Engineering. However, another engineering disciple has emerged in the world of Data and AI. It is called Machine Learning Engineering. It is about operationalization of ML Pipelines i.e. to train and deploy them in production. The process of operationalization is called as MLOps.
Like Data Engineering, ML Engineering/MLOps too have their own set of V’s. However, they differ since, they serve different purposes. The 3 V’s of ML Engineering are Velocity, Validation and Versioning. However, before diving into them, let’s first understand the common tasks of ML/MLOps Engineering.
Common tasks in ML Engineering/MLOps
In our introductory article on Machine Learning System Design, we elaborated on the various sub-systems of an ML Workflow. Having said that, it was strategic, applicable to architects/designers. Nonetheless, day to day tactical tasks for ML Engineers/MLOps Engineers revolve around the four aspects:
- Data Collection : Collecting and maintaining datasets for Training/Serving. Data collection spans sourcing new data, wrangling data from sources into a centralized repository, and cleaning data.
- Experimentation : Experimentation for continuous improvement of Models. ML engineers typically focus on improving ML performance, measured via metrics such as accuracy or mean-squared-error. Experiments can be data-driven or model-driven; for example, an engineer can create a new feature or change the model architecture from tree-based to neural network-based.
- Evaluation and Deployment : Evaluation and Deployment of ML Models. A model is typically evaluated by computing a metric (e.g., accuracy) over a collection of labeled data points hidden at training time, or a validation dataset, to see if its performance is better than what the currently-running production model achieved during its evaluation phase. Deployment involves reviewing the proposed change, possibly staging the change to increasing percentages of the population, or A/B testing on live users, and keeping records of any change to production in case of a necessary rollback.
- Monitoring and Response : Monitoring ML pipelines and responding to bugs involve tracking live metrics (via queries or dashboards), slicing and dicing sub-populations to investigate prediction quality, patching the model with non-ML heuristics for known failure modes, and finding in-the-wild failures and adding them to the evaluation set.
Across the above tasks, Velocity, Validation and Versioning is applicable in a variety of ways. Let’s dive slightly deeper into each of them.
Velocity
Since ML is so experimental in nature, it’s important to be able to prototype and iterate on ideas quickly (e.g. go from a
new idea to a trained model in a day). This is where easy access to quality data, efficient feature retrieval with Feature Stores and Orchestrated Experiments help.
Validation
ML Pipelines are a piece of software. Hence, all the standards applicable to standard software engineering are also applicable to ML Engineering or MLOps; including efficient validation. Since errors become more expensive to handle when users see them, it’s good to test changes, prune bad ideas, and proactively monitor pipelines for bugs as early as possible. This is usually carried out in Evaluation and Deployment stage, right before deploying the model to the target environment. For faster esponse, it’s advisable to integrate validation steps in ML Pipelines.
To know more, read our article on Data and Model Validation in ML Training Pipelines.
Versioning
Versioning in MLOps is more complicated than traditional software engineering. Here, we not only version code and dependencies, but also data, models and experiments. Versioning helps not only in efficient processes and documentation, but also proves to be a boon when things go wrong.
Since it’s impossible to anticipate all bugs before they occur, it’s helpful to store and manage multiple versions of production models and datasets for querying, debugging, and minimizing production pipeline downtime. ML engineers responded to buggy models in production by switching the model to a simpler, historical, or retrained version. Thus, Versioning augments both Velocity and Validation.
Conclusion
The 3 V’s are interlinked and enhance each other, as seen in the article. For instance, efficient versioning of data, models and experiments come in handy, thus increasing the velocity of new releases. Better velocity and versioning helps find bugs quicker, thus enhancing validation. Lastly, better validation techniques mean faster response to incidents, in turn affecting velocity. Hence, overall, they enhance the reliability of ML models.
We have borrowed ideas from the below Reference. However, the thoughts mentioned in the article, are author’s own. Finally, this is an article for information purposes. We do not claim any guarantees regarding its accuracy or completeness.
Reference: Ideas like Common Tasks in ML Engineering and 3V’s of MLOps have been borrowed from this paper: Operationalizing Machine Learning: An Interview Study
