When MLOps Is an Organizational and Communication Problem - Not a Tech Problem -...

Learn more

Might be useful

• CI/CD Framework – Continuous Integration and Continuous Delivery is the core framework of both software and machine learning solutions since improvements are continuously suggested and added to the existing code.
• Agile Methodology – The agile methodology is designed to make the process of continuous integration and improvement less cumbersome by dividing the end-to-end process into multiple sprints. The benefit of using sprints is that the result of every sprint has high transparency and is distributed across all teams and stakeholders involved in the project. This ensures that there are no last-minute surprises during the later stages of solution development. Overall, the agile methodology solves the following problems:
  • Lengthy development-to-deployment cycle – The primary delay occurs when the development team builds an entire solution and hands it over to the deployment team as a whole. There is a wide gap in this transfer since a lot of dependencies are often left unresolved. With the help of agile methodology, the hand-offs are sprint-wise and the dependencies are cleared off from the first few stages of the development itself.
  • Lack of sync between teams – Every data science project does not just include a development and deployment team. A lot of stakeholders like project managers, customer representatives, and other decision-makers are also involved. Without the implementation of MLOps, the visibility of an ML solution gets reduced manifold and the stakeholders only get a taste of it towards the later stages of development. This has the potential to mess up the entire project since business-oriented teams are known to provide expert advice that can turn around the whole solution. Through agile methodology, their inputs can be taken right from the start and considered across all sprints.

Read also

1. Use case identification: This is when the development and business teams collaborate with the client to understand and formulate the right use case for the problem at hand. This might involve the preparation of plans and software documentation, and getting them reviewed and approved from the client’s end.
2. Data engineering and processing: This is the first technical touchpoint where data engineers and data scientists look at the various sources from where data can be derived for solution development. Identifying the source is followed by building the pipeline that can streamline the data in the correct format and also check the data sanity in the process. This requires close collaboration between data scientists and data engineers, and also a few nods from different teams like the consultancy team who might be using the same data master.
3. ML solution building: This is the most intense component of the MLOps life cycle where the solution is both developed and deployed in continual cycles with the help of a CI/CD framework. There are multiple experiments involved in an ML solution with multiple versions of the model and data. Every version must therefore be well tracked such that one can easily refer to the optimal experiment and model version at later stages. Development teams often do not track these experiments with a standard tool and end up jumbling the process, increasing the work and time requirement and in all probability, reducing the optimal model performance. Every module that is successfully created should be deployed iteratively while leaving ample space for changing the core elements such as hyperparameters and model files.
4. Production deployment: Once the ML solution is built, tested, and deployed in local environments, it should be connected to and deployed on the production server which can be on public cloud, on-premise, or hybrid. This is followed by a short test that checks if the solution is working on the production server, which it usually does since the same solution has been tested previously in the local server. However, sometimes due to a mismatch in configurations, the test fails and the developers must promptly pinpoint the reason and find a fix for it.
5. Monitoring: After the solution is successfully deployed on the production server, it must be tested on a few batches of new data. Depending on the data type, the crucial time for monitoring can range from a few days to a few weeks. Some specific and pre-decided target metrics are used to identify if the model keeps serving the purpose. Triggers are used for prompt alerts when the performance is below expectations and calls for optimization by going back to the third stage of solution building.

Read also

• Long-chain of approvals – For every change that needs to be reflected on the production server, approval must be obtained from the relevant authorities. This takes a long time since the verification process is lengthy, ultimately delaying the development and deployment plans. This problem is not just about production servers, however, but also exists in terms of provisioning different company resources or integrating external solutions.
• Provisioning within budget – Sometimes, the development teams cannot use the company’s resources because of budget limitations or because the resource is shared across multiple teams. Sometimes, new resources are also required for ML solutions specifically. For instance, resources with high-powered computing or huge storage capacity. Such resources, even though vital for scaling ML solutions, fall out of most organization’s budgeting criteria. In that case, ML teams have to find a workaround that will often be suboptimal, to make the solution work with the same vigor if possible.
• ML stack that differs from the homegrown framework – The software deployment framework that most companies have been working on might be suboptimal or even irrelevant for deploying ML solutions. For instance, a python based ML solution might have to be deployed through a Java-based framework just to comply with the company’s existing system. This can create double work for the development and deployment teams since they have to replicate most of the codebase which is taxation on both resource and time. 
• Security go-ahead –  Every machine learning model or module is almost always part of a big repository. For example, a version of the random forest model is coded in the scikit-learn library, and users can simply use a function call to kickstart the model training. Most often, users are unaware of what the code is doing and which sources other than the data it is tapping into or sending out. Therefore, to ensure security, the code of encapsulated functions needs to be checked by the relevant security teams before the code is uploaded to the production server. Furthermore, on-cloud security APIs, the company’s security infrastructure, and licensing factors are time-intensive and need to be taken care of.
• Sync across all teams concerning data – A common data master obtained from a customer is used by multiple teams like sales teams, consultancy teams, and of course, the development teams. To avoid discrepancies in the data, a common set of column definitions and a common data source must be maintained. Even though this sounds doable, in real-world scenarios, due to the involvement of multiple contributors, this often becomes a challenge leading to a lot of back and forth between teams to resolve internally created data issues. An example of such an issue could be that after building the entire solution, the developer realizes that one of the columns is being populated differently by the customer onboarding team.

Learn more

• Virtual environment for ML stacks – For new companies and startups who have been implementing the agile methodology and have adopted most of the relevant frameworks, a backdated homegrown framework is not a problem. However, companies that are relatively old and function uniformly on a previously built framework, might not see the best results from ML teams in terms of resource optimization. This is because the teams will be busy figuring out how to best deploy their solution through the available framework. Once figured out, they have to repeat the suboptimal process for every solution that they want to deploy.
  A long-term fix for this is to invest resources and time into creating a separate ML stack that can integrate into the company framework, but also reduce work on the development front. A quick fix for this is to leverage virtual environments to deploy ML solutions for the end customer. Services such as Docker and Kubernetes are extremely useful in such cases. 
• Cost-benefit analysis – To reduce the long line of approvals and budget constraints, the Data Science development teams often need to delve into the business side and do a thorough cost-benefit analysis of limiting provisions vs the return on investment from working Data Science solutions that can run on those provisions. The teams might need to collaborate with the business operations team to get accurate feedback and data. The key decision-makers in organizations have either a short-term or a long-term profit-oriented view, and a cost-benefit analysis that promises growth can be the driving factor that opens up some of the bottlenecks. 
• Common key reference – To maintain transparency between multiple teams who operate on the master data from the customer, a reference key can be used since a common data source is not practical in situations where the dev teams have to duplicate and download the data multiple times. The common key for column definitions, might not be the ideal solution to avoid data discrepancy, but it is an improvement on the existing process where a data column can be easily misinterpreted, especially when the column names are non-explanatory.
• Using verified code sources: To shorten the time taken for a security check while uploading or approving machine learning libraries on production servers, the developers can restrict their code references to verified codebases such as TensorFlow and scikit-learn. Contrib libraries, if used, must be thoroughly checked by the developer to verify the input and output points. This is because, in case there is some security concern, the cycle of re-development and security check restarts and might insanely slow down the process.

• Hyperparameter versioning – Every ML model has to be tested with multiple sets of hyperparameter combinations before the optimal solution is found. However, that is not the primary challenge. Change in incoming data can dip the performance of the chosen combination and the hyperparameters must be re-tweaked. Even though the code and hyperparameters are controlled by the developers, the data is the independent factor that influences the controlled elements. It must be ensured that every version of the data and hyperparameters are tracked so that the optimal result can be found and reproduced with minimum hassle.
• Multiple experiments – With changing data, a lot of code also has to be modified across data processing, feature engineering, and model optimization. Such iterations cannot be planned and add to the time required in building the ML solution, impacting all the following stages and planned timelines.
• Testing and Validation – The solution must be tested and validated across multiple unseen datasets to understand if the solution can work in the production environment with incoming data. The challenge here is that the trained model may be skewed towards the training data set. To avoid this, testing and validation must be done in iterations during model creation itself. Given that the number of modeling iterations are not fixed, the time taken for testing with unseen datasets keeps increasing with the number of iterations.
• Post-deployment monitoring – The primary challenge here is that new data may not be consistent with historical data patterns. For instance, during the pandemic, the stock market curves defied all historical patterns and went down in a way a machine learning solution could not have predicted. Such external factors often come into play and the data must be collected, updated, and processed accordingly to maintain the recency of the solution.

Check also