It’s done!  Your predictive model is live, and you are already reaping the benefits. Whether your costs have been reduced, your revenue has gone up, or your profit margin has increased, so far, your model has been helpful to your business. So you’re thinking: It’s working, let’s forget about this model and move on to my next business problem. In our experience, this is not the best approach. Predictive models (statistical and machine learning) are based on assumptions and historical data. Even under normal circumstances, business conditions evolve over the long run, leading to changes in the underlying data. This is known as concept drift.

Concept drift can lead to a general erosion in performance in predictive models. Therefore, even under normal circumstances, we advise our customers to use safe-guards, include a human being in the loop, and check and update predictive models on an annual or semi-annual basis.

Even under the best of circumstances, when you review your models, you often realize that some of the assumptions behind these models are no longer valid.

This is because your models are based on a business-as-usual assumption where the environment is stable in the short term, and the historical data used to train the model is mostly representative of the real-life environment.

When either or both assumptions are no longer valid, the models built on those assumptions need to evolve.

What happens, however, when the world is turned upside down by a “black swan” event such as the Covid-19 pandemic we’re living through now?

In a situation like this one, and more than ever, it is critical that your business not accept the results of legacy models without applying a critical eye. You must ensure that your models are still accurate and are adapted to the current business environment.

What can happen to predictive models in the case of a sudden shock like COVID-19?

Depending on your business and how your predictive models are used, there are several possible answers.

Let’s review three possibilities that could be relevant to your business.

1. Your model assumptions and training data weren’t affected by the event

The first is when the assumptions and training data aren’t affected by the shock. In this case, you want to do a sanity check to confirm that your models are still performing appropriately.

After that, you can continue to use these models without any significant changes.

2. The model continues to perform, but only for part of the business

The second case is when your models still perform well for a portion of your business, but no longer fit another part.

For instance, consider a store that sells a broad array of products.  If some of those products are currently experiencing supply chain shortages, then a revenue forecast based on a healthy and robust supply chain will no longer be accurate for those products.

Products with normal supply levels, however, may still be a good fit for the revenue forecast model.  In this case, the model may need to be modified or selectively weighted to account for the partial mismatch.

3. Your training data and model assumptions no longer represent your business situation

The last scenario is when your models appear to be completely broken. This can happen because your current business situation is no longer well represented by the historical data, and the original model assumptions are not valid anymore.

Example scenario

Let’s look at an example to illustrate the concept. Let’s imagine that you manufacture a successful ice cream brand. Let’s also assume that you have built your demand forecast model based on historical demand, and you have found that seasonality is an essential factor in the forecast.

In the spring and summer, people buy a lot of ice cream.  In the fall and winter, demand is relatively low.  It seems like a straightforward and logical hypothesis for an ice cream business. After all, who doesn’t like ice cream during the warm summer months?

Let’s now imagine that your business has two distribution channels. The first channel is through retail stores where your ice cream is sold at local supermarkets. The second channel is through ice cream shops and restaurants.

As one might guess, sales through the retail channel will probably continue to be seasonal despite the quarantines. Purchasing patterns may change to a certain extent. For instance, consumers may buy bigger sizes, quarts vs. pint-sized ice cream, because they’re confined at home and don’t want to shop as often.  However, overall, people continue to buy more ice cream in the spring and summer than in fall and winter.

But if you look at your second distribution channel, where all the restaurants and shops closed, the model is incapable of predicting that those sales have dried up due to the pandemic.

For this second channel, you want to avoid basing your plans on your existing demand forecasting model because one of the key hypotheses that customers purchase more ice cream at restaurants and shops in the summer is no longer valid, at least not in the middle of a pandemic.

Conclusion

We are living through exceptional times. The underlying assumptions of the predictive models your business relies on are being tested and may no longer be valid.  Now, more than ever, you must examine your models and question the assumptions and training data upon which they were built.

To move forward, you need to sit with your data science team, compare the data and assumptions to the current business environment, and then decide what to do.  You can continue to use your models as is, use them for a part of their original scope, refine them (if you have enough data and understanding of the new business dynamics), or even pause them.

Even in more “normal” times, if there are even such things as normal times anymore, model drift is a fact of (ML and AI) life and one-off fixes won’t scale. To solve those issues and ensure your ML and AI models remain fresh and operational, the best approach is to leverage the concept of Machine Learning Operations, aka MLOps. MLOps processes and tools will allow your to programmatically manage your ML and AI models and ensure scale them effectively.

It may be challenging to get started with your MLOps journey, however. Neal Analytics has data science and AI experts that can help you through this journey. We offer assessment workshops to help you navigate the situation and make the best decision about your existing models and the best approach to manage them moving forward. Want to learn more?  Contact us today at contactus@nealanalytics.com

This updated article was initially published here and on LinkedIn on 5/12/2020.