How to avoid AI model drift with monitoring and management

Chapter 04

5 min read

AI models may start out strong and seem like they will produce valuable outputs, but if not properly monitored, even the most well-trained, unbiased AI model can “drift” from its original parameters and produce unwanted results once deployed. If an AI model’s training doesn’t align with incoming data, it can’t accurately interpret that data or use that data to reliably predict outcomes. If drift isn’t detected and mitigated quickly, it will only digress further, increasing the negative business impact. To ensure accurate AI throughout its lifecycle, you need to look at model drift as another key consideration in your overall strategy.

61% of businesses strategically scaling AI depend on large, accurate data sets. ¹

There are a few best practices when it comes to monitoring drift. First, it’s important to understand that model monitoring can require specific tools and skill sets. Having the right tools and data scientists in place is crucial. Next, you need to actively monitor all models that are in production from a central place. Having a centralized, holistic view can help break down silos and provide more transparency across the entire data lineage.

It’s also important to establish a consistent set of metrics for assessing the health of your AI models. Learning from what’s working and not working will allow your team to pivot and correct when needed. AI models also need to be monitored on an ongoing basis and not just viewed as one snapshot in time. A model’s health could change over time and cause greater drift in the future if those changes are not identified. Lastly, it is helpful to automate as much of the monitoring process as possible to scale across your organization. Automation can provide consistent and reliable notifications and provide more time for your teams to focus on model development instead of monitoring.

So, what if you have model drift? How can it be corrected?

How to correct drift in three steps

1. Estimate the impact. If you’re able to estimate the impact of model drift, you can better determine what to prioritize and what resources you need for repair.

2. Analyze the root cause of the drift. This is where a time-based analysis is helpful to see how drift numbers evolved and when. For example, if you run checks weekly, you can see how drift evolved each day. Analyzing timelines can also be helpful to determine if the drift was gradual or abrupt.

3. Resolve the drift issue. This involves retraining your model on a new training dataset that has more recent and relevant samples added to it. Ultimately, the goal is to get your models back into production quickly and correctly. If retraining the model doesn’t resolve the issue, then a new model may need to be built.²

An integrated approach can help your business track metrics continually and alert you to drift in accuracy and data consistency. You can also set targets and track them through development, validation and deployment.

An integrated data and AI platform also simplifies the steps it takes to identify business metrics that are affected by model drift. With its help, you’ll be able to minimize the impact of model degradation through automated drift monitoring.³

What actions can businesses take to help protect against model drift?

Choose your answer

Establish a regular cadence to re-train live models

Monitor for model results that deviate out of range

Compare deployed models with challenger models

All of the above

Protecting against model drift takes a holistic approach that includes active monitoring, validating outcomes by use of challenger models and re-training models periodically.

¹ The State of AI in 2020, McKinsey & Company, November 2020.

² Model Monitoring Best Practices: Maintaining Data Science at Scale (PDF, 1 MB ), Domino, August 2020.

³ Model Drift, IBM, June 2021.