Drift detection overview
Drift is the degradation of predictive performance over time. Watson OpenScale detects and highlights drift so that you can prevent errors in your model.
The Watson OpenScale drift monitor detects the drop in accuracy and the drop in data consistency in your model. A drop in either model accuracy or data consistency can lead to a negative impact on your business outcomes.
How it works
To identify drift, the Watson OpenScale drift monitor analyzes the behavior of your model and builds its own model to predict whether your model generates an accurate prediction for a data point. The drift detection model processes the payload data to identify the number of records that your model makes inaccurate predictions for and generate the predicted accuracy of your model. The predicted accuracy is compared to the base accuracy of your model during training to identify the drop in accuracy. Also, the Watson OpenScale identifies the drop in data consistency by analyzing the training data and extracting characteristics to compare to your model at run time.
For analysis, you can view different displays of drift monitor metrics that depend on whether you perform drift detection as part of batch processing.
Drift analysis for non-batch processing data
On the Watson OpenScale Insights dashboard, you can view a chart that displays the drift monitor metrics. By clicking a data point on the chart, you can display specific transactions that contribute to drift. On the transactions page, you can view transactions that are responsible for a drop in accuracy, a drop in data consistency, or both. You can also view the number of transactions that are identified and the features of your model that responsible for reduced accuracy or data consistency.
When you click the Number of transactions link, you can view the transaction details page to understand how the transactions are evaluated for drift. The transaction details page specifies a reason for the drop in accuracy or drop in data consistency and provides a recommendation to help fix the drift of your model. Each transaction that contributes to drift in your model is specified in a table that provides details about the transactions.
Drift analysis for batch processing data
For the large quantity of data that can be produced by the batch processor, you receive a count of records that contribute to a drop in accuracy, a drop in data consistency, and both. In addition to this summary display, you can run a specialized analysis notebook: Notebook for analyzing payload transactions causing drift.
Enabling notifications
To send email notifications, click the Share the recommendations button. To enable this feature, you must first connect to an SMTP server that is configured in IBM Cloud Pak for Data. For more information, see Enabling email notifications.
Supported drift metrics
Go further
Read about a scenario that uses drift:
Parent topic: Watson OpenScale