Monitoring ML models in production is often a tedious task. You can apply a simple retraining strategy based on monitoring the model’s performance: if your AUC drops by a given percentage, retrain. Although accurate, this approach requires you to obtain the ground truth for your predictions, which is not always fast, and certainly not “real-time.”
Instead of waiting for the ground truth, Dataiku’s Model Drift Monitoring plugin looks at the recent data the model has scored, and statistically compares it with the data on which the model was evaluated in the training phase. If these datasets are too different, the model may need to be retrained. And that wraps it up! We hope you found this roundup helpful as you explore new ways to take your data science and machine learning projects to the next level in 2021.
Are you ready to become a Dataiku Super User?