Model Drift in Machine Learning

Context and Introduction

Model drift refers to the degradation of model performance due to changes in data and relationships between input and output variables.

It is relatively common for model drift to impact an organization negatively over time or sometimes suddenly.

Illustration of model drift in forms of model accuracy degradation.

Types of Model Drift

Drift in machine learning occurs in two forms: concept drift and data drift.

Concept Drift

Concept drift happens when a new business scenario emerges, which was non-existent in the model building process.

Example: Users change their behavior over time, thus causing new patterns to emerge.

Data Drift

Data drift results from changing in features, which is typically unseen in the modeling phase.

Example: In spam detection, people may use synonyms of a word from the dictionary.

How to detect and mitigate Model Drift

In order to overcome the issue of model drift it is important to have a clear strategy on how to detect it and minimize its impact. In the following, some basic steps which will help in model drift detection and handling:

• Track metrics continually and get alerted on drift in accuracy and data consistency.
• Set targets and track them through development, validation and deployment.
• Simplify steps to identify business metrics affected by model drift.
• Minimize the impact of model degradation through automated retraining.

Illustration of how we can detect and minimize the impact of model drift.

References

How to detect Data Drift in Azure Machine Learning - Datasset to Mindset

Model Drift in Machine Learning

Model Drift in Machine Learning - How To Handle It In Big Data - KDnuggets