Realtime Concept Drift Detection for Machine Learning Classification Models

Realtime Concept Drift Detection for Machine Learning Classification Models

Concept drift can occur when the relationship between the input and output data changes i.e P(Y|X) changes while P(X) remains same. It happens when the behavior of the underlaying process has changed with respect to the model training time. This situation is not uncommon in real world situations, especially when it comes to ML models that involve human behavior, explicitly or implicitly. Please refer to my earlier post for more technical information on concept drift. I had a batch based implementation of several model drift detection algorithm as described in my earlier post. I implemented window based real time API for concept drift detection made them part of my Python package matumizi. The code is in my GitHub repo whakapai.

Concept Drift

Concept drift occurs when the conditional distribution P(Y|X) changes while P(X) doesn’t. it’s manifested with increasing error rate. Once concept drift occurs the model requires re training with recent data As an example, let’s consider a regression model for used car price. Due to changing economic condition, used car prices may generally be higher resulting in decreasing accuracy of the model. Chip Huyen has an excellent post on drift detection, worth reading

Drift can be detected intermittently or persistently. Only when the drift is persistent, model retraining should be considered. Intermediate drift detection is also valuable in the sense that it tells when the model prediction is unreliable because of some temporary change in the underlying process that the ML model represents.

Concept drift detection is only possible when real feedback following the model prediction is available. When feedback is not available, the only other alternative is to detect data drift i.e change in P(X). But data drift is not necessarily a good indicator of concept drift.

The following 4 algorithms have been implemented for real time concept drift detection. In this post I will show results for EDDM only

• DDM
• EDDM
• FHDDM
• ECDD

The drift detection is based on some statistics gathered from the binary error data. The input to the detection model is a binary which is 1 when error has occurred i.e model prediction is incorrect and 0 otherwise.

There is a parameter called warmup which defines the number of error samples to be used for statistics calculation. Here are the list of the important command line arguments. Please refer to my earlier post for guidance on choosing appropriate threshold parameter value

• threshold: Threshold value for drift detection (interpretation depends on the algorithm)
• warmup: Sample size for error statistics gathering
• wsize: Window size
• wpsize: Window processing step

Not every call to the detector return drift detection result. Nothing is returned during warmup. Later, result is returned every wpsize step.

You can checkpoint the drift detection model by providing a file path at the end of a session. For the next session you can restore the drift model from the saved drift model.

Drift Detection Result

Please consult the tutorial for details on how to call the drift detector. The driver code in sdriftd.py can be used as an example. The implementation is in sdrift.py. The binary error data is created artificially and save in file. The file is read one line at a time and the drift detector called.

In real life following the model prediction, a human will provide the real output as a feedback. Depending whether the model prediction is correct or not, 1 or 0 is sent to the drift detector. The drift detector is likely to be deployed as continuously running service.

Here is some output for 2000 model predictions. One plot shows the drift score.The other plot shows the occurrence of drift when the score goes below a threshold. It detects some intermittent drift, but from around 1600 predictions onwards, the drift is persistent. Please refer to the tutorial for the step details.

For the first run, all parameters should be specified along with file path where he drift model will be saved. For follow up runs, the parameters need not be specified, but the file path from where the drift model should be restored and the file path where the drift model should be saved should be specified. For details on how to use the other drift detection algorithms please consult the tutorial.

A drift detector service can be deployed for multiple classification models. In that case model ID needs to be provided for each call. Internally, the detector service will have to manage multiple detector models. Alternative you can deploy multiple drift detector services, one for each deployed classification model.

Wrapping Up

The concept drift detection algorithms mentioned here all for classification models. The input to the drift detector is a binary error signal. Concept drift for regression models could be detected with 2 sample statistics of regression error data. I will cover that in a separate post.