Data drift

The umbrella term in the literature is dataset shift (Quiñonero-Candela et al., eds., Dataset Shift in Machine Learning, MIT Press, 2009): the joint distribution P(X, Y) at deployment differs from the one in training. It is usually split into covariate shift, where P(X) changes but P(Y | X) stays the same (a new customer segment arrives); prior or label shift, where P(Y) changes (fraud becomes more common); and concept drift, where P(Y | X) itself changes, so the same inputs now call for different answers (what counts as spam evolves). Google's ML glossary defines concept drift as a shift in the relationship between features and the label that reduces model quality over time. Gama et al. (2014) classify drift by its timing as sudden, gradual, incremental or recurring (seasonal), and distinguish real drift, which changes the decision boundary, from virtual drift, which only changes the input distribution.

Detection compares live data with a reference window, usually the training or validation data. Common checks are per-feature statistical tests (Kolmogorov-Smirnov for numeric features, chi-squared for categorical), distance measures such as the population stability index, Jensen-Shannon or Wasserstein distance, drift in the model's output distribution, and, where labels arrive late, the actual decline in accuracy or error. Stream-learning detectors such as DDM and ADWIN watch the error rate and signal when it changes significantly. Univariate tests on many features raise many false alarms, so thresholds are usually tuned or corrected for multiple comparisons.

Data drift is related to but different from training-serving skew, where the pipeline itself produces different features at training and serving time; Google's production ML guidance treats both as monitoring targets and recommends applying the same statistical checks to training and serving data. Responses include scheduled or triggered retraining on recent data, weighting recent examples, online learning, and human review of whether the change is real or a data quality bug such as a broken upstream field.

For generative and language models the same idea appears as knowledge that ages: the world moves past the training cutoff, user behaviour changes, and prompts shift toward tasks absent from training.