Data drift vs Overfitting

Why they differ

Both show up as a model doing worse in real use than in testing, but overfitting is a flaw from training, while data drift comes from the world changing afterwards.

Data drift

Evaluation & metrics

The slow or sudden change in real-world data after a model goes live, so it no longer looks like what the model learned from.

Formal

A change over time in the inputs a model sees at inference, or in how those inputs relate to the right answer, compared with its training data; it lowers quality without any change to the model itself.

In plain English

Like an old city map that was right when it was printed, but the streets have been rebuilt since, so following it faithfully now gets you lost.

In practice

A model that guessed how many calls a support desk would get was trained before a new self-service website opened; afterwards the calls change in number and kind, and its guesses quietly go wrong.

Why it matters

A model that passed every test can still fail months later with no error message, so its live inputs and results must be watched and it must be retrained.

Overfitting

Machine learning basics

When a model learns its training examples by heart, including their noise, and then does badly on new cases it has not seen.

Formal

A failure of model training in which a model fits the training data too closely, so it scores well on that data but poorly on new data; found by testing on data held back from training.

In plain English

Like a student who learns last year's exam answers by heart, word for word, and then fails when the questions are slightly changed.

In practice

A pension fund's fraud model finds almost every false claim among the cases it was trained on, but misses most real fraud once live, because it learned details found only in those old cases.

Why it matters

It makes a model look better on paper than in real use, and a model that has learned examples by heart can later repeat private details from its training data.

Shared connections

Training data