Test set vs Validation set

Why they differ

The validation set is checked many times to steer choices while building; the test set is opened once, after every choice is made.

Test set

Evaluation & metrics

Examples locked away while a model is built and opened once at the end to give an honest final score.

Formal

A part of the labelled data that plays no role in model training or in choosing settings, used only after all choices are fixed to estimate how well the model will do in real use.

In plain English

Like the sealed envelope of exam questions that stays in the head teacher's safe until exam day, so nobody could have practised on them.

In practice

After weeks of adjusting a model that predicts which patients will miss hospital appointments, a data analyst in a region runs it once on 5,000 held-back appointments, reports that figure to management and does not tweak the model again.

Why it matters

If its examples leak into the training data, or it is checked again and again, the score flatters the model, and a weak system can be put into use on real people.

Validation set

Evaluation & metrics

Examples held back from training and checked again and again while building a model, to choose its settings and decide when to stop.

Formal

A part of the labelled data kept out of the training data and used during model training to compare hyperparameter choices and pick the best round to stop; because choices are tuned to it, its scores are not a fair final result.

In plain English

Like a cook tasting the sauce again and again while it simmers, adjusting salt and time, which helps steer the cooking but is not the guests' honest verdict on the finished dish.

In practice

A data analyst in a ministry tries five settings for a model that sorts public consultation responses by topic and keeps the one that scores best on 1,500 responses held back from training; the test set stays closed until then.

Why it matters

Without it, settings get chosen by looking at the test set, and the final score quietly stops being honest.

Shared connections

Model training Training data