Validation set

The validation set serves every decision made after the parameters have been fitted but before the model is frozen: choosing hyperparameters such as learning rate, regularisation strength or architecture size; choosing between model families; choosing the stopping epoch; choosing a decision threshold for a target precision or recall; and fitting post-hoc calibration such as Platt scaling or the temperature scaling of Guo et al. (2017). All of these are forms of learning from data, which is why they must use data that the parameters were not trained on and that the final test set does not share.

When data is scarce, a single split wastes examples and gives a noisy estimate, so k-fold cross-validation (typically k = 5 or 10) rotates the validation role across folds and averages the score; stratified folds keep class ratios constant. If the cross-validated score is also used to report performance after tuning, it is biased upward; nested cross-validation fixes this with an inner loop for tuning and an outer loop for estimation, a point made forcefully by Cawley and Talbot (2010). Time series need forward-chaining schemes such as scikit-learn's TimeSeriesSplit, where every validation fold lies after its training data, and grouped data needs group-aware folds.

Early stopping is the most visible use. The validation loss is evaluated periodically, training stops when it has not improved for a set number of evaluations (the patience), and the checkpoint with the best validation score is restored; Keras exposes this as restore_best_weights. Goodfellow et al. (§7.8) interpret early stopping as a form of regularisation. A common follow-up is to retrain on training plus validation data for the chosen number of steps, trading a held-out check for more data.

The validation score is an optimistically biased estimate of the selected model, and the bias grows with the number of configurations tried: taking the maximum over many noisy scores rewards configurations that got lucky on this particular sample, the winner's curse of model selection. Large hyperparameter sweeps can therefore overfit the validation set, which is why a separate test set is still required. In LLM fine-tuning frameworks the "eval" split, such as the eval_dataset in Hugging Face's Trainer, is a validation set in this sense. Terminology varies: NLP often says dev set, while clinical research uses "validation" for what machine learning calls testing.