Cross-validation

In k-fold cross-validation, as scikit-learn describes it, the training set is split into k smaller sets; for each fold a model is trained on the other k minus 1 folds and validated on the remaining one, and the reported score is the average over folds, often with its standard deviation. The idea goes back to Stone (1974), Cross-validatory choice and assessment of statistical predictions. Kohavi (1995) compared cross-validation and bootstrap on over half a million runs and concluded that for model selection on real-world datasets like his, ten-fold stratified cross-validation was the best method, even when computation allows more folds; scikit-learn likewise notes that 5 or 10 folds are generally preferred to leave-one-out, which uses n folds of one example and gives a high-variance estimate at great computational cost.

Variants handle structure in the data. Stratified k-fold keeps each fold's class proportions close to the whole set, which matters for imbalanced classification. Group k-fold keeps all examples from one group (a patient, a customer, a document) in the same fold, so the model is never tested on a group it trained on. For time series, TimeSeriesSplit uses expanding windows in which every test fold lies after its training data, since shuffling would let the model see the future. Repeated k-fold reruns the whole procedure with different random splits to reduce the variance of the estimate.

Cross-validation is mainly a tool for model selection and hyperparameter tuning (scikit-learn's GridSearchCV and RandomizedSearchCV run it internally). If the same cross-validated score is used both to pick the best configuration and to report performance, the estimate is optimistically biased; nested cross-validation adds an outer loop for honest estimation, or a separate untouched test set is kept for the final number.

Leakage is the most common error. scikit-learn stresses that preprocessing such as standardisation or feature selection must be learnt from the training folds only and applied to the held-out fold, which a Pipeline does automatically; selecting features on the full dataset before cross-validating can produce impressive scores on pure noise. For large deep learning models, full k-fold training is usually too expensive, so a single fixed validation set is the norm there.