Test set

The test set is the last of the three standard partitions (training, validation, test) and the only one whose purpose is estimation rather than fitting or selection. Split ratios such as 80/10/10 or 70/15/15 are conventions for moderate data sizes; what actually matters is the absolute number of test cases, since the width of a confidence interval shrinks with √n. For a metric near 90%, around 1,000 cases give roughly ±2 percentage points, and rare classes need enough positives on their own. Stratified splitting keeps class proportions equal across partitions, which matters when a class is rare.

Leakage is the main way a test set lies. Group leakage occurs when related records (the same patient, customer, device or document in several versions) land on both sides of the split; group-aware splitting (e.g. scikit-learn's GroupShuffleSplit or GroupKFold) prevents it. Temporal leakage occurs when a random split lets the model train on the future; forecasting and fraud models should be tested on a later time window than they were trained on. Near-duplicates, scraped mirrors and augmented copies need deduplication, often by hashing or MinHash. Preprocessing leakage arises when scalers, vocabularies or feature selection are fitted on the full data set before splitting. Kaufman et al. (2012) give a systematic treatment of leakage in data mining.

Every look at the test set leaks information into the modelling process. If results drive further changes, the test set has turned into a validation set and its score becomes optimistically biased. Kaggle's split between a public and a private leaderboard exists precisely because teams overfit the public part; Dwork et al. (2015) proposed a "reusable holdout" mechanism based on differential privacy to limit this. For large language models the problem appears as contamination, when public test items are already in the pretraining corpus, and on the evaluation side even the test labels themselves can be wrong: Northcutt et al. (2021) found label errors in at least 6% of the ImageNet validation set that serves as its de facto test set.

A test set also has to represent the deployment population, not just the collected data. Performance on an internal test split is internal validity; a test set drawn from another hospital, region or year measures transportability, which is why clinical prediction research distinguishes internal from external validation. Beware the terminology clash: in that literature "validation" usually means what machine learning calls testing, so a "validation cohort" in a medical paper is the held-out test set, not a tuning split.