{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/test-set","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/test-set/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/test-set/"},"term":{"en":"Test set","da":"Testsæt"},"aka":{"en":["holdout test set"],"da":["test set"]},"domain":["ai"],"cluster":"evaluation","layer":"training","status":"current","summary":{"en":"Examples locked away while a model is built and opened once at the end to give an honest final score.","da":"Eksempler, der låses væk, mens en model bygges, og åbnes én gang til sidst for at give et ærligt slutresultat."},"body":{"formal":{"en":"A part of the labelled data that plays no role in model training or in choosing settings, used only after all choices are fixed to estimate how well the model will do in real use.","da":"En del af de mærkede data, der ikke spiller nogen rolle i modeltræningen eller i valget af indstillinger, og som først bruges, når alle valg er låst, til at vurdere, hvor godt modellen vil klare sig i virkeligheden."},"plain":{"en":"Like the sealed envelope of exam questions that stays in the head teacher's safe until exam day, so nobody could have practised on them.","da":"Som den forseglede kuvert med eksamensspørgsmål, der ligger i rektors pengeskab til eksamensdagen, så ingen kan have øvet sig på dem."},"inPractice":{"en":"After weeks of adjusting a model that predicts which patients will miss hospital appointments, a data analyst in a region runs it once on 5,000 held-back appointments, reports that figure to management and does not tweak the model again.","da":"Efter ugers justering af en model, der forudsiger, hvilke patienter der udebliver fra deres aftaler på hospitalet, kører en dataanalytiker i en region den én gang på 5.000 tilbageholdte aftaler, rapporterer tallet til ledelsen og justerer ikke modellen igen."},"whyItMatters":{"en":"If its examples leak into the training data, or it is checked again and again, the score flatters the model, and a weak system can be put into use on real people.","da":"Hvis eksemplerne siver ind i træningsdata, eller testsættet tjekkes igen og igen, får modellen et for flot resultat, og et svagt system kan blive taget i brug over for rigtige mennesker."}},"deepDive":{"en":"The test set is the last of the three standard partitions (training, validation, test) and the only one whose purpose is estimation rather than fitting or selection. Split ratios such as 80/10/10 or 70/15/15 are conventions for moderate data sizes; what actually matters is the absolute number of test cases, since the width of a confidence interval shrinks with √n. For a metric near 90%, around 1,000 cases give roughly ±2 percentage points, and rare classes need enough positives on their own. Stratified splitting keeps class proportions equal across partitions, which matters when a class is rare.\n\nLeakage is the main way a test set lies. Group leakage occurs when related records (the same patient, customer, device or document in several versions) land on both sides of the split; group-aware splitting (e.g. scikit-learn's GroupShuffleSplit or GroupKFold) prevents it. Temporal leakage occurs when a random split lets the model train on the future; forecasting and fraud models should be tested on a later time window than they were trained on. Near-duplicates, scraped mirrors and augmented copies need deduplication, often by hashing or MinHash. Preprocessing leakage arises when scalers, vocabularies or feature selection are fitted on the full data set before splitting. Kaufman et al. (2012) give a systematic treatment of leakage in data mining.\n\nEvery look at the test set leaks information into the modelling process. If results drive further changes, the test set has turned into a validation set and its score becomes optimistically biased. Kaggle's split between a public and a private leaderboard exists precisely because teams overfit the public part; Dwork et al. (2015) proposed a \"reusable holdout\" mechanism based on differential privacy to limit this. For large language models the problem appears as contamination, when public test items are already in the pretraining corpus, and on the evaluation side even the test labels themselves can be wrong: Northcutt et al. (2021) found label errors in at least 6% of the ImageNet validation set that serves as its de facto test set.\n\nA test set also has to represent the deployment population, not just the collected data. Performance on an internal test split is internal validity; a test set drawn from another hospital, region or year measures transportability, which is why clinical prediction research distinguishes internal from external validation. Beware the terminology clash: in that literature \"validation\" usually means what machine learning calls testing, so a \"validation cohort\" in a medical paper is the held-out test set, not a tuning split.","da":"Testsættet er det sidste af de tre standardopdelinger (træning, validering, test) og det eneste, hvis formål er estimation frem for tilpasning eller udvælgelse. Fordelinger som 80/10/10 eller 70/15/15 er konventioner ved moderate datamængder; det, der reelt tæller, er det absolutte antal testtilfælde, fordi bredden af et konfidensinterval falder med √n. For et mål omkring 90 % giver cirka 1.000 tilfælde omtrent ±2 procentpoint, og sjældne klasser har brug for nok positive i sig selv. Stratificeret opdeling holder klasseandelene ens på tværs af opdelingerne, hvilket betyder noget, når en klasse er sjælden.\n\nLækage er den vigtigste måde, et testsæt lyver på. Gruppelækage opstår, når beslægtede poster - samme patient, kunde, enhed eller flere versioner af samme dokument - havner på begge sider af opdelingen; gruppebevidst opdeling (fx scikit-learns GroupShuffleSplit eller GroupKFold) forhindrer det. Tidslækage opstår, når en tilfældig opdeling lader modellen træne på fremtiden; prognose- og svindelmodeller bør testes på et senere tidsvindue end det, de er trænet på. Næsten-dubletter, spejlede kopier og augmenterede varianter kræver deduplikering, ofte med hashing eller MinHash. Lækage via forbehandling opstår, når skalering, ordforråd eller feature-udvælgelse tilpasses på hele datasættet før opdelingen. Kaufman m.fl. (2012) giver en systematisk gennemgang af lækage i data mining.\n\nHvert kig på testsættet lækker information ind i modelleringen. Hvis resultaterne styrer yderligere ændringer, er testsættet blevet til et valideringssæt, og scoren bliver for optimistisk. Kaggles opdeling i en offentlig og en privat rangliste findes netop, fordi hold overtilpasser den offentlige del; Dwork m.fl. (2015) foreslog en \"genbrugelig holdout\"-mekanisme baseret på differential privacy for at begrænse det. For store sprogmodeller viser problemet sig som kontaminering, når offentlige testopgaver allerede ligger i fortræningsdata, og selv testsættets labels kan være forkerte: Northcutt m.fl. (2021) fandt fejl i mindst 6 % af labels i ImageNets valideringssæt, som i praksis fungerer som dets testsæt.\n\nEt testsæt skal også repræsentere den population, modellen skal bruges på, og ikke kun de indsamlede data. Resultatet på et internt testsplit er intern validitet; et testsæt fra et andet hospital, en anden region eller et andet år måler overførbarhed, og derfor skelner klinisk prædiktionsforskning mellem intern og ekstern validering. Pas på terminologien: i den litteratur betyder \"validering\" som regel det, maskinlæring kalder test, så en \"valideringskohorte\" i en medicinsk artikel er det tilbageholdte testsæt og ikke et split til tuning."},"edges":[{"type":"requires","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/model-evaluation","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/training-data","why":{"en":"Training data is what the model learns from; the test set is kept strictly apart so it can measure the model fairly.","da":"Træningsdata er det, modellen lærer af; testsættet holdes strengt adskilt, så det kan måle modellen retfærdigt."},"confidence":"high","strength":"primary"},{"type":"contrasts-with","to":"ai/validation-set","why":{"en":"The validation set is checked many times to steer choices while building; the test set is opened once, after every choice is made.","da":"Valideringssættet tjekkes mange gange for at styre valg under opbygningen; testsættet åbnes én gang, når alle valg er truffet."},"confidence":"high","strength":"primary"}],"depth":2,"sources":[{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 5.3)","url":"https://www.deeplearningbook.org/","tier":"textbook","publisher":"MIT Press"},{"title":"Russell & Norvig, Artificial Intelligence: A Modern Approach","url":"https://aima.cs.berkeley.edu/","tier":"textbook","publisher":"Pearson"},{"title":"ISO/IEC 22989:2022, Artificial intelligence concepts and terminology","url":"https://www.iso.org/standard/74296.html","tier":"standard","publisher":"ISO/IEC"},{"title":"Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet?","url":"https://proceedings.mlr.press/v97/recht19a.html","tier":"reference","publisher":"ICML 2019 (PMLR 97)"},{"title":"Northcutt, Athalye & Mueller (2021), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks","url":"https://arxiv.org/abs/2103.14749","tier":"reference","publisher":"NeurIPS 2021 Datasets and Benchmarks"},{"title":"Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (§4.10, Test sets and Cross-validation)","url":"https://web.stanford.edu/~jurafsky/slp3/","tier":"textbook"}],"draft":true}