{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/validation-set","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/validation-set/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/validation-set/"},"term":{"en":"Validation set","da":"Valideringssæt"},"aka":{"en":["dev set","development set"],"da":["udviklingssæt","dev-sæt"]},"domain":["ai"],"cluster":"evaluation","layer":"training","status":"current","summary":{"en":"Examples held back from training and checked again and again while building a model, to choose its settings and decide when to stop.","da":"Eksempler, der holdes uden for træningen og tjekkes igen og igen for at vælge en models indstillinger og hvornår der skal stoppes."},"body":{"formal":{"en":"A part of the labelled data kept out of the training data and used during model training to compare hyperparameter choices and pick the best round to stop; because choices are tuned to it, its scores are not a fair final result.","da":"En del af de mærkede data, der holdes uden for træningsdata og bruges under modeltræningen til at sammenligne valg af hyperparametre og vælge den bedste runde at stoppe i; da valgene tilpasses til den, er dens resultater ikke et retfærdigt slutresultat."},"plain":{"en":"Like a cook tasting the sauce again and again while it simmers, adjusting salt and time, which helps steer the cooking but is not the guests' honest verdict on the finished dish.","da":"Som en kok, der smager på saucen igen og igen, mens den simrer, og justerer salt og tid - nyttigt at styre efter, men ikke gæsternes ærlige dom over den færdige ret."},"inPractice":{"en":"A data analyst in a ministry tries five settings for a model that sorts public consultation responses by topic and keeps the one that scores best on 1,500 responses held back from training; the test set stays closed until then.","da":"En dataanalytiker i et ministerium afprøver fem indstillinger for en model, der sorterer høringssvar efter emne, og beholder den, der scorer bedst på 1.500 høringssvar holdt uden for træningen; testsættet forbliver lukket indtil da."},"whyItMatters":{"en":"Without it, settings get chosen by looking at the test set, and the final score quietly stops being honest.","da":"Uden det vælges indstillinger ved at kigge på testsættet, og slutresultatet holder ubemærket op med at være ærligt."}},"deepDive":{"en":"The validation set serves every decision made after the parameters have been fitted but before the model is frozen: choosing hyperparameters such as learning rate, regularisation strength or architecture size; choosing between model families; choosing the stopping epoch; choosing a decision threshold for a target precision or recall; and fitting post-hoc calibration such as Platt scaling or the temperature scaling of Guo et al. (2017). All of these are forms of learning from data, which is why they must use data that the parameters were not trained on and that the final test set does not share.\n\nWhen data is scarce, a single split wastes examples and gives a noisy estimate, so k-fold cross-validation (typically k = 5 or 10) rotates the validation role across folds and averages the score; stratified folds keep class ratios constant. If the cross-validated score is also used to report performance after tuning, it is biased upward; nested cross-validation fixes this with an inner loop for tuning and an outer loop for estimation, a point made forcefully by Cawley and Talbot (2010). Time series need forward-chaining schemes such as scikit-learn's TimeSeriesSplit, where every validation fold lies after its training data, and grouped data needs group-aware folds.\n\nEarly stopping is the most visible use. The validation loss is evaluated periodically, training stops when it has not improved for a set number of evaluations (the patience), and the checkpoint with the best validation score is restored; Keras exposes this as restore_best_weights. Goodfellow et al. (§7.8) interpret early stopping as a form of regularisation. A common follow-up is to retrain on training plus validation data for the chosen number of steps, trading a held-out check for more data.\n\nThe validation score is an optimistically biased estimate of the selected model, and the bias grows with the number of configurations tried: taking the maximum over many noisy scores rewards configurations that got lucky on this particular sample, the winner's curse of model selection. Large hyperparameter sweeps can therefore overfit the validation set, which is why a separate test set is still required. In LLM fine-tuning frameworks the \"eval\" split, such as the eval_dataset in Hugging Face's Trainer, is a validation set in this sense. Terminology varies: NLP often says dev set, while clinical research uses \"validation\" for what machine learning calls testing.","da":"Valideringssættet bruges til alle de beslutninger, der træffes, efter at parametrene er tilpasset, men før modellen fryses: valg af hyperparametre som læringsrate, regulariseringsstyrke eller arkitekturstørrelse; valg mellem modelfamilier; valg af epoke at stoppe i; valg af beslutningstærskel for at ramme en ønsket præcision eller genkaldelse; og tilpasning af efterfølgende kalibrering som Platt scaling eller temperature scaling fra Guo m.fl. (2017). Alt dette er former for læring fra data, og derfor skal det ske på data, som parametrene ikke er trænet på, og som det endelige testsæt ikke deler.\n\nNår data er knappe, spilder en enkelt opdeling eksempler og giver et støjfyldt estimat, så k-fold-krydsvalidering (typisk k = 5 eller 10) lader valideringsrollen gå på skift mellem fold og tager gennemsnittet; stratificerede fold holder klasseforholdet konstant. Bruges den krydsvaliderede score også til at rapportere ydelse efter tuning, er den skævt opadtil; nested krydsvalidering løser det med en indre løkke til tuning og en ydre løkke til estimation, en pointe Cawley og Talbot (2010) gjorde meget tydeligt. Tidsserier kræver fremadskridende skemaer som scikit-learns TimeSeriesSplit, hvor hvert valideringsfold ligger efter sine træningsdata, og grupperede data kræver gruppebevidste fold.\n\nEarly stopping er den mest synlige anvendelse. Valideringstabet måles med jævne mellemrum, træningen stoppes, når det ikke er blevet bedre i et fastsat antal målinger (patience), og checkpointet med den bedste valideringsscore genindlæses - Keras tilbyder det som restore_best_weights. Goodfellow m.fl. (§7.8) tolker early stopping som en form for regularisering. En udbredt opfølgning er at træne igen på trænings- plus valideringsdata i det valgte antal skridt og dermed bytte et tilbageholdt tjek for flere data.\n\nValideringsscoren er et for optimistisk estimat af den valgte model, og skævheden vokser med antallet af afprøvede konfigurationer: at tage maksimum over mange støjfyldte scorer belønner konfigurationer, der var heldige med netop denne stikprøve - modeludvælgelsens winner's curse. Store hyperparametersøgninger kan derfor overtilpasse valideringssættet, og derfor er et separat testsæt stadig nødvendigt. I rammeværk til finjustering af LLM'er er \"eval\"-splittet, fx eval_dataset i Hugging Faces Trainer, et valideringssæt i denne betydning. Terminologien varierer: NLP siger ofte dev-sæt, mens klinisk forskning bruger \"validering\" om det, maskinlæring kalder test."},"edges":[{"type":"requires","to":"ai/hyperparameter","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/model-training","why":{"en":"It is checked throughout model training to steer choices, unlike the test set, which is kept apart until the end.","da":"Det tjekkes hele vejen gennem modeltræningen for at styre valgene, i modsætning til testsættet, der holdes adskilt til slutningen."},"confidence":"high","strength":"primary"},{"type":"mitigates","to":"ai/overfitting","why":{"en":"When the score on held-back examples stops rising while the training score keeps climbing, training is stopped before the model learns its examples by heart.","da":"Når resultatet på de tilbageholdte eksempler holder op med at stige, mens træningsresultatet bliver ved med at klatre, stoppes træningen, før modellen lærer sine eksempler udenad."},"confidence":"medium","strength":"normal"}],"depth":3,"sources":[{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 5.3 and 7.8)","url":"https://www.deeplearningbook.org/","tier":"textbook","publisher":"MIT Press"},{"title":"Russell & Norvig, Artificial Intelligence: A Modern Approach","url":"https://aima.cs.berkeley.edu/","tier":"textbook","publisher":"Pearson"},{"title":"Cawley & Talbot (2010), On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation","url":"https://jmlr.org/papers/v11/cawley10a.html","tier":"reference","publisher":"Journal of Machine Learning Research 11"},{"title":"Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (§4.10, Test sets and Cross-validation)","url":"https://web.stanford.edu/~jurafsky/slp3/","tier":"textbook"}],"draft":true}