{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/underfitting","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/underfitting/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/underfitting/"},"term":{"en":"Underfitting","da":"Undertilpasning (underfitting)"},"aka":{"en":[],"da":["underfitting"]},"domain":["ai"],"cluster":"training","layer":"training","status":"current","summary":{"en":"When a model is too simple or trained too little to catch the pattern, so it scores badly even on the examples it learned from.","da":"Når en model er for simpel eller trænet for lidt til at fange mønstret, så den scorer dårligt selv på de eksempler, den lærte af."},"body":{"formal":{"en":"A failure of model training in which the model cannot fit even its training data well, with high loss on the training data and on held-back data alike, usually because it has too few model parameters, too few epochs or too little useful information in its input.","da":"En fejl i modeltræning, hvor modellen ikke engang kan tilpasse sig sine træningsdata godt - højt tab både på træningsdata og på tilbageholdte data - typisk fordi den har for få modelparametre, for få epoker eller for lidt brugbar information i sit input."},"plain":{"en":"Like describing every animal as “has legs”; the rule is too rough to tell a dog from a table, even for the animals you studied.","da":"Som at beskrive alle dyr som “har ben” - reglen er for grov til at skelne en hund fra et bord, selv for de dyr, man har studeret."},"inPractice":{"en":"The owner of a small Danish garden centre forecasts sales with a straight line; it misses both the spring rush and the Christmas-tree peak even in last year's own figures, so she switches to a richer model.","da":"Ejeren af et lille dansk havecenter forudsiger salget med en ret linje; den rammer hverken forårstravlheden eller juletræssalget, selv i sidste års egne tal, så hun skifter til en rigere model."},"whyItMatters":{"en":"A model like this is wrong in a steady, predictable way, so it can look stable while giving poor decisions; fixing it means more capacity or more training, not more rules.","da":"En undertilpasset model tager fejl på en stabil, forudsigelig måde, så den kan se solid ud, mens den giver dårlige beslutninger; løsningen er mere kapacitet eller mere træning, ikke flere regler."}},"deepDive":{"en":"The signature of underfitting is a training loss that stays high and a validation loss close to it: the gap between them is small, but both are well above what the task allows, measured against a reference such as human-level performance, a strong baseline or the estimated irreducible (Bayes) error. Learning curves make the diagnosis concrete. Plotted against training-set size, an underfitting model's training and validation curves converge quickly at a poor level, so collecting more data does not help, whereas an overfitting model shows a wide gap that more data narrows. In the bias-variance decomposition of squared error, expected error = bias² + variance + irreducible noise, and underfitting is the high-bias regime (Goodfellow et al., §5.2 and §5.4).\n\nCauses fall into four groups. Capacity: the hypothesis class cannot represent the pattern, as with a linear model for a seasonal or interacting signal. Regularisation: weight decay, dropout, L1 penalties or data augmentation set so strong that they suppress real structure. Optimisation: too few steps, a learning rate far too small or so large that training stalls, stopping early on a noisy validation signal, vanishing gradients or poor initialisation. Information: the input features simply do not carry what is needed to predict the target. Bugs belong under optimisation in practice: misaligned labels, unscaled inputs, a frozen layer or a loss applied to the wrong tensor all look like underfitting. A standard sanity check is to try to overfit a single small batch: if the loss cannot be driven close to zero on a handful of examples, the problem is a bug or a capacity mismatch, not a lack of data.\n\nRemedies follow the cause: a larger or more expressive model, better features, weaker regularisation, longer training or a tuned learning-rate schedule. The classical picture of a U-shaped test-error curve over model capacity has been revised by the double-descent phenomenon (Belkin et al., 2019), in which test error falls again once models are large enough to interpolate the training data, so heavily overparameterised networks rarely underfit for lack of capacity. Underfitting at scale is instead about compute and data: Hoffmann et al. (2022) showed that several large language models of the time were undertrained for their size, and in single-epoch pretraining training and validation loss track each other closely, so the usual lever is more tokens or compute rather than regularisation.\n\nUnderfitting should be kept apart from neighbouring failures. Overfitting shows a large gap between training and validation loss; distribution shift shows good validation results but poor production results; label noise caps both curves at a level set by the data rather than the model. Because an underfit model errs consistently, its mistakes are systematic (a straight-line forecast that always misses the seasonal peak) and can look deceptively stable.","da":"Kendetegnet ved undertilpasning er et træningstab, der forbliver højt, og et valideringstab tæt på det: afstanden mellem dem er lille, men begge ligger et godt stykke over, hvad opgaven tillader, målt mod en reference som menneskeligt niveau, en stærk baseline eller den estimerede irreducible (Bayes-)fejl. Læringskurver gør diagnosen konkret. Tegnet op mod træningssættets størrelse mødes en undertilpasset models trænings- og valideringskurver hurtigt på et dårligt niveau, så flere data ikke hjælper, mens en overtilpasset model viser en bred kløft, som flere data indsnævrer. I bias-varians-dekomponeringen af kvadratfejl er forventet fejl = bias² + varians + irreducibel støj, og undertilpasning er regimet med høj bias (Goodfellow m.fl., §5.2 og §5.4).\n\nÅrsagerne falder i fire grupper. Kapacitet: hypoteseklassen kan ikke repræsentere mønstret, som når en lineær model skal fange et sæsonpræget signal eller vekselvirkninger. Regularisering: weight decay, dropout, L1-straf eller dataaugmentering er sat så kraftigt, at de undertrykker reel struktur. Optimering: for få skridt, en læringsrate der er alt for lille eller så stor, at træningen går i stå, tidligt stop på et støjfyldt valideringssignal, forsvindende gradienter eller dårlig initialisering. Information: inputtets features indeholder simpelthen ikke det, der skal til for at forudsige målet. Fejl i koden hører i praksis under optimering - forskudte labels, uskalerede input, et frosset lag eller et tab beregnet på den forkerte tensor ligner alle undertilpasning. Et standardtjek er at forsøge at overtilpasse en enkelt lille batch: kan tabet ikke presses tæt på nul på en håndfuld eksempler, er problemet en fejl eller manglende kapacitet, ikke mangel på data.\n\nLøsningen følger årsagen: en større eller mere udtryksfuld model, bedre features, svagere regularisering, længere træning eller en tunet læringsrateplan. Det klassiske billede af en U-formet testfejlkurve over modelkapacitet er blevet revideret af double descent (Belkin m.fl., 2019), hvor testfejlen falder igen, når modellerne er store nok til at interpolere træningsdata, så kraftigt overparametriserede netværk sjældent undertilpasser af mangel på kapacitet. Undertilpasning i stor skala handler i stedet om beregning og data: Hoffmann m.fl. (2022) viste, at flere af tidens store sprogmodeller var undertrænede i forhold til deres størrelse, og i fortræning med én epoke følger trænings- og valideringstab hinanden tæt, så det sædvanlige greb er flere tokens eller mere beregning snarere end regularisering.\n\nUndertilpasning bør holdes adskilt fra nabofejl. Overtilpasning viser en stor kløft mellem træning og validering; distributionsskift viser gode valideringsresultater, men dårlige resultater i drift; støj i labels lægger et loft over begge kurver, som bestemmes af data og ikke af modellen. Fordi en undertilpasset model tager fejl konsekvent, er dens fejl systematiske - en ret linje, der altid rammer ved siden af sæsontoppen - og kan virke vildledende stabile."},"edges":[{"type":"requires","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/loss-function","confidence":"high","strength":"normal"},{"type":"causes","to":"ai/hallucination","why":{"en":"A language model that has not learned enough fills gaps with fluent guesses more often.","da":"En sprogmodel, der ikke har lært nok, fylder oftere huller ud med flydende gæt."},"confidence":"low","strength":"minor"}],"depth":3,"sources":[{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 5.2 and 5.4)","url":"https://www.deeplearningbook.org/contents/ml.html","tier":"textbook","publisher":"MIT Press"},{"title":"Belkin et al. (2019), Reconciling modern machine-learning practice and the classical bias-variance trade-off","url":"https://doi.org/10.1073/pnas.1903070116","tier":"reference","publisher":"PNAS"},{"title":"Hoffmann et al. (2022), Training Compute-Optimal Large Language Models","url":"https://arxiv.org/abs/2203.15556","tier":"reference"}],"draft":true}