{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/sampling","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/sampling/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/sampling/"},"term":{"en":"Sampling","da":"Sampling (udtrækning af tokens)"},"aka":{"en":["decoding"],"da":["afkodning"]},"domain":["ai"],"cluster":"prompting","layer":"inference","status":"current","summary":{"en":"How a language model picks each next piece of text from its list of likely options, by a set rule or with some chance involved.","da":"Måden, en sprogmodel vælger hvert næste stykke tekst blandt sine sandsynlige muligheder - efter fast regel eller med tilfældighed."},"body":{"formal":{"en":"The step in inference where, after next-token prediction gives a chance for every possible token, one token is chosen - always the most likely, or drawn at random according to those chances after settings such as temperature and top-p reshape them - and the process repeats.","da":"Det trin i inferensen, hvor der, efter at forudsigelse af næste token har givet en chance for hvert muligt token, vælges ét token - altid det mest sandsynlige eller trukket tilfældigt efter de chancer, når indstillinger som temperatur og top-p har justeret dem - og processen gentages."},"plain":{"en":"Like choosing the next word in a story by drawing from a bag where common words have many slips and odd words have few - usually sensible, sometimes surprising.","da":"Som at vælge næste ord i en historie ved at trække fra en pose, hvor almindelige ord har mange sedler og sjældne ord få - oftest fornuftigt, af og til overraskende."},"inPractice":{"en":"Two case workers at a job centre ask the same chat assistant to draft a letter from identical notes and get differently worded drafts, because each word was picked with some chance involved.","da":"To sagsbehandlere på et jobcenter beder den samme chatassistent skrive et brev ud fra de samme noter og får forskelligt formulerede udkast, fordi hvert ord blev valgt med en smule tilfældighed."},"whyItMatters":{"en":"The choice of rule decides whether answers can be repeated exactly, which matters for testing and for tasks like code, and it shapes how varied or dull the text feels.","da":"Valget af regel afgør, om svar kan gentages nøjagtigt, hvilket betyder noget for test og for opgaver som kode, og det former, hvor afvekslende eller kedelig teksten føles."}},"deepDive":{"en":"At each decoding step the model produces a vector of logits over the vocabulary. A decoding strategy turns that vector into one chosen token, usually through a chain of logit processors: penalties and biases are applied, temperature rescales the logits, truncation rules such as top-k, top-p or min-p remove unlikely candidates, the remainder is renormalised with a softmax, and one token is drawn using a pseudo-random generator. Deterministic strategies skip the draw. The chosen token is appended, its key and value vectors are added to the KV cache, and the loop repeats until an end-of-sequence token, a stop sequence or the output-token limit.\n\nThe strategies form a family. Greedy decoding always picks the argmax. Beam search keeps the k highest-probability partial sequences and was standard in machine translation, but for open-ended generation Holtzman et al. (2020) showed that maximising likelihood produces bland, repetitive and degenerate text, because human text is not the most probable text. Pure sampling from the full distribution has the opposite problem: the long tail of individually unlikely tokens is collectively likely to be hit, and one bad token derails what follows. Truncation methods address this: top-k (used by Fan et al., 2018) keeps a fixed number of candidates, nucleus or top-p keeps the smallest set whose cumulative probability reaches p, and min-p (Nguyen et al., 2024) keeps tokens whose probability is at least a fraction of the top token's. Frequency and presence penalties discourage repetition, logit bias forces or bans specific tokens, and constrained decoding masks out tokens that would violate a grammar or JSON Schema.\n\nReproducibility is weaker than the settings suggest. Temperature 0 or greedy decoding removes intentional randomness, but hosted inference can still return different outputs for the same request. Floating-point addition is not associative, and GPU kernels choose different reduction orders depending on batch size and other load-dependent factors, so logits can differ in the last bits between runs and flip a near-tie. Thinking Machines Lab (2025) showed that batch-invariant kernels make repeated runs bit-identical at a throughput cost. Some APIs offer a seed parameter, but providers describe it as best-effort; mixture-of-experts routing and speculative decoding add further sources of variation.\n\nPractical guidance: use greedy or low temperature for extraction, classification and code where one correct answer exists; moderate temperature with top-p around 0.9 to 0.95 for prose; multiple samples plus voting or verification (self-consistency, best-of-n with a grader) when accuracy matters more than cost. Change one parameter at a time, evaluate over several samples per test case, and record decoding parameters alongside prompts so results can be reproduced as far as the platform allows.","da":"I hvert afkodningstrin giver modellen en vektor af logits over ordforrådet. En afkodningsstrategi gør vektoren til ét valgt token, typisk gennem en kæde af logit-processorer: Straffe og bias anvendes, temperaturen omskalerer logits, afskæringsregler som top-k, top-p eller min-p fjerner usandsynlige kandidater, resten normaliseres igen med en softmax, og ét token trækkes med en pseudotilfældig generator. Deterministiske strategier springer trækningen over. Det valgte token føjes til, dets key- og value-vektorer lægges i KV-cachen, og løkken gentages indtil et slut-token, en stopsekvens eller grænsen for output-tokens.\n\nStrategierne udgør en familie. Grådig afkodning vælger altid argmax. Beam search holder de k mest sandsynlige delsekvenser og var standard i maskinoversættelse, men for åben generering viste Holtzman m.fl. (2020), at maksimering af sandsynligheden giver flad, gentagende og degenereret tekst, fordi menneskelig tekst ikke er den mest sandsynlige tekst. Ren sampling fra hele fordelingen har det modsatte problem: Den lange hale af enkeltvis usandsynlige tokens rammes samlet set ofte, og ét dårligt token afsporer resten. Afskæringsmetoder løser det: Top-k (brugt af Fan m.fl., 2018) beholder et fast antal kandidater, nucleus- eller top-p-sampling beholder den mindste mængde, hvis samlede sandsynlighed når p, og min-p (Nguyen m.fl., 2024) beholder tokens, hvis sandsynlighed er mindst en vis andel af det mest sandsynlige tokens. Frequency- og presence-straffe modvirker gentagelser, logit bias tvinger eller forbyder bestemte tokens, og begrænset afkodning maskerer tokens, der ville bryde en grammatik eller et JSON Schema.\n\nReproducerbarheden er svagere, end indstillingerne antyder. Temperatur 0 eller grådig afkodning fjerner den tilsigtede tilfældighed, men hostet inferens kan stadig give forskelligt output på samme kald. Addition af flydende kommatal er ikke associativ, og GPU-kerner vælger forskellig rækkefølge for summeringer afhængigt af batchstørrelse og andre belastningsafhængige forhold, så logits kan afvige i de sidste bits mellem kørsler og vende et næsten-uafgjort valg. Thinking Machines Lab (2025) viste, at batch-invariante kerner gør gentagne kørsler bit-identiske på bekostning af gennemløb. Nogle API'er tilbyder en seed-parameter, men udbyderne beskriver den som best effort; routing i mixture of experts og spekulativ afkodning giver yderligere variation.\n\nPraktisk vejledning: Brug grådig afkodning eller lav temperatur til udtræk, klassifikation og kode, hvor der findes ét korrekt svar; moderat temperatur med top-p omkring 0,9 til 0,95 til prosa; flere træk plus afstemning eller verifikation (self-consistency, best-of-n med en bedømmer), når præcision betyder mere end pris. Ændr én parameter ad gangen, evaluer over flere træk pr. testtilfælde, og gem afkodningsparametre sammen med prompts, så resultater kan genskabes, så vidt platformen tillader det."},"edges":[{"type":"requires","to":"ai/next-token-prediction","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/inference","why":{"en":"Each time a language model writes a reply, sampling runs once for every token it adds.","da":"Hver gang en sprogmodel skriver et svar, kører sampling én gang for hvert token, den tilføjer."},"confidence":"high","strength":"primary"}],"depth":2,"sources":[{"title":"Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), chapter on large language models (sampling)","tier":"textbook"},{"title":"Holtzman et al. (2020), The Curious Case of Neural Text Degeneration","url":"https://arxiv.org/abs/1904.09751","tier":"reference"},{"title":"Thinking Machines Lab (2025), Defeating Nondeterminism in LLM Inference","url":"https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/","tier":"other","publisher":"Thinking Machines Lab"}],"draft":true}