{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/top-p-sampling","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/top-p-sampling/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/top-p-sampling/"},"term":{"en":"Top-p sampling","da":"Top-p-sampling"},"aka":{"en":["nucleus sampling"],"da":["nucleus sampling"]},"domain":["ai"],"cluster":"prompting","layer":"inference","status":"current","era":2019,"summary":{"en":"A rule letting a language model pick only among its most likely next options, until their chances add up to a set share like 90%.","da":"En regel, hvor en sprogmodel kun vælger blandt sine mest sandsynlige næste muligheder, til deres chancer tilsammen når fx 90 %."},"body":{"formal":{"en":"A form of sampling that sorts candidate tokens by chance, keeps the smallest group whose chances add up to at least p, and draws the next token from that group only, so the number of options grows or shrinks with how sure the model is.","da":"En form for sampling, der sorterer kandidat-tokens efter chance, beholder den mindste gruppe, hvis chancer tilsammen når mindst p, og kun trækker næste token fra den gruppe, så antallet af muligheder vokser eller bliver færre alt efter, hvor sikker modellen er."},"plain":{"en":"Like a quiz team that only considers answers they are fairly sure of - when one is obvious they go with it, when unsure they weigh several, and wild guesses never make the table.","da":"Som et quizhold, der kun overvejer svar, de er rimelig sikre på - når ét er oplagt, tager de det, når de er i tvivl, vejer de flere, og vilde gæt kommer aldrig på bordet."},"inPractice":{"en":"A developer in a municipality's IT department sees the letter-drafting assistant slip odd, off-topic phrases into letters; top-p is at 1, which lets any token be picked, so she lowers it to 0.9 and the stray phrases become rare.","da":"En udvikler i en kommunes IT-afdeling ser brevassistenten snige mærkelige, uvedkommende vendinger ind i breve; top-p står på 1, så ethvert token kan vælges, og hun sænker den til 0,9, hvorefter de skæve vendinger bliver sjældne."},"whyItMatters":{"en":"It is a main guard against the rare, odd word choices that make long text drift into nonsense; set badly, it gives either flat, repetitive text or stray words in letters sent to real people.","da":"Det er et af de vigtigste værn mod de sjældne, skæve ordvalg, der får lang tekst til at glide ud i vrøvl; sat forkert giver det enten flad, gentagende tekst eller vildfarne ord i breve til rigtige mennesker."}},"deepDive":{"en":"The algorithm, from Holtzman et al. (\"The Curious Case of Neural Text Degeneration\", ICLR 2020), is short. Sort the vocabulary by probability in descending order, compute the cumulative sum, keep the smallest prefix V(p) whose cumulative probability is at least p, set all other probabilities to zero, renormalise the survivors so they sum to 1, and sample from that truncated distribution. The kept set is called the nucleus. With p = 1 nothing is removed; as p approaches 0 only the single most likely token survives and the method becomes greedy decoding.\n\nIts motivation was a diagnosis of two opposite failures. Likelihood-maximising decoding such as beam search produces generic, repetitive text that falls into loops, while pure sampling from the full softmax regularly draws from the unreliable tail - tens of thousands of tokens that are each unlikely but together carry meaningful probability mass. Top-k sampling truncates the tail at a fixed number of candidates, but no fixed k fits every step: after \"The capital of France is\", almost all mass sits on one token, so k = 40 lets in nonsense, whereas at the start of a creative sentence hundreds of tokens are reasonable and k = 40 is too restrictive. Top-p adapts the candidate count to the shape of the distribution, which is its main advantage.\n\nIn practice p between about 0.9 and 0.95 is a common choice for open-ended text, while many APIs default to 1, which disables truncation and leaves variety to temperature. Because most implementations apply temperature before top-p, the two interact: raising temperature flattens the distribution and enlarges the nucleus, lowering it can shrink the nucleus to one token. Providers therefore advise adjusting one of them and leaving the other at its default, and some APIs reject requests that set both. Top-p can also be combined with top-k as an additional cap, and with repetition penalties.\n\nLimitations: when the model is uncertain and the distribution is flat, the nucleus can still contain hundreds of tokens including poor ones, since top-p only cuts by cumulative mass. Min-p sampling (Nguyen et al., 2024) instead keeps tokens whose probability is at least a set fraction of the top token's, scaling the cut-off with the model's confidence, and is available in several open-source inference engines. Top-p also does not prevent factual errors - a wrong answer can sit squarely inside the nucleus - and it provides no determinism; for reproducible pipelines, greedy decoding or constrained decoding is the relevant control.","da":"Algoritmen fra Holtzman m.fl. (\"The Curious Case of Neural Text Degeneration\", ICLR 2020) er kort. Sortér ordforrådet efter faldende sandsynlighed, beregn den kumulative sum, behold det mindste præfiks V(p), hvis samlede sandsynlighed er mindst p, sæt alle andre sandsynligheder til nul, normalisér de overlevende igen, så de summerer til 1, og træk fra den afskårne fordeling. Den mængde, der beholdes, kaldes kernen (nucleus). Med p = 1 fjernes intet; når p nærmer sig 0, overlever kun det mest sandsynlige token, og metoden bliver til grådig afkodning.\n\nMotivationen var en diagnose af to modsatte fejl. Afkodning, der maksimerer sandsynligheden, som beam search, giver generisk, gentagende tekst, der havner i løkker, mens ren sampling fra hele softmax-fordelingen jævnligt trækker fra den upålidelige hale - titusindvis af tokens, der hver for sig er usandsynlige, men tilsammen har betydelig sandsynlighedsmasse. Top-k-sampling skærer halen af ved et fast antal kandidater, men intet fast k passer til alle trin: Efter \"Frankrigs hovedstad er\" ligger næsten al massen på ét token, så k = 40 lukker vrøvl ind, mens hundredvis af tokens er rimelige i starten af en kreativ sætning, og k = 40 er for snævert. Top-p tilpasser antallet af kandidater til fordelingens form, og det er dens største fordel.\n\nI praksis er p omkring 0,9 til 0,95 et almindeligt valg til åben tekst, mens mange API'er har 1 som standard, hvilket slår afskæringen fra og overlader variationen til temperaturen. Fordi de fleste implementeringer anvender temperaturen før top-p, påvirker de to hinanden: En højere temperatur flader fordelingen ud og gør kernen større, en lavere kan skrumpe kernen til ét token. Udbyderne anbefaler derfor at justere den ene og lade den anden stå på standardværdien, og nogle API'er afviser kald, der sætter begge. Top-p kan også kombineres med top-k som et ekstra loft og med straf for gentagelser.\n\nBegrænsninger: Når modellen er usikker, og fordelingen er flad, kan kernen stadig rumme hundredvis af tokens, herunder dårlige, fordi top-p kun skærer efter samlet masse. Min-p-sampling (Nguyen m.fl., 2024) beholder i stedet tokens, hvis sandsynlighed er mindst en fastsat andel af det mest sandsynlige tokens, så afskæringen følger modellens sikkerhed, og den findes i flere open source-inferensmotorer. Top-p forhindrer heller ikke faktuelle fejl - et forkert svar kan ligge midt i kernen - og giver ingen determinisme; til reproducerbare pipelines er grådig afkodning eller begrænset afkodning det relevante håndtag."},"edges":[{"type":"requires","to":"ai/token","confidence":"high","strength":"normal"},{"type":"kind-of","to":"ai/sampling","why":{"en":"It is one particular rule for the sampling step, limiting the draw to the most likely group of tokens.","da":"Det er én bestemt regel for sampling-trinnet, der begrænser trækningen til den mest sandsynlige gruppe af tokens."},"confidence":"high","strength":"primary"}],"depth":1,"sources":[{"title":"Holtzman et al. (2020), The Curious Case of Neural Text Degeneration","url":"https://arxiv.org/abs/1904.09751","tier":"reference"},{"title":"Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), chapter on large language models","tier":"textbook"}],"draft":true}