Top-p sampling

The algorithm, from Holtzman et al. ("The Curious Case of Neural Text Degeneration", ICLR 2020), is short. Sort the vocabulary by probability in descending order, compute the cumulative sum, keep the smallest prefix V(p) whose cumulative probability is at least p, set all other probabilities to zero, renormalise the survivors so they sum to 1, and sample from that truncated distribution. The kept set is called the nucleus. With p = 1 nothing is removed; as p approaches 0 only the single most likely token survives and the method becomes greedy decoding.

Its motivation was a diagnosis of two opposite failures. Likelihood-maximising decoding such as beam search produces generic, repetitive text that falls into loops, while pure sampling from the full softmax regularly draws from the unreliable tail - tens of thousands of tokens that are each unlikely but together carry meaningful probability mass. Top-k sampling truncates the tail at a fixed number of candidates, but no fixed k fits every step: after "The capital of France is", almost all mass sits on one token, so k = 40 lets in nonsense, whereas at the start of a creative sentence hundreds of tokens are reasonable and k = 40 is too restrictive. Top-p adapts the candidate count to the shape of the distribution, which is its main advantage.

In practice p between about 0.9 and 0.95 is a common choice for open-ended text, while many APIs default to 1, which disables truncation and leaves variety to temperature. Because most implementations apply temperature before top-p, the two interact: raising temperature flattens the distribution and enlarges the nucleus, lowering it can shrink the nucleus to one token. Providers therefore advise adjusting one of them and leaving the other at its default, and some APIs reject requests that set both. Top-p can also be combined with top-k as an additional cap, and with repetition penalties.

Limitations: when the model is uncertain and the distribution is flat, the nucleus can still contain hundreds of tokens including poor ones, since top-p only cuts by cumulative mass. Min-p sampling (Nguyen et al., 2024) instead keeps tokens whose probability is at least a set fraction of the top token's, scaling the cut-off with the model's confidence, and is available in several open-source inference engines. Top-p also does not prevent factual errors - a wrong answer can sit squarely inside the nucleus - and it provides no determinism; for reproducible pipelines, greedy decoding or constrained decoding is the relevant control.