Sampling

At each decoding step the model produces a vector of logits over the vocabulary. A decoding strategy turns that vector into one chosen token, usually through a chain of logit processors: penalties and biases are applied, temperature rescales the logits, truncation rules such as top-k, top-p or min-p remove unlikely candidates, the remainder is renormalised with a softmax, and one token is drawn using a pseudo-random generator. Deterministic strategies skip the draw. The chosen token is appended, its key and value vectors are added to the KV cache, and the loop repeats until an end-of-sequence token, a stop sequence or the output-token limit.

The strategies form a family. Greedy decoding always picks the argmax. Beam search keeps the k highest-probability partial sequences and was standard in machine translation, but for open-ended generation Holtzman et al. (2020) showed that maximising likelihood produces bland, repetitive and degenerate text, because human text is not the most probable text. Pure sampling from the full distribution has the opposite problem: the long tail of individually unlikely tokens is collectively likely to be hit, and one bad token derails what follows. Truncation methods address this: top-k (used by Fan et al., 2018) keeps a fixed number of candidates, nucleus or top-p keeps the smallest set whose cumulative probability reaches p, and min-p (Nguyen et al., 2024) keeps tokens whose probability is at least a fraction of the top token's. Frequency and presence penalties discourage repetition, logit bias forces or bans specific tokens, and constrained decoding masks out tokens that would violate a grammar or JSON Schema.

Reproducibility is weaker than the settings suggest. Temperature 0 or greedy decoding removes intentional randomness, but hosted inference can still return different outputs for the same request. Floating-point addition is not associative, and GPU kernels choose different reduction orders depending on batch size and other load-dependent factors, so logits can differ in the last bits between runs and flip a near-tie. Thinking Machines Lab (2025) showed that batch-invariant kernels make repeated runs bit-identical at a throughput cost. Some APIs offer a seed parameter, but providers describe it as best-effort; mixture-of-experts routing and speculative decoding add further sources of variation.

Practical guidance: use greedy or low temperature for extraction, classification and code where one correct answer exists; moderate temperature with top-p around 0.9 to 0.95 for prose; multiple samples plus voting or verification (self-consistency, best-of-n with a grader) when accuracy matters more than cost. Change one parameter at a time, evaluate over several samples per test case, and record decoding parameters alongside prompts so results can be reproduced as far as the platform allows.