Next-token prediction

Formally, an autoregressive language model factorises the probability of a sequence as a product of conditionals, p(x1, ..., xn) = p(x1) · p(x2 | x1) · ... · p(xn | x1, ..., xn−1). At each position the network outputs a vector of logits, one per vocabulary entry (tens to hundreds of thousands of them), and a softmax turns these into a probability distribution over the next token. Training minimises the average negative log-likelihood of the actual next token, the cross-entropy loss; perplexity, the exponential of that loss, is the standard intrinsic metric.

Training is massively parallel because of teacher forcing and the causal mask. The whole ground-truth sequence is fed in at once, and the attention mask prevents position t from seeing positions after t, so one forward pass yields a loss term at every position simultaneously. Generation cannot be parallelised this way: each new token depends on the previous one, so producing n tokens requires n sequential forward passes. A KV cache avoids recomputing attention keys and values for earlier positions, making each step cost roughly proportional to the current length rather than recomputing the whole prefix, but the sequential dependency remains the main source of latency.

Several techniques attack that cost without changing the objective. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) lets a small draft model propose several tokens that the large model verifies in a single pass, accepting them with a rule that preserves the large model's output distribution exactly. Multi-token prediction heads (Gloeckle et al., 2024) train the model to predict several future tokens at once, which can be used for faster decoding. Diffusion-style language models replace left-to-right generation altogether but remain a minority approach.

The objective explains several documented quirks. Because each step conditions only on what came before, an early error is never revised, only continued; the model can talk itself into a wrong answer, which chain-of-thought and reasoning training partly exploit and partly counter. The "reversal curse" (Berglund et al., 2023) showed that models trained on "A is B" often fail to answer "B is A", a consequence of learning conditional continuations rather than symmetric facts. Next-token prediction is also why a model's confidence is not calibrated truth: a high-probability token is one that fits the pattern, which is the direct link to hallucination. Selecting the actual token from the distribution is a separate step, sampling, controlled by temperature, top-p and related settings.