Decoder

In the original transformer (Vaswani et al., 2017) each decoder block had three sublayers: masked multi-head self-attention over the tokens generated so far, cross-attention whose queries come from the decoder and whose keys and values come from the encoder output, and a position-wise feed-forward network, each wrapped in a residual connection and layer normalisation. A decoder-only model, introduced at scale by GPT (Radford et al., 2018, a 12-layer stack of about 117 million parameters), simply drops the cross-attention sublayer; the prompt and the continuation live in one sequence and are processed by the same causal stack.

The causal mask is what makes the stack autoregressive: attention scores for positions j > i are set to −∞ before the softmax, so the representation at position i depends only on tokens 1…i. That allows efficient training with teacher forcing, where all positions of a training sequence are predicted in parallel and the loss is the summed cross-entropy of each next token. At inference the model produces a probability distribution over the vocabulary for the next position, a decoding rule selects a token (greedy argmax, beam search, or sampling with temperature, top-k or top-p/nucleus sampling), the token is appended and the step repeats until an end-of-sequence token or a length limit.

Inference therefore has two phases with different bottlenecks. Prefill processes the whole prompt in one parallel pass and is compute-bound; it largely determines time to first token. Decode generates one token per forward pass and is bound by memory bandwidth, because all weights and the growing KV cache must be read for every token; total latency is roughly time to first token plus output length times time per output token. This is why providers usually price output tokens higher than input tokens. Speculative decoding (Leviathan et al.; Chen et al., 2023) lets a small draft model propose several tokens that the large model verifies in one pass, with an acceptance rule that preserves the large model's output distribution.

Decoder-only designs dominate large language models because a single objective, next-token prediction on raw text, scales well and one architecture covers both understanding and generation. Encoder-decoder models such as T5 and BART remain common where input and output are clearly separate, such as translation, speech recognition (Whisper) and some summarisation. Note that "decoder" is also used for a different thing in autoencoders and in latent diffusion, where it maps a latent code back to pixels; that decoder is not autoregressive.