Transformer

The core operation is scaled dot-product attention: each token's vector is projected into a query, a key and a value, and the output is softmax(QKᵀ / √d_k) · V, a weighted average of all values where the weights come from query-key similarity. Dividing by the square root of the key dimension keeps the dot products from saturating the softmax. Multi-head attention runs several of these in parallel on lower-dimensional projections and concatenates the results, letting different heads specialise (syntactic agreement, coreference, copying). Each block adds a position-wise feed-forward network, and both sub-layers are wrapped in residual connections with layer normalisation.

Vaswani et al. (2017) proposed an encoder-decoder model for machine translation: six encoder and six decoder layers, model dimension 512, eight heads, feed-forward width 2048, sinusoidal positional encodings, with the decoder using masked self-attention plus cross-attention to the encoder output. Attention itself is permutation-invariant, so position must be injected; later models replaced fixed sinusoids with learned embeddings, then relative schemes such as rotary position embeddings (RoPE) and ALiBi, which extend better to long sequences. Three families followed: encoder-only models such as BERT (bidirectional, used for classification and embeddings), decoder-only models such as GPT (causal mask, used for generation and now dominant for LLMs), and encoder-decoder models such as T5. The same block also underlies vision transformers, speech models and multimodal models, which convert patches or audio frames into token-like vectors.

Its advantage over recurrent networks is parallelism in training: every position is processed at once, and the path between any two tokens is a single attention step rather than a chain of recurrent updates, which eases learning long-range dependencies and scales efficiently on GPUs and TPUs. The cost is that attention compute grows quadratically with sequence length. Engineering responses include FlashAttention, which tiles the computation to avoid materialising the full attention matrix, sliding-window and sparse attention, grouped-query and multi-query attention to shrink the KV cache, and mixture-of-experts feed-forward layers to add parameters without proportional compute. Alternative architectures such as state-space models (for example Mamba) aim for linear scaling and are often combined with attention layers in hybrids.

Two common misconceptions: attention weights are not a reliable explanation of why a model produced an output, because information is also mixed through residual streams and feed-forward layers across many layers; and "transformer" names the architecture, not the training objective - the same design is trained as a masked-language model, a next-token predictor or a contrastive encoder depending on the task.