Attention mechanism

Attention was introduced by Bahdanau, Cho and Bengio (2014) as an alignment layer for recurrent encoder-decoder translation: instead of squeezing a source sentence into one fixed vector, the decoder computed a weighted average of all encoder states at each output step, with weights from a small feed-forward scoring network (additive attention). Luong et al. (2015) simplified the score to a dot product. Vaswani et al. (2017) then removed recurrence entirely and made attention the main operation of the transformer.

The transformer form is scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. Each token's hidden state is projected by learned matrices into a query, a key and a value vector; the query of token i is compared with the key of every token j, the scores are divided by √d_k so that the softmax does not saturate as the dimension grows, and the resulting weights mix the value vectors. Multi-head attention runs h such operations in parallel on lower-dimensional projections and concatenates them; the original base model used d_model = 512 with 8 heads of d_k = 64. Self-attention takes Q, K and V from the same sequence; cross-attention takes queries from one sequence (the decoder) and keys and values from another (the encoder output). A causal mask sets scores for future positions to −∞ so a decoder cannot look ahead. Attention itself is permutation-invariant, so position must be injected separately, originally with sinusoidal encodings and today usually with rotary position embeddings (RoPE) or relative biases such as ALiBi.

The cost is the defining constraint. The score matrix is n × n, so compute and naive memory grow as O(n²·d) in sequence length n. FlashAttention (Dao et al., 2022) keeps the exact result but tiles the computation so the full matrix is never written to GPU main memory, which removes the quadratic memory traffic but not the quadratic arithmetic. During generation the keys and values of earlier tokens are cached (the KV cache), and multi-query attention (Shazeer, 2019) and grouped-query attention (Ainslie et al., 2023) share key/value heads across query heads to shrink that cache. Sparse, sliding-window and linear-attention variants trade exactness or expressiveness for sub-quadratic cost.

Two misconceptions are common. Attention weights are not a reliable explanation of why a model produced an output: heads interact across many layers and residual paths, and several studies have shown that quite different weight patterns can give the same prediction. And a model that accepts a long context does not use all of it equally well; retrieval accuracy often drops for material in the middle of long inputs, so a large context window is not the same as reliable long-range attention.