Context window

The context window is the maximum sequence length, in tokens, over which a transformer computes attention in one forward pass. Input and output share the same budget: a model with a 200,000-token window that is asked for up to 8,000 output tokens can accept at most 192,000 tokens of input, and APIs reject or truncate requests that exceed it. Everything counts - system prompt, tool definitions, earlier turns, retrieved documents, images converted to tokens and, for reasoning models, the hidden reasoning tokens generated during the current turn.

The limit has three technical sources. Self-attention compares every token with every other token, so naive compute and memory grow quadratically with sequence length; kernels such as FlashAttention (Dao et al., 2022) reduce memory traffic but not the quadratic compute. Positional information must generalise: models trained with rotary position embeddings (RoPE) or ALiBi are usually pretrained on shorter sequences and then extended by continued training on long documents combined with rescaling of the positional frequencies. Finally, during generation the KV cache stores a key and a value vector per token, per layer and per KV head, so its size grows linearly with context length and often dominates GPU memory; grouped-query attention and cache quantisation exist largely to shrink it.

Advertised length and usable length are not the same. Liu et al. ("Lost in the Middle", 2023) showed that retrieval accuracy on multi-document question answering drops when the relevant passage sits in the middle of a long context rather than near the start or end, and synthetic benchmarks such as RULER report that many models' effective context is well below the nominal figure once tasks require more than finding a single needle. Anthropic describes the general effect as context rot: recall degrades as token count grows. More context is therefore not free even when it fits.

Applications manage the window explicitly. Chat front-ends drop or summarise the oldest turns (the failure in this term's example), agents compact long histories into summaries or offload notes to external memory, and RAG systems insert only the top-ranked chunks. Prompt caching reuses the computed prefix across calls to cut latency and cost but does not enlarge the window. The context window is also distinct from memory features that persist between sessions: those work by writing text to storage and re-inserting it into a later context, so they consume the same budget and inherit the same prompt-injection exposure as any other text placed there.