KV cache

In a decoder-only transformer, each attention layer projects every token into a query, a key and a value vector. Generating token t requires the attention of its query against the keys and values of all tokens 1…t. Because causal masking means earlier tokens' keys and values never change, they can be computed once and stored. Inference therefore splits into two phases: prefill, which processes the whole prompt in parallel and writes its K and V tensors into the cache, and decode, which produces one token per step, appends one new K/V entry per layer and reads the entire cache. Without the cache, each step would recompute attention inputs for the full prefix, turning linear per-token work into quadratic total work.

The memory cost is easy to estimate: bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per element. For a Llama-2-70B-style model (80 layers, 8 KV heads, head dimension 128) in 16-bit precision this is about 320 KiB per token, so a single 32,000-token context occupies roughly 10 GiB, and a batch of such requests can exceed the size of the weights themselves. Because decode must stream the whole cache from HBM on every step, long contexts make generation memory-bandwidth-bound and limit how many sequences a GPU can serve concurrently.

Most recent architecture and systems work attacks this cost. Multi-query attention (Shazeer, 2019) shares one K/V head across all query heads, and grouped-query attention (Ainslie et al., 2023) uses a small number of shared groups, cutting the cache by the ratio of query heads to KV heads. DeepSeek-V2's multi-head latent attention stores a compressed latent instead of full K/V. Sliding-window attention bounds the cache to the last W tokens, and the cache itself can be quantized to 8-bit or lower, trading some accuracy for capacity. On the systems side, vLLM's PagedAttention (Kwon et al., 2023) stores the cache in fixed-size blocks mapped through a block table, like virtual-memory pages, which removes most fragmentation from over-reserving contiguous buffers and allows copy-on-write sharing of common prefixes between sequences.

Several misconceptions are common. The KV cache is per request and lives only for the duration of generation unless a serving system deliberately keeps it; when it does, across requests, the feature is prefix or prompt caching. It is not a cache of answers, so it does not make identical questions free. Evicting or swapping cache blocks under memory pressure forces recomputation or preemption, which shows up as latency spikes. Finally, a shared cache across tenants creates a timing side channel, so multi-tenant serving systems should isolate cache reuse by customer.