{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/kv-cache","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/kv-cache/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/kv-cache/"},"term":{"en":"KV cache","da":"KV-cache"},"aka":{"en":["key-value cache"],"da":["key-value-cache"]},"domain":["ai"],"cluster":"ai-infrastructure","layer":"inference","status":"current","summary":{"en":"Memory where a language model keeps work it already did on earlier tokens, so each new word does not mean rereading everything.","da":"Hukommelse, hvor en sprogmodel gemmer arbejdet med tidligere tokens, så hvert nyt ord ikke kræver, at alt læses forfra."},"body":{"formal":{"en":"During inference in a transformer, the stored per-token numbers (called keys and values) that the attention mechanism compares against, for every token read so far; each new token adds its own and reuses the rest, so the store grows with the text.","da":"Under inferens i en transformer de gemte tal pr. token (kaldet keys og values), som attention-mekanismen sammenligner med, for hvert token læst indtil nu; hvert nyt token tilføjer sine egne og genbruger resten, så lageret vokser med teksten."},"plain":{"en":"Like keeping notes while reading a long book aloud - to say the next sentence you glance at your notes instead of starting again from page one.","da":"Som at tage noter, mens man læser en lang bog højt - for at sige næste sætning kigger man i noterne i stedet for at begynde forfra fra side ét."},"inPractice":{"en":"A Danish software house runs a legal chat tool for law firms; its operations engineer finds the GPU fills up not with model weights but with the KV cache of lawyers' long chats, which limits how many can use it at once.","da":"Et dansk softwarehus driver et juridisk chatværktøj for advokatfirmaer; driftsingeniøren opdager, at GPU'en ikke fyldes af modelvægte, men af KV-cache fra advokaternes lange samtaler, hvilket begrænser, hvor mange der kan bruge den samtidig."},"whyItMatters":{"en":"It is what makes writing long answers fast enough to use, and its memory cost is often the real limit on context length, speed and price.","da":"Den er det, der gør det hurtigt nok at skrive lange svar, og dens hukommelsesforbrug er ofte den reelle grænse for kontekstlængde, hastighed og pris."}},"deepDive":{"en":"In a decoder-only transformer, each attention layer projects every token into a query, a key and a value vector. Generating token t requires the attention of its query against the keys and values of all tokens 1…t. Because causal masking means earlier tokens' keys and values never change, they can be computed once and stored. Inference therefore splits into two phases: prefill, which processes the whole prompt in parallel and writes its K and V tensors into the cache, and decode, which produces one token per step, appends one new K/V entry per layer and reads the entire cache. Without the cache, each step would recompute attention inputs for the full prefix, turning linear per-token work into quadratic total work.\n\nThe memory cost is easy to estimate: bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per element. For a Llama-2-70B-style model (80 layers, 8 KV heads, head dimension 128) in 16-bit precision this is about 320 KiB per token, so a single 32,000-token context occupies roughly 10 GiB, and a batch of such requests can exceed the size of the weights themselves. Because decode must stream the whole cache from HBM on every step, long contexts make generation memory-bandwidth-bound and limit how many sequences a GPU can serve concurrently.\n\nMost recent architecture and systems work attacks this cost. Multi-query attention (Shazeer, 2019) shares one K/V head across all query heads, and grouped-query attention (Ainslie et al., 2023) uses a small number of shared groups, cutting the cache by the ratio of query heads to KV heads. DeepSeek-V2's multi-head latent attention stores a compressed latent instead of full K/V. Sliding-window attention bounds the cache to the last W tokens, and the cache itself can be quantized to 8-bit or lower, trading some accuracy for capacity. On the systems side, vLLM's PagedAttention (Kwon et al., 2023) stores the cache in fixed-size blocks mapped through a block table, like virtual-memory pages, which removes most fragmentation from over-reserving contiguous buffers and allows copy-on-write sharing of common prefixes between sequences.\n\nSeveral misconceptions are common. The KV cache is per request and lives only for the duration of generation unless a serving system deliberately keeps it; when it does, across requests, the feature is prefix or prompt caching. It is not a cache of answers, so it does not make identical questions free. Evicting or swapping cache blocks under memory pressure forces recomputation or preemption, which shows up as latency spikes. Finally, a shared cache across tenants creates a timing side channel, so multi-tenant serving systems should isolate cache reuse by customer.","da":"I en decoder-only-transformer projicerer hvert attention-lag hvert token til en query-, en key- og en value-vektor. For at generere token t skal dets query sammenlignes med keys og values for alle tokens 1…t. Da kausal maskering betyder, at tidligere tokens' keys og values aldrig ændrer sig, kan de beregnes én gang og gemmes. Inferens deles derfor i to faser: prefill, der behandler hele prompten parallelt og skriver dens K- og V-tensorer i cachen, og decode, der frembringer ét token pr. trin, tilføjer én ny K/V-post pr. lag og læser hele cachen. Uden cachen skulle hvert trin genberegne attention-input for hele præfikset, så lineært arbejde pr. token blev til kvadratisk samlet arbejde.\n\nHukommelsesforbruget er let at estimere: bytes pr. token = 2 (K og V) × antal lag × antal KV-heads × head-dimension × bytes pr. element. For en model af typen Llama-2-70B (80 lag, 8 KV-heads, head-dimension 128) med 16-bit-præcision giver det omkring 320 KiB pr. token, så en enkelt kontekst på 32.000 tokens fylder cirka 10 GiB, og en batch af sådanne forespørgsler kan fylde mere end selve vægtene. Fordi decode skal læse hele cachen fra HBM i hvert trin, gør lange kontekster genereringen begrænset af hukommelsesbåndbredden og begrænser, hvor mange sekvenser en GPU kan betjene samtidig.\n\nDet meste af det nyere arbejde med arkitektur og systemer går efter netop denne omkostning. Multi-query attention (Shazeer, 2019) deler ét K/V-head mellem alle query-heads, og grouped-query attention (Ainslie m.fl., 2023) bruger et lille antal fælles grupper, hvilket mindsker cachen med forholdet mellem query-heads og KV-heads. DeepSeek-V2's multi-head latent attention gemmer en komprimeret latent repræsentation i stedet for fulde K/V. Sliding-window attention begrænser cachen til de seneste W tokens, og selve cachen kan kvantiseres til 8 bit eller mindre, så man bytter lidt nøjagtighed for kapacitet. På systemsiden gemmer vLLM's PagedAttention (Kwon m.fl., 2023) cachen i blokke af fast størrelse, der slås op via en bloktabel ligesom sider i virtuel hukommelse; det fjerner det meste af den fragmentering, der opstår, når man reserverer sammenhængende buffere på forhånd, og gør det muligt at dele fælles præfikser mellem sekvenser med copy-on-write.\n\nFlere misforståelser går igen. KV-cachen hører til den enkelte forespørgsel og lever kun, mens der genereres, medmindre et serving-system bevidst gemmer den; gør det det på tværs af forespørgsler, hedder funktionen prefix caching eller prompt caching. Den er ikke en cache af svar, så identiske spørgsmål bliver ikke gratis. Når cacheblokke smides ud eller swappes under hukommelsespres, må de genberegnes, eller forespørgsler må sættes på pause, hvilket ses som spidser i latensen. Endelig skaber en cache, der deles mellem kunder, en timing-sidekanal, så serving-systemer med flere lejere bør isolere genbrug af cache pr. kunde."},"edges":[{"type":"requires","to":"ai/attention-mechanism","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/token","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/inference","why":{"en":"It only exists while a model is producing text, holding work between one token and the next.","da":"Den findes kun, mens en model frembringer tekst, og holder på arbejdet mellem ét token og det næste."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/model-serving","why":{"en":"Serving systems spend much of their effort fitting and sharing KV cache memory so more users fit on each GPU.","da":"Serving-systemer bruger en stor del af deres kræfter på at udnytte og dele KV-cache-hukommelse, så flere brugere kan være på hver GPU."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/quantization","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/throughput","confidence":"high","strength":"normal"}],"depth":3,"sources":[{"title":"Pope et al. (2022), Efficiently Scaling Transformer Inference","tier":"reference"},{"title":"Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention","tier":"reference"}],"draft":true}