{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/attention-mechanism","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/attention-mechanism/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/attention-mechanism/"},"term":{"en":"Attention mechanism","da":"Attention-mekanisme"},"aka":{"en":["attention","self-attention"],"da":["attention","self-attention"]},"domain":["ai"],"cluster":"model-architecture","layer":"model","status":"current","era":2014,"summary":{"en":"The step inside a model that decides, for each token, which other tokens in the input matter most right now.","da":"Det trin i en model, der for hvert token afgør, hvilke andre tokens i inputtet der betyder mest lige nu."},"body":{"formal":{"en":"A method in a neural network where each token's embedding is compared with every other token's, the match scores are turned into shares that add up to one, and the token takes on a mix of the others weighted by those shares.","da":"En metode i et neuralt netværk, hvor hvert tokens embedding sammenlignes med alle andre tokens, matchscorerne laves om til andele, der tilsammen giver én, og tokenet optager en blanding af de andre vægtet efter de andele."},"plain":{"en":"Like following one friend at a noisy party - out of all the voices in the room you pick up the few that matter to what you are hearing and let the rest fade.","da":"Som at følge med i, hvad én ven siger til en larmende fest - af alle stemmerne i rummet opfanger du de få, der har betydning, og lader resten glide i baggrunden."},"inPractice":{"en":"A case officer in a municipality asks a chat assistant whether a 120-page local plan allows a garage; attention lets the question draw directly on the one paragraph, far back in the text, that settles it.","da":"En sagsbehandler i en kommune spørger en chatassistent, om en lokalplan på 120 sider tillader en carport; attention lader spørgsmålet trække direkte på det ene afsnit langt tilbage i teksten, der afgør sagen."},"whyItMatters":{"en":"Every token is compared with every other, so the work grows much faster than the length of the input - the main reason a longer context window costs more time and money.","da":"Hvert token sammenlignes med alle de andre, så arbejdet vokser langt hurtigere end inputtets længde - hovedårsagen til, at et længere kontekstvindue koster mere tid og flere penge."}},"deepDive":{"en":"Attention was introduced by Bahdanau, Cho and Bengio (2014) as an alignment layer for recurrent encoder-decoder translation: instead of squeezing a source sentence into one fixed vector, the decoder computed a weighted average of all encoder states at each output step, with weights from a small feed-forward scoring network (additive attention). Luong et al. (2015) simplified the score to a dot product. Vaswani et al. (2017) then removed recurrence entirely and made attention the main operation of the transformer.\n\nThe transformer form is scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. Each token's hidden state is projected by learned matrices into a query, a key and a value vector; the query of token i is compared with the key of every token j, the scores are divided by √d_k so that the softmax does not saturate as the dimension grows, and the resulting weights mix the value vectors. Multi-head attention runs h such operations in parallel on lower-dimensional projections and concatenates them; the original base model used d_model = 512 with 8 heads of d_k = 64. Self-attention takes Q, K and V from the same sequence; cross-attention takes queries from one sequence (the decoder) and keys and values from another (the encoder output). A causal mask sets scores for future positions to −∞ so a decoder cannot look ahead. Attention itself is permutation-invariant, so position must be injected separately, originally with sinusoidal encodings and today usually with rotary position embeddings (RoPE) or relative biases such as ALiBi.\n\nThe cost is the defining constraint. The score matrix is n × n, so compute and naive memory grow as O(n²·d) in sequence length n. FlashAttention (Dao et al., 2022) keeps the exact result but tiles the computation so the full matrix is never written to GPU main memory, which removes the quadratic memory traffic but not the quadratic arithmetic. During generation the keys and values of earlier tokens are cached (the KV cache), and multi-query attention (Shazeer, 2019) and grouped-query attention (Ainslie et al., 2023) share key/value heads across query heads to shrink that cache. Sparse, sliding-window and linear-attention variants trade exactness or expressiveness for sub-quadratic cost.\n\nTwo misconceptions are common. Attention weights are not a reliable explanation of why a model produced an output: heads interact across many layers and residual paths, and several studies have shown that quite different weight patterns can give the same prediction. And a model that accepts a long context does not use all of it equally well; retrieval accuracy often drops for material in the middle of long inputs, so a large context window is not the same as reliable long-range attention.","da":"Attention blev introduceret af Bahdanau, Cho og Bengio (2014) som et alignment-lag i rekurrente encoder-decoder-modeller til maskinoversættelse: I stedet for at presse en kildesætning ned i én fast vektor beregnede decoderen ved hvert output-trin et vægtet gennemsnit af alle encoderens tilstande, med vægte fra et lille feed-forward-scoringsnetværk (additiv attention). Luong m.fl. (2015) forenklede scoren til et prikprodukt. Vaswani m.fl. (2017) fjernede derefter rekursionen helt og gjorde attention til transformerens centrale operation.\n\nTransformer-varianten er scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. Hvert tokens skjulte tilstand projiceres med lærte matricer til en query-, en key- og en value-vektor; query for token i sammenlignes med key for hvert token j, scorerne divideres med √d_k, så softmax ikke går i mætning, når dimensionen vokser, og de resulterende vægte blander value-vektorerne. Multi-head attention kører h sådanne operationer parallelt på projektioner med lavere dimension og sætter resultaterne sammen; den oprindelige basismodel brugte d_model = 512 med 8 hoveder à d_k = 64. Self-attention tager Q, K og V fra samme sekvens; cross-attention tager queries fra én sekvens (decoderen) og keys og values fra en anden (encoderens output). En kausal maske sætter scorerne for fremtidige positioner til −∞, så en decoder ikke kan kigge frem. Attention er i sig selv permutationsinvariant, så position skal tilføjes separat, oprindeligt med sinusformede kodninger og i dag typisk med rotary position embeddings (RoPE) eller relative bias som ALiBi.\n\nPrisen er den afgørende begrænsning. Scorematricen er n × n, så beregning og naivt hukommelsesforbrug vokser som O(n²·d) i sekvenslængden n. FlashAttention (Dao m.fl., 2022) giver det eksakte resultat, men opdeler beregningen i fliser, så hele matricen aldrig skrives til GPU'ens hovedhukommelse; det fjerner den kvadratiske hukommelsestrafik, men ikke den kvadratiske regnemængde. Under generering caches keys og values for tidligere tokens (KV-cachen), og multi-query attention (Shazeer, 2019) og grouped-query attention (Ainslie m.fl., 2023) deler key/value-hoveder mellem flere query-hoveder for at gøre cachen mindre. Sparse, sliding-window- og lineære attention-varianter bytter eksakthed eller udtrykskraft for en pris under det kvadratiske.\n\nTo misforståelser går igen. Attention-vægte er ikke en pålidelig forklaring på, hvorfor en model gav et bestemt output: Hovederne vekselvirker på tværs af mange lag og residualforbindelser, og flere studier har vist, at ret forskellige vægtmønstre kan give samme forudsigelse. Og at en model accepterer en lang kontekst, betyder ikke, at den udnytter det hele lige godt; genfindingen falder ofte for stof midt i lange input, så et stort kontekstvindue er ikke det samme som pålidelig attention over lange afstande."},"edges":[{"type":"requires","to":"ai/embedding","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/transformer","why":{"en":"Attention is the core building block that a transformer repeats layer after layer.","da":"Attention er den centrale byggesten, som en transformer gentager lag efter lag."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/context-window","why":{"en":"Attention compares every token with every other, so its cost sets how large a context window can practically be.","da":"Attention sammenligner hvert token med alle andre, så dens pris bestemmer, hvor stort et kontekstvindue reelt kan være."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/kv-cache","why":{"en":"A KV cache stores attention results for earlier tokens so they are not worked out again for each new token.","da":"En KV-cache gemmer attention-resultater for tidligere tokens, så de ikke skal regnes ud igen for hvert nyt token."},"confidence":"high","strength":"normal"}],"depth":2,"sources":[{"title":"Bahdanau, Cho & Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate","tier":"reference"},{"title":"Vaswani et al. (2017), Attention Is All You Need","tier":"reference"},{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 12.4)","tier":"textbook","publisher":"MIT Press"}],"draft":true}