{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/transformer","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/transformer/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/transformer/"},"term":{"en":"Transformer","da":"Transformer"},"aka":{"en":["transformer architecture"],"da":["transformer-arkitektur"]},"domain":["ai"],"cluster":"llm","layer":"model","status":"current","era":2017,"summary":{"en":"The neural network design behind today's language models, which weighs how every word in a text relates to every other word.","da":"Det neurale netværksdesign bag nutidens sprogmodeller, som vejer, hvordan hvert ord i en tekst hænger sammen med alle de andre."},"body":{"formal":{"en":"A neural network design from 2017 built around \"attention\", where the model scores how much each token should draw on every other token, and handles all tokens at once rather than one after another.","da":"Et neuralt netværksdesign fra 2017 bygget op om \"attention\", hvor modellen vurderer, hvor meget hvert token skal trække på alle andre tokens, og behandler alle tokens på én gang i stedet for ét ad gangen."},"plain":{"en":"Like reading a whole page at a glance and drawing lines between the words that belong together, instead of reading one word at a time.","da":"Som at læse en hel side på én gang og tegne streger mellem de ord, der hører sammen, i stedet for at læse ét ord ad gangen."},"inPractice":{"en":"When a ministry's translation tool handles “the bank refused the loan because it was too risky”, the transformer links “it” to “the loan”, so the Danish translation uses the word for “it” that fits the loan, not the bank.","da":"Når et ministeriums oversættelsesværktøj får en engelsk sætning om en bank, der afviste et lån, fordi “it” var for risikabelt, kobler transformeren “it” til lånet og ikke til banken, så det på dansk bliver “det” og ikke “den”."},"whyItMatters":{"en":"Because it can be trained quickly on huge amounts of text, it made large language models possible - and with them most of today's AI tools.","da":"Fordi den kan trænes hurtigt på enorme mængder tekst, gjorde den store sprogmodeller mulige - og dermed de fleste af nutidens AI-værktøjer."}},"deepDive":{"en":"The core operation is scaled dot-product attention: each token's vector is projected into a query, a key and a value, and the output is softmax(QKᵀ / √d_k) · V, a weighted average of all values where the weights come from query-key similarity. Dividing by the square root of the key dimension keeps the dot products from saturating the softmax. Multi-head attention runs several of these in parallel on lower-dimensional projections and concatenates the results, letting different heads specialise (syntactic agreement, coreference, copying). Each block adds a position-wise feed-forward network, and both sub-layers are wrapped in residual connections with layer normalisation.\n\nVaswani et al. (2017) proposed an encoder-decoder model for machine translation: six encoder and six decoder layers, model dimension 512, eight heads, feed-forward width 2048, sinusoidal positional encodings, with the decoder using masked self-attention plus cross-attention to the encoder output. Attention itself is permutation-invariant, so position must be injected; later models replaced fixed sinusoids with learned embeddings, then relative schemes such as rotary position embeddings (RoPE) and ALiBi, which extend better to long sequences. Three families followed: encoder-only models such as BERT (bidirectional, used for classification and embeddings), decoder-only models such as GPT (causal mask, used for generation and now dominant for LLMs), and encoder-decoder models such as T5. The same block also underlies vision transformers, speech models and multimodal models, which convert patches or audio frames into token-like vectors.\n\nIts advantage over recurrent networks is parallelism in training: every position is processed at once, and the path between any two tokens is a single attention step rather than a chain of recurrent updates, which eases learning long-range dependencies and scales efficiently on GPUs and TPUs. The cost is that attention compute grows quadratically with sequence length. Engineering responses include FlashAttention, which tiles the computation to avoid materialising the full attention matrix, sliding-window and sparse attention, grouped-query and multi-query attention to shrink the KV cache, and mixture-of-experts feed-forward layers to add parameters without proportional compute. Alternative architectures such as state-space models (for example Mamba) aim for linear scaling and are often combined with attention layers in hybrids.\n\nTwo common misconceptions: attention weights are not a reliable explanation of why a model produced an output, because information is also mixed through residual streams and feed-forward layers across many layers; and \"transformer\" names the architecture, not the training objective - the same design is trained as a masked-language model, a next-token predictor or a contrastive encoder depending on the task.","da":"Kerneoperationen er skaleret prikprodukt-attention: Hvert tokens vektor projiceres til en query, en key og en value, og outputtet er softmax(QKᵀ / √d_k) · V, et vægtet gennemsnit af alle values, hvor vægtene kommer af ligheden mellem query og key. Divisionen med kvadratroden af key-dimensionen forhindrer prikprodukterne i at mætte softmax-funktionen. Multi-head attention kører flere af disse parallelt på lavere-dimensionelle projektioner og sætter resultaterne sammen, så forskellige hoveder kan specialisere sig (syntaktisk kongruens, henvisninger, kopiering). Hver blok tilføjer et positionsvist feed-forward-netværk, og begge dellag er pakket ind i residualforbindelser med lagnormalisering.\n\nVaswani m.fl. (2017) foreslog en encoder-decoder-model til maskinoversættelse: seks encoder- og seks decoder-lag, modeldimension 512, otte hoveder, feed-forward-bredde 2048 og sinusformede positionskodninger, hvor decoderen brugte maskeret self-attention plus cross-attention til encoderens output. Attention i sig selv er ligeglad med rækkefølge, så positionen skal tilføres; senere modeller erstattede de faste sinuskurver med lærte embeddings og siden relative metoder som rotary position embeddings (RoPE) og ALiBi, der klarer lange sekvenser bedre. Tre familier fulgte: encoder-only-modeller som BERT (tovejs, brugt til klassifikation og embeddings), decoder-only-modeller som GPT (kausal maske, brugt til generering og nu dominerende for LLM'er) og encoder-decoder-modeller som T5. Den samme blok ligger også bag vision transformers, talemodeller og multimodale modeller, der omsætter billedfelter eller lydrammer til token-lignende vektorer.\n\nFordelen frem for rekurrente netværk er parallelitet i træningen: Alle positioner behandles på én gang, og vejen mellem to vilkårlige tokens er ét attention-trin i stedet for en kæde af rekurrente opdateringer, hvilket gør det lettere at lære afhængigheder over lange afstande og skalerer effektivt på GPU'er og TPU'er. Prisen er, at regnearbejdet for attention vokser kvadratisk med sekvenslængden. Ingeniørsvarene omfatter FlashAttention, der deler beregningen i fliser for at undgå at materialisere hele attention-matricen, sliding-window- og sparse attention, grouped-query og multi-query attention for at gøre KV-cachen mindre, og mixture of experts i feed-forward-lagene for at tilføje parametre uden tilsvarende regnearbejde. Alternative arkitekturer som state-space-modeller (fx Mamba) sigter mod lineær skalering og kombineres ofte med attention-lag i hybrider.\n\nTo udbredte misforståelser: Attention-vægte er ikke en pålidelig forklaring på, hvorfor en model gav et bestemt output, fordi information også blandes via residualstrømme og feed-forward-lag på tværs af mange lag; og \"transformer\" betegner arkitekturen, ikke træningsmålet - samme design trænes som maskeret sprogmodel, som forudsigelse af næste token eller som kontrastiv encoder afhængigt af opgaven."},"edges":[{"type":"requires","to":"ai/token","confidence":"high","strength":"normal"},{"type":"kind-of","to":"ai/neural-network","confidence":"high","strength":"normal"}],"depth":1,"sources":[{"title":"Vaswani et al. (2017), Attention Is All You Need","tier":"reference"},{"title":"Goodfellow, Bengio & Courville, Deep Learning","tier":"textbook","publisher":"MIT Press"}],"draft":true}