{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/backpropagation","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/backpropagation/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/backpropagation/"},"term":{"en":"Backpropagation","da":"Backpropagation"},"aka":{"en":["backprop","backward pass"],"da":["backprop","tilbagepropagering"]},"domain":["ai"],"cluster":"training","layer":"training","status":"current","era":1986,"summary":{"en":"The bookkeeping method that works out, layer by layer from the output back, how much each weight in a network added to its mistake.","da":"Den regnemetode, der lag for lag fra output og bagud finder ud af, hvor meget hver vægt i et netværk bidrog til fejlen."},"body":{"formal":{"en":"A method for computing, in one backward sweep through a neural network, how the loss would change if each weight changed; gradient descent then uses those numbers to update the model weights.","da":"En metode til i én baglæns gennemgang af et neuralt netværk at beregne, hvordan tabet ville ændre sig, hvis hver vægt ændrede sig; gradientnedstigning bruger derefter tallene til at opdatere modelvægtene."},"plain":{"en":"Like a kitchen following a burnt dish back through every cook who touched it, deciding how much blame each one carries, so each knows how much to change next time.","da":"Som et køkken, der sporer en brændt ret tilbage gennem hver kok, der rørte den, og afgør, hvor meget skyld hver bærer, så alle ved, hvor meget de skal ændre næste gang."},"inPractice":{"en":"A developer at a Danish logistics firm trains a network to read handwritten house numbers on parcels; when it reads a 7 as a 1, backpropagation passes that error back through every layer and gives each weight its share of the blame.","da":"En udvikler i en dansk logistikvirksomhed træner et netværk til at læse håndskrevne husnumre på pakker; når det læser et 7-tal som et 1-tal, sender backpropagation fejlen tilbage gennem alle lag og giver hver vægt sin del af skylden."},"whyItMatters":{"en":"Without a cheap way to share out blame, networks with many layers would take far too long to train; this method, together with fast hardware, is what made deep learning practical.","da":"Uden en billig måde at fordele skylden på ville netværk med mange lag tage alt for lang tid at træne; metoden er sammen med hurtig hardware det, der gjorde deep learning praktisk mulig."}},"deepDive":{"en":"Backpropagation is reverse-mode automatic differentiation applied to a network's computational graph. The forward pass computes and stores each layer's pre-activations z and activations a; the backward pass then applies the chain rule from the scalar loss towards the input. For a fully connected layer l the error signal is δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ), and the weight gradient is ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ. Because the loss is a single scalar, one backward sweep yields the partial derivative for every parameter at a cost of only a small constant multiple of the forward pass, whereas finite differences would need one extra forward pass per parameter, which means billions of passes for a modern model. A widely used rule of thumb puts training compute at about 6N FLOPs per token for a model with N parameters: roughly 2N forward and 4N backward.\n\nThe idea predates its fame. Seppo Linnainmaa described reverse-mode differentiation in 1970, Paul Werbos proposed applying it to neural networks in his 1974 thesis, and Rumelhart, Hinton and Williams popularised it in Nature in 1986 by showing that it learns useful internal representations in hidden layers. A frequent misconception is that backpropagation is the learning algorithm; it only computes gradients, and an optimiser such as SGD or Adam decides how to change the weights with them.\n\nThe main engineering cost is memory. Every intermediate activation needed for the backward pass must be kept until it is used, so memory grows with depth, batch size and sequence length. Activation (gradient) checkpointing stores only some activations and recomputes the rest during the backward pass, cutting memory to roughly the square root of the number of layers at the price of an extra partial forward pass (Chen et al., 2016). Mixed-precision training uses loss scaling so that small FP16 gradients do not underflow to zero. In PyTorch, loss.backward() walks a dynamically recorded graph and accumulates into each parameter's .grad, which is why gradients must be zeroed between steps unless accumulation is intended.\n\nGradients multiplied through many layers can shrink or grow exponentially. Vanishing gradients, analysed by Hochreiter (1991) and Bengio et al. (1994), made deep sigmoid networks and recurrent networks trained by backpropagation through time hard to train; exploding gradients cause divergence. The standard remedies (ReLU-family activations, variance-preserving initialisation (Glorot, He), residual connections, normalisation layers, LSTM gating and gradient-norm clipping) are all ways of keeping the backward signal well conditioned. Operations with no useful derivative, such as sampling or rounding, need surrogates like the straight-through estimator or score-function (REINFORCE) gradients.","da":"Backpropagation er automatisk differentiation i baglæns tilstand (reverse mode) anvendt på et netværks beregningsgraf. Det forlæns pass beregner og gemmer hvert lags præaktiveringer z og aktiveringer a; det baglæns pass anvender derefter kædereglen fra det skalare tab mod inputtet. For et fuldt forbundet lag l er fejlsignalet δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ), og vægtgradienten er ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ. Fordi tabet er én skalar, giver ét baglæns gennemløb den partielle afledede for hver eneste parameter til en pris, der kun er et lille konstant multiplum af det forlæns pass, mens endelige differenser ville kræve et ekstra forlæns pass pr. parameter - milliarder af gennemløb for en moderne model. En udbredt tommelfingerregel sætter træningsberegningen til cirka 6N FLOP pr. token for en model med N parametre: omkring 2N forlæns og 4N baglæns.\n\nIdéen er ældre end dens berømmelse. Seppo Linnainmaa beskrev differentiation i baglæns tilstand i 1970, Paul Werbos foreslog at bruge den på neurale netværk i sin afhandling fra 1974, og Rumelhart, Hinton og Williams gjorde den kendt i Nature i 1986 ved at vise, at den lærer nyttige interne repræsentationer i skjulte lag. En udbredt misforståelse er, at backpropagation er selve læringsalgoritmen; den beregner kun gradienter, og en optimeringsalgoritme som SGD eller Adam afgør, hvordan vægtene ændres ud fra dem.\n\nDen største tekniske omkostning er hukommelse. Hver mellemliggende aktivering, som det baglæns pass skal bruge, må gemmes, til den bruges, så hukommelsen vokser med dybde, batchstørrelse og sekvenslængde. Activation (gradient) checkpointing gemmer kun nogle aktiveringer og genberegner resten under det baglæns pass, hvilket skærer hukommelsen ned til omtrent kvadratroden af antallet af lag mod prisen af et ekstra delvist forlæns pass (Chen m.fl., 2016). Træning med blandet præcision bruger loss scaling, så små FP16-gradienter ikke underflower til nul. I PyTorch gennemløber loss.backward() en dynamisk optaget graf og akkumulerer i hver parameters .grad, og derfor skal gradienterne nulstilles mellem skridt, medmindre akkumulering er tilsigtet.\n\nGradienter, der ganges gennem mange lag, kan skrumpe eller vokse eksponentielt. Forsvindende gradienter, analyseret af Hochreiter (1991) og Bengio m.fl. (1994), gjorde dybe sigmoid-netværk og rekurrente netværk trænet med backpropagation through time svære at træne; eksploderende gradienter får træningen til at divergere. Standardmodtrækkene - aktiveringer i ReLU-familien, varians-bevarende initialisering (Glorot, He), residualforbindelser, normaliseringslag, LSTM-gates og klipning af gradientnormen - er alle måder at holde det baglæns signal velkonditioneret. Operationer uden brugbar afledt, som sampling eller afrunding, kræver surrogater som straight-through-estimatoren eller score-function-gradienter (REINFORCE)."},"edges":[{"type":"requires","to":"ai/neural-network","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/gradient-descent","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/model-weights","why":{"en":"Its whole output is one number per weight saying which way that weight should move.","da":"Hele dens resultat er ét tal pr. vægt, der siger, hvilken vej vægten skal flyttes."},"confidence":"high","strength":"normal"}],"depth":4,"sources":[{"title":"Rumelhart, Hinton & Williams (1986), Learning representations by back-propagating errors","url":"https://doi.org/10.1038/323533a0","tier":"reference","publisher":"Nature 323, 533-536"},{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 6.5)","url":"https://www.deeplearningbook.org/contents/mlp.html","tier":"textbook","publisher":"MIT Press"},{"title":"Chen et al. (2016), Training Deep Nets with Sublinear Memory Cost","url":"https://arxiv.org/abs/1604.06174","tier":"reference"},{"title":"Kaplan et al. (2020), Scaling Laws for Neural Language Models","url":"https://arxiv.org/abs/2001.08361","tier":"reference"}],"draft":true}