{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/gradient-descent","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/gradient-descent/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/gradient-descent/"},"term":{"en":"Gradient descent","da":"Gradientnedstigning (gradient descent)"},"aka":{"en":["stochastic gradient descent"],"da":["gradient descent"]},"domain":["ai"],"cluster":"training","layer":"training","status":"current","summary":{"en":"The step-by-step method most models learn by, which nudges every setting a little in whichever direction makes the error shrink.","da":"Den trinvise metode, de fleste modeller lærer med - skub hver indstilling en smule i den retning, der gør fejlen mindre."},"body":{"formal":{"en":"A method that lowers the value of a loss function by repeatedly working out which way each model parameter should move to reduce the loss, then moving all of them a small step that way; the size of the step is a hyperparameter.","da":"En metode, der sænker værdien af en tabsfunktion ved gentagne gange at regne ud, hvilken vej hver modelparameter skal flyttes for at mindske tabet, og så flytte dem alle et lille skridt i den retning; skridtets størrelse er en hyperparameter."},"plain":{"en":"Like walking down a foggy hill with your eyes closed; you feel which way the ground slopes under your feet and take a small step downhill, again and again, until it is flat.","da":"Som at gå ned ad en tåget bakke med lukkede øjne - du mærker, hvilken vej jorden hælder under fødderne, og tager et lille skridt nedad, igen og igen, til det bliver fladt."},"inPractice":{"en":"An analyst in a ministry trains a model to sort incoming letters by topic; the error jumps up and down instead of falling, because gradient descent is taking steps that are too large, so she lowers the step size and starts again.","da":"En analytiker i et ministerium træner en model til at sortere indkomne breve efter emne; fejlen hopper op og ned i stedet for at falde, fordi gradientnedstigning tager for store skridt, så hun sænker skridtstørrelsen og starter forfra."},"whyItMatters":{"en":"Almost every modern neural network is trained this way, so its choices, such as step size and how many rounds, decide whether training succeeds, stalls or wastes large amounts of costly computing.","da":"Næsten alle moderne neurale netværk trænes sådan, så valgene - skridtstørrelse, antal runder - afgør, om træningen lykkes, går i stå eller spilder store mængder dyr regnekraft."}},"deepDive":{"en":"The basic update is θₜ₊₁ = θₜ − η ∇L(θₜ), where θ is the parameter vector, L the loss and η the learning rate. The negative gradient is the direction of steepest local decrease, an idea usually credited to Cauchy (1847). For a convex loss whose gradient is L-Lipschitz (L-smooth), a fixed step η ≤ 1/L guarantees that the loss never increases and converges at rate O(1/t), and strong convexity gives linear convergence; on a quadratic, any η above 2/L along the sharpest direction makes the iterates diverge, which is the oscillating, exploding loss seen when the learning rate is set too high. Neural-network losses are non-convex, so these guarantees hold only locally, and in high dimensions saddle points and flat regions are a bigger obstacle than poor local minima (Dauphin et al., 2014).\n\nStochastic gradient descent replaces the full gradient with an estimate from a mini-batch. Its theory goes back to the stochastic approximation of Robbins and Monro (1951), whose conditions Σηₜ = ∞ and Σηₜ² < ∞ explain why learning rates are decayed over training. Gradient noise is not only a cost: it helps the iterates escape saddle points and is thought to bias training towards flatter, better-generalising solutions.\n\nAlmost all deep learning uses refinements of plain SGD. Momentum (Polyak's heavy-ball method, 1964) and Nesterov's accelerated gradient (1983) accumulate a velocity to damp oscillation across narrow valleys. Adaptive methods scale each coordinate by its gradient history: AdaGrad (Duchi et al., 2011), RMSProp (Hinton's 2012 lecture notes) and Adam (Kingma & Ba, 2015), whose widely used defaults are β₁ = 0.9, β₂ = 0.999 and ε = 10⁻⁸. AdamW (Loshchilov & Hutter, 2019) decouples weight decay from the adaptive update and is the default optimiser for most transformer training. Adam-type optimisers keep two extra tensors per parameter, so optimiser state is often larger than the weights themselves, a major term in training memory budgets. Second-order methods such as Newton's method or L-BFGS converge in fewer steps but are impractical at the scale of billions of parameters.\n\nIn practice the learning-rate schedule matters as much as the optimiser: a linear warmup over the first steps, then cosine or linear decay, is standard for large models, and a short learning-rate range test is a cheap way to find a workable η. Clipping the global gradient norm (often at 1.0 in LLM training) contains occasional loss spikes. Gradient descent should be kept distinct from its neighbours: the loss function defines what is minimised, backpropagation computes the gradient, and gradient descent decides the step.","da":"Den grundlæggende opdatering er θₜ₊₁ = θₜ − η ∇L(θₜ), hvor θ er parametervektoren, L tabet og η læringsraten. Den negative gradient er retningen med det stejleste lokale fald, en idé der normalt tilskrives Cauchy (1847). For et konvekst tab, hvis gradient er L-Lipschitz (L-glat), garanterer et fast skridt η ≤ 1/L, at tabet aldrig stiger og konvergerer med rate O(1/t), og stærk konveksitet giver lineær konvergens; på en kvadratisk funktion får ethvert η over 2/L i den skarpeste retning iterationerne til at divergere, hvilket er det svingende, eksploderende tab, man ser, når læringsraten er sat for højt. Tabet for neurale netværk er ikke-konvekst, så garantierne gælder kun lokalt, og i høje dimensioner er saddelpunkter og flade områder en større forhindring end dårlige lokale minima (Dauphin m.fl., 2014).\n\nStokastisk gradientnedstigning (SGD) erstatter den fulde gradient med et estimat fra en mini-batch. Teorien går tilbage til Robbins og Monros stokastiske approksimation (1951), hvis betingelser Σηₜ = ∞ og Σηₜ² < ∞ forklarer, hvorfor læringsraten nedtrappes under træningen. Støjen i gradienten er ikke kun en omkostning: den hjælper iterationerne væk fra saddelpunkter og menes at trække træningen mod fladere løsninger, der generaliserer bedre.\n\nNæsten al deep learning bruger forfinelser af ren SGD. Momentum (Polyaks heavy-ball-metode, 1964) og Nesterovs accelererede gradient (1983) opbygger en hastighed, der dæmper svingninger på tværs af smalle dale. Adaptive metoder skalerer hver koordinat efter dens gradienthistorik: AdaGrad (Duchi m.fl., 2011), RMSProp (Hintons forelæsningsnoter fra 2012) og Adam (Kingma & Ba, 2015), hvis udbredte standardværdier er β₁ = 0,9, β₂ = 0,999 og ε = 10⁻⁸. AdamW (Loshchilov & Hutter, 2019) afkobler weight decay fra den adaptive opdatering og er standardoptimeringsalgoritmen til det meste transformertræning. Optimeringsalgoritmer af Adam-typen gemmer to ekstra tensorer pr. parameter, så optimeringstilstanden ofte fylder mere end selve vægtene og er en stor post i hukommelsesbudgettet for træning. Andenordensmetoder som Newtons metode eller L-BFGS konvergerer i færre skridt, men er upraktiske ved milliarder af parametre.\n\nI praksis betyder læringsrateplanen lige så meget som optimeringsalgoritmen: lineær opvarmning over de første skridt og derefter cosinus- eller lineær nedtrapning er standard for store modeller, og en kort læringsrate-test (range test) er en billig måde at finde et brugbart η. Klipning af den samlede gradientnorm (ofte ved 1,0 i LLM-træning) holder sporadiske tabsspidser i skak. Gradientnedstigning bør holdes adskilt fra naboerne: tabsfunktionen definerer, hvad der minimeres, backpropagation beregner gradienten, og gradientnedstigning bestemmer skridtet."},"edges":[{"type":"requires","to":"ai/loss-function","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/model-parameter","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/model-training","why":{"en":"It is the engine inside model training that turns measured errors into changes to the model.","da":"Den er motoren i modeltræningen, der omsætter målte fejl til ændringer i modellen."},"confidence":"high","strength":"primary"}],"depth":3,"sources":[{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 4 and 8)","url":"https://www.deeplearningbook.org/","tier":"textbook","publisher":"MIT Press"},{"title":"ISO/IEC 22989:2022, Artificial intelligence concepts and terminology","url":"https://www.iso.org/standard/74296.html","tier":"standard","publisher":"ISO/IEC"},{"title":"Robbins & Monro (1951), A Stochastic Approximation Method","url":"https://doi.org/10.1214/aoms/1177729586","tier":"reference","publisher":"The Annals of Mathematical Statistics"},{"title":"Kingma & Ba (2015), Adam, A Method for Stochastic Optimization","url":"https://arxiv.org/abs/1412.6980","tier":"reference","publisher":"ICLR 2015"},{"title":"Loshchilov & Hutter (2019), Decoupled Weight Decay Regularization","url":"https://arxiv.org/abs/1711.05101","tier":"reference","publisher":"ICLR 2019"}],"draft":true}