{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/batch-size","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/batch-size/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/batch-size/"},"term":{"en":"Batch size","da":"Batchstørrelse"},"aka":{"en":["mini-batch size"],"da":["batch size"]},"domain":["ai"],"cluster":"training","layer":"training","status":"current","summary":{"en":"How many examples a model looks at together before it updates itself once during model training.","da":"Hvor mange eksempler en model ser på samlet, før den opdaterer sig selv én gang under modeltræning."},"body":{"formal":{"en":"The number of examples from the training data grouped into one batch; the loss is averaged over the batch and gradient descent makes one update per batch, so it sets both memory use and how noisy each step is.","da":"Antallet af eksempler fra træningsdata, der samles i én batch; tabet beregnes som gennemsnit over batchen, og gradientnedstigning laver én opdatering pr. batch, så værdien styrer både hukommelsesforbrug og hvor ujævnt hvert skridt er."},"plain":{"en":"Like a teacher marking homework; correcting after every single sheet is slow and jumpy, waiting for the whole class is steady but needs a big desk.","da":"Som en lærer, der retter lektier - at rette efter hvert eneste ark er langsomt og hoppende, at vente på hele klassen er roligt, men kræver et stort bord."},"inPractice":{"en":"A data analyst at a water utility trains a leak-spotting model on years of meter readings; the run stops with an out-of-memory error on the utility's only GPU, so she halves the batch size and lowers the step size to match.","da":"En dataanalytiker på et vandværk træner en model, der skal finde lækager, på flere års målerdata; kørslen stopper med en fejl om for lidt hukommelse på vandværkets eneste GPU, så hun halverer batchstørrelsen og sænker skridtstørrelsen tilsvarende."},"whyItMatters":{"en":"It decides how much hardware a run needs and how fast and stable learning is, which drives both cost and final quality.","da":"Den afgør, hvor meget hardware en kørsel kræver, og hvor hurtig og stabil læringen er, hvilket styrer både pris og endelig kvalitet."}},"deepDive":{"en":"Batch size B spans a spectrum from full-batch gradient descent (B = N, the whole training set) through mini-batch training to pure stochastic gradient descent (B = 1). The mini-batch gradient is an unbiased estimate of the full gradient whose variance falls roughly in proportion to 1/B, and one epoch contains ⌈N/B⌉ optimiser steps. Vision models commonly use tens to a few thousand images per batch; large language models count batches in tokens, and GPT-3's largest model was trained with batches of about 3.2 million tokens. In distributed training the number that matters is the global (effective) batch: per-device micro-batch × gradient-accumulation steps × number of data-parallel replicas.\n\nBatch size and learning rate are coupled. Goyal et al. (2017) proposed the linear scaling rule (multiply the learning rate by k when the batch grows by k, with a warmup phase at the start) and used it to train ResNet-50 on ImageNet with a batch of 8,192 in one hour. The rule breaks down beyond a critical batch size, which McCandlish et al. (2018) linked to the gradient noise scale: below it, doubling B roughly halves the number of steps needed; above it, extra samples per step mostly add compute without saving steps. The critical batch size tends to grow as the loss falls, which is one reason large training runs ramp the batch size up during training.\n\nThe effect on generalisation is debated. Keskar et al. (2017) reported that large batches tend to converge to sharp minima with a generalisation gap, while Hoffer et al. (2017) showed much of that gap closes when the number of updates and the learning-rate schedule are adjusted, and Smith et al. (2018) showed that increasing the batch size can substitute for decaying the learning rate. The practical lesson is that batch size cannot be changed in isolation; the learning rate, warmup and schedule must be retuned together.\n\nMemory is usually the binding constraint, because stored activations grow linearly with B (and with sequence length). The standard fix for out-of-memory errors is to reduce the micro-batch and add gradient accumulation, summing gradients over several forward and backward passes before one optimiser step, which preserves the effective batch and hence the optimisation dynamics. The exception is batch normalisation, whose statistics are computed per micro-batch and degrade with very small batches; group or layer normalisation avoids that dependency. Batch sizes that are multiples of 8 help tensor-core utilisation on NVIDIA GPUs, whereas the preference for powers of two is largely folklore. Inference batching is a separate concern, trading latency for throughput in model serving.","da":"Batchstørrelsen B spænder fra gradientnedstigning på hele datasættet (B = N) over mini-batch-træning til ren stokastisk gradientnedstigning (B = 1). Mini-batch-gradienten er et middelret estimat af den fulde gradient, hvis varians falder omtrent proportionalt med 1/B, og én epoke rummer ⌈N/B⌉ optimeringsskridt. Billedmodeller bruger typisk fra titals til nogle tusinde billeder pr. batch; store sprogmodeller måler batches i tokens, og GPT-3's største model blev trænet med batches på cirka 3,2 millioner tokens. Ved distribueret træning er det den globale (effektive) batch, der tæller: mikrobatch pr. enhed × antal skridt med gradientakkumulering × antal dataparallelle replikaer.\n\nBatchstørrelse og læringsrate hænger sammen. Goyal m.fl. (2017) foreslog den lineære skaleringsregel - gang læringsraten med k, når batchen vokser med faktor k, med en opvarmningsfase i starten - og brugte den til at træne ResNet-50 på ImageNet med en batch på 8.192 på én time. Reglen holder ikke over en kritisk batchstørrelse, som McCandlish m.fl. (2018) koblede til gradientens støjskala: under den halverer en fordobling af B omtrent antallet af nødvendige skridt; over den tilføjer ekstra eksempler pr. skridt mest beregning uden at spare skridt. Den kritiske batchstørrelse har en tendens til at vokse, efterhånden som tabet falder, hvilket er én grund til, at store træningskørsler øger batchstørrelsen undervejs.\n\nEffekten på generalisering er omdiskuteret. Keskar m.fl. (2017) rapporterede, at store batches har tendens til at konvergere mod skarpe minima med en generaliseringskløft, mens Hoffer m.fl. (2017) viste, at meget af kløften lukkes, når antallet af opdateringer og læringsrateplanen justeres, og Smith m.fl. (2018) viste, at en voksende batchstørrelse kan erstatte en faldende læringsrate. Den praktiske lære er, at batchstørrelsen ikke kan ændres isoleret; læringsrate, opvarmning og plan skal tunes igen samlet.\n\nHukommelse er som regel den bindende begrænsning, fordi de gemte aktiveringer vokser lineært med B (og med sekvenslængden). Standardløsningen på fejl om for lidt hukommelse er at mindske mikrobatchen og tilføje gradientakkumulering, hvor gradienter summeres over flere forlæns-baglæns gennemløb før ét optimeringsskridt, så den effektive batch og dermed optimeringsdynamikken bevares. Undtagelsen er batchnormalisering, hvis statistik beregnes pr. mikrobatch og forringes ved meget små batches; gruppe- eller lagnormalisering undgår den afhængighed. Batchstørrelser, der er multipla af 8, hjælper udnyttelsen af tensor-kerner på NVIDIA-GPU'er, mens forkærligheden for potenser af to mest er folklore. Batching ved inferens er en separat sag, hvor latenstid byttes for gennemløb i model serving."},"edges":[{"type":"requires","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"kind-of","to":"ai/hyperparameter","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/learning-rate","why":{"en":"The two are tuned together; a larger batch gives steadier steps, so the learning rate is often raised along with it.","da":"De to indstilles sammen; en større batch giver mere stabile skridt, så læringsraten ofte hæves sammen med den."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/gradient-descent","why":{"en":"Gradient descent makes one step per batch, so the batch size decides how many steps an epoch contains.","da":"Gradientnedstigning tager ét skridt pr. batch, så batchstørrelsen afgør, hvor mange skridt en epoke indeholder."},"confidence":"high","strength":"primary"}],"depth":2,"sources":[{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 8.1.3)","url":"https://www.deeplearningbook.org/contents/optimization.html","tier":"textbook","publisher":"MIT Press"},{"title":"Goyal et al. (2017), Accurate, Large Minibatch SGD, Training ImageNet in 1 Hour","url":"https://arxiv.org/abs/1706.02677","tier":"reference"},{"title":"Keskar et al. (2017), On Large-Batch Training for Deep Learning, Generalization Gap and Sharp Minima","url":"https://arxiv.org/abs/1609.04836","tier":"reference","publisher":"ICLR 2017"},{"title":"Brown et al. (2020), Language Models are Few-Shot Learners (GPT-3)","url":"https://arxiv.org/abs/2005.14165","tier":"reference"}],"draft":true}