Batch size

Batch size B spans a spectrum from full-batch gradient descent (B = N, the whole training set) through mini-batch training to pure stochastic gradient descent (B = 1). The mini-batch gradient is an unbiased estimate of the full gradient whose variance falls roughly in proportion to 1/B, and one epoch contains ⌈N/B⌉ optimiser steps. Vision models commonly use tens to a few thousand images per batch; large language models count batches in tokens, and GPT-3's largest model was trained with batches of about 3.2 million tokens. In distributed training the number that matters is the global (effective) batch: per-device micro-batch × gradient-accumulation steps × number of data-parallel replicas.

Batch size and learning rate are coupled. Goyal et al. (2017) proposed the linear scaling rule (multiply the learning rate by k when the batch grows by k, with a warmup phase at the start) and used it to train ResNet-50 on ImageNet with a batch of 8,192 in one hour. The rule breaks down beyond a critical batch size, which McCandlish et al. (2018) linked to the gradient noise scale: below it, doubling B roughly halves the number of steps needed; above it, extra samples per step mostly add compute without saving steps. The critical batch size tends to grow as the loss falls, which is one reason large training runs ramp the batch size up during training.

The effect on generalisation is debated. Keskar et al. (2017) reported that large batches tend to converge to sharp minima with a generalisation gap, while Hoffer et al. (2017) showed much of that gap closes when the number of updates and the learning-rate schedule are adjusted, and Smith et al. (2018) showed that increasing the batch size can substitute for decaying the learning rate. The practical lesson is that batch size cannot be changed in isolation; the learning rate, warmup and schedule must be retuned together.

Memory is usually the binding constraint, because stored activations grow linearly with B (and with sequence length). The standard fix for out-of-memory errors is to reduce the micro-batch and add gradient accumulation, summing gradients over several forward and backward passes before one optimiser step, which preserves the effective batch and hence the optimisation dynamics. The exception is batch normalisation, whose statistics are computed per micro-batch and degrade with very small batches; group or layer normalisation avoids that dependency. Batch sizes that are multiples of 8 help tensor-core utilisation on NVIDIA GPUs, whereas the preference for powers of two is largely folklore. Inference batching is a separate concern, trading latency for throughput in model serving.