Learning rate

In stochastic gradient descent the update is w <- w - eta * g, where g is the gradient of the loss on a mini-batch and eta is the learning rate. For full-batch gradient descent on a smooth loss whose curvature is bounded by L, any eta below 2/L guarantees that each step lowers the loss, which is why too large a rate shows up as oscillation or a loss that explodes to NaN, and too small a rate as slow progress or getting stuck on plateaus. Learning rate and batch size interact: Goyal et al. (2017) trained ImageNet models with mini-batches of 8,192 by scaling the learning rate linearly with batch size and adding a gradual warm-up over the first epochs.

Schedules change eta over training. Common ones are step decay, exponential decay, cosine annealing, one-cycle and linear warm-up followed by decay, which is standard for transformers because early updates with an adaptive optimiser are unstable. Smith (2017) proposed cyclical learning rates that oscillate between bounds, together with a learning-rate range test for finding sensible bounds. PyTorch implements these in torch.optim.lr_scheduler (for example StepLR, CosineAnnealingLR, OneCycleLR, LinearLR), stepped once per batch or per epoch depending on the scheduler.

Adaptive optimisers keep a global learning rate but scale it per parameter. Adam (Kingma and Ba, 2015) divides a running mean of gradients by the square root of a running mean of squared gradients; its paper defaults are a learning rate of 0.001, beta1 = 0.9, beta2 = 0.999 and epsilon = 1e-8. Adaptive scaling reduces, but does not remove, sensitivity to the base rate, and it changes how weight decay behaves, which motivated AdamW. In practice the learning rate is the first hyperparameter to tune, usually on a logarithmic grid, and a rate that works for pretraining is typically too high for fine-tuning, where values one or two orders of magnitude smaller are common.