Gradient descent

The basic update is θₜ₊₁ = θₜ − η ∇L(θₜ), where θ is the parameter vector, L the loss and η the learning rate. The negative gradient is the direction of steepest local decrease, an idea usually credited to Cauchy (1847). For a convex loss whose gradient is L-Lipschitz (L-smooth), a fixed step η ≤ 1/L guarantees that the loss never increases and converges at rate O(1/t), and strong convexity gives linear convergence; on a quadratic, any η above 2/L along the sharpest direction makes the iterates diverge, which is the oscillating, exploding loss seen when the learning rate is set too high. Neural-network losses are non-convex, so these guarantees hold only locally, and in high dimensions saddle points and flat regions are a bigger obstacle than poor local minima (Dauphin et al., 2014).

Stochastic gradient descent replaces the full gradient with an estimate from a mini-batch. Its theory goes back to the stochastic approximation of Robbins and Monro (1951), whose conditions Σηₜ = ∞ and Σηₜ² < ∞ explain why learning rates are decayed over training. Gradient noise is not only a cost: it helps the iterates escape saddle points and is thought to bias training towards flatter, better-generalising solutions.

Almost all deep learning uses refinements of plain SGD. Momentum (Polyak's heavy-ball method, 1964) and Nesterov's accelerated gradient (1983) accumulate a velocity to damp oscillation across narrow valleys. Adaptive methods scale each coordinate by its gradient history: AdaGrad (Duchi et al., 2011), RMSProp (Hinton's 2012 lecture notes) and Adam (Kingma & Ba, 2015), whose widely used defaults are β₁ = 0.9, β₂ = 0.999 and ε = 10⁻⁸. AdamW (Loshchilov & Hutter, 2019) decouples weight decay from the adaptive update and is the default optimiser for most transformer training. Adam-type optimisers keep two extra tensors per parameter, so optimiser state is often larger than the weights themselves, a major term in training memory budgets. Second-order methods such as Newton's method or L-BFGS converge in fewer steps but are impractical at the scale of billions of parameters.

In practice the learning-rate schedule matters as much as the optimiser: a linear warmup over the first steps, then cosine or linear decay, is standard for large models, and a short learning-rate range test is a cheap way to find a workable η. Clipping the global gradient norm (often at 1.0 in LLM training) contains occasional loss spikes. Gradient descent should be kept distinct from its neighbours: the loss function defines what is minimised, backpropagation computes the gradient, and gradient descent decides the step.