Backpropagation

Backpropagation is reverse-mode automatic differentiation applied to a network's computational graph. The forward pass computes and stores each layer's pre-activations z and activations a; the backward pass then applies the chain rule from the scalar loss towards the input. For a fully connected layer l the error signal is δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ), and the weight gradient is ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ. Because the loss is a single scalar, one backward sweep yields the partial derivative for every parameter at a cost of only a small constant multiple of the forward pass, whereas finite differences would need one extra forward pass per parameter, which means billions of passes for a modern model. A widely used rule of thumb puts training compute at about 6N FLOPs per token for a model with N parameters: roughly 2N forward and 4N backward.

The idea predates its fame. Seppo Linnainmaa described reverse-mode differentiation in 1970, Paul Werbos proposed applying it to neural networks in his 1974 thesis, and Rumelhart, Hinton and Williams popularised it in Nature in 1986 by showing that it learns useful internal representations in hidden layers. A frequent misconception is that backpropagation is the learning algorithm; it only computes gradients, and an optimiser such as SGD or Adam decides how to change the weights with them.

The main engineering cost is memory. Every intermediate activation needed for the backward pass must be kept until it is used, so memory grows with depth, batch size and sequence length. Activation (gradient) checkpointing stores only some activations and recomputes the rest during the backward pass, cutting memory to roughly the square root of the number of layers at the price of an extra partial forward pass (Chen et al., 2016). Mixed-precision training uses loss scaling so that small FP16 gradients do not underflow to zero. In PyTorch, loss.backward() walks a dynamically recorded graph and accumulates into each parameter's .grad, which is why gradients must be zeroed between steps unless accumulation is intended.

Gradients multiplied through many layers can shrink or grow exponentially. Vanishing gradients, analysed by Hochreiter (1991) and Bengio et al. (1994), made deep sigmoid networks and recurrent networks trained by backpropagation through time hard to train; exploding gradients cause divergence. The standard remedies (ReLU-family activations, variance-preserving initialisation (Glorot, He), residual connections, normalisation layers, LSTM gating and gradient-norm clipping) are all ways of keeping the backward signal well conditioned. Operations with no useful derivative, such as sampling or rounding, need surrogates like the straight-through estimator or score-function (REINFORCE) gradients.