Bias-variance trade-off

For squared-error loss, the expected error of a learned predictor at a point x, averaged over training sets drawn from the same distribution, decomposes exactly into bias squared (how far the average prediction is from the true function), variance (how much individual predictions scatter around that average), and the irreducible noise variance of the target. Geman, Bienenstock and Doursat (1992) brought this decomposition into neural network research as the bias/variance dilemma: a flexible, nonparametric estimator has low bias but needs very large samples to keep its variance down, while a constrained one has low variance but may be systematically wrong. For other losses, such as 0-1 classification loss, there is no single clean additive decomposition, and several competing definitions exist.

In classical practice the trade-off is steered through model capacity: polynomial degree, tree depth, number of neighbours in k-nearest neighbours, or the strength of a regularization penalty. Plotting test error against capacity gives the textbook U-shaped curve, with underfitting on the left, overfitting on the right, and the best model at the bottom, found with a validation set or cross-validation. Ensembles act on the variance term directly: bagging and random forests average many high-variance trees, while boosting mainly reduces bias by adding weak learners in sequence.

Belkin, Hsu, Ma and Mandal (2019) showed that the U-curve is only the first part of a longer picture they called double descent. Test error peaks near the interpolation threshold, where the model has just enough capacity to fit the training data exactly, and then falls again as capacity keeps growing, often below the classical minimum. The effect appears for random-feature models, decision-tree ensembles and neural networks, and helps explain why heavily overparameterised deep networks that reach zero training error can still generalise. The decomposition itself remains true; what changes is the assumption that variance must rise monotonically with parameter count, since implicit regularisation from the training procedure (for example stochastic gradient descent finding minimum-norm solutions) keeps variance in check.