Underfitting

The signature of underfitting is a training loss that stays high and a validation loss close to it: the gap between them is small, but both are well above what the task allows, measured against a reference such as human-level performance, a strong baseline or the estimated irreducible (Bayes) error. Learning curves make the diagnosis concrete. Plotted against training-set size, an underfitting model's training and validation curves converge quickly at a poor level, so collecting more data does not help, whereas an overfitting model shows a wide gap that more data narrows. In the bias-variance decomposition of squared error, expected error = bias² + variance + irreducible noise, and underfitting is the high-bias regime (Goodfellow et al., §5.2 and §5.4).

Causes fall into four groups. Capacity: the hypothesis class cannot represent the pattern, as with a linear model for a seasonal or interacting signal. Regularisation: weight decay, dropout, L1 penalties or data augmentation set so strong that they suppress real structure. Optimisation: too few steps, a learning rate far too small or so large that training stalls, stopping early on a noisy validation signal, vanishing gradients or poor initialisation. Information: the input features simply do not carry what is needed to predict the target. Bugs belong under optimisation in practice: misaligned labels, unscaled inputs, a frozen layer or a loss applied to the wrong tensor all look like underfitting. A standard sanity check is to try to overfit a single small batch: if the loss cannot be driven close to zero on a handful of examples, the problem is a bug or a capacity mismatch, not a lack of data.

Remedies follow the cause: a larger or more expressive model, better features, weaker regularisation, longer training or a tuned learning-rate schedule. The classical picture of a U-shaped test-error curve over model capacity has been revised by the double-descent phenomenon (Belkin et al., 2019), in which test error falls again once models are large enough to interpolate the training data, so heavily overparameterised networks rarely underfit for lack of capacity. Underfitting at scale is instead about compute and data: Hoffmann et al. (2022) showed that several large language models of the time were undertrained for their size, and in single-epoch pretraining training and validation loss track each other closely, so the usual lever is more tokens or compute rather than regularisation.

Underfitting should be kept apart from neighbouring failures. Overfitting shows a large gap between training and validation loss; distribution shift shows good validation results but poor production results; label noise caps both curves at a level set by the data rather than the model. Because an underfit model errs consistently, its mistakes are systematic (a straight-line forecast that always misses the seasonal peak) and can look deceptively stable.