Regularization

Explicit penalty methods add a term to the training objective. L2 regularization adds lambda times the sum of squared weights; in linear regression this is ridge regression (Hoerl and Kennard, 1970), a special case of Tikhonov regularization, and it shrinks all coefficients smoothly toward zero. L1 regularization adds lambda times the sum of absolute weights; in linear regression this is the lasso (Tibshirani, 1996), which drives some coefficients to exactly zero and so performs feature selection. Elastic net mixes the two. In scikit-learn the strength is the alpha argument of Ridge and Lasso, and it is normally chosen by cross-validation (RidgeCV, LassoCV). From a Bayesian view, L2 corresponds to a Gaussian prior on the weights and L1 to a Laplace prior, so the penalised solution is a maximum a posteriori estimate.

Weight decay multiplies weights by a factor slightly below one at every update. For plain stochastic gradient descent this is equivalent to an L2 penalty, but Loshchilov and Hutter (2019) showed that it is not equivalent for adaptive optimisers such as Adam, where the L2 gradient is rescaled per parameter. Their decoupled version, AdamW, applies decay directly to the weights and is now the default choice for training transformers; PyTorch exposes it as torch.optim.AdamW with a weight_decay argument.

Deep learning adds implicit and structural regularisers. Dropout (Srivastava et al., 2014) randomly zeroes units during training, which approximates averaging an ensemble of thinned networks; at test time all units are used with rescaled activations. Early stopping halts training when validation loss stops improving and, for quadratic losses, behaves much like an L2 penalty. Data augmentation, label smoothing, batch normalisation noise, parameter sharing in convolutional networks, and the implicit bias of stochastic gradient descent toward flat or low-norm solutions also regularise. The strength of every regulariser is itself a hyperparameter and moves the model along the bias-variance trade-off: too little leaves overfitting, too much causes underfitting.