Loss function

Supervised training is usually framed as empirical risk minimisation: choose parameters θ to minimise (1/N) Σᵢ ℓ(f(xᵢ; θ), yᵢ) + λΩ(θ), where ℓ is the per-example loss and Ω an optional regulariser such as the squared L2 norm behind weight decay. Terminology is loose: some texts reserve "loss" for a single example, "cost" for the average and "objective" for the full expression including regularisation, but most code uses "loss" for all three. Training typically sees the average over a mini-batch, which is an unbiased estimate of the full-data average.

Most standard losses are negative log-likelihoods under an assumed noise model. Mean squared error corresponds to Gaussian noise and estimates the conditional mean; mean absolute error corresponds to Laplace noise, estimates the median and is more robust to outliers; the Huber loss is quadratic near zero and linear beyond a threshold. For classification, cross-entropy (log loss) −log p(y|x) under a softmax or sigmoid output is the default, and minimising it is equivalent to minimising the KL divergence from the data distribution to the model's. Language models use token-level cross-entropy, and perplexity is simply exp of the mean cross-entropy per token. Other families include the hinge loss of support vector machines, contrastive losses such as InfoNCE used to train embedding models, and KL terms in knowledge distillation and in the RLHF penalty that keeps a policy near its reference model.

The loss is usually a differentiable surrogate for what is actually wanted. Accuracy and F1 are piecewise constant in the parameters and give zero gradient almost everywhere, so models are trained on cross-entropy and thresholded afterwards. Cross-entropy is a proper scoring rule and in principle rewards calibrated probabilities, though large networks are often overconfident in practice (Guo et al., 2017). Class imbalance and asymmetric costs are handled by per-class weights, by the focal loss of Lin et al. (2017), which multiplies cross-entropy by (1 − pₜ)^γ with γ = 2 as the common default, or by keeping the loss unweighted and moving the decision threshold. Label smoothing replaces one-hot targets with slightly softened ones to discourage overconfidence.

Implementation details cause real bugs. PyTorch's CrossEntropyLoss and BCEWithLogitsLoss expect raw logits and apply log-softmax or sigmoid internally using the numerically stable log-sum-exp trick; feeding them probabilities that have already been through softmax silently degrades training. Monitoring training and validation loss side by side is the primary diagnostic for underfitting and overfitting. Finally, whatever the loss rewards is what the model optimises, including loopholes; a mis-specified loss is the supervised-learning counterpart of reward hacking in reinforcement learning.