Supervised learning

Formally, supervised learning assumes a dataset of pairs (xᵢ, yᵢ) drawn from an unknown joint distribution P(X, Y) and seeks a function f from a hypothesis class that minimises expected loss E[L(f(X), Y)]. Since P is unknown, training minimises the empirical average over the sample, typically with regularisation. The output type sets the task: a discrete label gives classification (loss usually cross-entropy), a continuous value gives regression (loss usually squared or absolute error), and structured outputs such as sequences, bounding boxes or segmentation masks give structured prediction.

Evaluation relies on held-out labelled data. k-fold cross-validation (k = 5 or 10 is conventional) estimates generalisation when data is scarce; with enough data a fixed train/validation/test split is used. Metrics must match the decision: accuracy is misleading under class imbalance, where precision, recall, F1, ROC-AUC or precision-recall curves are more informative, and regression is judged with MAE, RMSE or calibrated prediction intervals.

The label is the bottleneck. Labelling is expensive, slow and error-prone, and annotator disagreement is often genuine ambiguity rather than carelessness; measuring inter-annotator agreement (Cohen's or Fleiss' kappa) shows the ceiling a model can meaningfully reach. Techniques to reduce the burden include active learning (the model asks for labels on its most uncertain examples), weak supervision (noisy labels from heuristics and rules combined statistically), semi-supervised learning (a small labelled set plus a large unlabelled one), and starting from a pretrained model via transfer learning. A subtle failure is label leakage, where the label is recorded using information that the model also sees as input.

Supervised learning learns the labelling process, not the underlying truth. If past alert triage was inconsistent, or loan decisions reflected discrimination, the model reproduces it faithfully and appears accurate against the same biased labels. It also assumes the mapping from inputs to labels stays stable; when attackers or markets change behaviour, performance degrades silently until someone relabels fresh data.

The boundaries with neighbouring paradigms are about where the target signal comes from. In unsupervised learning there is no target; in self-supervised learning the target is derived automatically from the input itself (a masked word, the next token); in reinforcement learning the only signal is a scalar reward that arrives after actions, without the correct action ever being shown. Instruction tuning of language models is supervised learning on human-written prompt and response pairs.