Regression

The name comes from Francis Galton's observation that children of unusually tall parents tend to be closer to average height, published in 1886 as "Regression towards mediocrity in hereditary stature". The workhorse is linear regression fitted by ordinary least squares: minimise the sum of squared residuals, which for a design matrix X and target vector y has the closed-form solution β = (XᵀX)⁻¹Xᵀy, the normal equations. Under the Gauss-Markov assumptions (a model linear in its parameters, errors with zero mean given the inputs, uncorrelated errors with constant variance, no perfect multicollinearity) OLS is the best linear unbiased estimator. In practice it is solved with QR or SVD decompositions rather than an explicit matrix inverse, for numerical stability.

Regularised variants trade a little bias for lower variance: ridge regression (Hoerl and Kennard, 1970) adds an L2 penalty, lasso (Tibshirani, 1996) an L1 penalty that sets some coefficients exactly to zero, and elastic net combines both. Non-linear regression is handled by feature transformations, splines, generalised additive models, tree ensembles such as gradient boosting, or neural networks with a linear output unit. Poisson and other generalised linear models suit counts and rates; time-series forecasting adds temporal structure (autoregression, seasonality) and must be validated on strictly later periods.

The loss function encodes what an error costs. Mean squared error targets the conditional mean and is dominated by outliers; mean absolute error targets the conditional median and is more robust; Huber loss is quadratic near zero and linear in the tails. Reported metrics include RMSE and MAE in the target's own units, MAPE (undefined when the true value is zero and asymmetric in its penalties), and R², the share of variance explained, which can be negative on test data when a model does worse than predicting the mean.

A point forecast is incomplete without uncertainty. Quantile regression, using the pinball loss, predicts chosen quantiles such as the 10th and 90th percentile directly; conformal prediction wraps any model to give intervals with finite-sample coverage guarantees under exchangeability. Common failures are heteroscedasticity (error that grows with the level, invalidating constant-width intervals), extrapolation beyond the range seen in training, where tree models go flat and linear models continue straight lines indefinitely, and confusing correlation in coefficients with causal effect.

Two naming traps: logistic regression is a classification method, and "regression" in software testing means a previously working feature breaking, which is unrelated. Regression also differs from classification only in the output type; ordinal targets such as ratings from 1 to 5 sit in between and can be modelled either way.