Gradient boosting

Boosting builds an additive model F(x) = sum of nu * h_m(x) stage by stage. Friedman (2001, Annals of Statistics) framed it as gradient descent in function space: at each stage a new weak learner h_m, typically a small regression tree, is fitted to the negative gradient of the loss with respect to the current predictions (the pseudo-residuals) and added with a shrinkage factor, the learning rate nu. For squared error the pseudo-residuals are simply the residuals; other differentiable losses give log-loss for classification or Huber and quantile losses for robust regression. AdaBoost (Freund and Schapire, 1997) is an earlier boosting method that turns out to be a special case with exponential loss.

The main hyperparameters interact: a smaller learning rate needs more trees but usually generalises better; tree depth (often 3 to 8 levels) controls how many features can interact; and subsampling rows or columns per tree (stochastic gradient boosting, Friedman 2002) adds regularisation. Because each tree corrects the previous ones, boosting mainly reduces bias and, unlike a random forest, can overfit as trees are added, so the number of rounds is set by early stopping on a validation set.

Modern libraries made the method fast and dominant on tabular data. XGBoost (Chen and Guestrin, KDD 2016) added a second-order approximation of the loss, explicit L1 and L2 penalties on leaf weights, sparsity-aware split finding and cache-aware parallel construction. LightGBM (Ke et al., 2017) uses histogram-based splits and leaf-wise growth, and CatBoost (Prokhorenkova et al., 2018) uses ordered boosting and native handling of categorical features. scikit-learn provides HistGradientBoostingClassifier and Regressor in the same histogram style.

Benchmarks such as Grinsztajn et al. (2022) found tree-based ensembles still outperform deep learning on typical medium-sized tabular datasets. The costs are sequential training that parallelises only within a tree, more sensitive tuning than random forests, and predictions that cannot extrapolate beyond the range seen in training.