Deep learning

The defining idea is representation learning: instead of engineers hand-crafting features (edge detectors, MFCCs for audio, n-gram counts for text) and feeding them to a shallow model, a deep network learns a hierarchy of features end to end from raw input, with all layers optimised jointly by backpropagation against a single loss. Depth matters because composing many simple nonlinear transformations can represent some functions far more compactly than a shallow network of comparable size.

The ideas are old; what changed around 2012 was the combination of large labelled datasets, GPU computation and a handful of training techniques. AlexNet (Krizhevsky, Sutskever and Hinton) won the ImageNet ILSVRC 2012 challenge with a top-5 error of about 15.3 percent against about 26.2 percent for the runner-up, trained on two consumer GPUs and using ReLU activations, dropout and data augmentation. Speech recognition had seen a similar shift a year or two earlier. ResNet (He et al., 2015) introduced residual (skip) connections, which let networks of 100+ layers train by giving gradients a direct path backwards. The transformer (Vaswani et al., 2017) then replaced recurrence with attention and became the default architecture for language, and increasingly for vision and audio.

Since around 2020, empirical scaling laws have shaped practice: Kaplan et al. (2020) and Hoffmann et al. (2022, "Chinchilla") showed that loss falls roughly as a power law in parameters, data and compute, with Chinchilla suggesting on the order of 20 training tokens per parameter for compute-optimal training. This drove the move to foundation models trained once with self-supervision and adapted many times through fine-tuning or prompting.

Known weaknesses follow from the method. Models are data- and compute-hungry, are poorly calibrated out of distribution, are vulnerable to adversarial examples (small, crafted input perturbations that flip outputs), and can exploit spurious correlations in the training data, for example a hospital-specific marker on X-rays rather than the pathology. Explainability remains partial. Many tabular problems are still solved better by gradient-boosted trees, so "deep" is not automatically "better".

Terminology: deep learning is a subset of machine learning that uses neural networks with multiple hidden layers; there is no fixed layer threshold. The neural network is the model family, while deep learning refers to the practice of training deep instances of it at scale, including the surrounding toolkit of optimisers, normalisation, regularisation and accelerator hardware.