Knowledge distillation

The idea predates deep learning - Buciluă, Caruana and Niculescu-Mizil described "model compression" in 2006 - but the standard formulation is Hinton, Vinyals and Dean (2015). The teacher's logits z are passed through a softmax with temperature T, p_i = exp(z_i/T) / Σ_j exp(z_j/T); a T above 1 flattens the distribution and exposes the "dark knowledge" in the relative probabilities of wrong classes (that a 7 looks more like a 1 than like an 8). The student is trained on a weighted sum of the ordinary cross-entropy against hard labels and the KL divergence between teacher and student softened distributions, with the soft term multiplied by T² so its gradients keep a comparable scale as T changes.

Variants differ in what is transferred. Response-based distillation matches output distributions; feature-based distillation (FitNets, 2014) also matches intermediate activations through learned projections; relation-based methods match similarities between examples. For language models, token-level distillation matches the teacher's next-token distribution at every position, which requires access to its logits and a shared tokenizer. Sequence-level distillation (Kim and Rush, 2016) instead trains the student on text the teacher generates, which only needs black-box API access. Much of what is loosely called distillation in the LLM world is this second kind: supervised fine-tuning on synthetic teacher outputs, including chain-of-thought traces, as with the smaller models DeepSeek released in 2025 that were fine-tuned on reasoning samples from DeepSeek-R1.

Results can be strong but are bounded. DistilBERT (Sanh et al., 2019) removed half of BERT-base's layers, was about 40% smaller and 60% faster, and kept roughly 97% of its GLUE score. The student inherits the teacher's errors and biases, often degrades most on rare or long-tail inputs that the transfer set did not cover, and usually ends up below the teacher on the distilled task. A very large capacity gap between teacher and student can also make distillation less effective than using an intermediate-sized teacher.

Distillation is distinct from its neighbours. Quantization keeps the same architecture and weights but stores them at lower precision; pruning removes weights or structures from the same network; transfer learning adapts one model to a new task. These techniques are frequently stacked: distil to a smaller architecture, then quantize for deployment. Distillation also has a security and legal side. Model extraction attacks are effectively unauthorised distillation through a public API, and the terms of several commercial providers prohibit using outputs to develop competing models, so provenance of synthetic training data is a licensing question as much as a technical one.