Small language model (SLM)

There is no standard threshold for "small"; the label is relative to the frontier and in practice means a dense decoder-only transformer that fits in the memory of a single consumer GPU, laptop or phone. The memory arithmetic sets the boundary: weights need parameters × bytes per parameter, so a 3.8-billion-parameter model takes about 7.6 GB at 16 bits and under 2 GB at 4 bits, plus a KV cache that grows with context length, number of layers and key/value heads. The Phi-3 technical report (Abdin et al., 2024) is a reference point: Phi-3-mini has 3.8 billion parameters, was trained on 3.3 trillion tokens, and a 4-bit quantized version occupying about 1.8 GB generated more than 12 tokens per second on an iPhone 14 with an A16 chip.

Three techniques make small models competitive. First, data quality: the Phi line started with Phi-1 (1.3 billion parameters, "Textbooks Are All You Need", 2023), trained on filtered "textbook-quality" web data and synthetic exercises generated by a larger model. Second, overtraining: instead of the Chinchilla-optimal ratio of roughly 20 tokens per parameter, small models are trained on many trillions of tokens, which costs more once but makes every later inference cheaper. Third, knowledge distillation (Hinton, Vinyals and Dean, 2015), where the student is trained to match a larger teacher's output distribution, typically via a KL-divergence loss on temperature-softened logits rather than only on hard labels; Gemma 2's smaller models and Meta's Llama 3.2 1B and 3B models used distillation, the latter combined with pruning. Quantization and pruning then shrink the result further for on-device runtimes such as llama.cpp, ONNX Runtime or Core ML.

The trade-offs are predictable. Stored factual knowledge scales with parameter count, so small models hallucinate facts more readily and are best paired with retrieval or tightly scoped tasks; multi-step reasoning, long-context use, instruction following in complex prompts and performance in lower-resource languages such as Danish tend to degrade first, and English benchmark scores such as MMLU overstate how well a small model will do on Danish case notes. Fine-tuning a small model with LoRA on a few thousand in-domain examples is a common way to close the gap for a narrow task, and cascades that try a small model first and escalate uncertain cases to a larger one are a common serving pattern.

On privacy, local inference means prompts and outputs need not leave the device, which removes a processor relationship and third-country transfer questions for that step, but the organisation remains data controller under the GDPR, and device encryption, access control and the handling of stored outputs still have to be addressed. Small models are also easier to extract and redistribute once deployed to endpoints, which matters if the fine-tuned weights encode sensitive training data.