Knowledge distillation vs Transfer learning

Why they differ

Transfer learning reuses the same model for a new task; distillation moves skill into a different, smaller model.

AI hardware & serving

Training a small “student” model to copy the answers of a large “teacher” model, so it keeps much of the skill at a fraction of the size.

Formal

A form of model training in which a smaller student model learns from the full output of a larger teacher model - the chance it gives every possible answer, not just its top pick - or from text the teacher writes, instead of only from labelled training data.

In plain English

Like a pupil who learns not only which answer the teacher picks but how sure the teacher is about each option, and ends up almost as good after far less study.

In practice

A data scientist at Skattestyrelsen has a large model answer 100,000 typical questions about tax deductions, then trains a small model on those answers so the self-service chat runs cheaply on the agency's own servers.

Why it matters

It is how many fast, cheap models are made, and it also means a rival can copy much of a model's skill just by collecting its answers, which is why many providers ban this in their terms.

Transfer learning

Machine learning basics

Reusing what a model already learned on one big task as the starting point for a new, related task, instead of starting from zero.

Formal

A way of doing model training in which a model trained on a large, general body of training data is taken as the start for a new task, and only adjusted with a smaller amount of task data.

In plain English

Like a car driver learning to drive a van; steering, traffic rules and road sense carry over, so only the size and the mirrors need practice.

In practice

An engineer at a water utility takes an image model trained on millions of everyday photos and teaches it to spot cracks in sewer inspection videos, using only 3,000 labelled frames.

Why it matters

It lets small teams build useful models without huge data or budgets, but the new model also inherits the old one's blind spots, biases and any hidden tampering.

Shared connections

Model training