Instruction tuning vs Pretraining

Why they differ

Pretraining gives broad knowledge from raw text; instruction tuning afterwards teaches the model to act on requests.

Training & optimisation

Extra training on many written requests paired with good answers, which turns a text-continuing base model into one that follows orders.

Formal

A form of fine-tuning in which a language model that has finished pretraining is trained on sets of instructions and example responses, so that it answers a request instead of merely continuing the text it was given.

In plain English

Like a well-read new employee who knows a lot but must be shown, through many worked examples, how the office expects a request to be handled.

In practice

A customer-service lead at a Danish online shop tests an open model with “Write a polite reply refusing the refund”; the base version just adds more customer complaints, while the version that has had instruction tuning writes the reply.

Why it matters

It is what makes a raw model usable as an assistant, and the example answers chosen here shape its tone, its refusals and much of its safety.

Pretraining

Training & optimisation

The first, huge and costly stage of teaching a model, reading vast amounts of text to pick up general patterns before any special task.

Formal

The initial stage of model training in which a model learns from a very large data set that mostly carries no answers, usually by self-supervised learning such as guessing the next token; the result is a general base that later fine-tuning adapts.

In plain English

Like a small child who hears years of everyday talk before starting school; nobody explains the grammar, yet the child picks up how the language works.

In practice

A research group at a Danish university spends months on a large national research computer pretraining a model on billions of words of public Danish text; the resulting base model can continue any text but does not yet follow instructions well.

Why it matters

What goes into this stage, including copyrighted, personal or poisoned text, is baked into every product built on the model and is very hard to take out.