Instruction tuning

Mechanically, instruction tuning is supervised fine-tuning with the same next-token cross-entropy objective as pretraining, applied to curated examples of the form instruction (plus optional input) → desired response, or to multi-turn conversations. Conversations are serialised with a chat template that marks roles with special tokens (ChatML's <|im_start|> and <|im_end|> is one widespread example), and the loss is usually computed only on the assistant's tokens; in Hugging Face code the prompt positions get the label −100 so they are ignored. Learning rates are far smaller than in pretraining and one to a few epochs are typical. A model must later be prompted with exactly the template it was tuned on, since a mismatched template measurably degrades output.

The technique took shape in 2021. FLAN (Wei et al.) took a 137-billion-parameter pretrained model, rephrased more than 60 NLP data sets into natural-language instruction templates, and showed that the tuned model beat zero-shot GPT-3 175B on 20 of 25 evaluated data sets, with gains growing with model scale and the number of task clusters. T0 (Sanh et al., 2021) reached similar conclusions, and the 2022 Flan collection scaled to 1,836 tasks. InstructGPT (Ouyang et al., 2022) used about 13,000 training prompts with human-written demonstrations as the supervised stage before RLHF. Cheaper recipes followed: Self-Instruct bootstrapped instructions from a model's own outputs, Stanford Alpaca used 52,000 examples generated by an OpenAI model, and LIMA (Zhou et al., 2023) reported strong results from only 1,000 carefully curated examples, supporting the view that tuning mostly teaches format and style while knowledge comes from pretraining.

That view has practical consequences. Instruction data teaching facts the base model does not know is learned slowly and has been linked to more hallucination (Gekhman et al., 2024), so domain knowledge is often better supplied through continued pretraining or retrieval. Heavy tuning on a narrow domain causes catastrophic forgetting of general ability, which recipes counter by mixing in general instruction data. Distilling responses from a stronger proprietary model can conflict with that provider's terms of use.

Instruction tuning is also where much safety behaviour is set, and it is fragile. Qi et al. (2023) showed that fine-tuning an aligned model on as few as ten adversarially designed examples could largely remove its refusals, and even benign fine-tuning data eroded safety somewhat. Downstream fine-tuning of an aligned model therefore needs its own safety evaluation. In the standard pipeline instruction tuning is followed by preference optimisation such as RLHF or DPO, which uses comparisons between answers rather than single reference answers.