{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/instruction-tuning","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/instruction-tuning/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/instruction-tuning/"},"term":{"en":"Instruction tuning","da":"Instruktionstilpasning (instruction tuning)"},"aka":{"en":["supervised fine-tuning","SFT"],"da":["instruction tuning","SFT"]},"domain":["ai"],"cluster":"training","layer":"training","status":"current","era":2021,"summary":{"en":"Extra training on many written requests paired with good answers, which turns a text-continuing base model into one that follows orders.","da":"Ekstra træning på skrevne forespørgsler med gode svar, som gør en basismodel, der blot fortsætter tekst, til en, der følger ordrer."},"body":{"formal":{"en":"A form of fine-tuning in which a language model that has finished pretraining is trained on sets of instructions and example responses, so that it answers a request instead of merely continuing the text it was given.","da":"En form for finjustering, hvor en fortrænet sprogmodel trænes på sæt af instruktioner og eksempelsvar, så den besvarer en forespørgsel i stedet for blot at fortsætte den tekst, den fik."},"plain":{"en":"Like a well-read new employee who knows a lot but must be shown, through many worked examples, how the office expects a request to be handled.","da":"Som en ny medarbejder, der har læst meget, men gennem mange eksempler må have vist, hvordan kontoret forventer, at en opgave løses."},"inPractice":{"en":"A customer-service lead at a Danish online shop tests an open model with “Write a polite reply refusing the refund”; the base version just adds more customer complaints, while the version that has had instruction tuning writes the reply.","da":"En kundeservicechef i en dansk webshop tester en åben model med “Skriv et høfligt svar, der afviser at give pengene tilbage”; basisversionen skriver bare flere kundeklager, mens den version, der har fået instruktionstilpasning, skriver svaret."},"whyItMatters":{"en":"It is what makes a raw model usable as an assistant, and the example answers chosen here shape its tone, its refusals and much of its safety.","da":"Det er det, der gør en rå model brugbar som assistent, og de eksempelsvar, der vælges her, former dens tone, dens afvisninger og meget af dens sikkerhed."}},"deepDive":{"en":"Mechanically, instruction tuning is supervised fine-tuning with the same next-token cross-entropy objective as pretraining, applied to curated examples of the form instruction (plus optional input) → desired response, or to multi-turn conversations. Conversations are serialised with a chat template that marks roles with special tokens (ChatML's <|im_start|> and <|im_end|> is one widespread example), and the loss is usually computed only on the assistant's tokens; in Hugging Face code the prompt positions get the label −100 so they are ignored. Learning rates are far smaller than in pretraining and one to a few epochs are typical. A model must later be prompted with exactly the template it was tuned on, since a mismatched template measurably degrades output.\n\nThe technique took shape in 2021. FLAN (Wei et al.) took a 137-billion-parameter pretrained model, rephrased more than 60 NLP data sets into natural-language instruction templates, and showed that the tuned model beat zero-shot GPT-3 175B on 20 of 25 evaluated data sets, with gains growing with model scale and the number of task clusters. T0 (Sanh et al., 2021) reached similar conclusions, and the 2022 Flan collection scaled to 1,836 tasks. InstructGPT (Ouyang et al., 2022) used about 13,000 training prompts with human-written demonstrations as the supervised stage before RLHF. Cheaper recipes followed: Self-Instruct bootstrapped instructions from a model's own outputs, Stanford Alpaca used 52,000 examples generated by an OpenAI model, and LIMA (Zhou et al., 2023) reported strong results from only 1,000 carefully curated examples, supporting the view that tuning mostly teaches format and style while knowledge comes from pretraining.\n\nThat view has practical consequences. Instruction data teaching facts the base model does not know is learned slowly and has been linked to more hallucination (Gekhman et al., 2024), so domain knowledge is often better supplied through continued pretraining or retrieval. Heavy tuning on a narrow domain causes catastrophic forgetting of general ability, which recipes counter by mixing in general instruction data. Distilling responses from a stronger proprietary model can conflict with that provider's terms of use.\n\nInstruction tuning is also where much safety behaviour is set, and it is fragile. Qi et al. (2023) showed that fine-tuning an aligned model on as few as ten adversarially designed examples could largely remove its refusals, and even benign fine-tuning data eroded safety somewhat. Downstream fine-tuning of an aligned model therefore needs its own safety evaluation. In the standard pipeline instruction tuning is followed by preference optimisation such as RLHF or DPO, which uses comparisons between answers rather than single reference answers.","da":"Mekanisk er instruktionstilpasning superviseret finjustering med samme krydsentropimål for næste token som i fortræningen, anvendt på kuraterede eksempler af formen instruktion (plus eventuelt input) → ønsket svar eller på samtaler over flere ture. Samtaler serialiseres med en chatskabelon, der markerer roller med særlige tokens (ChatML's <|im_start|> og <|im_end|> er et udbredt eksempel), og tabet beregnes som regel kun på assistentens tokens; i Hugging Face-kode får promptpositionerne labelen −100, så de ignoreres. Læringsraterne er langt mindre end i fortræningen, og en til få epoker er typisk. En model skal efterfølgende promptes med præcis den skabelon, den er tilpasset med, fordi en afvigende skabelon målbart forringer output.\n\nTeknikken tog form i 2021. FLAN (Wei m.fl.) tog en fortrænet model med 137 milliarder parametre, omformulerede mere end 60 NLP-datasæt til instruktionsskabeloner i naturligt sprog og viste, at den tilpassede model slog zero-shot GPT-3 175B på 20 af 25 evaluerede datasæt, med gevinster der voksede med modelstørrelsen og antallet af opgaveklynger. T0 (Sanh m.fl., 2021) kom til lignende konklusioner, og Flan-samlingen fra 2022 skalerede op til 1.836 opgaver. InstructGPT (Ouyang m.fl., 2022) brugte omkring 13.000 træningsprompts med menneskeskrevne eksempelsvar som det supervisede trin før RLHF. Billigere opskrifter fulgte: Self-Instruct genererede instruktioner ud fra en models egne output, Stanford Alpaca brugte 52.000 eksempler genereret af en OpenAI-model, og LIMA (Zhou m.fl., 2023) rapporterede stærke resultater med kun 1.000 omhyggeligt udvalgte eksempler, hvilket understøtter synspunktet om, at tilpasningen mest lærer format og stil, mens viden kommer fra fortræningen.\n\nDet synspunkt har praktiske konsekvenser. Instruktionsdata, der lærer modellen fakta, som basismodellen ikke kender, læres langsomt og er blevet forbundet med flere hallucinationer (Gekhman m.fl., 2024), så domæneviden tilføres ofte bedre via fortsat fortræning eller retrieval. Kraftig tilpasning til et snævert domæne giver katastrofal glemsel af generelle evner, hvilket opskrifter modvirker ved at blande generelle instruktionsdata ind. At destillere svar fra en stærkere proprietær model kan være i strid med udbyderens brugsvilkår.\n\nInstruktionstilpasning er også der, hvor meget af sikkerhedsadfærden fastlægges, og den er skrøbelig. Qi m.fl. (2023) viste, at finjustering af en aligned model på så få som ti bevidst designede eksempler i vid udstrækning kunne fjerne dens afvisninger, og at selv harmløse finjusteringsdata svækkede sikkerheden noget. Videre finjustering af en aligned model kræver derfor sin egen sikkerhedsevaluering. I standardforløbet følges instruktionstilpasning af præferenceoptimering som RLHF eller DPO, der bruger sammenligninger mellem svar i stedet for enkelte referencesvar."},"edges":[{"type":"requires","to":"ai/pretraining","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/large-language-model","confidence":"high","strength":"normal"},{"type":"kind-of","to":"ai/fine-tuning","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/pretraining","why":{"en":"Pretraining gives broad knowledge from raw text; instruction tuning afterwards teaches the model to act on requests.","da":"Fortræning giver bred viden fra rå tekst; instruktionstilpasning lærer bagefter modellen at handle på forespørgsler."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/rlhf","why":{"en":"Instruction tuning usually comes first; RLHF then refines the answers using human preferences.","da":"Instruktionstilpasning kommer som regel først; RLHF finpudser derefter svarene ud fra menneskelige præferencer."},"confidence":"high","strength":"primary"}],"depth":3,"sources":[{"title":"Wei et al. (2021), Finetuned Language Models Are Zero-Shot Learners","tier":"reference"},{"title":"Ouyang et al. (2022), Training language models to follow instructions with human feedback","tier":"reference"},{"title":"Qi et al. (2023), Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!","url":"https://arxiv.org/abs/2310.03693","tier":"reference","publisher":"arXiv"},{"title":"Gekhman et al. (2024), Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?","url":"https://arxiv.org/abs/2405.05904","tier":"reference","publisher":"arXiv"}],"draft":true}