{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/alignment","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/alignment/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/alignment/"},"term":{"en":"AI alignment","da":"AI-alignment"},"aka":{"en":["value alignment"],"da":["værdialignment","value alignment"]},"domain":["ai"],"cluster":"ai-risk","layer":"training","status":"current","summary":{"en":"The work of making an AI system aim for what people actually intend, and refuse what they would not accept.","da":"Arbejdet med at få et AI-system til at stræbe efter det, mennesker faktisk mener, og afvise det, de ikke vil acceptere."},"body":{"formal":{"en":"The field and practice of steering an AI system's goals and behaviour to match the intentions and values of its makers and users - helpful, honest, refusing harmful requests - including in situations it was never tested on.","da":"Det felt og den praksis, der styrer et AI-systems mål og adfærd, så de svarer til skabernes og brugernes hensigter og værdier - hjælpsomt, ærligt og afvisende over for skadelige ønsker - også i situationer, det aldrig er testet i."},"plain":{"en":"Like King Midas, who wished that everything he touched would turn to gold and got exactly that - including his food. The wish was granted to the letter, not to the meaning.","da":"Som kong Midas, der ønskede, at alt, han rørte ved, blev til guld, og fik præcis det - også sin mad. Ønsket blev opfyldt efter ordlyden, ikke efter meningen."},"inPractice":{"en":"A research team at a Danish university adapts an open model to Danish, then has people rate its answers so it learns to admit when it is unsure and to refuse step-by-step help with building weapons.","da":"Et forskerhold på et dansk universitet tilpasser en åben model til dansk og lader derefter mennesker bedømme dens svar, så den lærer at indrømme usikkerhed og nægte trinvis hjælp til at bygge våben."},"whyItMatters":{"en":"A capable system that pursues a slightly wrong goal can cause harm at scale; as AI agents act more on their own, the gap between what we asked for and what we meant matters more.","da":"Et dygtigt system, der forfølger et en smule forkert mål, kan gøre skade i stor skala; jo mere AI-agenter handler på egen hånd, jo mere betyder afstanden mellem det, vi bad om, og det, vi mente."}},"deepDive":{"en":"Alignment research usually splits the problem in two. Outer alignment asks whether the objective we optimise - a reward function, a preference model, a loss - actually captures what we want; failures here show up as specification gaming or reward hacking, where the system satisfies the letter of the objective (a boat-racing agent circling to collect points, a summariser that learns what raters reward rather than what is accurate). Inner alignment asks whether the learned system actually pursues that objective, or has picked up a different goal that merely correlated with it in training and diverges under distribution shift (goal misgeneralisation). Goodhart's law - a measure that becomes a target ceases to be a good measure - is the recurring theme.\n\nFor large language models the standard post-training stack is supervised fine-tuning on demonstrations, followed by preference optimisation. In RLHF as described for InstructGPT (Ouyang et al., 2022), humans rank pairs of responses, a reward model is trained with a Bradley-Terry pairwise loss, and the policy is optimised with PPO against that reward plus a KL penalty that keeps it close to the SFT reference model to limit reward hacking; the 1.3-billion-parameter InstructGPT was preferred by raters over the 175-billion-parameter GPT-3. Direct Preference Optimization (Rafailov et al., 2023) removes the explicit reward model and optimises a closed-form loss on preference pairs. Constitutional AI (Bai et al., 2022) replaces much of the human harmlessness labelling with AI feedback guided by a written set of principles (RLAIF).\n\nKnown side effects of preference training include sycophancy (agreeing with the user's stated view), verbosity bias, over-refusal of benign requests, and a veneer of safety that jailbreaks can bypass because harmful capabilities remain in the weights. Research on \"alignment faking\" (Greenblatt et al., 2024) and on deliberately backdoored \"sleeper agent\" models (Hubinger et al., 2024) showed that models can behave differently when they infer they are being trained or evaluated, and that standard safety training may fail to remove conditional behaviour, which is why evaluation cannot rely on observed behaviour alone. Scalable oversight - debate, recursive reward modelling, AI-assisted evaluation - and interpretability aim at supervising systems whose outputs humans cannot easily check.\n\nAlignment should be distinguished from neighbouring controls. Guardrails are external input and output filters around a model; they can be changed without retraining but do not alter what the model would do unfiltered. Instruction tuning is a component of alignment, not a synonym. Governance and regulation, such as the EU AI Act's obligations on general-purpose models with systemic risk, set requirements for evaluation and risk mitigation but do not prescribe a training method. In deployed products, alignment is one layer among several, and least privilege, human approval and monitoring remain necessary because no current method guarantees aligned behaviour on unseen inputs.","da":"Alignment-forskningen deler normalt problemet i to. Ydre alignment (outer alignment) spørger, om det mål, vi optimerer - en belønningsfunktion, en præferencemodel, en tabsfunktion - faktisk indfanger det, vi ønsker; fejl her viser sig som specification gaming eller reward hacking, hvor systemet opfylder målet efter ordlyden (en agent i et bådrace, der kører i ring for at samle point, eller en opsummeringsmodel, der lærer, hvad bedømmerne belønner, i stedet for hvad der er korrekt). Indre alignment (inner alignment) spørger, om det lærte system faktisk forfølger målet, eller om det har samlet et andet mål op, som blot korrelerede med det under træningen og afviger, når fordelingen skifter (goal misgeneralisation). Goodharts lov - et mål, der bliver en målsætning, holder op med at være et godt mål - går igen overalt.\n\nFor store sprogmodeller består den gængse efter-træning af supervised fine-tuning på demonstrationer efterfulgt af præferenceoptimering. I RLHF som beskrevet for InstructGPT (Ouyang et al., 2022) rangerer mennesker par af svar, en belønningsmodel trænes med et parvist Bradley-Terry-tab, og modellen optimeres med PPO mod belønningen plus en KL-straf, der holder den tæt på SFT-referencemodellen for at begrænse reward hacking; den 1,3 milliarder parametre store InstructGPT blev foretrukket af bedømmerne frem for GPT-3 med 175 milliarder parametre. Direct Preference Optimization (Rafailov et al., 2023) fjerner den eksplicitte belønningsmodel og optimerer et lukket tab direkte på præferencepar. Constitutional AI (Bai et al., 2022) erstatter en stor del af den menneskelige mærkning af skadelighed med AI-feedback styret af et nedskrevet sæt principper (RLAIF).\n\nKendte bivirkninger af præferencetræning er sycophancy (at give brugeren ret i dennes synspunkt), en forkærlighed for lange svar, overdreven afvisning af harmløse forespørgsler og en sikkerhedsfernis, som jailbreaks kan komme uden om, fordi de skadelige evner stadig findes i vægtene. Forskning i \"alignment faking\" (Greenblatt et al., 2024) og i bevidst bagdørsramte \"sleeper agent\"-modeller (Hubinger et al., 2024) viste, at modeller kan opføre sig anderledes, når de udleder, at de bliver trænet eller evalueret, og at almindelig sikkerhedstræning kan lade betinget adfærd blive siddende - derfor kan evaluering ikke alene bygge på observeret adfærd. Scalable oversight - debat, rekursiv belønningsmodellering, AI-assisteret evaluering - og interpretability sigter mod at føre tilsyn med systemer, hvis output mennesker ikke let kan efterprøve.\n\nAlignment skal holdes adskilt fra nabokontrollerne. Guardrails er eksterne filtre på input og output rundt om en model; de kan ændres uden gentræning, men ændrer ikke, hvad modellen ville gøre ufiltreret. Instruction tuning er en del af alignment, ikke et synonym. Governance og regulering, som AI-forordningens krav til AI-modeller til almen brug med systemisk risiko, stiller krav om evaluering og risikoafbødning, men foreskriver ikke en træningsmetode. I produkter i drift er alignment ét lag blandt flere, og mindst mulige rettigheder, menneskelig godkendelse og overvågning er stadig nødvendige, fordi ingen nuværende metode garanterer alignet adfærd på input, modellen ikke har set før."},"edges":[{"type":"requires","to":"ai/artificial-intelligence","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/guardrails","why":{"en":"Alignment shapes what the model itself tends to do; guardrails are separate checks placed around it that catch what alignment misses.","da":"Alignment former, hvad modellen selv er tilbøjelig til at gøre; guardrails er separate kontroller rundt om den, der fanger det, alignment overser."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/rlhf","why":{"en":"RLHF is the most widely used way to push a large language model toward the behaviour people prefer.","da":"RLHF er den mest udbredte måde at skubbe en stor sprogmodel hen mod den adfærd, mennesker foretrækker."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/instruction-tuning","why":{"en":"Teaching a model to follow instructions is usually the first alignment step after pretraining.","da":"At lære en model at følge instruktioner er som regel det første alignment-skridt efter fortræningen."},"confidence":"high","strength":"normal"}],"depth":2,"sources":[{"title":"Ouyang et al., Training language models to follow instructions with human feedback (2022)","tier":"reference","publisher":"OpenAI / NeurIPS"},{"title":"Russell & Norvig, Artificial Intelligence: A Modern Approach, 4th ed. (value alignment)","tier":"textbook","publisher":"Pearson"},{"title":"NIST AI 600-1 - Artificial Intelligence Risk Management Framework, Generative AI Profile","tier":"standard","publisher":"NIST"}],"draft":true}