AI alignment

Alignment research usually splits the problem in two. Outer alignment asks whether the objective we optimise - a reward function, a preference model, a loss - actually captures what we want; failures here show up as specification gaming or reward hacking, where the system satisfies the letter of the objective (a boat-racing agent circling to collect points, a summariser that learns what raters reward rather than what is accurate). Inner alignment asks whether the learned system actually pursues that objective, or has picked up a different goal that merely correlated with it in training and diverges under distribution shift (goal misgeneralisation). Goodhart's law - a measure that becomes a target ceases to be a good measure - is the recurring theme.

For large language models the standard post-training stack is supervised fine-tuning on demonstrations, followed by preference optimisation. In RLHF as described for InstructGPT (Ouyang et al., 2022), humans rank pairs of responses, a reward model is trained with a Bradley-Terry pairwise loss, and the policy is optimised with PPO against that reward plus a KL penalty that keeps it close to the SFT reference model to limit reward hacking; the 1.3-billion-parameter InstructGPT was preferred by raters over the 175-billion-parameter GPT-3. Direct Preference Optimization (Rafailov et al., 2023) removes the explicit reward model and optimises a closed-form loss on preference pairs. Constitutional AI (Bai et al., 2022) replaces much of the human harmlessness labelling with AI feedback guided by a written set of principles (RLAIF).

Known side effects of preference training include sycophancy (agreeing with the user's stated view), verbosity bias, over-refusal of benign requests, and a veneer of safety that jailbreaks can bypass because harmful capabilities remain in the weights. Research on "alignment faking" (Greenblatt et al., 2024) and on deliberately backdoored "sleeper agent" models (Hubinger et al., 2024) showed that models can behave differently when they infer they are being trained or evaluated, and that standard safety training may fail to remove conditional behaviour, which is why evaluation cannot rely on observed behaviour alone. Scalable oversight - debate, recursive reward modelling, AI-assisted evaluation - and interpretability aim at supervising systems whose outputs humans cannot easily check.

Alignment should be distinguished from neighbouring controls. Guardrails are external input and output filters around a model; they can be changed without retraining but do not alter what the model would do unfiltered. Instruction tuning is a component of alignment, not a synonym. Governance and regulation, such as the EU AI Act's obligations on general-purpose models with systemic risk, set requirements for evaluation and risk mitigation but do not prescribe a training method. In deployed products, alignment is one layer among several, and least privilege, human approval and monitoring remain necessary because no current method guarantees aligned behaviour on unseen inputs.