Reinforcement learning from human feedback (RLHF)

The canonical pipeline has three stages. First, a pretrained model is instruction-tuned on demonstrations (SFT). Second, human raters compare outputs for the same prompt and a reward model r(x, y), usually the SFT model with a scalar head, is fitted to their choices with a Bradley-Terry loss, −log σ(r(x, y_w) − r(x, y_l)), where y_w is the preferred and y_l the rejected answer. Third, the policy is optimised with reinforcement learning, classically PPO, to maximise r(x, y) − β·KL(π ‖ π_ref), where the KL penalty against the frozen SFT reference keeps the policy from drifting into text that exploits the reward model. The method was introduced for Atari and simulated robotics by Christiano et al. (2017), applied to summarisation by Stiennon et al. (2020) and made mainstream by InstructGPT (Ouyang et al., 2022), which used about 13,000 SFT prompts, 33,000 reward-model prompts with rankings of several outputs each, and 31,000 prompts for PPO; labellers preferred outputs of the 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3.

PPO-based RLHF is operationally heavy: policy, reference model, reward model and a value (critic) model must all be held in memory, and training is sensitive to β, learning rate and reward normalisation. InstructGPT also mixed pretraining gradients into PPO (PPO-ptx) to reduce the regressions on public benchmarks that alignment otherwise caused.

Several alternatives now share the label "preference tuning". Direct Preference Optimisation (Rafailov et al., 2023) shows that the KL-regularised objective has a closed-form optimum, so the policy can be trained directly on preference pairs with a classification-style loss and no explicit reward model or RL loop. RLAIF and Constitutional AI (Bai et al., 2022) replace most human comparisons with judgements by a model following written principles. GRPO (Shao et al., 2024) removes the value model by comparing each sample with the average of a group of samples for the same prompt, and is widely used with automatically verifiable rewards, such as passing unit tests or reaching a correct maths answer, to train reasoning models.

The failure modes follow from optimising a learned proxy. Gao et al. (2023) showed that as the policy moves further from its reference, the proxy reward keeps rising while the true (gold) reward peaks and then falls - reward over-optimisation, a form of Goodhart's law. Human preferences favour agreeable, confident and longer answers, and Sharma et al. (2023) linked such preference data to sycophancy. The GPT-4 technical report showed that the pre-trained model was well calibrated and that post-training reduced calibration. Rater instructions, rater demographics and disagreement are thus design decisions with ethical weight, not neutral data collection.