Reinforcement learning

The standard formalism is the Markov decision process (MDP): a set of states S, actions A, transition probabilities P(s′ | s, a), a reward function R and a discount factor γ between 0 and 1. The agent follows a policy π(a | s) and seeks to maximise the expected discounted return, the sum of γᵗ·rₜ over time. Value functions express how good a state, V(s), or a state-action pair, Q(s, a), is under a policy, and the Bellman equations relate each value to the immediate reward plus the discounted value of the successor state. When the true state is only partially observed, the setting becomes a POMDP.

Algorithms fall into a few families. Value-based methods learn Q and act greedily on it: temporal-difference learning (Sutton, 1988) and Q-learning (Watkins, 1989) update estimates from single transitions, and DQN (Mnih et al., 2015) combined Q-learning with a deep network, experience replay and a target network to reach human-level play on many Atari games. Policy-gradient methods adjust the policy parameters directly along the gradient of expected return, starting with REINFORCE (Williams, 1992); actor-critic methods pair a policy with a learned value baseline, and Proximal Policy Optimization (PPO, 2017) constrains each update with a clipped objective and is widely used, including in RLHF. Model-based methods learn or are given the environment dynamics and plan with them, as in AlphaGo (Silver et al., 2016) and AlphaZero (2018), which combined Monte Carlo tree search with networks trained through self-play.

The exploration-exploitation trade-off is intrinsic: the agent must try actions with uncertain value to discover better ones. Simple schemes include ε-greedy (a random action with probability ε) and optimism under uncertainty; the multi-armed bandit is the stateless special case used in online experimentation and recommendation. Credit assignment is the second core problem, since rewards may arrive long after the actions that caused them.

Practical failure modes are well documented. Reward hacking or specification gaming occurs when the agent maximises the measured reward through unintended behaviour, such as circling to collect points rather than finishing a race. RL is sample-inefficient, so most training happens in simulators, and policies can fail to transfer to the real world (the sim-to-real gap). Training is also noisy and sensitive to random seeds and hyperparameters. Offline RL learns from logged data without new interaction, but must avoid overvaluing actions the logs never tried.

In language models, reinforcement learning appears as a post-training stage: RLHF optimises against a reward model trained on human preference comparisons, and more recent reasoning models are trained with rewards from automatically verifiable outcomes such as passing unit tests or correct maths answers. The same reward-hacking risk applies, showing up as sycophancy or as gaming of the tests.