Definition
Reinforcement learning from human feedback (RLHF) trains a policy against a reward that is learned from human preference judgements, instead of a reward written by hand. People compare pairs of outputs, a reward model is fit so that preferred outputs score higher, and the policy is optimised with reinforcement learning to maximise . A KL penalty keeps the policy close to a reference model. The method comes from Christiano et al., "Deep Reinforcement Learning from Human Preferences" (NeurIPS 2017). It was applied to language models by Ziegler et al. (2019) and Stiennon et al., "Learning to Summarize from Human Feedback" (NeurIPS 2020). It became the standard alignment recipe with Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT, NeurIPS 2022).
A pretrained LM is trained to continue internet text, not to be helpful. Asked a question, it may continue with more questions, make things up, or produce toxic text, because all of these are plausible continuations. Supervised fine-tuning (SFT) on human-written answers helps, but demonstrations are expensive, and imitation copies the demonstrator's mistakes along with their skill. Many qualities people care about are also easier to judge than to write: "which of these two summaries is better?" is a quick question even when writing a good summary is slow. RLHF turns these cheap judgements into a training signal.
Christiano, Leike, Brown, Martic, Legg and Amodei first showed this on Atari and MuJoCo. Humans watched pairs of 1–2 second clips and said which was closer to the goal. A reward model learned from these comparisons, and an RL agent trained on the learned reward. Feedback on less than 1% of the agent's interactions was enough. A simulated robot learned to do a backflip from about 900 human comparisons, a behaviour that is hard to specify as a reward function. Ziegler et al. (2019) moved the recipe to GPT-2 and added the KL penalty to the pretrained model. Stiennon et al. (2020) scaled it to summarisation of Reddit TL;DR posts with 1.3B and 6.7B models: labellers preferred the RLHF summaries over the human reference summaries. InstructGPT applied the same pipeline to general instruction following, and a 1.3B InstructGPT model was preferred over the 175B GPT-3, which has 100 times more parameters.
Stage 1: supervised fine-tuning
Start from a pretrained LM and fine-tune it on prompt–demonstration pairs with the ordinary next-token loss (see Supervised fine-tuning (SFT)). InstructGPT used about 13k training prompts, taken from the OpenAI API and written by labellers, each with a labeller-written answer. The result, , plays three roles later. It initialises the policy, it is the frozen reference for the KL penalty, and (at 6B) it initialises the reward model.
Stage 2: the reward model
- the Bradley–Terry model (Bradley & Terry, 1952): only reward differences matter, so is identified up to a constant per prompt
- the SFT transformer with its unembedding replaced by a scalar head, read at the last token
- InstructGPT showed labellers 4 to 9 responses per prompt and had them ranked, which gives pairs per prompt
- all pairs from one prompt are put into one batch element. Treating them as independent samples made the RM overfit within one epoch
This is logistic regression on reward differences. The RM never sees an absolute score, only which of two outputs won. Ranking outputs costs about as much labeller time as comparing two, but it yields comparisons: 6 for , 36 for . InstructGPT trained on about 33k prompts. It used a 6B RM because 175B RMs trained unstably and cost far more in the RL loop. Before RL, the reward is shifted so that labeller demonstrations score 0 on average, since a per-prompt constant has no effect on the comparisons.
Stage 3: RL with a KL penalty
- the policy being trained, initialised from
- KL coefficient. In expectation the penalty is ; InstructGPT used
- "PPO-ptx": mixing in pretraining gradients reduced the regressions on public NLP benchmarks (the "alignment tax"); is plain PPO
- summed over the generated tokens; in practice applied as a per-token reward , with added at the final token
Each generated answer is one episode. The tokens are the actions, and the reward is sparse: the RM score arrives at the last token, while the KL penalty is paid along the way. The optimiser is Proximal policy optimization (Schulman et al., 2017). PPO clips the probability ratio between the new and old policy so that each update stays close to the policy that generated the samples. A value head, initialised from the RM, estimates returns for generalised advantage estimation. This means four networks are in memory at once: the policy, the reference, the reward model and the value model.
The KL-regularised objective has a closed-form optimum, which is useful for intuition and is the starting point for Direct preference optimization (DPO):
- : the reward is ignored
- puts all mass on , which is exactly what exploits RM errors
- a sum over all possible responses, which is intractable. PPO approximates by sampling instead
Reward hacking and the KL coefficient
The reward model is a proxy. It was trained on outputs from , and it is only accurate near that distribution. RL is an optimiser, and it will find any input on which the proxy is wrong in its favour. This is Goodhart's law: once the measure becomes the target, it stops being a good measure. Typical exploits in language models are longer answers, confident-sounding filler, flattery and agreement with the user, and repeated phrases that the RM happens to like. Stiennon et al. saw this directly. As they optimised harder against their RM, its score kept rising while the human ratings of the summaries peaked and then fell.
- the gain in "true" reward, measured by a separate gold RM that labelled the proxy RM's training data
- distance travelled from the initial policy; best-of- sampling and RL both trace a rise then a fall
- fitted coefficients that change smoothly with proxy-RM size: larger RMs overoptimise later
Gao, Schulman and Hilton, "Scaling Laws for Reward Model Overoptimization" (ICML 2023), made this quantitative with a synthetic setup: a 6B "gold" RM stands in for humans, and smaller proxy RMs are trained on its labels. The proxy score rises monotonically with . The gold score follows the curves above: it peaks and then declines. In their RL runs, adding a KL penalty mostly changed how far along the gold-vs-KL curve training went, rather than moving the curve up. It acted much like early stopping. So is best understood as a budget. It sets how far from the policy may move, and the right budget is the one where the gold curve peaks. More and better preference data push that peak further out. So does collecting fresh comparisons on the current policy's outputs (Stiennon et al. and Bai et al. 2022 iterated this way).
Worked example: one ranking, one episode
| pair (winner, loser) | r_w − r_l | −log σ(r_w − r_l) |
|---|---|---|
| A, B | 0.9 | 0.3412 |
| A, C | 1.5 | 0.2014 |
| A, D | 2.2 | 0.1051 |
| B, C | 0.6 | 0.4375 |
| B, D | 1.3 | 0.2410 |
| C, D | 0.7 | 0.4032 |
| mean over the 6 = C(4,2) pairs | 0.2882 |
Rewarding one 4-token answer in PPO (β = 0.02, RM score 0.85)
- 1Log-probabilities of the sampled tokens under the policy: . Under the frozen SFT model: .
- 2Per-token log-ratios . Their sum, , is a one-sample estimate of the sequence KL.
- 3Per-token rewards log-ratio , then add the RM score at the last token: .
- 4Episode return . The KL term is small here. It grows when the policy puts much more probability than the SFT model on tokens it likes, which is exactly what an exploit looks like.
- 5For intuition about : with over three answers with rewards , the optimum is at (KL 0.084), at (KL 0.323), at (KL 0.915), and nearly one-hot at (KL 1.609, which is ).
1import itertools, torch, torch.nn.functional as F23def rm_loss(scores):4 """scores: (K,) RM outputs for K answers to one prompt, sorted best → worst.5 All C(K,2) pairs share one batch element, as in InstructGPT."""6 pairs = list(itertools.combinations(range(scores.numel()), 2)) # (winner, loser) in rank order7 w = torch.tensor([i for i, _ in pairs]); l = torch.tensor([j for _, j in pairs])8 return -F.logsigmoid(scores[w] - scores[l]).mean()910print(rm_loss(torch.tensor([1.3, 0.4, -0.2, -0.9]))) # tensor(0.2882), matches the worked example1112def shaped_rewards(logp_policy, logp_ref, rm_score, mask, beta=0.02):13 """Per-token reward: -beta * (log pi - log pi_ref), plus the RM score on the last real token.14 logp_*: (B, T) log-probs of the sampled tokens; mask: (B, T) 1 for response tokens."""15 kl = (logp_policy - logp_ref) * mask16 rewards = -beta * kl17 last = mask.sum(1).long() - 1 # index of the final response token18 rewards[torch.arange(rewards.size(0)), last] += rm_score19 return rewards, kl.sum(1) # rewards (B, T), sequence KL estimate (B,)2021def ppo_policy_loss(logp_new, logp_old, advantages, mask, clip=0.2):22 ratio = torch.exp(logp_new - logp_old) # pi_new / pi_old per token23 unclipped = ratio * advantages24 clipped = torch.clamp(ratio, 1 - clip, 1 + clip) * advantages25 return -(torch.min(unclipped, clipped) * mask).sum() / mask.sum()2627# one PPO iteration, schematically:28# y ~ policy.generate(x) (sampling: the expensive part)29# logp_old, values = policy(x, y), value_net(x, y); logp_ref = sft(x, y); s = reward_model(x, y)30# rewards, kl = shaped_rewards(logp_old, logp_ref, s, mask)31# advantages, returns = gae(rewards, values, mask, gamma=1.0, lam=0.95)32# for a few epochs over minibatches: loss = ppo_policy_loss(...) + c_v * value_loss(...)Use when
- +Learns from judgements, which are cheaper and more reliable than demonstrations for many qualities
- +Can push past the quality of the demonstrations; InstructGPT 1.3B was preferred over GPT-3 175B
- +The reward model is reusable: for best-of-n, for rejection sampling, and as an evaluator
- +On-policy RL corrects the model's own mistakes rather than imitating someone else's
Avoid when
- −Complex and expensive: four models in memory and generation inside the training loop
- −Reward hacking: the policy exploits RM errors (length, flattery, style) once it moves far from π_SFT
- −Sensitive to β, the PPO hyperparameters, and the RM's size and data coverage
- −Alignment tax on standard benchmarks unless pretraining data is mixed in (PPO-ptx)
Check yourself
Why does InstructGPT put all C(K,2) comparisons from one ranked prompt into a single batch element instead of shuffling them as independent examples?
Check yourself
You halve the KL coefficient β and the reward-model score of your PPO policy goes up a lot, but human raters now prefer it less often. What is the most likely explanation?
Remember
- ✓RLHF has three stages: SFT on demonstrations; a reward model trained with on human comparisons; then PPO on .
- ✓Lineage: Christiano et al. 2017 (Atari/MuJoCo), Ziegler et al. 2019 and Stiennon et al. 2020 (summarisation), and InstructGPT 2022 (13k SFT, 33k RM and 31k PPO prompts; a 6B RM; ; PPO-ptx).
- ✓The KL-regularised optimum is ; sets how far the policy may move from the data the RM was trained on.
- ✓Overoptimisation: the proxy reward rises with KL while the true reward peaks and then falls. Gao et al. fit the true reward as with .
- ✓Watch held-out preference judgements, length and KL, not the RM score. Fresh comparisons on policy samples extend the safe KL budget.