RLHF

RLHF fine-tunes an LM in three stages: SFT on demonstrations, a reward model fit to human comparisons with the Bradley–Terry loss −log σ(r_w − r_l), then PPO on r − β·log(π/π_SFT). The KL term limits reward hacking: pushed too far, the proxy reward rises while true quality falls.

16 min read1378 wordsadvanced2 viewsupdated 2026-10-13

Definition

Reinforcement learning from human feedback (RLHF) trains a policy against a reward that is learned from human preference judgements, instead of a reward written by hand. People compare pairs of outputs, a reward model rθ(x,y)r_\theta(x,y) is fit so that preferred outputs score higher, and the policy is optimised with reinforcement learning to maximise rθr_\theta. A KL penalty keeps the policy close to a reference model. The method comes from Christiano et al., "Deep Reinforcement Learning from Human Preferences" (NeurIPS 2017). It was applied to language models by Ziegler et al. (2019) and Stiennon et al., "Learning to Summarize from Human Feedback" (NeurIPS 2020). It became the standard alignment recipe with Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT, NeurIPS 2022).

A pretrained LM is trained to continue internet text, not to be helpful. Asked a question, it may continue with more questions, make things up, or produce toxic text, because all of these are plausible continuations. Supervised fine-tuning (SFT) on human-written answers helps, but demonstrations are expensive, and imitation copies the demonstrator's mistakes along with their skill. Many qualities people care about are also easier to judge than to write: "which of these two summaries is better?" is a quick question even when writing a good summary is slow. RLHF turns these cheap judgements into a training signal.

Christiano, Leike, Brown, Martic, Legg and Amodei first showed this on Atari and MuJoCo. Humans watched pairs of 1–2 second clips and said which was closer to the goal. A reward model learned from these comparisons, and an RL agent trained on the learned reward. Feedback on less than 1% of the agent's interactions was enough. A simulated robot learned to do a backflip from about 900 human comparisons, a behaviour that is hard to specify as a reward function. Ziegler et al. (2019) moved the recipe to GPT-2 and added the KL penalty to the pretrained model. Stiennon et al. (2020) scaled it to summarisation of Reddit TL;DR posts with 1.3B and 6.7B models: labellers preferred the RLHF summaries over the human reference summaries. InstructGPT applied the same pipeline to general instruction following, and a 1.3B InstructGPT model was preferred over the 175B GPT-3, which has 100 times more parameters.

Stage 1: supervised fine-tuning

Start from a pretrained LM and fine-tune it on prompt–demonstration pairs with the ordinary next-token loss (see Supervised fine-tuning (SFT)). InstructGPT used about 13k training prompts, taken from the OpenAI API and written by labellers, each with a labeller-written answer. The result, πSFT\pi^{SFT}, plays three roles later. It initialises the policy, it is the frozen reference for the KL penalty, and (at 6B) it initialises the reward model.

Stage 2: the reward model

P(yw≻yl∣x)=σ(rθ(x,yw)−rθ(x,yl)),LRM(θ)=−1(K2) E(x,yw,yl)[log⁡σ(rθ(x,yw)−rθ(x,yl))]P(y_w\succ y_l\mid x)=\sigma\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big),\qquad \mathcal L_{RM}(\theta)=-\frac{1}{\binom K2}\,\mathbb E_{(x,y_w,y_l)}\Big[\log\sigma\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big)\Big]
σ(rw−rl)\sigma(r_w-r_l)
the Bradley–Terry model (Bradley & Terry, 1952): only reward differences matter, so rθr_\theta is identified up to a constant per prompt
rθ(x,y)r_\theta(x,y)
the SFT transformer with its unembedding replaced by a scalar head, read at the last token
KK
InstructGPT showed labellers 4 to 9 responses per prompt and had them ranked, which gives (K2)\binom K2 pairs per prompt
1/(K2)1/\binom K2
all pairs from one prompt are put into one batch element. Treating them as independent samples made the RM overfit within one epoch

This is logistic regression on reward differences. The RM never sees an absolute score, only which of two outputs won. Ranking KK outputs costs about as much labeller time as comparing two, but it yields (K2)\binom K2 comparisons: 6 for K=4K=4, 36 for K=9K=9. InstructGPT trained on about 33k prompts. It used a 6B RM because 175B RMs trained unstably and cost far more in the RL loop. Before RL, the reward is shifted so that labeller demonstrations score 0 on average, since a per-prompt constant has no effect on the comparisons.

InstructGPT (Ouyang et al. 2022): three stages, each with its own data1 · SFTprompt x from theAPI / labelerslabeler writes thedesired answer yfine-tune GPT-3 on(x, y): next-token CE≈13k prompts with demonstrations2 · reward modelsample K = 4 to 9answers for prompt xlabeler ranks thembest → worsttrain 6B RM r_θ(x, y):−log σ(r(y_w) − r(y_l))≈33k prompts, C(K,2) pairs each3 · RL with PPOpolicy π_RL samplesy for a new prompt xreward r_θ(x,y) − β·log(π_RL/π_SFT), β=0.02PPO update of π_RL(value net from the RM)≈31k prompts, no labels neededhand-offsSFT model → initialises the RM (6B, unembedding replaced by a scalar head)SFT model → initialises π_RL and stays frozen as the KL reference π_SFTRM → frozen reward in stage 3; also initialises PPO's value functionobjective: E[r_θ(x,y)] − β·KL(π_RL ‖ π_SFT) + γ·E_pretrain[log π_RL] (PPO-ptx)
fig.The InstructGPT pipeline. Stage 1 fine-tunes GPT-3 on about 13k demonstrations. Stage 2 has labellers rank 4–9 sampled answers for about 33k prompts and fits a 6B reward model with the Bradley–Terry loss on every pair. Stage 3 runs PPO on about 31k prompts: the policy samples an answer, the frozen RM scores it, and a per-token penalty β·log(π_RL/π_SFT) with β = 0.02 is subtracted. The SFT model initialises the RM, the policy and the frozen KL reference. The RM also initialises PPO's value network.

Stage 3: RL with a KL penalty

max⁡ϕ Ex∼D, y∼πϕRL(⋅∣x)[rθ(x,y)−βlog⁡πϕRL(y∣x)πSFT(y∣x)]  +  γ Ex∼Dpretrain[log⁡πϕRL(x)]\max_{\phi}\ \mathbb E_{x\sim\mathcal D,\ y\sim\pi^{RL}_\phi(\cdot\mid x)}\Big[r_\theta(x,y)-\beta\log\frac{\pi^{RL}_\phi(y\mid x)}{\pi^{SFT}(y\mid x)}\Big]\;+\;\gamma\,\mathbb E_{x\sim\mathcal D_{pretrain}}\big[\log\pi^{RL}_\phi(x)\big]
πϕRL\pi^{RL}_\phi
the policy being trained, initialised from πSFT\pi^{SFT}
β\beta
KL coefficient. In expectation the penalty is β KL(πRL∥πSFT)\beta\,\mathrm{KL}(\pi^{RL}\|\pi^{SFT}); InstructGPT used β=0.02\beta=0.02
γ\gamma
"PPO-ptx": mixing in pretraining gradients reduced the regressions on public NLP benchmarks (the "alignment tax"); γ=0\gamma=0 is plain PPO
log⁡πRLπSFT\log\frac{\pi^{RL}}{\pi^{SFT}}
summed over the generated tokens; in practice applied as a per-token reward −β(log⁡πRL(yt)−log⁡πSFT(yt))-\beta(\log\pi^{RL}(y_t)-\log\pi^{SFT}(y_t)), with rθr_\theta added at the final token

Each generated answer is one episode. The tokens are the actions, and the reward is sparse: the RM score arrives at the last token, while the KL penalty is paid along the way. The optimiser is Proximal policy optimization (Schulman et al., 2017). PPO clips the probability ratio between the new and old policy so that each update stays close to the policy that generated the samples. A value head, initialised from the RM, estimates returns for generalised advantage estimation. This means four networks are in memory at once: the policy, the reference, the reward model and the value model.

The KL-regularised objective has a closed-form optimum, which is useful for intuition and is the starting point for Direct preference optimization (DPO):

π∗(y∣x)=1Z(x) πSFT(y∣x) exp⁡ ⁣(rθ(x,y)/β)\pi^\ast(y\mid x)=\frac{1}{Z(x)}\,\pi^{SFT}(y\mid x)\,\exp\!\big(r_\theta(x,y)/\beta\big)
β→∞\beta\to\infty
π∗=πSFT\pi^\ast=\pi^{SFT}: the reward is ignored
β→0\beta\to0
π∗\pi^\ast puts all mass on arg⁡max⁡yrθ(x,y)\arg\max_y r_\theta(x,y), which is exactly what exploits RM errors
Z(x)Z(x)
a sum over all possible responses, which is intractable. PPO approximates π∗\pi^\ast by sampling instead

Reward hacking and the KL coefficient

The reward model is a proxy. It was trained on outputs from πSFT\pi^{SFT}, and it is only accurate near that distribution. RL is an optimiser, and it will find any input on which the proxy is wrong in its favour. This is Goodhart's law: once the measure becomes the target, it stops being a good measure. Typical exploits in language models are longer answers, confident-sounding filler, flattery and agreement with the user, and repeated phrases that the RM happens to like. Stiennon et al. saw this directly. As they optimised harder against their RM, its score kept rising while the human ratings of the summaries peaked and then fell.

RgoldRL(d)=d (αRL−βRLlog⁡d),RgoldBoN(d)=d (αBoN−βBoN d),d=KL(π ∥ πinit)R_{gold}^{RL}(d)=d\,\big(\alpha_{RL}-\beta_{RL}\log d\big),\qquad R_{gold}^{BoN}(d)=d\,\big(\alpha_{BoN}-\beta_{BoN}\,d\big),\qquad d=\sqrt{\mathrm{KL}(\pi\,\|\,\pi_{init})}
RgoldR_{gold}
the gain in "true" reward, measured by a separate gold RM that labelled the proxy RM's training data
dd
distance travelled from the initial policy; best-of-nn sampling and RL both trace a rise then a fall
α, β\alpha,\ \beta
fitted coefficients that change smoothly with proxy-RM size: larger RMs overoptimise later

Gao, Schulman and Hilton, "Scaling Laws for Reward Model Overoptimization" (ICML 2023), made this quantitative with a synthetic setup: a 6B "gold" RM stands in for humans, and smaller proxy RMs are trained on its labels. The proxy score rises monotonically with dd. The gold score follows the curves above: it peaks and then declines. In their RL runs, adding a KL penalty mostly changed how far along the gold-vs-KL curve training went, rather than moving the curve up. It acted much like early stopping. So β\beta is best understood as a budget. It sets how far from πSFT\pi^{SFT} the policy may move, and the right budget is the one where the gold curve peaks. More and better preference data push that peak further out. So does collecting fresh comparisons on the current policy's outputs (Stiennon et al. and Bai et al. 2022 iterated this way).

reward gain over π_ref vs √KL(π_β ‖ π_ref)01201234proxy r̂ (what PPO sees)gold r* (what people wanted)response length ℓ0.00.20.40.60.81.0□ π_ref · ■ π_β · mean ℓ = 0.46reward vs ℓ: gold · RM 1.33ℓ − 0.09ℓ²RM fit on 1,000 pairs: r̂ = 0.96q + 1.33ℓ − 0.09ℓ² (gold: 1q + 2.4ℓ − 3ℓ²)RM pair accuracy: 97.8% on π_ref samples, 29.0% on π_β samplesβ = 0.100: KL = 3.08 nats (√KL 1.76), proxy +1.212, gold +1.037best gold at β = 0.112 (√KL 1.66): +1.038; the best possible is +1.112near the gold optimum along this curve77 responses (7 quality × 11 length) · π_β ∝ π_ref·exp(r̂/β), the optimum PPO with a KL penalty targets
comparisons from
fig.Overoptimisation, computed exactly on a toy. There are 77 responses on a grid of 7 quality levels × 11 lengths. Gold reward is r* = q + 2.4ℓ − 3ℓ², so the best length is 0.4. π_ref prefers short answers (ℓ ≈ 0.25). A Bradley–Terry RM r̂ = w₁q + w₂ℓ + w₃ℓ² is fit by maximum likelihood on noisy comparisons. The policy for each β is the exact KL-regularised optimum π_β ∝ π_ref·exp(r̂/β). Defaults: 1,000 pairs drawn from π_ref. The fit is r̂ = 0.96q + 1.33ℓ − 0.09ℓ², which orders 97.8% of π_ref pairs correctly but misses the curvature, because π_ref has almost no long answers. At β = 0.100 the policy moves KL = 3.08 nats (√KL 1.76), the mean length rises to 0.46, the proxy gains +1.212 and gold +1.037, close to the gold peak of +1.038 at β = 0.112. On π_β's own samples the RM is right only 29.0% of the time. At β = 0.01 (√KL 4.44) the proxy climbs to +1.876 while gold collapses to +0.032. Every answer is now maximal length, and the RM's pair accuracy on π_β is 0.0%. At β = 1, proxy and gold agree (+0.406 vs +0.411). Switching the comparisons to ½ π_ref + ½ uniform gives r̂ = 0.95q + 1.74ℓ − 2.19ℓ². Then even β = 0.01 is safe (√KL 1.76, gold +1.106, against +1.112 for the best response).

Worked example: one ranking, one episode

pair (winner, loser)r_w − r_l−log σ(r_w − r_l)
A, B0.90.3412
A, C1.50.2014
A, D2.20.1051
B, C0.60.4375
B, D1.30.2410
C, D0.70.4032
mean over the 6 = C(4,2) pairs0.2882
Reward-model loss on one prompt with K = 4 ranked answers A ≻ B ≻ C ≻ D and current scores r = (1.3, 0.4, −0.2, −0.9)

Rewarding one 4-token answer in PPO (β = 0.02, RM score 0.85)

  1. 1Log-probabilities of the sampled tokens under the policy: (−0.9,−1.6,−0.4,−2.1)(-0.9,-1.6,-0.4,-2.1). Under the frozen SFT model: (−1.2,−1.5,−0.9,−2.0)(-1.2,-1.5,-0.9,-2.0).
  2. 2Per-token log-ratios log⁡πRL−log⁡πSFT=(0.3,−0.1,0.5,−0.1)\log\pi^{RL}-\log\pi^{SFT}=(0.3,-0.1,0.5,-0.1). Their sum, 0.60.6, is a one-sample estimate of the sequence KL.
  3. 3Per-token rewards −β⋅-\beta\cdot log-ratio =(−0.006, 0.002, −0.010, 0.002)=(-0.006,\,0.002,\,-0.010,\,0.002), then add the RM score at the last token: (−0.006, 0.002, −0.010, 0.852)(-0.006,\,0.002,\,-0.010,\,0.852).
  4. 4Episode return 0.838=0.85−0.02⋅0.60.838=0.85-0.02\cdot0.6. The KL term is small here. It grows when the policy puts much more probability than the SFT model on tokens it likes, which is exactly what an exploit looks like.
  5. 5For intuition about β\beta: with πSFT=(0.5,0.3,0.2)\pi^{SFT}=(0.5,0.3,0.2) over three answers with rewards (0,1,2)(0,1,2), the optimum π∗∝πSFTer/β\pi^\ast\propto\pi^{SFT}e^{r/\beta} is (0.325,0.322,0.353)(0.325,0.322,0.353) at β=2\beta=2 (KL 0.084), (0.179,0.292,0.529)(0.179,0.292,0.529) at β=1\beta=1 (KL 0.323), (0.037,0.163,0.801)(0.037,0.163,0.801) at β=0.5\beta=0.5 (KL 0.915), and nearly one-hot at β=0.1\beta=0.1 (KL 1.609, which is log⁡(1/0.2)\log(1/0.2)).
python
1import itertools, torch, torch.nn.functional as F
2
3def rm_loss(scores):
4 """scores: (K,) RM outputs for K answers to one prompt, sorted best → worst.
5 All C(K,2) pairs share one batch element, as in InstructGPT."""
6 pairs = list(itertools.combinations(range(scores.numel()), 2)) # (winner, loser) in rank order
7 w = torch.tensor([i for i, _ in pairs]); l = torch.tensor([j for _, j in pairs])
8 return -F.logsigmoid(scores[w] - scores[l]).mean()
9
10print(rm_loss(torch.tensor([1.3, 0.4, -0.2, -0.9]))) # tensor(0.2882), matches the worked example
11
12def shaped_rewards(logp_policy, logp_ref, rm_score, mask, beta=0.02):
13 """Per-token reward: -beta * (log pi - log pi_ref), plus the RM score on the last real token.
14 logp_*: (B, T) log-probs of the sampled tokens; mask: (B, T) 1 for response tokens."""
15 kl = (logp_policy - logp_ref) * mask
16 rewards = -beta * kl
17 last = mask.sum(1).long() - 1 # index of the final response token
18 rewards[torch.arange(rewards.size(0)), last] += rm_score
19 return rewards, kl.sum(1) # rewards (B, T), sequence KL estimate (B,)
20
21def ppo_policy_loss(logp_new, logp_old, advantages, mask, clip=0.2):
22 ratio = torch.exp(logp_new - logp_old) # pi_new / pi_old per token
23 unclipped = ratio * advantages
24 clipped = torch.clamp(ratio, 1 - clip, 1 + clip) * advantages
25 return -(torch.min(unclipped, clipped) * mask).sum() / mask.sum()
26
27# one PPO iteration, schematically:
28# y ~ policy.generate(x) (sampling: the expensive part)
29# logp_old, values = policy(x, y), value_net(x, y); logp_ref = sft(x, y); s = reward_model(x, y)
30# rewards, kl = shaped_rewards(logp_old, logp_ref, s, mask)
31# advantages, returns = gae(rewards, values, mask, gamma=1.0, lam=0.95)
32# for a few epochs over minibatches: loss = ppo_policy_loss(...) + c_v * value_loss(...)
The three RLHF losses in PyTorch: Bradley–Terry RM loss over K ranked answers, the KL-shaped per-token reward, and PPO's clipped policy loss (written for clarity; not run here)

Use when

  • +Learns from judgements, which are cheaper and more reliable than demonstrations for many qualities
  • +Can push past the quality of the demonstrations; InstructGPT 1.3B was preferred over GPT-3 175B
  • +The reward model is reusable: for best-of-n, for rejection sampling, and as an evaluator
  • +On-policy RL corrects the model's own mistakes rather than imitating someone else's

Avoid when

  • −Complex and expensive: four models in memory and generation inside the training loop
  • −Reward hacking: the policy exploits RM errors (length, flattery, style) once it moves far from π_SFT
  • −Sensitive to β, the PPO hyperparameters, and the RM's size and data coverage
  • −Alignment tax on standard benchmarks unless pretraining data is mixed in (PPO-ptx)

Check yourself

Why does InstructGPT put all C(K,2) comparisons from one ranked prompt into a single batch element instead of shuffling them as independent examples?

Check yourself

You halve the KL coefficient β and the reward-model score of your PPO policy goes up a lot, but human raters now prefer it less often. What is the most likely explanation?

Remember

  • ✓RLHF has three stages: SFT on demonstrations; a reward model trained with −log⁡σ(rθ(x,yw)−rθ(x,yl))-\log\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)) on human comparisons; then PPO on rθ(x,y)−βlog⁡(πRL/πSFT)r_\theta(x,y)-\beta\log(\pi^{RL}/\pi^{SFT}).
  • ✓Lineage: Christiano et al. 2017 (Atari/MuJoCo), Ziegler et al. 2019 and Stiennon et al. 2020 (summarisation), and InstructGPT 2022 (13k SFT, 33k RM and 31k PPO prompts; a 6B RM; β=0.02\beta=0.02; PPO-ptx).
  • ✓The KL-regularised optimum is π∗∝πSFTexp⁡(r/β)\pi^\ast\propto\pi^{SFT}\exp(r/\beta); β\beta sets how far the policy may move from the data the RM was trained on.
  • ✓Overoptimisation: the proxy reward rises with KL while the true reward peaks and then falls. Gao et al. fit the true reward as d(α−βlog⁡d)d(\alpha-\beta\log d) with d=KLd=\sqrt{\mathrm{KL}}.
  • ✓Watch held-out preference judgements, length and KL, not the RM score. Fresh comparisons on policy samples extend the safe KL budget.
← back to Deep Learning roadmap