# RLHF

> RLHF fine-tunes an LM in three stages: SFT on demonstrations, a reward model fit to human comparisons with the Bradley–Terry loss −log σ(r_w − r_l), then PPO on r − β·log(π/π_SFT). The KL term limits reward hacking: pushed too far, the proxy reward rises while true quality falls.

Source: https://www.mlfoundry.in/learn/rlhf  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 16 min read · updated 2026-10-13

**Reinforcement learning from human feedback (RLHF):** trains a policy against a reward that is *learned from human preference judgements*, instead of a reward written by hand. People compare pairs of outputs, a **reward model** $r_\theta(x,y)$ is fit so that preferred outputs score higher, and the policy is optimised with reinforcement learning to maximise $r_\theta$. A KL penalty keeps the policy close to a reference model. The method comes from Christiano et al., "Deep Reinforcement Learning from Human Preferences" (NeurIPS 2017). It was applied to language models by Ziegler et al. (2019) and Stiennon et al., "Learning to Summarize from Human Feedback" (NeurIPS 2020). It became the standard alignment recipe with Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT, NeurIPS 2022).

A pretrained LM is trained to continue internet text, not to be helpful. Asked a question, it may continue with more questions, make things up, or produce toxic text, because all of these are plausible continuations. **Supervised fine-tuning (SFT)** on human-written answers helps, but demonstrations are expensive, and imitation copies the demonstrator's mistakes along with their skill. Many qualities people care about are also easier to *judge* than to *write*: "which of these two summaries is better?" is a quick question even when writing a good summary is slow. RLHF turns these cheap judgements into a training signal.

Christiano, Leike, Brown, Martic, Legg and Amodei first showed this on Atari and MuJoCo. Humans watched pairs of 1–2 second clips and said which was closer to the goal. A reward model learned from these comparisons, and an RL agent trained on the learned reward. Feedback on less than 1% of the agent's interactions was enough. A simulated robot learned to do a backflip from about 900 human comparisons, a behaviour that is hard to specify as a reward function. Ziegler et al. (2019) moved the recipe to GPT-2 and added the KL penalty to the pretrained model. Stiennon et al. (2020) scaled it to summarisation of Reddit TL;DR posts with 1.3B and 6.7B models: labellers preferred the RLHF summaries over the human reference summaries. InstructGPT applied the same pipeline to general instruction following, and a 1.3B InstructGPT model was preferred over the 175B GPT-3, which has 100 times more parameters.

## Stage 1: supervised fine-tuning

Start from a pretrained LM and fine-tune it on prompt–demonstration pairs with the ordinary next-token loss (see **Supervised fine-tuning (SFT)**). InstructGPT used about 13k training prompts, taken from the OpenAI API and written by labellers, each with a labeller-written answer. The result, $\pi^{SFT}$, plays three roles later. It initialises the policy, it is the frozen reference for the KL penalty, and (at 6B) it initialises the reward model.

## Stage 2: the reward model

$$
P(y_w\succ y_l\mid x)=\sigma\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big),\qquad \mathcal L_{RM}(\theta)=-\frac{1}{\binom K2}\,\mathbb E_{(x,y_w,y_l)}\Big[\log\sigma\big(r_\theta(x,y_w)-r_\theta(x,y_l)\big)\Big]
$$

- $\sigma(r_w-r_l)$: the **Bradley–Terry** model (Bradley & Terry, 1952): only reward *differences* matter, so $r_\theta$ is identified up to a constant per prompt
- $r_\theta(x,y)$: the SFT transformer with its unembedding replaced by a scalar head, read at the last token
- $K$: InstructGPT showed labellers 4 to 9 responses per prompt and had them ranked, which gives $\binom K2$ pairs per prompt
- $1/\binom K2$: all pairs from one prompt are put into one batch element. Treating them as independent samples made the RM overfit within one epoch

This is logistic regression on reward differences. The RM never sees an absolute score, only which of two outputs won. Ranking $K$ outputs costs about as much labeller time as comparing two, but it yields $\binom K2$ comparisons: 6 for $K=4$, 36 for $K=9$. InstructGPT trained on about 33k prompts. It used a 6B RM because 175B RMs trained unstably and cost far more in the RL loop. Before RL, the reward is shifted so that labeller demonstrations score 0 on average, since a per-prompt constant has no effect on the comparisons.

> Figure: The InstructGPT pipeline. Stage 1 fine-tunes GPT-3 on about 13k demonstrations. Stage 2 has labellers rank 4–9 sampled answers for about 33k prompts and fits a 6B reward model with the Bradley–Terry loss on every pair. Stage 3 runs PPO on about 31k prompts: the policy samples an answer, the frozen RM scores it, and a per-token penalty β·log(π_RL/π_SFT) with β = 0.02 is subtracted. The SFT model initialises the RM, the policy and the frozen KL reference. The RM also initialises PPO's value network.

## Stage 3: RL with a KL penalty

$$
\max_{\phi}\ \mathbb E_{x\sim\mathcal D,\ y\sim\pi^{RL}_\phi(\cdot\mid x)}\Big[r_\theta(x,y)-\beta\log\frac{\pi^{RL}_\phi(y\mid x)}{\pi^{SFT}(y\mid x)}\Big]\;+\;\gamma\,\mathbb E_{x\sim\mathcal D_{pretrain}}\big[\log\pi^{RL}_\phi(x)\big]
$$

- $\pi^{RL}_\phi$: the policy being trained, initialised from $\pi^{SFT}$
- $\beta$: KL coefficient. In expectation the penalty is $\beta\,\mathrm{KL}(\pi^{RL}\|\pi^{SFT})$; InstructGPT used $\beta=0.02$
- $\gamma$: "PPO-ptx": mixing in pretraining gradients reduced the regressions on public NLP benchmarks (the "alignment tax"); $\gamma=0$ is plain PPO
- $\log\frac{\pi^{RL}}{\pi^{SFT}}$: summed over the generated tokens; in practice applied as a per-token reward $-\beta(\log\pi^{RL}(y_t)-\log\pi^{SFT}(y_t))$, with $r_\theta$ added at the final token

Each generated answer is one episode. The tokens are the actions, and the reward is sparse: the RM score arrives at the last token, while the KL penalty is paid along the way. The optimiser is **Proximal policy optimization** (Schulman et al., 2017). PPO clips the probability ratio between the new and old policy so that each update stays close to the policy that generated the samples. A value head, initialised from the RM, estimates returns for generalised advantage estimation. This means four networks are in memory at once: the policy, the reference, the reward model and the value model.

The KL-regularised objective has a closed-form optimum, which is useful for intuition and is the starting point for **Direct preference optimization (DPO)**:

$$
\pi^\ast(y\mid x)=\frac{1}{Z(x)}\,\pi^{SFT}(y\mid x)\,\exp\!\big(r_\theta(x,y)/\beta\big)
$$

- $\beta\to\infty$: $\pi^\ast=\pi^{SFT}$: the reward is ignored
- $\beta\to0$: $\pi^\ast$ puts all mass on $\arg\max_y r_\theta(x,y)$, which is exactly what exploits RM errors
- $Z(x)$: a sum over all possible responses, which is intractable. PPO approximates $\pi^\ast$ by sampling instead

## Reward hacking and the KL coefficient

The reward model is a proxy. It was trained on outputs from $\pi^{SFT}$, and it is only accurate near that distribution. RL is an optimiser, and it will find any input on which the proxy is wrong in its favour. This is Goodhart's law: once the measure becomes the target, it stops being a good measure. Typical exploits in language models are longer answers, confident-sounding filler, flattery and agreement with the user, and repeated phrases that the RM happens to like. Stiennon et al. saw this directly. As they optimised harder against their RM, its score kept rising while the human ratings of the summaries peaked and then fell.

$$
R_{gold}^{RL}(d)=d\,\big(\alpha_{RL}-\beta_{RL}\log d\big),\qquad R_{gold}^{BoN}(d)=d\,\big(\alpha_{BoN}-\beta_{BoN}\,d\big),\qquad d=\sqrt{\mathrm{KL}(\pi\,\|\,\pi_{init})}
$$

- $R_{gold}$: the gain in "true" reward, measured by a separate gold RM that labelled the proxy RM's training data
- $d$: distance travelled from the initial policy; best-of-$n$ sampling and RL both trace a rise then a fall
- $\alpha,\ \beta$: fitted coefficients that change smoothly with proxy-RM size: larger RMs overoptimise later

Gao, Schulman and Hilton, "Scaling Laws for Reward Model Overoptimization" (ICML 2023), made this quantitative with a synthetic setup: a 6B "gold" RM stands in for humans, and smaller proxy RMs are trained on its labels. The proxy score rises monotonically with $d$. The gold score follows the curves above: it peaks and then declines. In their RL runs, adding a KL penalty mostly changed how far along the gold-vs-KL curve training went, rather than moving the curve up. It acted much like early stopping. So $\beta$ is best understood as a *budget*. It sets how far from $\pi^{SFT}$ the policy may move, and the right budget is the one where the gold curve peaks. More and better preference data push that peak further out. So does collecting fresh comparisons on the current policy's outputs (Stiennon et al. and Bai et al. 2022 iterated this way).

> Figure: Overoptimisation, computed exactly on a toy. There are 77 responses on a grid of 7 quality levels × 11 lengths. Gold reward is r* = q + 2.4ℓ − 3ℓ², so the best length is 0.4. π_ref prefers short answers (ℓ ≈ 0.25). A Bradley–Terry RM r̂ = w₁q + w₂ℓ + w₃ℓ² is fit by maximum likelihood on noisy comparisons. The policy for each β is the exact KL-regularised optimum π_β ∝ π_ref·exp(r̂/β). Defaults: 1,000 pairs drawn from π_ref. The fit is r̂ = 0.96q + 1.33ℓ − 0.09ℓ², which orders 97.8% of π_ref pairs correctly but misses the curvature, because π_ref has almost no long answers. At β = 0.100 the policy moves KL = 3.08 nats (√KL 1.76), the mean length rises to 0.46, the proxy gains +1.212 and gold +1.037, close to the gold peak of +1.038 at β = 0.112. On π_β's own samples the RM is right only 29.0% of the time. At β = 0.01 (√KL 4.44) the proxy climbs to +1.876 while gold collapses to +0.032. Every answer is now maximal length, and the RM's pair accuracy on π_β is 0.0%. At β = 1, proxy and gold agree (+0.406 vs +0.411). Switching the comparisons to ½ π_ref + ½ uniform gives r̂ = 0.95q + 1.74ℓ − 2.19ℓ². Then even β = 0.01 is safe (√KL 1.76, gold +1.106, against +1.112 for the best response).

## Worked example: one ranking, one episode

Reward-model loss on one prompt with K = 4 ranked answers A ≻ B ≻ C ≻ D and current scores r = (1.3, 0.4, −0.2, −0.9)

| pair (winner, loser) | r_w − r_l | −log σ(r_w − r_l) |
| --- | --- | --- |
| A, B | 0.9 | 0.3412 |
| A, C | 1.5 | 0.2014 |
| A, D | 2.2 | 0.1051 |
| B, C | 0.6 | 0.4375 |
| B, D | 1.3 | 0.2410 |
| C, D | 0.7 | 0.4032 |
| mean over the 6 = C(4,2) pairs |  | 0.2882 |

**Rewarding one 4-token answer in PPO (β = 0.02, RM score 0.85)**

1. Log-probabilities of the sampled tokens under the policy: $(-0.9,-1.6,-0.4,-2.1)$. Under the frozen SFT model: $(-1.2,-1.5,-0.9,-2.0)$.
2. Per-token log-ratios $\log\pi^{RL}-\log\pi^{SFT}=(0.3,-0.1,0.5,-0.1)$. Their sum, $0.6$, is a one-sample estimate of the sequence KL.
3. Per-token rewards $-\beta\cdot$ log-ratio $=(-0.006,\,0.002,\,-0.010,\,0.002)$, then add the RM score at the last token: $(-0.006,\,0.002,\,-0.010,\,0.852)$.
4. Episode return $0.838=0.85-0.02\cdot0.6$. The KL term is small here. It grows when the policy puts much more probability than the SFT model on tokens it likes, which is exactly what an exploit looks like.
5. For intuition about $\beta$: with $\pi^{SFT}=(0.5,0.3,0.2)$ over three answers with rewards $(0,1,2)$, the optimum $\pi^\ast\propto\pi^{SFT}e^{r/\beta}$ is $(0.325,0.322,0.353)$ at $\beta=2$ (KL 0.084), $(0.179,0.292,0.529)$ at $\beta=1$ (KL 0.323), $(0.037,0.163,0.801)$ at $\beta=0.5$ (KL 0.915), and nearly one-hot at $\beta=0.1$ (KL 1.609, which is $\log(1/0.2)$).

```python
import itertools, torch, torch.nn.functional as F

def rm_loss(scores):
    """scores: (K,) RM outputs for K answers to one prompt, sorted best → worst.
    All C(K,2) pairs share one batch element, as in InstructGPT."""
    pairs = list(itertools.combinations(range(scores.numel()), 2))   # (winner, loser) in rank order
    w = torch.tensor([i for i, _ in pairs]); l = torch.tensor([j for _, j in pairs])
    return -F.logsigmoid(scores[w] - scores[l]).mean()

print(rm_loss(torch.tensor([1.3, 0.4, -0.2, -0.9])))   # tensor(0.2882), matches the worked example

def shaped_rewards(logp_policy, logp_ref, rm_score, mask, beta=0.02):
    """Per-token reward: -beta * (log pi - log pi_ref), plus the RM score on the last real token.
    logp_*: (B, T) log-probs of the sampled tokens; mask: (B, T) 1 for response tokens."""
    kl = (logp_policy - logp_ref) * mask
    rewards = -beta * kl
    last = mask.sum(1).long() - 1                       # index of the final response token
    rewards[torch.arange(rewards.size(0)), last] += rm_score
    return rewards, kl.sum(1)                           # rewards (B, T), sequence KL estimate (B,)

def ppo_policy_loss(logp_new, logp_old, advantages, mask, clip=0.2):
    ratio = torch.exp(logp_new - logp_old)              # pi_new / pi_old per token
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1 - clip, 1 + clip) * advantages
    return -(torch.min(unclipped, clipped) * mask).sum() / mask.sum()

# one PPO iteration, schematically:
#   y ~ policy.generate(x)                         (sampling: the expensive part)
#   logp_old, values = policy(x, y), value_net(x, y);  logp_ref = sft(x, y);  s = reward_model(x, y)
#   rewards, kl = shaped_rewards(logp_old, logp_ref, s, mask)
#   advantages, returns = gae(rewards, values, mask, gamma=1.0, lam=0.95)
#   for a few epochs over minibatches:  loss = ppo_policy_loss(...) + c_v * value_loss(...)
```

The three RLHF losses in PyTorch: Bradley–Terry RM loss over K ranked answers, the KL-shaped per-token reward, and PPO's clipped policy loss (written for clarity; not run here)

> **A rising reward curve is not progress:** The RM score is the one number PPO is guaranteed to improve. Track something it was *not* trained on. Use held-out human or strong-model preference judgements against the SFT model, plus simple statistics such as response length, refusal rate and repetition. In the figure, the RM orders 97.8% of reference pairs correctly, yet on the samples of an over-optimised policy it is right 0.0% of the time. If the KL climbs steadily and the answers get longer, suspect reward hacking before you celebrate.

> **The reference is a trust region on data, not on weights:** The KL penalty does not say "stay near the old weights". It says "do not put probability on outputs the SFT model finds unlikely". Those are exactly the outputs where the RM has no training data. That is why the reference must be the model whose samples the RM was trained on. It is also why refreshing the RM with comparisons of the current policy's outputs lets you safely spend a larger KL budget.

> **Variations:** Best-of-$n$ sampling against the RM is the simplest use of a reward model, with no RL at all. Its KL from the base policy is about $\log n-(n-1)/n$. Llama 2 combined rejection-sampling fine-tuning with PPO. Constitutional AI and RLAIF replace human labels with model-generated preference labels. **Direct preference optimization (DPO)** removes the RM and PPO and optimises the same KL-regularised objective with a classification loss. **GRPO** drops the value network and uses group-normalised rewards as advantages.

**Strengths**

- Learns from judgements, which are cheaper and more reliable than demonstrations for many qualities
- Can push past the quality of the demonstrations; InstructGPT 1.3B was preferred over GPT-3 175B
- The reward model is reusable: for best-of-n, for rejection sampling, and as an evaluator
- On-policy RL corrects the model's own mistakes rather than imitating someone else's

**Limitations**

- Complex and expensive: four models in memory and generation inside the training loop
- Reward hacking: the policy exploits RM errors (length, flattery, style) once it moves far from π_SFT
- Sensitive to β, the PPO hyperparameters, and the RM's size and data coverage
- Alignment tax on standard benchmarks unless pretraining data is mixed in (PPO-ptx)

**Question:** Why does InstructGPT put all C(K,2) comparisons from one ranked prompt into a single batch element instead of shuffling them as independent examples?

- A. Because the Bradley–Terry loss is undefined for single pairs
- B. Because the pairs are highly correlated; treating them as independent made the reward model overfit within one epoch
- C. Because PPO requires whole rankings rather than pairs
- D. To make the reward model output probabilities instead of scores

**Answer:** B. Because the pairs are highly correlated; treating them as independent made the reward model overfit within one epoch — Each answer appears in $K-1$ pairs, so shuffled pairs let the RM see the same completions many times per epoch, and it overfit. Processing one prompt's $\binom K2$ comparisons together, with the loss averaged over them, needs only $K$ forward passes and avoided the overfitting.

**Question:** You halve the KL coefficient β and the reward-model score of your PPO policy goes up a lot, but human raters now prefer it less often. What is the most likely explanation?

- A. The PPO clip range is too small
- B. The policy moved further from π_SFT, into outputs where the reward model is wrong in its favour (overoptimisation)
- C. The reward model is underfitting its training pairs
- D. The SFT model was trained for too many epochs

**Answer:** B. The policy moved further from π_SFT, into outputs where the reward model is wrong in its favour (overoptimisation) — A smaller $\beta$ allows a larger KL budget. The proxy reward always rises with KL, but the true reward peaks and then falls, as Stiennon et al. observed and Gao et al. fit as $d(\alpha-\beta\log d)$. Restore a larger $\beta$, stop earlier, or improve the RM with comparisons collected on the new policy's outputs.

**Key points**

- RLHF has three stages: SFT on demonstrations; a reward model trained with $-\log\sigma(r_\theta(x,y_w)-r_\theta(x,y_l))$ on human comparisons; then PPO on $r_\theta(x,y)-\beta\log(\pi^{RL}/\pi^{SFT})$.
- Lineage: Christiano et al. 2017 (Atari/MuJoCo), Ziegler et al. 2019 and Stiennon et al. 2020 (summarisation), and InstructGPT 2022 (13k SFT, 33k RM and 31k PPO prompts; a 6B RM; $\beta=0.02$; PPO-ptx).
- The KL-regularised optimum is $\pi^\ast\propto\pi^{SFT}\exp(r/\beta)$; $\beta$ sets how far the policy may move from the data the RM was trained on.
- Overoptimisation: the proxy reward rises with KL while the true reward peaks and then falls. Gao et al. fit the true reward as $d(\alpha-\beta\log d)$ with $d=\sqrt{\mathrm{KL}}$.
- Watch held-out preference judgements, length and KL, not the RM score. Fresh comparisons on policy samples extend the safe KL budget.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
