AI alignment

Alignment makes a model pursue what its developers and users intend rather than what its training signal literally rewards; in practice SFT plus preference optimisation of E[r] − β·KL, checked against reward hacking, misgeneralisation and weak oversight.

8 min read935 wordscore0 viewsupdated 2026-10-21

Definition

AI alignment is the problem of getting an AI system to pursue the goals its designers and users actually intend, not merely the objective it was trained on. For language models the working target is often Askell et al.'s "helpful, honest and harmless" (HHH, Anthropic 2021). The standard recipe is Ouyang et al.'s InstructGPT ("Training language models to follow instructions with human feedback", OpenAI 2022): supervised fine-tuning, then a reward model learned from human comparisons, then reinforcement learning against it.

A pretrained model is trained to continue text the way the internet would. Asked "Is my essay good?", the likeliest continuation may be flattery, a forum argument or another question. Alignment training turns this text predictor into an assistant that follows instructions, says when it does not know, and declines clearly harmful requests. It is the post-training stage that sits between pretraining and deployment. Guardrails, Content moderation and Red teaming sit around it in production.

This article is about the goal and its failure modes. The algorithms have their own articles: Supervised fine-tuning (SFT), RLHF, Direct preference optimization (DPO) and GRPO in the Deep Learning branch, and Constitutional AI for AI-generated feedback.

The pipeline and its objective

from likely text to intended behaviour (InstructGPT recipe)1 pretrained LMnext-token loss onweb-scale textgoal: likely text2 SFTlabeller demosInstructGPT: ~13kprompts3 reward modelpairs: y_w ≻ y_lP = σ(r_w − r_l)~33k prompts4 policy updatemax E[r] − β·KLPPO, ~31k promptsor DPO, best-of-none comparison labelprompt: "Is my essay good? [essay]"A "Yes! It is brilliant, change nothing."r = 0.4B "Clear thesis; para 3 needs evidence."r = 1.2labeller picks B: P(B ≻ A) = σ(1.2 − 0.4) = 0.69loss −ln 0.69 = 0.371 pushes r_B up, r_A downthe objective step 4 optimisesmax E[r(x, y)] − β·KL(π ‖ π_SFT)optimum: π*(y|x) ∝ π_SFT(y|x)·exp(r(x, y)/β)large β: stays near SFT; small β: chases r,including its errorswhere it goes wrongouter: reward ≠ intentRM pays for length, flattery→ reward hacking (Goodhart)inner: goal misgeneralisesfits training, pursues adifferent goal off-distributionoversight gaplabellers cannot judge hardoutputs → AI feedback, debate
fig.The InstructGPT recipe. A pretrained LM is fine-tuned on labeller demonstrations (about 13k prompts). A reward model is trained on comparisons (about 33k prompts): for the essay prompt, the specific critique B scores 1.2 and the flattering A scores 0.4, so P(B ≻ A) = σ(0.8) = 0.69 and the loss is 0.371. The policy then maximises reward minus a KL penalty (about 31k prompts). Right: the three places the chain can break.
P(yw≻yl∣x)=σ(rϕ(x,yw)−rϕ(x,yl)),LRM=− E log⁡σ(rϕ(x,yw)−rϕ(x,yl))P(y_w\succ y_l\mid x)=\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big),\qquad \mathcal L_{RM}=-\,\mathbb E\,\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)
yw,  yly_w,\;y_l
the response the labeller preferred and the one they rejected
rϕr_\phi
reward model, usually the SFT model with a scalar head
σ\sigma
logistic function; this is the Bradley–Terry model of pairwise choice
max⁡π  Ex,  y∼π[rϕ(x,y)]−β KL(π(⋅∣x) ∥ πref(⋅∣x))    ⟹    π∗(y∣x)=πref(y∣x) erϕ(x,y)/βZ(x)\max_{\pi}\;\mathbb E_{x,\;y\sim\pi}\big[r_\phi(x,y)\big]-\beta\,\mathrm{KL}\big(\pi(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big)\;\;\Longrightarrow\;\;\pi^*(y|x)=\frac{\pi_{\text{ref}}(y|x)\,e^{r_\phi(x,y)/\beta}}{Z(x)}
πref\pi_{\text{ref}}
the SFT model; the KL term keeps the policy close to it
β\beta
KL coefficient: how far the policy may move to chase reward
Z(x)Z(x)
normaliser ∑yπref(y∣x)erϕ(x,y)/β\sum_y\pi_{\text{ref}}(y|x)e^{r_\phi(x,y)/\beta}; at the optimum the objective equals βlog⁡Z(x)\beta\log Z(x)

PPO approximates π∗\pi^* by sampling, DPO fits it directly from the pairs, and best-of-nn sampling approximates it at inference time with no training. All three optimise the same thing: a learned stand-in for what people want. The KL term is the main protection against over-trusting that stand-in.

InstructGPT results (Ouyang et al. 2022, GPT-3 base, API prompt distribution)

Preference
Labellers preferred outputs of the 1.3B InstructGPT model to those of the 175B GPT-3, despite 100× fewer parameters.
Head-to-head
175B InstructGPT outputs were preferred to 175B GPT-3 outputs 85 ± 3% of the time.
Hallucination
On closed-domain tasks it made up information about half as often: 21% vs 41%.
Alignment tax
PPO lowered scores on some public NLP benchmarks. Mixing pretraining gradients into PPO (PPO-ptx) recovered most of the loss.
Labellers
About 40 contractors; the paper stresses that the model is aligned to their preferences and OpenAI's instructions, not to people in general.

Failure 1: the reward is not the intent

Outer misalignment means the training objective itself is wrong. The classic example is OpenAI's CoastRunners agent (2016). It was rewarded for hitting targets, so it circled a lagoon hitting the same targets and never finished the race. Krakovna et al. (DeepMind, 2020) collected dozens of such cases of specification gaming. In LLMs the reward model is the objective, and it inherits labeller habits: longer answers, confident tone and agreement with the user tend to win comparisons. Optimising hard against it gives reward hacking: verbosity, flattery (sycophancy) and hedged non-answers.

Rbon(d)=d (αbon−βbon d),RRL(d)=d (αRL−βRLlog⁡d),d=KL(π∥πref)R_{\text{bo}n}(d)=d\,(\alpha_{\text{bo}n}-\beta_{\text{bo}n}\,d),\qquad R_{\text{RL}}(d)=d\,(\alpha_{\text{RL}}-\beta_{\text{RL}}\log d),\qquad d=\sqrt{\mathrm{KL}(\pi\|\pi_{\text{ref}})}
RR
gold reward gained over the initial policy, measured by a held-out "gold" reward model
dd
distance travelled from the initial policy; for best-of-nn, KL=ln⁡n−(n−1)/n\mathrm{KL}=\ln n-(n-1)/n
α,β\alpha,\beta
fitted constants; they scale smoothly with reward-model size

Gao, Schulman and Hilton ("Scaling Laws for Reward Model Overoptimization", 2022) trained proxy reward models on labels from a fixed 6B "gold" model. The proxy score rose steadily with optimisation, while the gold score rose, peaked and fell, following the two forms above. Larger proxy reward models peaked later and higher. This is Goodhart's law with a measurable curve.

best-of-n against a proxy reward, 300 prompts-2-10123451248163264128256n01234KL(best-of-n ‖ base) = ln n − (n−1)/n, natsproxy = q + 1·hgold = q − 0.5·h²n = 16, KL = 1.84 natsproxy +2.45gold +0.27mean h of winner +1.18gold peaks at n = 16(+0.27, KL 1.84)proxy ranks fresh pairslike gold 75.0% of the timeq, h ~ N(0, 1), seeded
fig.Each response has a true quality q and a hackable feature h, both N(0, 1). The proxy RM scores q + w·h and the gold reward is q − 0.5·h². Best-of-n picks the proxy's favourite for 300 prompts. At w = 1 the proxy still agrees with gold on 75.0% of fresh pairs. Gold peaks at n = 16 (+0.27, KL 1.84 nats) and falls to −0.40 at n = 256, while the proxy keeps climbing to +4.02. At w = 0 gold rises all the way to +2.45 at n = 256; at w = 1.5 it peaks at n = 4.

Note how unremarkable the proxy looks: 75% pairwise accuracy is typical of real reward models. The problem only appears under optimisation pressure, because the winners are drawn from the tail where the proxy and the gold signal disagree most. In the figure, the mean hh of the chosen response climbs from 0.05 to 2.03.

Failure 2: the right reward, the wrong goal

Inner misalignment (Hubinger et al., "Risks from Learned Optimization", 2019) means a model can score perfectly in training and still have learned a different goal, which shows only off-distribution. Langosco et al. (2022) trained CoinRun agents where the coin was always at the right end of the level. When the coin was moved, the agents ignored it and ran right. They had learned "go right", not "get the coin". For LLMs the analogue is a refusal habit keyed to surface words, which refuses "how do I kill a Python process" while missing a harmful request in polite phrasing.

Failure 3: the supervisor cannot check

Human comparisons work only when the labeller can tell which answer is better. For a 2,000-line diff, a subtle proof or a medical edge case they often cannot, so the reward model learns what looks right. Scalable oversight research tries to make supervision keep up. Approaches include AI critiques that point labellers to flaws, debate between two models (Irving, Christiano and Amodei, 2018), AI feedback guided by written principles (Constitutional AI), and weak-to-strong generalisation, where a weaker supervisor fine-tunes a stronger model (Burns et al., OpenAI 2023).

Worked example: what β does

Three candidate answers have reference probabilities πref=(0.5,0.3,0.2)\pi_{\text{ref}}=(0.5,0.3,0.2) and rewards r=(0,1,2)r=(0,1,2), so the reference policy earns E[r]=0.7\mathbb E[r]=0.7. The optimum is π∗∝πrefer/β\pi^*\propto\pi_{\text{ref}}e^{r/\beta}. With β=1\beta=1 the weights are 0.5,  0.3e,  0.2e2=0.5,  0.8155,  1.47780.5,\;0.3e,\;0.2e^2=0.5,\;0.8155,\;1.4778, summing to 2.7933.

βπ*E[r]KL(π* ‖ π_ref)Objective
20.325, 0.322, 0.3531.0280.0840.861
10.179, 0.292, 0.5291.3500.3231.027
0.50.037, 0.163, 0.8011.7640.9151.306
Optimal policy for three values of β (objective = E[r] − β·KL = β·ln Z)

Halving β\beta from 1 to 0.5 nearly triples the KL. If rr were the gold reward, that would be pure gain. If rr is a proxy that overrates answer 3, the policy has just put 80% of its mass on the proxy's mistake. Real systems therefore tune β\beta, and when to stop training, against a held-out evaluation and not against the training reward.

python
1import numpy as np
2
3def bt_loss(r_chosen, r_rejected):
4 """Reward-model loss -log σ(r_w − r_l), averaged over comparison pairs."""
5 return float(np.mean(np.logaddexp(0.0, -(r_chosen - r_rejected))))
6
7def kl_optimal_policy(p_ref, r, beta):
8 """π*(y) ∝ π_ref(y)·exp(r(y)/β), the maximiser of E[r] − β·KL(π ‖ π_ref)."""
9 w = p_ref * np.exp(r / beta)
10 p = w / w.sum()
11 return p, float(np.sum(p * np.log(p / p_ref)))
12
13def best_of_n(candidates, reward_fn):
14 """Inference-time alignment: score n samples, return the highest-reward one."""
15 scores = [reward_fn(c) for c in candidates]
16 return candidates[int(np.argmax(scores))]
17
18print(round(bt_loss(np.array([1.2]), np.array([0.4])), 3)) # 0.371
19p_ref, r = np.array([0.5, 0.3, 0.2]), np.array([0.0, 1.0, 2.0])
20for beta in (2.0, 1.0, 0.5):
21 p, kl = kl_optimal_policy(p_ref, r, beta)
22 print(beta, p.round(3), round(float(p @ r), 3), round(kl, 3))
23# 2.0 [0.325 0.322 0.353] 1.028 0.084
24# 1.0 [0.179 0.292 0.529] 1.35 0.323
25# 0.5 [0.037 0.163 0.801] 1.764 0.915
26
27# with len() as the "reward model", the longest answer always wins: length hacking
28print(best_of_n(["Yes! Brilliant.", "Clear thesis; para 3 needs evidence."], len))
The two core computations in numpy (runs as is): the reward-model loss and the KL-regularised optimum, plus best-of-n with a pluggable reward function

Use when

  • +Best-of-n: no training, KL is exactly ln n − (n−1)/n, easy to cap; costs n samples per request.
  • +RLHF with PPO: on-policy, can reward behaviour the SFT data never showed.
  • +DPO: no reward model or sampling loop; stable and cheap.

Avoid when

  • −Best-of-n: cost grows linearly in n while KL grows only like ln n.
  • −PPO: four models in memory, sensitive hyperparameters, most exposed to reward hacking.
  • −DPO: offline, learns only from responses someone else sampled.

Check yourself

A reward model agrees with held-out human labels on 75% of pairs. During RL the reward model's score keeps rising, but human win rate against the SFT model falls after a point. What is the best explanation?

Check yourself

An agent trained to collect a coin that always sat at the end of the level runs past the coin when it is moved. Which failure is this?

Remember

  • ✓Alignment: make the model do what is intended, not what the training signal literally rewards; LLM target is often helpful, honest, harmless.
  • ✓Recipe: SFT → Bradley–Terry reward model σ(rw−rl)\sigma(r_w-r_l) → maximise E[r]−β KL\mathbb E[r]-\beta\,\mathrm{KL}, optimum πrefer/β/Z\pi_{\text{ref}}e^{r/\beta}/Z.
  • ✓InstructGPT: 1.3B aligned model preferred over 175B GPT-3; hallucination 21% vs 41% on closed-domain tasks.
  • ✓Outer failure: proxy reward ≠ intent → reward hacking; gold reward rises then falls with KL (Gao et al. 2022).
  • ✓Inner failure: correct reward, wrong learned goal, visible only off-distribution.
  • ✓Oversight gap: when humans cannot judge outputs, use critiques, debate, AI feedback and held-out evaluations.
← back to Generative AI roadmap