Definition
AI alignment is the problem of getting an AI system to pursue the goals its designers and users actually intend, not merely the objective it was trained on. For language models the working target is often Askell et al.'s "helpful, honest and harmless" (HHH, Anthropic 2021). The standard recipe is Ouyang et al.'s InstructGPT ("Training language models to follow instructions with human feedback", OpenAI 2022): supervised fine-tuning, then a reward model learned from human comparisons, then reinforcement learning against it.
A pretrained model is trained to continue text the way the internet would. Asked "Is my essay good?", the likeliest continuation may be flattery, a forum argument or another question. Alignment training turns this text predictor into an assistant that follows instructions, says when it does not know, and declines clearly harmful requests. It is the post-training stage that sits between pretraining and deployment. Guardrails, Content moderation and Red teaming sit around it in production.
This article is about the goal and its failure modes. The algorithms have their own articles: Supervised fine-tuning (SFT), RLHF, Direct preference optimization (DPO) and GRPO in the Deep Learning branch, and Constitutional AI for AI-generated feedback.
The pipeline and its objective
- the response the labeller preferred and the one they rejected
- reward model, usually the SFT model with a scalar head
- logistic function; this is the Bradley–Terry model of pairwise choice
- the SFT model; the KL term keeps the policy close to it
- KL coefficient: how far the policy may move to chase reward
- normaliser ; at the optimum the objective equals
PPO approximates by sampling, DPO fits it directly from the pairs, and best-of- sampling approximates it at inference time with no training. All three optimise the same thing: a learned stand-in for what people want. The KL term is the main protection against over-trusting that stand-in.
InstructGPT results (Ouyang et al. 2022, GPT-3 base, API prompt distribution)
- Preference
- Labellers preferred outputs of the 1.3B InstructGPT model to those of the 175B GPT-3, despite 100× fewer parameters.
- Head-to-head
- 175B InstructGPT outputs were preferred to 175B GPT-3 outputs 85 ± 3% of the time.
- Hallucination
- On closed-domain tasks it made up information about half as often: 21% vs 41%.
- Alignment tax
- PPO lowered scores on some public NLP benchmarks. Mixing pretraining gradients into PPO (PPO-ptx) recovered most of the loss.
- Labellers
- About 40 contractors; the paper stresses that the model is aligned to their preferences and OpenAI's instructions, not to people in general.
Failure 1: the reward is not the intent
Outer misalignment means the training objective itself is wrong. The classic example is OpenAI's CoastRunners agent (2016). It was rewarded for hitting targets, so it circled a lagoon hitting the same targets and never finished the race. Krakovna et al. (DeepMind, 2020) collected dozens of such cases of specification gaming. In LLMs the reward model is the objective, and it inherits labeller habits: longer answers, confident tone and agreement with the user tend to win comparisons. Optimising hard against it gives reward hacking: verbosity, flattery (sycophancy) and hedged non-answers.
- gold reward gained over the initial policy, measured by a held-out "gold" reward model
- distance travelled from the initial policy; for best-of-,
- fitted constants; they scale smoothly with reward-model size
Gao, Schulman and Hilton ("Scaling Laws for Reward Model Overoptimization", 2022) trained proxy reward models on labels from a fixed 6B "gold" model. The proxy score rose steadily with optimisation, while the gold score rose, peaked and fell, following the two forms above. Larger proxy reward models peaked later and higher. This is Goodhart's law with a measurable curve.
Note how unremarkable the proxy looks: 75% pairwise accuracy is typical of real reward models. The problem only appears under optimisation pressure, because the winners are drawn from the tail where the proxy and the gold signal disagree most. In the figure, the mean of the chosen response climbs from 0.05 to 2.03.
Failure 2: the right reward, the wrong goal
Inner misalignment (Hubinger et al., "Risks from Learned Optimization", 2019) means a model can score perfectly in training and still have learned a different goal, which shows only off-distribution. Langosco et al. (2022) trained CoinRun agents where the coin was always at the right end of the level. When the coin was moved, the agents ignored it and ran right. They had learned "go right", not "get the coin". For LLMs the analogue is a refusal habit keyed to surface words, which refuses "how do I kill a Python process" while missing a harmful request in polite phrasing.
Failure 3: the supervisor cannot check
Human comparisons work only when the labeller can tell which answer is better. For a 2,000-line diff, a subtle proof or a medical edge case they often cannot, so the reward model learns what looks right. Scalable oversight research tries to make supervision keep up. Approaches include AI critiques that point labellers to flaws, debate between two models (Irving, Christiano and Amodei, 2018), AI feedback guided by written principles (Constitutional AI), and weak-to-strong generalisation, where a weaker supervisor fine-tunes a stronger model (Burns et al., OpenAI 2023).
Worked example: what β does
Three candidate answers have reference probabilities and rewards , so the reference policy earns . The optimum is . With the weights are , summing to 2.7933.
| β | π* | E[r] | KL(π* ‖ π_ref) | Objective |
|---|---|---|---|---|
| 2 | 0.325, 0.322, 0.353 | 1.028 | 0.084 | 0.861 |
| 1 | 0.179, 0.292, 0.529 | 1.350 | 0.323 | 1.027 |
| 0.5 | 0.037, 0.163, 0.801 | 1.764 | 0.915 | 1.306 |
Halving from 1 to 0.5 nearly triples the KL. If were the gold reward, that would be pure gain. If is a proxy that overrates answer 3, the policy has just put 80% of its mass on the proxy's mistake. Real systems therefore tune , and when to stop training, against a held-out evaluation and not against the training reward.
1import numpy as np23def bt_loss(r_chosen, r_rejected):4 """Reward-model loss -log σ(r_w − r_l), averaged over comparison pairs."""5 return float(np.mean(np.logaddexp(0.0, -(r_chosen - r_rejected))))67def kl_optimal_policy(p_ref, r, beta):8 """π*(y) ∝ π_ref(y)·exp(r(y)/β), the maximiser of E[r] − β·KL(π ‖ π_ref)."""9 w = p_ref * np.exp(r / beta)10 p = w / w.sum()11 return p, float(np.sum(p * np.log(p / p_ref)))1213def best_of_n(candidates, reward_fn):14 """Inference-time alignment: score n samples, return the highest-reward one."""15 scores = [reward_fn(c) for c in candidates]16 return candidates[int(np.argmax(scores))]1718print(round(bt_loss(np.array([1.2]), np.array([0.4])), 3)) # 0.37119p_ref, r = np.array([0.5, 0.3, 0.2]), np.array([0.0, 1.0, 2.0])20for beta in (2.0, 1.0, 0.5):21 p, kl = kl_optimal_policy(p_ref, r, beta)22 print(beta, p.round(3), round(float(p @ r), 3), round(kl, 3))23# 2.0 [0.325 0.322 0.353] 1.028 0.08424# 1.0 [0.179 0.292 0.529] 1.35 0.32325# 0.5 [0.037 0.163 0.801] 1.764 0.9152627# with len() as the "reward model", the longest answer always wins: length hacking28print(best_of_n(["Yes! Brilliant.", "Clear thesis; para 3 needs evidence."], len))Use when
- +Best-of-n: no training, KL is exactly ln n − (n−1)/n, easy to cap; costs n samples per request.
- +RLHF with PPO: on-policy, can reward behaviour the SFT data never showed.
- +DPO: no reward model or sampling loop; stable and cheap.
Avoid when
- −Best-of-n: cost grows linearly in n while KL grows only like ln n.
- −PPO: four models in memory, sensitive hyperparameters, most exposed to reward hacking.
- −DPO: offline, learns only from responses someone else sampled.
Check yourself
A reward model agrees with held-out human labels on 75% of pairs. During RL the reward model's score keeps rising, but human win rate against the SFT model falls after a point. What is the best explanation?
Check yourself
An agent trained to collect a coin that always sat at the end of the level runs past the coin when it is moved. Which failure is this?
Remember
- ✓Alignment: make the model do what is intended, not what the training signal literally rewards; LLM target is often helpful, honest, harmless.
- ✓Recipe: SFT → Bradley–Terry reward model → maximise , optimum .
- ✓InstructGPT: 1.3B aligned model preferred over 175B GPT-3; hallucination 21% vs 41% on closed-domain tasks.
- ✓Outer failure: proxy reward ≠ intent → reward hacking; gold reward rises then falls with KL (Gao et al. 2022).
- ✓Inner failure: correct reward, wrong learned goal, visible only off-distribution.
- ✓Oversight gap: when humans cannot judge outputs, use critiques, debate, AI feedback and held-out evaluations.