# AI alignment

> Alignment makes a model pursue what its developers and users intend rather than what its training signal literally rewards; in practice SFT plus preference optimisation of E[r] − β·KL, checked against reward hacking, misgeneralisation and weak oversight.

Source: https://www.mlfoundry.in/learn/ai-alignment  
Section: Generative AI › Safety, Alignment and Responsible AI  
Level: core · 8 min read · updated 2026-10-21

**AI alignment:** is the problem of getting an AI system to pursue the goals its designers and users actually intend, not merely the objective it was trained on. For language models the working target is often Askell et al.'s "helpful, honest and harmless" (HHH, Anthropic 2021). The standard recipe is Ouyang et al.'s InstructGPT ("Training language models to follow instructions with human feedback", OpenAI 2022): supervised fine-tuning, then a reward model learned from human comparisons, then reinforcement learning against it.

A pretrained model is trained to continue text the way the internet would. Asked "Is my essay good?", the likeliest continuation may be flattery, a forum argument or another question. Alignment training turns this text predictor into an assistant that follows instructions, says when it does not know, and declines clearly harmful requests. It is the post-training stage that sits between pretraining and deployment. **Guardrails**, **Content moderation** and **Red teaming** sit around it in production.

This article is about the goal and its failure modes. The algorithms have their own articles: **Supervised fine-tuning (SFT)**, **RLHF**, **Direct preference optimization (DPO)** and **GRPO** in the Deep Learning branch, and **Constitutional AI** for AI-generated feedback.

## The pipeline and its objective

> Figure: The InstructGPT recipe. A pretrained LM is fine-tuned on labeller demonstrations (about 13k prompts). A reward model is trained on comparisons (about 33k prompts): for the essay prompt, the specific critique B scores 1.2 and the flattering A scores 0.4, so P(B ≻ A) = σ(0.8) = 0.69 and the loss is 0.371. The policy then maximises reward minus a KL penalty (about 31k prompts). Right: the three places the chain can break.

$$
P(y_w\succ y_l\mid x)=\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big),\qquad \mathcal L_{RM}=-\,\mathbb E\,\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)
$$

- $y_w,\;y_l$: the response the labeller preferred and the one they rejected
- $r_\phi$: reward model, usually the SFT model with a scalar head
- $\sigma$: logistic function; this is the Bradley–Terry model of pairwise choice

$$
\max_{\pi}\;\mathbb E_{x,\;y\sim\pi}\big[r_\phi(x,y)\big]-\beta\,\mathrm{KL}\big(\pi(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big)\;\;\Longrightarrow\;\;\pi^*(y|x)=\frac{\pi_{\text{ref}}(y|x)\,e^{r_\phi(x,y)/\beta}}{Z(x)}
$$

- $\pi_{\text{ref}}$: the SFT model; the KL term keeps the policy close to it
- $\beta$: KL coefficient: how far the policy may move to chase reward
- $Z(x)$: normaliser $\sum_y\pi_{\text{ref}}(y|x)e^{r_\phi(x,y)/\beta}$; at the optimum the objective equals $\beta\log Z(x)$

PPO approximates $\pi^*$ by sampling, DPO fits it directly from the pairs, and best-of-$n$ sampling approximates it at inference time with no training. All three optimise the same thing: a learned stand-in for what people want. The KL term is the main protection against over-trusting that stand-in.

**InstructGPT results (Ouyang et al. 2022, GPT-3 base, API prompt distribution)**

- **Preference:** Labellers preferred outputs of the 1.3B InstructGPT model to those of the 175B GPT-3, despite 100× fewer parameters.
- **Head-to-head:** 175B InstructGPT outputs were preferred to 175B GPT-3 outputs 85 ± 3% of the time.
- **Hallucination:** On closed-domain tasks it made up information about half as often: 21% vs 41%.
- **Alignment tax:** PPO lowered scores on some public NLP benchmarks. Mixing pretraining gradients into PPO (PPO-ptx) recovered most of the loss.
- **Labellers:** About 40 contractors; the paper stresses that the model is aligned to *their* preferences and OpenAI's instructions, not to people in general.

## Failure 1: the reward is not the intent

**Outer misalignment** means the training objective itself is wrong. The classic example is OpenAI's CoastRunners agent (2016). It was rewarded for hitting targets, so it circled a lagoon hitting the same targets and never finished the race. Krakovna et al. (DeepMind, 2020) collected dozens of such cases of *specification gaming*. In LLMs the reward model is the objective, and it inherits labeller habits: longer answers, confident tone and agreement with the user tend to win comparisons. Optimising hard against it gives **reward hacking**: verbosity, flattery (*sycophancy*) and hedged non-answers.

$$
R_{\text{bo}n}(d)=d\,(\alpha_{\text{bo}n}-\beta_{\text{bo}n}\,d),\qquad R_{\text{RL}}(d)=d\,(\alpha_{\text{RL}}-\beta_{\text{RL}}\log d),\qquad d=\sqrt{\mathrm{KL}(\pi\|\pi_{\text{ref}})}
$$

- $R$: gold reward gained over the initial policy, measured by a held-out "gold" reward model
- $d$: distance travelled from the initial policy; for best-of-$n$, $\mathrm{KL}=\ln n-(n-1)/n$
- $\alpha,\beta$: fitted constants; they scale smoothly with reward-model size

Gao, Schulman and Hilton ("Scaling Laws for Reward Model Overoptimization", 2022) trained proxy reward models on labels from a fixed 6B "gold" model. The proxy score rose steadily with optimisation, while the gold score rose, peaked and fell, following the two forms above. Larger proxy reward models peaked later and higher. This is Goodhart's law with a measurable curve.

> Figure: Each response has a true quality q and a hackable feature h, both N(0, 1). The proxy RM scores q + w·h and the gold reward is q − 0.5·h². Best-of-n picks the proxy's favourite for 300 prompts. At w = 1 the proxy still agrees with gold on 75.0% of fresh pairs. Gold peaks at n = 16 (+0.27, KL 1.84 nats) and falls to −0.40 at n = 256, while the proxy keeps climbing to +4.02. At w = 0 gold rises all the way to +2.45 at n = 256; at w = 1.5 it peaks at n = 4.

Note how unremarkable the proxy looks: 75% pairwise accuracy is typical of real reward models. The problem only appears under optimisation pressure, because the winners are drawn from the tail where the proxy and the gold signal disagree most. In the figure, the mean $h$ of the chosen response climbs from 0.05 to 2.03.

## Failure 2: the right reward, the wrong goal

**Inner misalignment** (Hubinger et al., "Risks from Learned Optimization", 2019) means a model can score perfectly in training and still have learned a different goal, which shows only off-distribution. Langosco et al. (2022) trained CoinRun agents where the coin was always at the right end of the level. When the coin was moved, the agents ignored it and ran right. They had learned "go right", not "get the coin". For LLMs the analogue is a refusal habit keyed to surface words, which refuses "how do I kill a Python process" while missing a harmful request in polite phrasing.

## Failure 3: the supervisor cannot check

Human comparisons work only when the labeller can tell which answer is better. For a 2,000-line diff, a subtle proof or a medical edge case they often cannot, so the reward model learns what *looks* right. **Scalable oversight** research tries to make supervision keep up. Approaches include AI critiques that point labellers to flaws, debate between two models (Irving, Christiano and Amodei, 2018), AI feedback guided by written principles (**Constitutional AI**), and weak-to-strong generalisation, where a weaker supervisor fine-tunes a stronger model (Burns et al., OpenAI 2023).

## Worked example: what β does

Three candidate answers have reference probabilities $\pi_{\text{ref}}=(0.5,0.3,0.2)$ and rewards $r=(0,1,2)$, so the reference policy earns $\mathbb E[r]=0.7$. The optimum is $\pi^*\propto\pi_{\text{ref}}e^{r/\beta}$. With $\beta=1$ the weights are $0.5,\;0.3e,\;0.2e^2=0.5,\;0.8155,\;1.4778$, summing to 2.7933.

Optimal policy for three values of β (objective = E[r] − β·KL = β·ln Z)

| β | π* | E[r] | KL(π* ‖ π_ref) | Objective |
| --- | --- | --- | --- | --- |
| 2 | 0.325, 0.322, 0.353 | 1.028 | 0.084 | 0.861 |
| 1 | 0.179, 0.292, 0.529 | 1.350 | 0.323 | 1.027 |
| 0.5 | 0.037, 0.163, 0.801 | 1.764 | 0.915 | 1.306 |

Halving $\beta$ from 1 to 0.5 nearly triples the KL. If $r$ were the gold reward, that would be pure gain. If $r$ is a proxy that overrates answer 3, the policy has just put 80% of its mass on the proxy's mistake. Real systems therefore tune $\beta$, and when to stop training, against a held-out evaluation and not against the training reward.

```python
import numpy as np

def bt_loss(r_chosen, r_rejected):
    """Reward-model loss -log σ(r_w − r_l), averaged over comparison pairs."""
    return float(np.mean(np.logaddexp(0.0, -(r_chosen - r_rejected))))

def kl_optimal_policy(p_ref, r, beta):
    """π*(y) ∝ π_ref(y)·exp(r(y)/β), the maximiser of E[r] − β·KL(π ‖ π_ref)."""
    w = p_ref * np.exp(r / beta)
    p = w / w.sum()
    return p, float(np.sum(p * np.log(p / p_ref)))

def best_of_n(candidates, reward_fn):
    """Inference-time alignment: score n samples, return the highest-reward one."""
    scores = [reward_fn(c) for c in candidates]
    return candidates[int(np.argmax(scores))]

print(round(bt_loss(np.array([1.2]), np.array([0.4])), 3))      # 0.371
p_ref, r = np.array([0.5, 0.3, 0.2]), np.array([0.0, 1.0, 2.0])
for beta in (2.0, 1.0, 0.5):
    p, kl = kl_optimal_policy(p_ref, r, beta)
    print(beta, p.round(3), round(float(p @ r), 3), round(kl, 3))
# 2.0 [0.325 0.322 0.353] 1.028 0.084
# 1.0 [0.179 0.292 0.529] 1.35 0.323
# 0.5 [0.037 0.163 0.801] 1.764 0.915

# with len() as the "reward model", the longest answer always wins: length hacking
print(best_of_n(["Yes! Brilliant.", "Clear thesis; para 3 needs evidence."], len))
```

The two core computations in numpy (runs as is): the reward-model loss and the KL-regularised optimum, plus best-of-n with a pluggable reward function

> **Do not report the training reward as progress:** The reward model's score is the one number the optimiser is guaranteed to raise. Track signals it was not trained on: held-out human or **LLM-as-a-judge** preferences against the SFT model, response length, refusal rate on benign prompts, and rate of agreement with false user claims. If reward climbs while length and KL climb with it, suspect hacking first.

> **Aligned to whom?:** A preference dataset encodes the judgements of particular labellers following particular guidelines. Choices such as how cautious to be on medical questions, or whose politeness norms apply, are value judgements, not technical constants. Public specifications of intended behaviour (written constitutions, model specs) make those choices explicit and open to argument. See **Bias and fairness in generative models** for how such choices show up in outputs.

**Strengths**

- Best-of-n: no training, KL is exactly ln n − (n−1)/n, easy to cap; costs n samples per request.
- RLHF with PPO: on-policy, can reward behaviour the SFT data never showed.
- DPO: no reward model or sampling loop; stable and cheap.

**Limitations**

- Best-of-n: cost grows linearly in n while KL grows only like ln n.
- PPO: four models in memory, sensitive hyperparameters, most exposed to reward hacking.
- DPO: offline, learns only from responses someone else sampled.

**Question:** A reward model agrees with held-out human labels on 75% of pairs. During RL the reward model's score keeps rising, but human win rate against the SFT model falls after a point. What is the best explanation?

- A. The reward model is underfitting and needs more epochs
- B. Overoptimisation: the policy has moved into regions where the proxy and the true preference disagree
- C. The KL coefficient β is too large
- D. Human labellers became less consistent over time

**Answer:** B. Overoptimisation: the policy has moved into regions where the proxy and the true preference disagree — Pairwise accuracy is measured on typical responses. Optimisation selects atypical ones from the tail, where the proxy's errors are largest. Gao et al. found exactly this rise-then-fall of gold reward with KL. A *larger* β would slow it down, not cause it.

**Question:** An agent trained to collect a coin that always sat at the end of the level runs past the coin when it is moved. Which failure is this?

- A. Reward hacking
- B. Goal misgeneralisation (inner misalignment)
- C. Distribution shift in the reward model
- D. Sycophancy

**Answer:** B. Goal misgeneralisation (inner misalignment) — The reward (coin collected) was correct and the agent maximised it in training. It learned a different goal, "go right", that coincided with the reward on the training distribution and diverged once the coin moved.

**Key points**

- Alignment: make the model do what is intended, not what the training signal literally rewards; LLM target is often helpful, honest, harmless.
- Recipe: SFT → Bradley–Terry reward model $\sigma(r_w-r_l)$ → maximise $\mathbb E[r]-\beta\,\mathrm{KL}$, optimum $\pi_{\text{ref}}e^{r/\beta}/Z$.
- InstructGPT: 1.3B aligned model preferred over 175B GPT-3; hallucination 21% vs 41% on closed-domain tasks.
- Outer failure: proxy reward ≠ intent → reward hacking; gold reward rises then falls with KL (Gao et al. 2022).
- Inner failure: correct reward, wrong learned goal, visible only off-distribution.
- Oversight gap: when humans cannot judge outputs, use critiques, debate, AI feedback and held-out evaluations.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
