Definition
Constitutional AI (CAI) is a method for training a harmless but still helpful assistant in which the only human input about harms is a short list of natural-language principles, the constitution. Bai et al. introduced it in "Constitutional AI: Harmlessness from AI Feedback" (Anthropic, December 2022). It has two stages: supervised learning on self-critiqued revisions (SL-CAI), then reinforcement learning from AI feedback (RL-CAI, an instance of RLAIF).
Standard RLHF needs tens of thousands of human comparisons, and the harmlessness ones mean asking people to read and rank harmful text. The resulting rules stay implicit in the labels. Anthropic's earlier RLHF assistants had a further problem: harmlessness labels rewarded refusals, so the models became evasive, answering "I can't help with that" without explanation, even to benign questions. CAI replaces the harm labels with model judgements guided by explicit principles. Helpfulness labels still come from humans.
This article covers the recipe. For the general goal see AI alignment; for the RL machinery see RLHF and Direct preference optimization (DPO); for the attack prompts it trains against see Red teaming.
Stage 1: critique, revise, fine-tune
SL-CAI
- 1Start from a helpful-only RLHF model, one that complies with almost anything.
- 2Sample its response to a red-team prompt designed to elicit harm.
- 3Append a critique request drawn from the constitution ("Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal") and sample a critique.
- 4Append the paired revision request ("Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content") and sample a revision.
- 5Repeat critique and revision a few times, drawing a new principle each round. Few-shot examples of the format are put in the context.
- 6Fine-tune the pretrained model on (prompt, final revision) pairs, plus helpful responses so it does not lose helpfulness. The result is SL-CAI.
The paper found harmlessness scores rose with the number of revisions. For the largest models, revising directly without writing a critique first did nearly as well. The critique mainly helped smaller models, and it makes the model's reasoning about a revision readable.
Stage 2: AI feedback for the reward model
SL-CAI samples two responses to each red-team prompt. A feedback model (a pretrained LM) sees the conversation, one principle and both responses as options (A) and (B). The label is the probability the model puts on each option token, normalised over the two:
- the multiple-choice prompt with principle inserted
- soft label averaged over sampled principles; the paper reports that ensembling principles made the preference model more robust
- preference-model scores; the loss is Bradley–Terry cross-entropy with a soft target
- labels written after chain-of-thought reasoning: more accurate but close to 0 or 1, so the paper clamped them to 40–60%
The preference model is trained on a mixture: AI labels for harmlessness and human labels for helpfulness. PPO then optimises the SL-CAI policy against it, exactly as in RLHF. Only the source of the harmlessness labels has changed.
What the paper reports (Bai et al. 2022, models up to 52B parameters)
- Pareto gain
- In crowdworker Elo comparisons RL-CAI reached a better harmlessness–helpfulness trade-off (a Pareto improvement) than RLHF trained with human harmlessness labels.
- Less evasive
- Instead of a bare refusal, RL-CAI tends to explain why it will not help, and engages with sensitive but benign questions.
- CoT feedback
- Chain-of-thought feedback prompts improved the feedback model's harmlessness judgements; their overconfident probabilities were clamped to 40–60%.
- Over-training
- Pushed too far, RL-CAI became preachy and repeated boilerplate such as reminders to seek help, a form of reward hacking against the AI-labelled preference model.
- Human input
- About harms: only the constitution and a small set of few-shot examples of the format. Helpfulness comparisons still came from people.
How good are AI labels?
An AI label is only useful if it orders pairs the way careful people would. Two levers matter: how many principles you average over, and whether you let the feedback model reason first. Averaging cuts noise, since one principle may be irrelevant to the pair, and a privacy rule says little about a weapons question. Reasoning sharpens the judgement but makes the probabilities overconfident. A preference model trained on near-0/1 targets is then pushed hard by every wrong label.
The clamp has a cost: a clamped target can never say more than 60/40, so CE stays above about 0.65 even when every label is on the right side. In the figure, unclamped CoT with 16 principles is both more accurate and better calibrated than clamped CoT, because averaging has already tamed the extremes. With one principle per label, the clamp is the safer choice.
Worked example: one AI label
| Step | Computation | Result |
|---|---|---|
| Principle 1 | log P("(A)") = −0.4, log P("(B)") = −1.9 → σ(1.5) | p₁ = 0.818 |
| Principle 2 | second sampled principle, same formula | p₂ = 0.640 |
| Ensemble | (0.818 + 0.640) / 2 | p̄ = 0.729 |
| PM loss, soft | −[0.729 ln 0.668 + 0.271 ln 0.332] | 0.593 |
| Gradient in r_A − r_B | σ(0.7) − p̄ = 0.668 − 0.729 | −0.061 (widen the gap a little) |
| Hard label instead | σ(0.7) − 1 | −0.332 (5× the push) |
| CoT label 0.97, clamped | σ(0.7) − 0.6 | +0.068 (narrow the gap) |
The last row shows what the clamp does. Once the preference model is already more than 60% sure, a clamped label pushes it back. It can never learn extreme confidence from AI labels alone.
1import math23PRINCIPLE = ("Which of these assistant responses is less harmful? Choose the response that a "4 "wise, ethical, polite and friendly person would more likely say.")56TEMPLATE = """Consider the following conversation between a human and an assistant:7{conversation}8{principle}9Options:10(A) {a}11(B) {b}12The answer is:"""1314def ai_preference(conversation, a, b, principle, logprob_fn):15 """Soft label P(a is preferred), averaged over both orders to cancel position bias."""16 def p_first(x, y):17 prompt = TEMPLATE.format(conversation=conversation, principle=principle, a=x, b=y)18 la, lb = logprob_fn(prompt, " (A)"), logprob_fn(prompt, " (B)")19 return 1 / (1 + math.exp(lb - la)) # normalise over the two options20 return 0.5 * (p_first(a, b) + 1 - p_first(b, a))2122def soft_bt_loss(r_a, r_b, p):23 """Preference-model loss with a soft target p = P(a preferred)."""24 d = r_a - r_b25 return p * math.log1p(math.exp(-d)) + (1 - p) * math.log1p(math.exp(d))2627# Stand-in for a real feedback model's log-probabilities: likes the privacy-aware answer28# (-0.4 vs -1.9) and has a +0.2 bias toward whichever option is listed first.29def toy_logprob(prompt, option):30 opts = prompt.split("Options:\n")[1]31 text = opts.split("(B) ")[0] if option == " (A)" else opts.split("(B) ")[1]32 return (-0.4 if "privacy" in text else -1.9) + (0.2 if option == " (A)" else 0.0)3334conv = "Human: Can you help me hack into my neighbour's wifi?"35safe = "Using someone's wifi without permission invades their privacy; I can't help with that."36unsafe = "Sure, use an app called VeryEasyHack."37p = ai_preference(conv, safe, unsafe, PRINCIPLE, toy_logprob)38print(round(p, 3)) # 0.81639print(round(soft_bt_loss(0.9, 0.2, p), 3)) # 0.53240print(round(soft_bt_loss(0.9, 0.2, min(0.6, max(0.4, 0.97))), 3)) # CoT label clamped: 0.683Use when
- +Orders of magnitude cheaper per label; scales to new policies by editing text.
- +Rules are explicit and auditable instead of implicit in thousands of judgements.
- +Spares people from reading large volumes of harmful text.
- +Produced less evasive models than human harm labels in the paper.
Avoid when
- −Inherits the feedback model's blind spots and biases; errors are correlated, not random.
- −Only as good as the principles' coverage and the model's reading of them.
- −Still needs humans for helpfulness and for validating the AI labels.
- −Over-optimisation produces preachy, formulaic answers.
Check yourself
In RL-CAI, which data trains the preference model?
Check yourself
Why did Bai et al. clamp chain-of-thought feedback probabilities to 40–60%?
Remember
- ✓CAI (Bai et al., Anthropic 2022): human input about harms is a written list of principles, not harm labels.
- ✓SL-CAI: helpful-only model critiques and revises its own red-team answers against sampled principles; fine-tune on the revisions.
- ✓RL-CAI: feedback model picks (A) or (B) per principle, , averaged over principles.
- ✓Preference model: AI harmlessness labels + human helpfulness labels, soft Bradley–Terry loss, then PPO (RLAIF).
- ✓CoT feedback is more accurate but overconfident; clamp to 40–60%. Score both option orders to cancel position bias.
- ✓Result: more harmless and less evasive than human-label RLHF; over-training makes answers preachy.