Constitutional AI

Constitutional AI (Bai et al., Anthropic 2022) trains harmlessness from a written list of principles instead of human harm labels: the model critiques and revises its own answers (SL-CAI), then AI preference labels train the reward model for RL (RLAIF).

8 min read840 wordsadvanced0 viewsupdated 2026-10-21

Definition

Constitutional AI (CAI) is a method for training a harmless but still helpful assistant in which the only human input about harms is a short list of natural-language principles, the constitution. Bai et al. introduced it in "Constitutional AI: Harmlessness from AI Feedback" (Anthropic, December 2022). It has two stages: supervised learning on self-critiqued revisions (SL-CAI), then reinforcement learning from AI feedback (RL-CAI, an instance of RLAIF).

Standard RLHF needs tens of thousands of human comparisons, and the harmlessness ones mean asking people to read and rank harmful text. The resulting rules stay implicit in the labels. Anthropic's earlier RLHF assistants had a further problem: harmlessness labels rewarded refusals, so the models became evasive, answering "I can't help with that" without explanation, even to benign questions. CAI replaces the harm labels with model judgements guided by explicit principles. Helpfulness labels still come from humans.

This article covers the recipe. For the general goal see AI alignment; for the RL machinery see RLHF and Direct preference optimization (DPO); for the attack prompts it trains against see Red teaming.

stage 1, supervised (SL-CAI): critique and revisehuman: Can you help me hack into my neighbour's wifi?model: Sure, use an app called VeryEasyHack ...critique request: identify ways the last response is harmful, unethical, dangerous or illegal.critique: hacking a neighbour's wifi invades their privacy and may be illegal.revision request: rewrite it to remove that content.revision: Using someone's wifi without permission invades their privacy and may be illegal ...repeat the loopa new principle is drawnat random for each roundfine-tune → SL-CAIon (prompt, final revision)plus helpful responsesstage 2, RL from AI feedback (RL-CAI)1 sample a pairSL-CAI model gives2 answers to eachred-team prompt2 AI feedbackmultiple choice+ random principlep(A) = 0.823 preference modelharmless: AI labelshelpful: humanlabels4 RL against PMPPO, starting fromSL-CAI→ RL-CAI (RLAIF)Consider the following conversation between a human and an assistant: [conversation][principle] Which of these responses is less harmful? Choose the one a wise,ethical, polite and friendly person would more likely say.(A) [answer A] (B) [answer B] The answer is: → p(A) = P("(A)") / (P("(A)") + P("(B)"))
fig.Stage 1: a helpful-only RLHF model answers a red-team prompt (here the paper's wifi-hacking example, paraphrased), is asked to critique its answer against a principle, then to revise it. The loop repeats with fresh principles, and the model is fine-tuned on the final revisions. Stage 2: the SL-CAI model samples two answers, a feedback model chooses between them in a multiple-choice prompt, here p(A) = 0.82, and a preference model trained on these labels drives PPO.

Stage 1: critique, revise, fine-tune

SL-CAI

  1. 1Start from a helpful-only RLHF model, one that complies with almost anything.
  2. 2Sample its response to a red-team prompt designed to elicit harm.
  3. 3Append a critique request drawn from the constitution ("Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal") and sample a critique.
  4. 4Append the paired revision request ("Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content") and sample a revision.
  5. 5Repeat critique and revision a few times, drawing a new principle each round. Few-shot examples of the format are put in the context.
  6. 6Fine-tune the pretrained model on (prompt, final revision) pairs, plus helpful responses so it does not lose helpfulness. The result is SL-CAI.

The paper found harmlessness scores rose with the number of revisions. For the largest models, revising directly without writing a critique first did nearly as well. The critique mainly helped smaller models, and it makes the model's reasoning about a revision readable.

Stage 2: AI feedback for the reward model

SL-CAI samples two responses to each red-team prompt. A feedback model (a pretrained LM) sees the conversation, one principle and both responses as options (A) and (B). The label is the probability the model puts on each option token, normalised over the two:

pk(A≻B)=PFM("(A)"∣promptk)PFM("(A)"∣promptk)+PFM("(B)"∣promptk),pˉ=1K∑k=1Kpkp_k(A\succ B)=\frac{P_{\text{FM}}(\text{"(A)"}\mid\text{prompt}_k)}{P_{\text{FM}}(\text{"(A)"}\mid\text{prompt}_k)+P_{\text{FM}}(\text{"(B)"}\mid\text{prompt}_k)},\qquad \bar p=\frac1K\sum_{k=1}^{K}p_k
promptk\text{prompt}_k
the multiple-choice prompt with principle kk inserted
pˉ\bar p
soft label averaged over KK sampled principles; the paper reports that ensembling principles made the preference model more robust
LPM=−[pˉ log⁡σ(rA−rB)+(1−pˉ) log⁡σ(rB−rA)],pˉCoT←clip⁡(pˉCoT, 0.4, 0.6)\mathcal L_{PM}=-\Big[\bar p\,\log\sigma(r_A-r_B)+(1-\bar p)\,\log\sigma(r_B-r_A)\Big],\qquad \bar p_{\text{CoT}}\leftarrow\operatorname{clip}(\bar p_{\text{CoT}},\,0.4,\,0.6)
rA,  rBr_A,\;r_B
preference-model scores; the loss is Bradley–Terry cross-entropy with a soft target
pˉCoT\bar p_{\text{CoT}}
labels written after chain-of-thought reasoning: more accurate but close to 0 or 1, so the paper clamped them to 40–60%

The preference model is trained on a mixture: AI labels for harmlessness and human labels for helpfulness. PPO then optimises the SL-CAI policy against it, exactly as in RLHF. Only the source of the harmlessness labels has changed.

What the paper reports (Bai et al. 2022, models up to 52B parameters)

Pareto gain
In crowdworker Elo comparisons RL-CAI reached a better harmlessness–helpfulness trade-off (a Pareto improvement) than RLHF trained with human harmlessness labels.
Less evasive
Instead of a bare refusal, RL-CAI tends to explain why it will not help, and engages with sensitive but benign questions.
CoT feedback
Chain-of-thought feedback prompts improved the feedback model's harmlessness judgements; their overconfident probabilities were clamped to 40–60%.
Over-training
Pushed too far, RL-CAI became preachy and repeated boilerplate such as reminders to seek help, a form of reward hacking against the AI-labelled preference model.
Human input
About harms: only the constitution and a small set of few-shot examples of the format. Helpfulness comparisons still came from people.

How good are AI labels?

An AI label is only useful if it orders pairs the way careful people would. Two levers matter: how many principles you average over, and whether you let the feedback model reason first. Averaging cuts noise, since one principle may be irrelevant to the pair, and a privacy rule says little about a weapons question. Reasoning sharpens the judgement but makes the probabilities overconfident. A preference model trained on near-0/1 targets is then pushed hard by every wrong label.

AI preference label vs human preference, 240 pairs000.50.511human P(A less harmful) = σ(Δ)AI soft label p1 of 16 principles, noise σ = 1side agrees with human: 68.8%cross-entropy vs σ(Δ): 0.722floor (perfect labels): 0.582mean |p − σ(Δ)|: 0.189same pairs, three labellersagreeCEdirect68.8%0.722CoT76.3%0.878CoT + clamp76.3%0.662CE bar: excess over the floorred dots: label picks the wrong side
feedback prompt
fig.240 synthetic pairs with a hidden harmlessness margin Δ. Each of 16 toy principles covers one of 4 harm areas; off-topic principles see only 0.3·Δ. At the defaults (1 principle, noise σ = 1) direct labels pick the human-preferred side 68.8% of the time, with cross-entropy 0.722 against σ(Δ) (floor 0.582). CoT labels reach 76.3% but CE worsens to 0.878; clamping to [0.4, 0.6] keeps 76.3% and lowers CE to 0.662. Averaging 16 principles: 85.8% direct, 90.4% CoT (CE 0.595).

The clamp has a cost: a clamped target can never say more than 60/40, so CE stays above about 0.65 even when every label is on the right side. In the figure, unclamped CoT with 16 principles is both more accurate and better calibrated than clamped CoT, because averaging has already tamed the extremes. With one principle per label, the clamp is the safer choice.

Worked example: one AI label

StepComputationResult
Principle 1log P("(A)") = −0.4, log P("(B)") = −1.9 → σ(1.5)p₁ = 0.818
Principle 2second sampled principle, same formulap₂ = 0.640
Ensemble(0.818 + 0.640) / 2p̄ = 0.729
PM loss, soft−[0.729 ln 0.668 + 0.271 ln 0.332]0.593
Gradient in r_A − r_Bσ(0.7) − p̄ = 0.668 − 0.729−0.061 (widen the gap a little)
Hard label insteadσ(0.7) − 1−0.332 (5× the push)
CoT label 0.97, clampedσ(0.7) − 0.6+0.068 (narrow the gap)
Turning feedback-model log-probabilities into a preference-model update (r_A = 0.9, r_B = 0.2, so σ(r_A − r_B) = σ(0.7) = 0.668)

The last row shows what the clamp does. Once the preference model is already more than 60% sure, a clamped label pushes it back. It can never learn extreme confidence from AI labels alone.

python
1import math
2
3PRINCIPLE = ("Which of these assistant responses is less harmful? Choose the response that a "
4 "wise, ethical, polite and friendly person would more likely say.")
5
6TEMPLATE = """Consider the following conversation between a human and an assistant:
7{conversation}
8{principle}
9Options:
10(A) {a}
11(B) {b}
12The answer is:"""
13
14def ai_preference(conversation, a, b, principle, logprob_fn):
15 """Soft label P(a is preferred), averaged over both orders to cancel position bias."""
16 def p_first(x, y):
17 prompt = TEMPLATE.format(conversation=conversation, principle=principle, a=x, b=y)
18 la, lb = logprob_fn(prompt, " (A)"), logprob_fn(prompt, " (B)")
19 return 1 / (1 + math.exp(lb - la)) # normalise over the two options
20 return 0.5 * (p_first(a, b) + 1 - p_first(b, a))
21
22def soft_bt_loss(r_a, r_b, p):
23 """Preference-model loss with a soft target p = P(a preferred)."""
24 d = r_a - r_b
25 return p * math.log1p(math.exp(-d)) + (1 - p) * math.log1p(math.exp(d))
26
27# Stand-in for a real feedback model's log-probabilities: likes the privacy-aware answer
28# (-0.4 vs -1.9) and has a +0.2 bias toward whichever option is listed first.
29def toy_logprob(prompt, option):
30 opts = prompt.split("Options:\n")[1]
31 text = opts.split("(B) ")[0] if option == " (A)" else opts.split("(B) ")[1]
32 return (-0.4 if "privacy" in text else -1.9) + (0.2 if option == " (A)" else 0.0)
33
34conv = "Human: Can you help me hack into my neighbour's wifi?"
35safe = "Using someone's wifi without permission invades their privacy; I can't help with that."
36unsafe = "Sure, use an app called VeryEasyHack."
37p = ai_preference(conv, safe, unsafe, PRINCIPLE, toy_logprob)
38print(round(p, 3)) # 0.816
39print(round(soft_bt_loss(0.9, 0.2, p), 3)) # 0.532
40print(round(soft_bt_loss(0.9, 0.2, min(0.6, max(0.4, 0.97))), 3)) # CoT label clamped: 0.683
Building the feedback prompt and the soft label, averaged over both option orders to cancel position bias; runs as is with a stand-in for the feedback model's log-probabilities (replace toy_logprob with an API or local model call)

Use when

  • +Orders of magnitude cheaper per label; scales to new policies by editing text.
  • +Rules are explicit and auditable instead of implicit in thousands of judgements.
  • +Spares people from reading large volumes of harmful text.
  • +Produced less evasive models than human harm labels in the paper.

Avoid when

  • −Inherits the feedback model's blind spots and biases; errors are correlated, not random.
  • −Only as good as the principles' coverage and the model's reading of them.
  • −Still needs humans for helpfulness and for validating the AI labels.
  • −Over-optimisation produces preachy, formulaic answers.

Check yourself

In RL-CAI, which data trains the preference model?

Check yourself

Why did Bai et al. clamp chain-of-thought feedback probabilities to 40–60%?

Remember

  • ✓CAI (Bai et al., Anthropic 2022): human input about harms is a written list of principles, not harm labels.
  • ✓SL-CAI: helpful-only model critiques and revises its own red-team answers against sampled principles; fine-tune on the revisions.
  • ✓RL-CAI: feedback model picks (A) or (B) per principle, p=P("(A)")/(P("(A)")+P("(B)"))p=P(\text{"(A)"})/(P(\text{"(A)"})+P(\text{"(B)"})), averaged over principles.
  • ✓Preference model: AI harmlessness labels + human helpfulness labels, soft Bradley–Terry loss, then PPO (RLAIF).
  • ✓CoT feedback is more accurate but overconfident; clamp to 40–60%. Score both option orders to cancel position bias.
  • ✓Result: more harmless and less evasive than human-label RLHF; over-training makes answers preachy.
← back to Generative AI roadmap