# Constitutional AI

> Constitutional AI (Bai et al., Anthropic 2022) trains harmlessness from a written list of principles instead of human harm labels: the model critiques and revises its own answers (SL-CAI), then AI preference labels train the reward model for RL (RLAIF).

Source: https://www.mlfoundry.in/learn/constitutional-ai  
Section: Generative AI › Safety, Alignment and Responsible AI  
Level: advanced · 8 min read · updated 2026-10-21

**Constitutional AI (CAI):** is a method for training a harmless but still helpful assistant in which the only human input about harms is a short list of natural-language principles, the *constitution*. Bai et al. introduced it in "Constitutional AI: Harmlessness from AI Feedback" (Anthropic, December 2022). It has two stages: supervised learning on self-critiqued revisions (SL-CAI), then reinforcement learning from AI feedback (RL-CAI, an instance of RLAIF).

Standard **RLHF** needs tens of thousands of human comparisons, and the harmlessness ones mean asking people to read and rank harmful text. The resulting rules stay implicit in the labels. Anthropic's earlier RLHF assistants had a further problem: harmlessness labels rewarded refusals, so the models became *evasive*, answering "I can't help with that" without explanation, even to benign questions. CAI replaces the harm labels with model judgements guided by explicit principles. Helpfulness labels still come from humans.

This article covers the recipe. For the general goal see **AI alignment**; for the RL machinery see **RLHF** and **Direct preference optimization (DPO)**; for the attack prompts it trains against see **Red teaming**.

> Figure: Stage 1: a helpful-only RLHF model answers a red-team prompt (here the paper's wifi-hacking example, paraphrased), is asked to critique its answer against a principle, then to revise it. The loop repeats with fresh principles, and the model is fine-tuned on the final revisions. Stage 2: the SL-CAI model samples two answers, a feedback model chooses between them in a multiple-choice prompt, here p(A) = 0.82, and a preference model trained on these labels drives PPO.

## Stage 1: critique, revise, fine-tune

**SL-CAI**

1. Start from a **helpful-only** RLHF model, one that complies with almost anything.
2. Sample its response to a red-team prompt designed to elicit harm.
3. Append a *critique request* drawn from the constitution ("Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal") and sample a critique.
4. Append the paired *revision request* ("Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content") and sample a revision.
5. Repeat critique and revision a few times, drawing a new principle each round. Few-shot examples of the format are put in the context.
6. Fine-tune the pretrained model on (prompt, final revision) pairs, plus helpful responses so it does not lose helpfulness. The result is SL-CAI.

The paper found harmlessness scores rose with the number of revisions. For the largest models, revising directly without writing a critique first did nearly as well. The critique mainly helped smaller models, and it makes the model's reasoning about a revision readable.

## Stage 2: AI feedback for the reward model

SL-CAI samples two responses to each red-team prompt. A feedback model (a pretrained LM) sees the conversation, one principle and both responses as options (A) and (B). The label is the probability the model puts on each option token, normalised over the two:

$$
p_k(A\succ B)=\frac{P_{\text{FM}}(\text{"(A)"}\mid\text{prompt}_k)}{P_{\text{FM}}(\text{"(A)"}\mid\text{prompt}_k)+P_{\text{FM}}(\text{"(B)"}\mid\text{prompt}_k)},\qquad \bar p=\frac1K\sum_{k=1}^{K}p_k
$$

- $\text{prompt}_k$: the multiple-choice prompt with principle $k$ inserted
- $\bar p$: soft label averaged over $K$ sampled principles; the paper reports that ensembling principles made the preference model more robust

$$
\mathcal L_{PM}=-\Big[\bar p\,\log\sigma(r_A-r_B)+(1-\bar p)\,\log\sigma(r_B-r_A)\Big],\qquad \bar p_{\text{CoT}}\leftarrow\operatorname{clip}(\bar p_{\text{CoT}},\,0.4,\,0.6)
$$

- $r_A,\;r_B$: preference-model scores; the loss is Bradley–Terry cross-entropy with a soft target
- $\bar p_{\text{CoT}}$: labels written after chain-of-thought reasoning: more accurate but close to 0 or 1, so the paper clamped them to 40–60%

The preference model is trained on a mixture: AI labels for harmlessness and human labels for helpfulness. PPO then optimises the SL-CAI policy against it, exactly as in **RLHF**. Only the source of the harmlessness labels has changed.

**What the paper reports (Bai et al. 2022, models up to 52B parameters)**

- **Pareto gain:** In crowdworker Elo comparisons RL-CAI reached a better harmlessness–helpfulness trade-off (a Pareto improvement) than RLHF trained with human harmlessness labels.
- **Less evasive:** Instead of a bare refusal, RL-CAI tends to explain why it will not help, and engages with sensitive but benign questions.
- **CoT feedback:** Chain-of-thought feedback prompts improved the feedback model's harmlessness judgements; their overconfident probabilities were clamped to 40–60%.
- **Over-training:** Pushed too far, RL-CAI became preachy and repeated boilerplate such as reminders to seek help, a form of reward hacking against the AI-labelled preference model.
- **Human input:** About harms: only the constitution and a small set of few-shot examples of the format. Helpfulness comparisons still came from people.

## How good are AI labels?

An AI label is only useful if it orders pairs the way careful people would. Two levers matter: how many principles you average over, and whether you let the feedback model reason first. Averaging cuts noise, since one principle may be irrelevant to the pair, and a privacy rule says little about a weapons question. Reasoning sharpens the judgement but makes the probabilities overconfident. A preference model trained on near-0/1 targets is then pushed hard by every wrong label.

> Figure: 240 synthetic pairs with a hidden harmlessness margin Δ. Each of 16 toy principles covers one of 4 harm areas; off-topic principles see only 0.3·Δ. At the defaults (1 principle, noise σ = 1) direct labels pick the human-preferred side 68.8% of the time, with cross-entropy 0.722 against σ(Δ) (floor 0.582). CoT labels reach 76.3% but CE worsens to 0.878; clamping to [0.4, 0.6] keeps 76.3% and lowers CE to 0.662. Averaging 16 principles: 85.8% direct, 90.4% CoT (CE 0.595).

The clamp has a cost: a clamped target can never say more than 60/40, so CE stays above about 0.65 even when every label is on the right side. In the figure, unclamped CoT with 16 principles is both more accurate and better calibrated than clamped CoT, because averaging has already tamed the extremes. With one principle per label, the clamp is the safer choice.

## Worked example: one AI label

Turning feedback-model log-probabilities into a preference-model update (r_A = 0.9, r_B = 0.2, so σ(r_A − r_B) = σ(0.7) = 0.668)

| Step | Computation | Result |
| --- | --- | --- |
| Principle 1 | log P("(A)") = −0.4, log P("(B)") = −1.9 → σ(1.5) | p₁ = 0.818 |
| Principle 2 | second sampled principle, same formula | p₂ = 0.640 |
| Ensemble | (0.818 + 0.640) / 2 | p̄ = 0.729 |
| PM loss, soft | −[0.729 ln 0.668 + 0.271 ln 0.332] | 0.593 |
| Gradient in r_A − r_B | σ(0.7) − p̄ = 0.668 − 0.729 | −0.061 (widen the gap a little) |
| Hard label instead | σ(0.7) − 1 | −0.332 (5× the push) |
| CoT label 0.97, clamped | σ(0.7) − 0.6 | +0.068 (narrow the gap) |

The last row shows what the clamp does. Once the preference model is already more than 60% sure, a clamped label pushes it back. It can never learn extreme confidence from AI labels alone.

```python
import math

PRINCIPLE = ("Which of these assistant responses is less harmful? Choose the response that a "
             "wise, ethical, polite and friendly person would more likely say.")

TEMPLATE = """Consider the following conversation between a human and an assistant:
{conversation}
{principle}
Options:
(A) {a}
(B) {b}
The answer is:"""

def ai_preference(conversation, a, b, principle, logprob_fn):
    """Soft label P(a is preferred), averaged over both orders to cancel position bias."""
    def p_first(x, y):
        prompt = TEMPLATE.format(conversation=conversation, principle=principle, a=x, b=y)
        la, lb = logprob_fn(prompt, " (A)"), logprob_fn(prompt, " (B)")
        return 1 / (1 + math.exp(lb - la))          # normalise over the two options
    return 0.5 * (p_first(a, b) + 1 - p_first(b, a))

def soft_bt_loss(r_a, r_b, p):
    """Preference-model loss with a soft target p = P(a preferred)."""
    d = r_a - r_b
    return p * math.log1p(math.exp(-d)) + (1 - p) * math.log1p(math.exp(d))

# Stand-in for a real feedback model's log-probabilities: likes the privacy-aware answer
# (-0.4 vs -1.9) and has a +0.2 bias toward whichever option is listed first.
def toy_logprob(prompt, option):
    opts = prompt.split("Options:\n")[1]
    text = opts.split("(B) ")[0] if option == " (A)" else opts.split("(B) ")[1]
    return (-0.4 if "privacy" in text else -1.9) + (0.2 if option == " (A)" else 0.0)

conv = "Human: Can you help me hack into my neighbour's wifi?"
safe = "Using someone's wifi without permission invades their privacy; I can't help with that."
unsafe = "Sure, use an app called VeryEasyHack."
p = ai_preference(conv, safe, unsafe, PRINCIPLE, toy_logprob)
print(round(p, 3))                                   # 0.816
print(round(soft_bt_loss(0.9, 0.2, p), 3))           # 0.532
print(round(soft_bt_loss(0.9, 0.2, min(0.6, max(0.4, 0.97))), 3))   # CoT label clamped: 0.683
```

Building the feedback prompt and the soft label, averaged over both option orders to cancel position bias; runs as is with a stand-in for the feedback model's log-probabilities (replace toy_logprob with an API or local model call)

> **The feedback model has its own biases:** An LLM judge prefers the option listed first, longer answers and its own style (see **LLM-as-a-judge**). In CAI these biases become the reward. Score both option orders and average, as in the code. Check AI labels against a small human-labelled set before training on them, and watch for the preachy, boilerplate-heavy outputs that signal the policy has found the judge's soft spots.

> **The constitution is the interface:** Because the rules are text, they can be read, debated and changed without relabelling data. Anthropic published the principles used for Claude in May 2023, drawing on sources such as the UN Universal Declaration of Human Rights and other labs' rule sets. Later in 2023 it trialled "Collective Constitutional AI", drafting principles from about a thousand members of the US public. The principles still have to be interpreted by a model, so a clear rule does not guarantee the behaviour it describes.

**Strengths**

- Orders of magnitude cheaper per label; scales to new policies by editing text.
- Rules are explicit and auditable instead of implicit in thousands of judgements.
- Spares people from reading large volumes of harmful text.
- Produced less evasive models than human harm labels in the paper.

**Limitations**

- Inherits the feedback model's blind spots and biases; errors are correlated, not random.
- Only as good as the principles' coverage and the model's reading of them.
- Still needs humans for helpfulness and for validating the AI labels.
- Over-optimisation produces preachy, formulaic answers.

**Question:** In RL-CAI, which data trains the preference model?

- A. Only AI labels, for both helpfulness and harmlessness
- B. AI labels for harmlessness mixed with human labels for helpfulness
- C. Human labels for harmlessness, AI labels for helpfulness
- D. The SL-CAI revisions, used as chosen responses

**Answer:** B. AI labels for harmlessness mixed with human labels for helpfulness — The paper replaced only the harmlessness comparisons with AI feedback. Helpfulness comparisons still came from crowdworkers. The revisions train the SL-CAI policy, not the preference model.

**Question:** Why did Bai et al. clamp chain-of-thought feedback probabilities to 40–60%?

- A. To make the labels agree more often with humans
- B. Because CoT made the feedback model's probabilities near 0 or 1, and training on such targets made the RL policy worse
- C. To reduce the number of tokens the feedback model generates
- D. Because the preference model cannot represent probabilities outside that range

**Answer:** B. Because CoT made the feedback model's probabilities near 0 or 1, and training on such targets made the RL policy worse — Clamping changes no label's side, so agreement is unchanged. It limits how hard any one label, right or wrong, can push the preference model. In the worked example a clamped 0.97 label even nudges the gap down once the model is above 60%.

**Key points**

- CAI (Bai et al., Anthropic 2022): human input about harms is a written list of principles, not harm labels.
- SL-CAI: helpful-only model critiques and revises its own red-team answers against sampled principles; fine-tune on the revisions.
- RL-CAI: feedback model picks (A) or (B) per principle, $p=P(\text{"(A)"})/(P(\text{"(A)"})+P(\text{"(B)"}))$, averaged over principles.
- Preference model: AI harmlessness labels + human helpfulness labels, soft Bradley–Terry loss, then PPO (RLAIF).
- CoT feedback is more accurate but overconfident; clamp to 40–60%. Score both option orders to cancel position bias.
- Result: more harmless and less evasive than human-label RLHF; over-training makes answers preachy.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
