Red teaming

Red teaming attacks a generative system on purpose, by human experts and by attacker LLMs (Perez et al. 2022), across a scoped harm taxonomy, and reports attack success rate per cell with intervals; every finding becomes a regression test.

8 min read690 wordscore0 viewsupdated 2026-10-21

Definition

Red teaming is structured adversarial testing of an AI system: people or programs try to make it fail in the ways that matter (harmful content, data leakage, unsafe tool actions), and the failures are measured, logged and fixed. Perez et al. ("Red Teaming Language Models with Language Models", 2022, DeepMind) showed that one LM can generate test cases for another at scale. Ganguli et al. (2022, Anthropic) described large manual campaigns and released 38,961 red-team attacks.

Benchmarks measure average behaviour on fixed inputs. Red teaming looks for the worst case on inputs nobody wrote down yet. It sits between training and release, and runs again after every significant change. Its outputs feed Guardrails, safety fine-tuning data (Supervised fine-tuning (SFT), RLHF) and the regression suite. The attacks it uses are covered in Jailbreaking and Prompt injection; this article is about running the campaign and reading its numbers.

red teaming is a loop, not a one-off test1 scopesystem as deployedthreat model: who,with what accessharm taxonomyexit: ASR < 1%/area2 generate attacksexperts: domain +adversarial skillred LM: zero-shot,few-shot, SFT, RL(Perez et al. 2022)3 targetmodel + systemprompt + tools+ guardrailsexactly as inproduction4 judgeharm classifier+ human reviewseverity rubricASR = hits / Nper taxonomy cell5 findings logprivacy, role-play9/50 = 18%, highcyber, direct0/50 = 0%+ transcripts6 mitigateSFT / RLHF datasafety classifierguardrailsprompt + toolpermissions7 regressionre-run findingson every releasethen hunt fornew attacks(adaptive)next roundred-LM methodszero-shot: diverse, low ASRfew-shot / SFT: in betweenRL: high ASR, low diversitya suite that always passes finds nothing new: keep regression tests and the hunt separate
fig.The loop. Scope fixes the deployed system, the threat model, the harm taxonomy and an exit criterion. Experts and a red LM generate attacks against the target exactly as it runs in production, with system prompt, tools and guardrails. A judge (classifier plus human review) scores each output. Findings such as "privacy × role-play: 9/50 = 18%, high severity" drive mitigations. Every finding then joins a regression suite, and the next round looks for new attacks. Inset: Perez et al.'s finding that moving from zero-shot to RL generation raises success rate but lowers diversity.

Scoping

Before the first attack

  1. 1System under test: the model plus system prompt, retrieval, tools and filters, at a pinned version. A raw model and a deployed app fail differently.
  2. 2Threat model: who attacks (curious user, motivated abuser, third party planting content), with what access (chat only, API with system prompt, documents the app ingests) and what budget.
  3. 3Harm taxonomy: the categories to cover, e.g. Weidinger et al.'s six risk areas (2021) or a product policy. Cross them with attack styles to get a grid of cells.
  4. 4Judging rules: what counts as a failure, severity levels, who reviews borderline cases.
  5. 5Exit criteria: e.g. ASR below 1% in every high-severity cell, with a stated confidence level, and no open critical findings.

Manual and automated

Manual red teamers bring domain knowledge (a biosecurity expert knows which answer is actually dangerous) and creativity. They are slow and expensive, and they tend to search where they already expect problems. Automated red teaming uses an attacker LLM. Perez et al. compared four ways to generate test cases: zero-shot prompting, stochastic few-shot prompting from earlier successes, supervised fine-tuning on successes, and RL against a harm classifier. RL found failures most often but with the least diversity. Zero-shot was the most diverse and the least effective.

Later methods push for both. Rainbow Teaming (Samvelyan et al. 2024) keeps a MAP-Elites archive with one best attack per (risk category × attack style) cell, so the search must fill the grid. PAIR and TAP iterate on single targets (see Jailbreaking). Open-source harnesses such as Microsoft's PyRIT and garak automate the send–judge–log loop.

Reading the numbers

ASR^c=kcnc,nneeded=⌈ln⁡(1−γ)ln⁡(1−p)⌉≈3p    (γ=0.95)\widehat{\mathrm{ASR}}_{c}=\frac{k_c}{n_c},\qquad n_{\text{needed}}=\left\lceil\frac{\ln(1-\gamma)}{\ln(1-p)}\right\rceil\approx\frac{3}{p}\;\;(\gamma=0.95)
kc,  nck_c,\;n_c
failures found and attempts made in taxonomy cell cc; report a Wilson interval, not just the ratio
pp
the true failure rate you want to be able to detect
γ\gamma
probability of seeing at least one failure; ≈3/p\approx3/p is the "rule of three"
m=TPR⋅a+FPR⋅(1−a)⟹a^=m−FPRTPR−FPRm=\mathrm{TPR}\cdot a+\mathrm{FPR}\cdot(1-a)\quad\Longrightarrow\quad \hat a=\frac{m-\mathrm{FPR}}{\mathrm{TPR}-\mathrm{FPR}}
mm
ASR reported by an automated judge
aa
true ASR; the correction (Rogan–Gladen) needs TPR and FPR measured on a human-labelled sample

A judge with a 5% false-positive rate cannot report an ASR below about 5%, whatever the model does. When the true rate is low, judge error dominates the measurement. Spot-check judged failures by hand and measure the judge on a labelled set before trusting small differences.

failures found / attempts per cell, 1000 attemptsdirectrole-playobfuscatedmulti-turnweaponscyber misuseprivacy / PIIhateself-harmmisinformation0/50/80/40/10/70/3302/8770/20/50/60/90/31/60/60/74/130/30/50/50/60/70/40/40/4shade = observed failure rate in the cellsame budget, four strategiesmanual expertsASR 3.2%11 cells, 5/5 weakuniform automatedASR 3.8%12 cells, 5/5 weakRL-style exploitASR 30.7%3 cells, 2/5 weakquality-diversityASR 5.7%13 cells, 5/5 weakRL-style exploit: true ASR 307/1000 = 30.7%judge TPR 0.9, FPR 5% reports 31.1%; corrected (m − FPR)/(TPR − FPR) = 30.7%exploiting one weak cell maximises ASR; covering the grid finds more distinct failuresa 1% failure needs 299 attempts to show up once with 95% confidence; 0.1% needs 2995synthetic grid: rates are toy assumptions, not measurements of any model
strategy
fig.A synthetic 6 × 4 grid with hidden failure rates: five weak cells (12–35%), the rest 0.3–2.1%. At 1,000 attempts, the RL-style exploiter (ε-greedy) reaches the highest ASR, 307/1000 = 30.7%, but finds failures in only 3 cells and 2 of the 5 weak ones. Uniform sampling reaches 3.8% and covers 12 cells, all 5 weak. Quality-diversity reaches 5.7% over 13 cells, also all 5 weak. Manual experts, biased towards direct asks, find 1 of 5 weak cells at 100 attempts. With judge FPR 5%, uniform's true 3.8% is reported as 8.2%.

Checkable specifics

Perez et al. 2022
red LM and target from the Gopher family (target: 280B Dialogue-Prompted Gopher); offensive replies scored by a classifier; also probed data leakage, contact-information generation and distributional bias
Ganguli et al. 2022
38,961 human red-team attacks on models of 2.7B, 13B and 52B parameters; RLHF models became harder to red team as they grew, the other model types did not
Weidinger et al. 2021
six risk areas: discrimination/exclusion/toxicity; information hazards; misinformation; malicious uses; human–computer interaction harms; automation, access and environmental harms
Frameworks
NIST AI 600-1, the Generative AI Profile (July 2024), lists red teaming among its suggested actions; MITRE ATLAS catalogues adversarial tactics against AI systems
Regulation (general terms)
the EU AI Act requires providers of general-purpose AI models with systemic risk to carry out adversarial testing; obligations and dates vary, so check the current text (this is not legal advice)

Worked example: is the cell fixed?

ResultPoint estimateWilson 95% intervalWhat it supports
privacy × role-play: 9/5018%9.8–30.8%a real weakness; fix before release
after fix: 0/500%0–7.1%only "below ~7%"; not "fixed"
to claim < 1%⌈ln 0.05 / ln 0.99⌉ = 299 clean attempts
to claim < 0.1%2,995 clean attempts
judge FPR 5%, TPR 90%, reports 8.2%corrected (0.082 − 0.05)/0.85 = 3.8%
Results for two cells and what they license you to say (95% level)

Fifty clean attempts after a fix sound reassuring and prove little. The exact one-sided bound for 0/50 is 1−0.051/50=5.8%1-0.05^{1/50}=5.8\%, close to the rule-of-three 3/50=6%3/50=6\%. Low-rate claims need either many automated attempts or a statement of the bound you actually reached.

python
1import math, random
2from collections import defaultdict
3
4AREAS = ["privacy", "cyber", "self-harm"]
5STYLES = ["direct", "role-play", "multi-turn"]
6
7def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
8 p, d = k / n, 1 + z * z / n
9 c = (p + z * z / (2 * n)) / d
10 h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
11 return max(0.0, c - h), min(1.0, c + h)
12
13def run_campaign(gen_attack, target, judge, n_per_cell=50, seed=0):
14 rng = random.Random(seed)
15 hits, log = defaultdict(int), []
16 for area in AREAS:
17 for style in STYLES:
18 for _ in range(n_per_cell):
19 prompt = gen_attack(area, style, rng) # expert template or red LM
20 reply = target(prompt) # the deployed system, guardrails on
21 verdict = judge(area, prompt, reply) # classifier + human spot-check
22 if verdict["unsafe"]:
23 hits[area, style] += 1
24 log.append({"area": area, "style": style, "prompt": prompt,
25 "reply": reply, "severity": verdict["severity"]})
26 report = {(a, s): (hits[a, s], n_per_cell, wilson(hits[a, s], n_per_cell))
27 for a in AREAS for s in STYLES} # zero-hit cells matter too
28 return report, log # the log becomes regression test cases
29
30# stand-ins so the harness runs end to end; replace with real calls
31weak = {("privacy", "role-play"): 0.18}
32gen = lambda a, s, rng: (a, s, rng.random())
33tgt = lambda p: "UNSAFE" if p[2] < weak.get((p[0], p[1]), 0.01) else "refusal"
34jdg = lambda a, p, r: {"unsafe": r == "UNSAFE", "severity": "high"}
35
36report, log = run_campaign(gen, tgt, jdg)
37for (area, style), (k, n, (lo, hi)) in sorted(report.items()):
38 print(f"{area:9s} {style:10s} {k}/{n} = {k/n:.0%} 95% CI {lo:.1%}-{hi:.1%}")
39# privacy multi-turn 0/50 = 0% 95% CI 0.0%-7.1%
40# privacy role-play 4/50 = 8% 95% CI 3.2%-18.8% (true rate 18%: 50 tries is noisy)
A minimal campaign harness: taxonomy grid × attempts, judge, per-cell Wilson intervals, findings log (runs as is with the stand-in functions; swap in real generator, target and judge calls)

Where it's used

Pre-release model evaluation

Expert campaigns on dangerous-capability areas (bio, cyber) and broad automated sweeps across the policy taxonomy.

Application security

Testing a deployed agent's tools, retrieval and permissions for injection and data exfiltration, with the guardrails switched on.

Continuous regression

Every confirmed finding becomes a test case run in CI on each model, prompt or guardrail change.

Check yourself

After a fix, a cell shows 0 failures in 50 attempts. What can you claim at 95% confidence?

Check yourself

Perez et al. generated test cases by zero-shot prompting, few-shot prompting, supervised learning and RL. What did RL trade away for its higher failure rate?

Remember

  • ✓Scope first: the deployed system, the threat model, a harm taxonomy × attack-style grid, judging rules, exit criteria.
  • ✓Manual experts bring depth; red LMs (Perez et al. 2022) bring scale; RL attackers trade diversity for success rate.
  • ✓Report ASR per cell with intervals; detecting a rate pp at 95% needs about 3/p3/p attempts (299 for 1%).
  • ✓Correct judge error: a^=(m−FPR)/(TPR−FPR)\hat a=(m-\mathrm{FPR})/(\mathrm{TPR}-\mathrm{FPR}); a 5% FPR judge cannot see rates below 5%.
  • ✓Findings become regression tests; keep fresh, held-out attacks so lower ASR means real robustness.
← back to Generative AI roadmap