Definition
Red teaming is structured adversarial testing of an AI system: people or programs try to make it fail in the ways that matter (harmful content, data leakage, unsafe tool actions), and the failures are measured, logged and fixed. Perez et al. ("Red Teaming Language Models with Language Models", 2022, DeepMind) showed that one LM can generate test cases for another at scale. Ganguli et al. (2022, Anthropic) described large manual campaigns and released 38,961 red-team attacks.
Benchmarks measure average behaviour on fixed inputs. Red teaming looks for the worst case on inputs nobody wrote down yet. It sits between training and release, and runs again after every significant change. Its outputs feed Guardrails, safety fine-tuning data (Supervised fine-tuning (SFT), RLHF) and the regression suite. The attacks it uses are covered in Jailbreaking and Prompt injection; this article is about running the campaign and reading its numbers.
Scoping
Before the first attack
- 1System under test: the model plus system prompt, retrieval, tools and filters, at a pinned version. A raw model and a deployed app fail differently.
- 2Threat model: who attacks (curious user, motivated abuser, third party planting content), with what access (chat only, API with system prompt, documents the app ingests) and what budget.
- 3Harm taxonomy: the categories to cover, e.g. Weidinger et al.'s six risk areas (2021) or a product policy. Cross them with attack styles to get a grid of cells.
- 4Judging rules: what counts as a failure, severity levels, who reviews borderline cases.
- 5Exit criteria: e.g. ASR below 1% in every high-severity cell, with a stated confidence level, and no open critical findings.
Manual and automated
Manual red teamers bring domain knowledge (a biosecurity expert knows which answer is actually dangerous) and creativity. They are slow and expensive, and they tend to search where they already expect problems. Automated red teaming uses an attacker LLM. Perez et al. compared four ways to generate test cases: zero-shot prompting, stochastic few-shot prompting from earlier successes, supervised fine-tuning on successes, and RL against a harm classifier. RL found failures most often but with the least diversity. Zero-shot was the most diverse and the least effective.
Later methods push for both. Rainbow Teaming (Samvelyan et al. 2024) keeps a MAP-Elites archive with one best attack per (risk category × attack style) cell, so the search must fill the grid. PAIR and TAP iterate on single targets (see Jailbreaking). Open-source harnesses such as Microsoft's PyRIT and garak automate the send–judge–log loop.
Reading the numbers
- failures found and attempts made in taxonomy cell ; report a Wilson interval, not just the ratio
- the true failure rate you want to be able to detect
- probability of seeing at least one failure; is the "rule of three"
- ASR reported by an automated judge
- true ASR; the correction (Rogan–Gladen) needs TPR and FPR measured on a human-labelled sample
A judge with a 5% false-positive rate cannot report an ASR below about 5%, whatever the model does. When the true rate is low, judge error dominates the measurement. Spot-check judged failures by hand and measure the judge on a labelled set before trusting small differences.
Checkable specifics
- Perez et al. 2022
- red LM and target from the Gopher family (target: 280B Dialogue-Prompted Gopher); offensive replies scored by a classifier; also probed data leakage, contact-information generation and distributional bias
- Ganguli et al. 2022
- 38,961 human red-team attacks on models of 2.7B, 13B and 52B parameters; RLHF models became harder to red team as they grew, the other model types did not
- Weidinger et al. 2021
- six risk areas: discrimination/exclusion/toxicity; information hazards; misinformation; malicious uses; human–computer interaction harms; automation, access and environmental harms
- Frameworks
- NIST AI 600-1, the Generative AI Profile (July 2024), lists red teaming among its suggested actions; MITRE ATLAS catalogues adversarial tactics against AI systems
- Regulation (general terms)
- the EU AI Act requires providers of general-purpose AI models with systemic risk to carry out adversarial testing; obligations and dates vary, so check the current text (this is not legal advice)
Worked example: is the cell fixed?
| Result | Point estimate | Wilson 95% interval | What it supports |
|---|---|---|---|
| privacy × role-play: 9/50 | 18% | 9.8–30.8% | a real weakness; fix before release |
| after fix: 0/50 | 0% | 0–7.1% | only "below ~7%"; not "fixed" |
| to claim < 1% | ⌈ln 0.05 / ln 0.99⌉ = 299 clean attempts | ||
| to claim < 0.1% | 2,995 clean attempts | ||
| judge FPR 5%, TPR 90%, reports 8.2% | corrected (0.082 − 0.05)/0.85 = 3.8% |
Fifty clean attempts after a fix sound reassuring and prove little. The exact one-sided bound for 0/50 is , close to the rule-of-three . Low-rate claims need either many automated attempts or a statement of the bound you actually reached.
1import math, random2from collections import defaultdict34AREAS = ["privacy", "cyber", "self-harm"]5STYLES = ["direct", "role-play", "multi-turn"]67def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:8 p, d = k / n, 1 + z * z / n9 c = (p + z * z / (2 * n)) / d10 h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d11 return max(0.0, c - h), min(1.0, c + h)1213def run_campaign(gen_attack, target, judge, n_per_cell=50, seed=0):14 rng = random.Random(seed)15 hits, log = defaultdict(int), []16 for area in AREAS:17 for style in STYLES:18 for _ in range(n_per_cell):19 prompt = gen_attack(area, style, rng) # expert template or red LM20 reply = target(prompt) # the deployed system, guardrails on21 verdict = judge(area, prompt, reply) # classifier + human spot-check22 if verdict["unsafe"]:23 hits[area, style] += 124 log.append({"area": area, "style": style, "prompt": prompt,25 "reply": reply, "severity": verdict["severity"]})26 report = {(a, s): (hits[a, s], n_per_cell, wilson(hits[a, s], n_per_cell))27 for a in AREAS for s in STYLES} # zero-hit cells matter too28 return report, log # the log becomes regression test cases2930# stand-ins so the harness runs end to end; replace with real calls31weak = {("privacy", "role-play"): 0.18}32gen = lambda a, s, rng: (a, s, rng.random())33tgt = lambda p: "UNSAFE" if p[2] < weak.get((p[0], p[1]), 0.01) else "refusal"34jdg = lambda a, p, r: {"unsafe": r == "UNSAFE", "severity": "high"}3536report, log = run_campaign(gen, tgt, jdg)37for (area, style), (k, n, (lo, hi)) in sorted(report.items()):38 print(f"{area:9s} {style:10s} {k}/{n} = {k/n:.0%} 95% CI {lo:.1%}-{hi:.1%}")39# privacy multi-turn 0/50 = 0% 95% CI 0.0%-7.1%40# privacy role-play 4/50 = 8% 95% CI 3.2%-18.8% (true rate 18%: 50 tries is noisy)Where it's used
Pre-release model evaluation
Expert campaigns on dangerous-capability areas (bio, cyber) and broad automated sweeps across the policy taxonomy.
Application security
Testing a deployed agent's tools, retrieval and permissions for injection and data exfiltration, with the guardrails switched on.
Continuous regression
Every confirmed finding becomes a test case run in CI on each model, prompt or guardrail change.
Check yourself
After a fix, a cell shows 0 failures in 50 attempts. What can you claim at 95% confidence?
Check yourself
Perez et al. generated test cases by zero-shot prompting, few-shot prompting, supervised learning and RL. What did RL trade away for its higher failure rate?
Remember
- ✓Scope first: the deployed system, the threat model, a harm taxonomy × attack-style grid, judging rules, exit criteria.
- ✓Manual experts bring depth; red LMs (Perez et al. 2022) bring scale; RL attackers trade diversity for success rate.
- ✓Report ASR per cell with intervals; detecting a rate at 95% needs about attempts (299 for 1%).
- ✓Correct judge error: ; a 5% FPR judge cannot see rates below 5%.
- ✓Findings become regression tests; keep fresh, held-out attacks so lower ASR means real robustness.