# Red teaming

> Red teaming attacks a generative system on purpose, by human experts and by attacker LLMs (Perez et al. 2022), across a scoped harm taxonomy, and reports attack success rate per cell with intervals; every finding becomes a regression test.

Source: https://www.mlfoundry.in/learn/red-teaming  
Section: Generative AI › Safety, Alignment and Responsible AI  
Level: core · 8 min read · updated 2026-10-21

**Red teaming:** is structured adversarial testing of an AI system: people or programs try to make it fail in the ways that matter (harmful content, data leakage, unsafe tool actions), and the failures are measured, logged and fixed. Perez et al. ("Red Teaming Language Models with Language Models", 2022, DeepMind) showed that one LM can generate test cases for another at scale. Ganguli et al. (2022, Anthropic) described large manual campaigns and released 38,961 red-team attacks.

Benchmarks measure average behaviour on fixed inputs. Red teaming looks for the worst case on inputs nobody wrote down yet. It sits between training and release, and runs again after every significant change. Its outputs feed **Guardrails**, safety fine-tuning data (**Supervised fine-tuning (SFT)**, **RLHF**) and the regression suite. The attacks it uses are covered in **Jailbreaking** and **Prompt injection**; this article is about running the campaign and reading its numbers.

> Figure: The loop. Scope fixes the deployed system, the threat model, the harm taxonomy and an exit criterion. Experts and a red LM generate attacks against the target exactly as it runs in production, with system prompt, tools and guardrails. A judge (classifier plus human review) scores each output. Findings such as "privacy × role-play: 9/50 = 18%, high severity" drive mitigations. Every finding then joins a regression suite, and the next round looks for new attacks. Inset: Perez et al.'s finding that moving from zero-shot to RL generation raises success rate but lowers diversity.

## Scoping

**Before the first attack**

1. **System under test**: the model *plus* system prompt, retrieval, tools and filters, at a pinned version. A raw model and a deployed app fail differently.
2. **Threat model**: who attacks (curious user, motivated abuser, third party planting content), with what access (chat only, API with system prompt, documents the app ingests) and what budget.
3. **Harm taxonomy**: the categories to cover, e.g. Weidinger et al.'s six risk areas (2021) or a product policy. Cross them with attack styles to get a grid of cells.
4. **Judging rules**: what counts as a failure, severity levels, who reviews borderline cases.
5. **Exit criteria**: e.g. ASR below 1% in every high-severity cell, with a stated confidence level, and no open critical findings.

## Manual and automated

Manual red teamers bring domain knowledge (a biosecurity expert knows which answer is actually dangerous) and creativity. They are slow and expensive, and they tend to search where they already expect problems. Automated red teaming uses an attacker LLM. Perez et al. compared four ways to generate test cases: zero-shot prompting, stochastic few-shot prompting from earlier successes, supervised fine-tuning on successes, and RL against a harm classifier. RL found failures most often but with the least diversity. Zero-shot was the most diverse and the least effective.

Later methods push for both. Rainbow Teaming (Samvelyan et al. 2024) keeps a MAP-Elites archive with one best attack per (risk category × attack style) cell, so the search must fill the grid. PAIR and TAP iterate on single targets (see **Jailbreaking**). Open-source harnesses such as Microsoft's PyRIT and garak automate the send–judge–log loop.

## Reading the numbers

$$
\widehat{\mathrm{ASR}}_{c}=\frac{k_c}{n_c},\qquad n_{\text{needed}}=\left\lceil\frac{\ln(1-\gamma)}{\ln(1-p)}\right\rceil\approx\frac{3}{p}\;\;(\gamma=0.95)
$$

- $k_c,\;n_c$: failures found and attempts made in taxonomy cell $c$; report a Wilson interval, not just the ratio
- $p$: the true failure rate you want to be able to detect
- $\gamma$: probability of seeing at least one failure; $\approx3/p$ is the "rule of three"

$$
m=\mathrm{TPR}\cdot a+\mathrm{FPR}\cdot(1-a)\quad\Longrightarrow\quad \hat a=\frac{m-\mathrm{FPR}}{\mathrm{TPR}-\mathrm{FPR}}
$$

- $m$: ASR reported by an automated judge
- $a$: true ASR; the correction (Rogan–Gladen) needs TPR and FPR measured on a human-labelled sample

A judge with a 5% false-positive rate cannot report an ASR below about 5%, whatever the model does. When the true rate is low, judge error dominates the measurement. Spot-check judged failures by hand and measure the judge on a labelled set before trusting small differences.

> Figure: A synthetic 6 × 4 grid with hidden failure rates: five weak cells (12–35%), the rest 0.3–2.1%. At 1,000 attempts, the RL-style exploiter (ε-greedy) reaches the highest ASR, 307/1000 = 30.7%, but finds failures in only 3 cells and 2 of the 5 weak ones. Uniform sampling reaches 3.8% and covers 12 cells, all 5 weak. Quality-diversity reaches 5.7% over 13 cells, also all 5 weak. Manual experts, biased towards direct asks, find 1 of 5 weak cells at 100 attempts. With judge FPR 5%, uniform's true 3.8% is reported as 8.2%.

**Checkable specifics**

- **Perez et al. 2022:** red LM and target from the Gopher family (target: 280B Dialogue-Prompted Gopher); offensive replies scored by a classifier; also probed data leakage, contact-information generation and distributional bias
- **Ganguli et al. 2022:** 38,961 human red-team attacks on models of 2.7B, 13B and 52B parameters; RLHF models became harder to red team as they grew, the other model types did not
- **Weidinger et al. 2021:** six risk areas: discrimination/exclusion/toxicity; information hazards; misinformation; malicious uses; human–computer interaction harms; automation, access and environmental harms
- **Frameworks:** NIST AI 600-1, the Generative AI Profile (July 2024), lists red teaming among its suggested actions; MITRE ATLAS catalogues adversarial tactics against AI systems
- **Regulation (general terms):** the EU AI Act requires providers of general-purpose AI models with systemic risk to carry out adversarial testing; obligations and dates vary, so check the current text (this is not legal advice)

## Worked example: is the cell fixed?

Results for two cells and what they license you to say (95% level)

| Result | Point estimate | Wilson 95% interval | What it supports |
| --- | --- | --- | --- |
| privacy × role-play: 9/50 | 18% | 9.8–30.8% | a real weakness; fix before release |
| after fix: 0/50 | 0% | 0–7.1% | only "below ~7%"; not "fixed" |
| to claim < 1% |  |  | ⌈ln 0.05 / ln 0.99⌉ = 299 clean attempts |
| to claim < 0.1% |  |  | 2,995 clean attempts |
| judge FPR 5%, TPR 90%, reports 8.2% |  |  | corrected (0.082 − 0.05)/0.85 = 3.8% |

Fifty clean attempts after a fix sound reassuring and prove little. The exact one-sided bound for 0/50 is $1-0.05^{1/50}=5.8\%$, close to the rule-of-three $3/50=6\%$. Low-rate claims need either many automated attempts or a statement of the bound you actually reached.

```python
import math, random
from collections import defaultdict

AREAS = ["privacy", "cyber", "self-harm"]
STYLES = ["direct", "role-play", "multi-turn"]

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p, d = k / n, 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, c - h), min(1.0, c + h)

def run_campaign(gen_attack, target, judge, n_per_cell=50, seed=0):
    rng = random.Random(seed)
    hits, log = defaultdict(int), []
    for area in AREAS:
        for style in STYLES:
            for _ in range(n_per_cell):
                prompt = gen_attack(area, style, rng)        # expert template or red LM
                reply = target(prompt)                       # the deployed system, guardrails on
                verdict = judge(area, prompt, reply)         # classifier + human spot-check
                if verdict["unsafe"]:
                    hits[area, style] += 1
                    log.append({"area": area, "style": style, "prompt": prompt,
                                "reply": reply, "severity": verdict["severity"]})
    report = {(a, s): (hits[a, s], n_per_cell, wilson(hits[a, s], n_per_cell))
              for a in AREAS for s in STYLES}       # zero-hit cells matter too
    return report, log   # the log becomes regression test cases

# stand-ins so the harness runs end to end; replace with real calls
weak = {("privacy", "role-play"): 0.18}
gen = lambda a, s, rng: (a, s, rng.random())
tgt = lambda p: "UNSAFE" if p[2] < weak.get((p[0], p[1]), 0.01) else "refusal"
jdg = lambda a, p, r: {"unsafe": r == "UNSAFE", "severity": "high"}

report, log = run_campaign(gen, tgt, jdg)
for (area, style), (k, n, (lo, hi)) in sorted(report.items()):
    print(f"{area:9s} {style:10s} {k}/{n} = {k/n:.0%}  95% CI {lo:.1%}-{hi:.1%}")
# privacy   multi-turn 0/50 = 0%  95% CI 0.0%-7.1%
# privacy   role-play  4/50 = 8%  95% CI 3.2%-18.8%   (true rate 18%: 50 tries is noisy)
```

A minimal campaign harness: taxonomy grid × attempts, judge, per-cell Wilson intervals, findings log (runs as is with the stand-in functions; swap in real generator, target and judge calls)

> **Optimising ASR rewards one trick:** An automated attacker scored only on success rate converges on the single weakest cell and repeats it. The campaign then reports a high ASR, 30.7% in the viz, while most of the grid goes untested. Score red-team generators on coverage and diversity too (cells with a finding, distinct attack clusters, self-BLEU), as Rainbow Teaming does with its archive.

> **Red teaming is not a certificate:** A campaign shows that failures exist, not that they do not. Its results depend on the threat model, the attackers' skill and budget, and the judge. Publish those alongside the numbers. Keep a held-out set of attacks that were never used for mitigation, so a fall in ASR reflects real robustness and not training on the test.

- **Pre-release model evaluation:** Expert campaigns on dangerous-capability areas (bio, cyber) and broad automated sweeps across the policy taxonomy.
- **Application security:** Testing a deployed agent's tools, retrieval and permissions for injection and data exfiltration, with the guardrails switched on.
- **Continuous regression:** Every confirmed finding becomes a test case run in CI on each model, prompt or guardrail change.

**Question:** After a fix, a cell shows 0 failures in 50 attempts. What can you claim at 95% confidence?

- A. The failure is fixed
- B. The failure rate is below about 6–7%
- C. The failure rate is below 1%
- D. Nothing at all

**Answer:** B. The failure rate is below about 6–7% — The one-sided exact bound is $1-0.05^{1/50}=5.8\%$ (rule of three: $3/50=6\%$); the two-sided Wilson upper limit is 7.1%. To claim below 1% you need about 299 clean attempts.

**Question:** Perez et al. generated test cases by zero-shot prompting, few-shot prompting, supervised learning and RL. What did RL trade away for its higher failure rate?

- A. Speed
- B. Diversity of test cases
- C. Use of a classifier
- D. Access to the target model

**Answer:** B. Diversity of test cases — RL against the harm classifier concentrated on the attacks that worked best and produced less varied test cases. Zero-shot generation was the most diverse but found failures least often. Mixing methods, or optimising for diversity explicitly, covers more of the space.

**Key points**

- Scope first: the deployed system, the threat model, a harm taxonomy × attack-style grid, judging rules, exit criteria.
- Manual experts bring depth; red LMs (Perez et al. 2022) bring scale; RL attackers trade diversity for success rate.
- Report ASR per cell with intervals; detecting a rate $p$ at 95% needs about $3/p$ attempts (299 for 1%).
- Correct judge error: $\hat a=(m-\mathrm{FPR})/(\mathrm{TPR}-\mathrm{FPR})$; a 5% FPR judge cannot see rates below 5%.
- Findings become regression tests; keep fresh, held-out attacks so lower ASR means real robustness.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
