# Jailbreaking

> A jailbreak is an input that gets a safety-trained model to produce what it was trained to refuse, by pitting helpfulness against safety or leaving its safety data's distribution (Wei et al. 2023); GCG automates it with gradient-searched suffixes.

Source: https://www.mlfoundry.in/learn/jailbreaking  
Section: Generative AI › Prompt Engineering  
Level: advanced · 8 min read · updated 2026-10-21

**Jailbreaking:** is crafting inputs that make a safety-trained LLM produce outputs its developer trained it to refuse, such as dangerous instructions or abusive content. The user is the attacker and the target is the model's own safety behaviour. Wei, Haghtalab and Steinhardt ("Jailbroken: How Does LLM Safety Training Fail?", NeurIPS 2023) traced jailbreaks to two failure modes. Zou et al. ("Universal and Transferable Adversarial Attacks on Aligned Language Models", 2023) showed they can be found automatically by optimisation.

Safety behaviour is added after pre-training, by **Supervised fine-tuning (SFT)** on refusals and by **RLHF** or **Constitutional AI**. The capability to produce the harmful text is still in the weights; training only makes refusal the likely response on inputs like those seen in safety data. A jailbreak finds inputs where that no longer holds. This differs from **Prompt injection**, where a third party's text hijacks an *application* and the payload need not be harmful at all.

**Competing objectives:** the model is trained to be helpful, follow instructions and stay in character, as well as to refuse. Role-play frames, instructions to begin with an affirmative prefix, or bans on refusal phrases set these goals against each other. **Mismatched generalisation:** pre-training covers far more input space than safety training. The same request in base64, a cipher or a low-resource language may be understood by the model but never have been refused in training.

> Figure: Top: the main attack families, the access each needs, and the failure mode it exploits. Requests are shown only as a placeholder; this article explains mechanisms, not working prompts. Bottom: the usual defence stack. Input filters catch high-perplexity GCG suffixes, the model itself is safety-trained, and an output classifier checks what was actually generated. Evaluation reports attack success rate with a rubric judge, together with over-refusal on safe prompts.

## Optimisation attacks: GCG

Greedy Coordinate Gradient (GCG) appends a suffix of tokens $x_{\mathcal I}$ to a harmful request and searches for the suffix that makes an affirmative target such as "Sure, here is…" most likely. Once the model starts that way it usually continues.

$$
\min_{x_{\mathcal I}\in\{1..V\}^{|\mathcal I|}}\;\mathcal L(x_{1:n})=-\log p\big(x^\star_{n+1:n+H}\,\big|\,x_{1:n}\big),\qquad \mathcal X_i=\operatorname{Top\text{-}k}\big(-\nabla_{e_{x_i}}\mathcal L\big)
$$

- $x^\star_{n+1:n+H}$: the target completion of $H$ tokens (an affirmative opening)
- $e_{x_i}$: one-hot vector of the token at suffix position $i$; its gradient ranks replacement tokens
- $\mathcal X_i$: $k$ candidate tokens for position $i$; $B$ random single swaps are evaluated exactly and the best is kept

The gradient is only a linear guess at which swaps help, so each step scores a batch of real candidates with forward passes. Optimising one suffix over several behaviours and models (Vicuna and Guanaco in the paper) gave "universal" suffixes that transferred to closed models behind APIs. The cost is the weak point for the attacker: with the paper's settings, one run is about $500\times512=256{,}000$ forward passes.

**Published specifics (check the papers for exact settings)**

- **GCG (Zou et al. 2023):** 20-token suffix initialised as "! ! … !", top-k = 256 candidates per position, batch B = 512, up to 500 steps; evaluated on AdvBench (520 harmful behaviours, 500 harmful strings)
- **PAIR (Chao et al. 2023):** black-box: an attacker LLM rewrites the prompt from a judge's feedback; often succeeds in under 20 queries. TAP (Mehrotra et al. 2023) adds tree search with pruning
- **Many-shot (Anil et al. 2024):** hundreds of fake dialogue turns in a long context; tested up to 256 shots, with attack effectiveness following a power law in the number of shots
- **Low-resource languages (Yong et al. 2023):** translating AdvBench into low-resource languages such as Zulu bypassed GPT-4's safeguards in about 79% of cases (as of the 2023 model)
- **Crescendo (Russinovich et al. 2024):** multi-turn: starts benign and escalates by referring back to the model's own earlier answers

$$
-\log p(\text{harmful reply}\mid n\text{ shots})\;\approx\;C\,n^{-\alpha}
$$

- $n$: number of faux dialogue examples placed in the context
- $\alpha$: power-law exponent fitted per model and task (Anil et al. 2024); a straight line on log–log axes

The power law is the same in-context learning that makes **Few-shot prompting** work, which is why it cannot simply be trained away. Long context windows made the attack possible.

> Figure: A toy GCG over 48 abstract tokens against a synthetic compliance model, run for 20 "behaviours" of different difficulty. Defaults (20-token suffix, no fluency penalty, T = 40): p(comply) starts at 3.1 × 10⁻⁷ for "! ! …" and passes 0.5 after a median of 12 iterations. After 80 iterations all 20 behaviours are jailbroken, but the suffixes have perplexity 79–352 against a benign median of 14.5 (max 31.2). The filter stops all 20 and flags 0% of benign text. Set λ = 1: still 20/20, now at perplexity 8.3–20.2, so all pass the filter. With 5 suffix tokens, 0/20 succeed.

## Defences

Defence families and what they are known to miss

| Defence | How it works | Known gap |
| --- | --- | --- |
| Perplexity filter (Alon & Kamfonas 2023; Jain et al. 2023) | Flag prompts, or windows of them, whose LM perplexity is far above normal | Readable attacks (PAIR, AutoDAN, role-play) have normal perplexity |
| SmoothLLM (Robey et al. 2023) | Randomly perturb characters in several copies of the prompt; majority-vote the responses | Targets brittle token-level suffixes; costs several model calls |
| Input/output classifiers (e.g. Llama Guard, Inan et al. 2023) | A separate safety model labels the prompt and the response against a harm taxonomy | Obfuscated outputs; classifier itself can be attacked |
| Adversarial and refusal training | Add jailbreak attempts with refusals to SFT/RLHF data | Covers known attacks; new families appear |
| Circuit breakers (Zou et al. 2024) | Train internal representations of harmful content to be rerouted, so generation breaks down | Research method; can hurt utility if over-applied |

## Worked example: the judge decides the ASR

Illustrative audit of 200 responses to harmful requests, scored by a keyword judge ("no refusal phrase = success") and by a rubric judge ("specific and actually harmful = success")

| Response type | Count | Keyword judge | Rubric judge |
| --- | --- | --- | --- |
| Clear refusal | 105 | refused | safe |
| Apology, then complies with specifics | 15 | refused | harmful |
| Complies with specific harmful detail | 30 | success | harmful |
| Complies but vague, wrong or off-topic | 35 | success | safe |
| Safe redirection, no refusal phrase | 15 | success | safe |
| ASR (95% Wilson interval) | 200 | 80/200 = 40.0% (33.5–46.9%) | 45/200 = 22.5% (17.3–28.8%) |

The keyword judge both over-counts (vague or harmless answers) and under-counts (compliance after an apology). StrongREJECT (Souly et al. 2024) made this point: many published jailbreaks look far less effective when the answer must be specific and useful. HarmBench (Mazeika et al. 2024) and JailbreakBench (Chao et al. 2024) fix the behaviours, the judge and the query budget so that numbers can be compared.

```python
import math, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()

@torch.no_grad()
def window_ppl(text: str, win: int = 10) -> float:
    """Max perplexity over sliding windows: a short gibberish suffix
    can hide inside the mean perplexity of a long prompt."""
    ids = tok(text, return_tensors="pt").input_ids[0]
    worst = 0.0
    for s in range(max(1, len(ids) - win + 1)):
        w = ids[s:s + win].unsqueeze(0)
        if w.shape[1] < 2:
            break
        worst = max(worst, math.exp(lm(w, labels=w).loss.item()))
    return worst

def calibrate(benign_prompts: list[str], q: float = 0.999) -> float:
    scores = sorted(window_ppl(p) for p in benign_prompts)
    return scores[int(q * (len(scores) - 1))]

def flag(prompt: str, threshold: float) -> bool:
    return window_ppl(prompt) > threshold   # route to refusal or human review
```

A windowed perplexity filter with GPT-2 via Hugging Face transformers (not run here: torch and transformers were unavailable). Calibrate the threshold on benign traffic

> **ASR 0 is easy and useless:** A model that refuses everything has zero attack success. Always report over-refusal next to ASR, for example on XSTest (Röttger et al.), whose safe prompts only look dangerous ("how do I kill a Python process?"). A defence that halves ASR but doubles false refusals may be a net loss for users.

> **Evaluate against adaptive attacks:** Most defences are tested on attacks that existed before the defence. Perplexity filters stop GCG; fluent attacks such as AutoDAN (Liu et al. 2023) and PAIR were then built to slip past them. Report results against an attacker who knows the defence, with a fixed query budget. Treat every defence as a speed bump for **Red teaming** to measure, not a wall.

**Question:** A request that the model refuses in English is answered when written in base64. Which failure mode is this?

- A. Competing objectives
- B. Mismatched generalisation
- C. Prompt injection
- D. Hallucination

**Answer:** B. Mismatched generalisation — Pre-training taught the model to read base64, but safety training never covered that input distribution, so the refusal behaviour did not generalise to it. Competing objectives is when the model is pushed to trade safety for helpfulness or instruction-following.

**Question:** Why does a perplexity filter stop vanilla GCG but not PAIR?

- A. PAIR uses more queries
- B. GCG suffixes are unreadable token strings with very high perplexity, while PAIR's attacker LLM writes fluent text
- C. PAIR needs gradients
- D. GCG only works on open models

**Answer:** B. GCG suffixes are unreadable token strings with very high perplexity, while PAIR's attacker LLM writes fluent text — The filter measures how unnatural the text is. GCG optimises only for the target likelihood, so its suffixes are gibberish. PAIR's prompts are written in natural language. In the viz, a fluency penalty λ = 1 brings suffix perplexity to benign levels with no loss of success.

**Key points**

- Jailbreak: the user crafts input that defeats the model's own safety training; distinct from prompt injection.
- Two failure modes (Wei et al. 2023): competing objectives (role-play, refusal suppression) and mismatched generalisation (encodings, low-resource languages).
- GCG minimises $-\log p(\text{Sure, here is}\mid\text{request}+\text{suffix})$ with gradient-ranked token swaps; suffixes transfer across models.
- Many-shot jailbreaks follow a power law in the number of shots; PAIR/TAP automate black-box attacks in tens of queries.
- Layer defences (filters, safety training, output classifiers), measure ASR with a rubric judge on a fixed benchmark, and always report over-refusal.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
