Definition
Jailbreaking is crafting inputs that make a safety-trained LLM produce outputs its developer trained it to refuse, such as dangerous instructions or abusive content. The user is the attacker and the target is the model's own safety behaviour. Wei, Haghtalab and Steinhardt ("Jailbroken: How Does LLM Safety Training Fail?", NeurIPS 2023) traced jailbreaks to two failure modes. Zou et al. ("Universal and Transferable Adversarial Attacks on Aligned Language Models", 2023) showed they can be found automatically by optimisation.
Safety behaviour is added after pre-training, by Supervised fine-tuning (SFT) on refusals and by RLHF or Constitutional AI. The capability to produce the harmful text is still in the weights; training only makes refusal the likely response on inputs like those seen in safety data. A jailbreak finds inputs where that no longer holds. This differs from Prompt injection, where a third party's text hijacks an application and the payload need not be harmful at all.
Competing objectives: the model is trained to be helpful, follow instructions and stay in character, as well as to refuse. Role-play frames, instructions to begin with an affirmative prefix, or bans on refusal phrases set these goals against each other. Mismatched generalisation: pre-training covers far more input space than safety training. The same request in base64, a cipher or a low-resource language may be understood by the model but never have been refused in training.
Optimisation attacks: GCG
Greedy Coordinate Gradient (GCG) appends a suffix of tokens to a harmful request and searches for the suffix that makes an affirmative target such as "Sure, here is…" most likely. Once the model starts that way it usually continues.
- the target completion of tokens (an affirmative opening)
- one-hot vector of the token at suffix position ; its gradient ranks replacement tokens
- candidate tokens for position ; random single swaps are evaluated exactly and the best is kept
The gradient is only a linear guess at which swaps help, so each step scores a batch of real candidates with forward passes. Optimising one suffix over several behaviours and models (Vicuna and Guanaco in the paper) gave "universal" suffixes that transferred to closed models behind APIs. The cost is the weak point for the attacker: with the paper's settings, one run is about forward passes.
Published specifics (check the papers for exact settings)
- GCG (Zou et al. 2023)
- 20-token suffix initialised as "! ! … !", top-k = 256 candidates per position, batch B = 512, up to 500 steps; evaluated on AdvBench (520 harmful behaviours, 500 harmful strings)
- PAIR (Chao et al. 2023)
- black-box: an attacker LLM rewrites the prompt from a judge's feedback; often succeeds in under 20 queries. TAP (Mehrotra et al. 2023) adds tree search with pruning
- Many-shot (Anil et al. 2024)
- hundreds of fake dialogue turns in a long context; tested up to 256 shots, with attack effectiveness following a power law in the number of shots
- Low-resource languages (Yong et al. 2023)
- translating AdvBench into low-resource languages such as Zulu bypassed GPT-4's safeguards in about 79% of cases (as of the 2023 model)
- Crescendo (Russinovich et al. 2024)
- multi-turn: starts benign and escalates by referring back to the model's own earlier answers
- number of faux dialogue examples placed in the context
- power-law exponent fitted per model and task (Anil et al. 2024); a straight line on log–log axes
The power law is the same in-context learning that makes Few-shot prompting work, which is why it cannot simply be trained away. Long context windows made the attack possible.
Defences
| Defence | How it works | Known gap |
|---|---|---|
| Perplexity filter (Alon & Kamfonas 2023; Jain et al. 2023) | Flag prompts, or windows of them, whose LM perplexity is far above normal | Readable attacks (PAIR, AutoDAN, role-play) have normal perplexity |
| SmoothLLM (Robey et al. 2023) | Randomly perturb characters in several copies of the prompt; majority-vote the responses | Targets brittle token-level suffixes; costs several model calls |
| Input/output classifiers (e.g. Llama Guard, Inan et al. 2023) | A separate safety model labels the prompt and the response against a harm taxonomy | Obfuscated outputs; classifier itself can be attacked |
| Adversarial and refusal training | Add jailbreak attempts with refusals to SFT/RLHF data | Covers known attacks; new families appear |
| Circuit breakers (Zou et al. 2024) | Train internal representations of harmful content to be rerouted, so generation breaks down | Research method; can hurt utility if over-applied |
Worked example: the judge decides the ASR
| Response type | Count | Keyword judge | Rubric judge |
|---|---|---|---|
| Clear refusal | 105 | refused | safe |
| Apology, then complies with specifics | 15 | refused | harmful |
| Complies with specific harmful detail | 30 | success | harmful |
| Complies but vague, wrong or off-topic | 35 | success | safe |
| Safe redirection, no refusal phrase | 15 | success | safe |
| ASR (95% Wilson interval) | 200 | 80/200 = 40.0% (33.5–46.9%) | 45/200 = 22.5% (17.3–28.8%) |
The keyword judge both over-counts (vague or harmless answers) and under-counts (compliance after an apology). StrongREJECT (Souly et al. 2024) made this point: many published jailbreaks look far less effective when the answer must be specific and useful. HarmBench (Mazeika et al. 2024) and JailbreakBench (Chao et al. 2024) fix the behaviours, the judge and the query budget so that numbers can be compared.
1import math, torch2from transformers import AutoModelForCausalLM, AutoTokenizer34tok = AutoTokenizer.from_pretrained("gpt2")5lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()67@torch.no_grad()8def window_ppl(text: str, win: int = 10) -> float:9 """Max perplexity over sliding windows: a short gibberish suffix10 can hide inside the mean perplexity of a long prompt."""11 ids = tok(text, return_tensors="pt").input_ids[0]12 worst = 0.013 for s in range(max(1, len(ids) - win + 1)):14 w = ids[s:s + win].unsqueeze(0)15 if w.shape[1] < 2:16 break17 worst = max(worst, math.exp(lm(w, labels=w).loss.item()))18 return worst1920def calibrate(benign_prompts: list[str], q: float = 0.999) -> float:21 scores = sorted(window_ppl(p) for p in benign_prompts)22 return scores[int(q * (len(scores) - 1))]2324def flag(prompt: str, threshold: float) -> bool:25 return window_ppl(prompt) > threshold # route to refusal or human reviewCheck yourself
A request that the model refuses in English is answered when written in base64. Which failure mode is this?
Check yourself
Why does a perplexity filter stop vanilla GCG but not PAIR?
Remember
- ✓Jailbreak: the user crafts input that defeats the model's own safety training; distinct from prompt injection.
- ✓Two failure modes (Wei et al. 2023): competing objectives (role-play, refusal suppression) and mismatched generalisation (encodings, low-resource languages).
- ✓GCG minimises with gradient-ranked token swaps; suffixes transfer across models.
- ✓Many-shot jailbreaks follow a power law in the number of shots; PAIR/TAP automate black-box attacks in tens of queries.
- ✓Layer defences (filters, safety training, output classifiers), measure ASR with a rubric judge on a fixed benchmark, and always report over-refusal.