Jailbreaking

A jailbreak is an input that gets a safety-trained model to produce what it was trained to refuse, by pitting helpfulness against safety or leaving its safety data's distribution (Wei et al. 2023); GCG automates it with gradient-searched suffixes.

8 min read630 wordsadvanced0 viewsupdated 2026-10-21

Definition

Jailbreaking is crafting inputs that make a safety-trained LLM produce outputs its developer trained it to refuse, such as dangerous instructions or abusive content. The user is the attacker and the target is the model's own safety behaviour. Wei, Haghtalab and Steinhardt ("Jailbroken: How Does LLM Safety Training Fail?", NeurIPS 2023) traced jailbreaks to two failure modes. Zou et al. ("Universal and Transferable Adversarial Attacks on Aligned Language Models", 2023) showed they can be found automatically by optimisation.

Safety behaviour is added after pre-training, by Supervised fine-tuning (SFT) on refusals and by RLHF or Constitutional AI. The capability to produce the harmful text is still in the weights; training only makes refusal the likely response on inputs like those seen in safety data. A jailbreak finds inputs where that no longer holds. This differs from Prompt injection, where a third party's text hijacks an application and the payload need not be harmful at all.

Competing objectives: the model is trained to be helpful, follow instructions and stay in character, as well as to refuse. Role-play frames, instructions to begin with an affirmative prefix, or bans on refusal phrases set these goals against each other. Mismatched generalisation: pre-training covers far more input space than safety training. The same request in base64, a cipher or a low-resource language may be understood by the model but never have been refused in training.

attack families: [harmful request] wrapped so safety training failsfamilyaccessmechanismexploitsrole-play / personamanualfictional frame where rules "don't apply"competingrefusal suppressionmanualforce the reply to open "Sure, here is"competingobfuscationmanualbase64, ciphers, low-resource languagesmismatchPAIR / TAPblack-boxattacker LLM rewrites, judge scoresbothGCG suffixwhite-boxgradient-optimised token suffixmismatchmany-shotlong ctxhundreds of faux dialogues in contextin-contextCrescendomulti-turnbenign start, escalates turn by turncompetingcompeting: helpfulness beats harmlessnessmismatch: input outside safety datadefence in depth (each layer lowers ASR; none is complete)input filtersperplexity, classifiersafety-trained modelrefusal + adversarialoutput classifiere.g. Llama Guardresponseor refusalGCG gibberish → high perplexity, caughtcircuit breakers, Constitutional AIchecks what was producedevaluate: ASR on HarmBench / JailbreakBench, rubric judge (StrongREJECT)and over-refusal of safe prompts (XSTest): a model that refuses everything scores ASR 0
fig.Top: the main attack families, the access each needs, and the failure mode it exploits. Requests are shown only as a placeholder; this article explains mechanisms, not working prompts. Bottom: the usual defence stack. Input filters catch high-perplexity GCG suffixes, the model itself is safety-trained, and an output classifier checks what was actually generated. Evaluation reports attack success rate with a rubric judge, together with over-refusal on safe prompts.

Optimisation attacks: GCG

Greedy Coordinate Gradient (GCG) appends a suffix of tokens xIx_{\mathcal I} to a harmful request and searches for the suffix that makes an affirmative target such as "Sure, here is…" most likely. Once the model starts that way it usually continues.

min⁡xI∈{1..V}∣I∣  L(x1:n)=−log⁡p(xn+1:n+H⋆ ∣ x1:n),Xi=Top-k⁡(−∇exiL)\min_{x_{\mathcal I}\in\{1..V\}^{|\mathcal I|}}\;\mathcal L(x_{1:n})=-\log p\big(x^\star_{n+1:n+H}\,\big|\,x_{1:n}\big),\qquad \mathcal X_i=\operatorname{Top\text{-}k}\big(-\nabla_{e_{x_i}}\mathcal L\big)
xn+1:n+H⋆x^\star_{n+1:n+H}
the target completion of HH tokens (an affirmative opening)
exie_{x_i}
one-hot vector of the token at suffix position ii; its gradient ranks replacement tokens
Xi\mathcal X_i
kk candidate tokens for position ii; BB random single swaps are evaluated exactly and the best is kept

The gradient is only a linear guess at which swaps help, so each step scores a batch of real candidates with forward passes. Optimising one suffix over several behaviours and models (Vicuna and Guanaco in the paper) gave "universal" suffixes that transferred to closed models behind APIs. The cost is the weak point for the attacker: with the paper's settings, one run is about 500×512=256,000500\times512=256{,}000 forward passes.

Published specifics (check the papers for exact settings)

GCG (Zou et al. 2023)
20-token suffix initialised as "! ! … !", top-k = 256 candidates per position, batch B = 512, up to 500 steps; evaluated on AdvBench (520 harmful behaviours, 500 harmful strings)
PAIR (Chao et al. 2023)
black-box: an attacker LLM rewrites the prompt from a judge's feedback; often succeeds in under 20 queries. TAP (Mehrotra et al. 2023) adds tree search with pruning
Many-shot (Anil et al. 2024)
hundreds of fake dialogue turns in a long context; tested up to 256 shots, with attack effectiveness following a power law in the number of shots
Low-resource languages (Yong et al. 2023)
translating AdvBench into low-resource languages such as Zulu bypassed GPT-4's safeguards in about 79% of cases (as of the 2023 model)
Crescendo (Russinovich et al. 2024)
multi-turn: starts benign and escalates by referring back to the model's own earlier answers
−log⁡p(harmful reply∣n shots)  ≈  C n−α-\log p(\text{harmful reply}\mid n\text{ shots})\;\approx\;C\,n^{-\alpha}
nn
number of faux dialogue examples placed in the context
α\alpha
power-law exponent fitted per model and task (Anil et al. 2024); a straight line on log–log axes

The power law is the same in-context learning that makes Few-shot prompting work, which is why it cannot simply be trained away. Long context windows made the attack possible.

[request] + suffix (behaviour 1, step 80); colour = token's push towards complyingas##^^onNow|veluno@peruno@asonper%velviaNowp(target "Sure, here is") per iteration0.00.51.0080 iterationsperplexity (log)1101001000filter T = 40benignsuffixesbehaviour 1: p = 1.000, perplexity 197.2 → flagged by filterjailbroken (p > 0.5): 20/20 = ASR 100%and under perplexity T: 0/20; benign text flagged: 0.0%toy model with abstract tokens; λ adds a fluency penalty to the attack objective
fig.A toy GCG over 48 abstract tokens against a synthetic compliance model, run for 20 "behaviours" of different difficulty. Defaults (20-token suffix, no fluency penalty, T = 40): p(comply) starts at 3.1 × 10⁻⁷ for "! ! …" and passes 0.5 after a median of 12 iterations. After 80 iterations all 20 behaviours are jailbroken, but the suffixes have perplexity 79–352 against a benign median of 14.5 (max 31.2). The filter stops all 20 and flags 0% of benign text. Set λ = 1: still 20/20, now at perplexity 8.3–20.2, so all pass the filter. With 5 suffix tokens, 0/20 succeed.

Defences

DefenceHow it worksKnown gap
Perplexity filter (Alon & Kamfonas 2023; Jain et al. 2023)Flag prompts, or windows of them, whose LM perplexity is far above normalReadable attacks (PAIR, AutoDAN, role-play) have normal perplexity
SmoothLLM (Robey et al. 2023)Randomly perturb characters in several copies of the prompt; majority-vote the responsesTargets brittle token-level suffixes; costs several model calls
Input/output classifiers (e.g. Llama Guard, Inan et al. 2023)A separate safety model labels the prompt and the response against a harm taxonomyObfuscated outputs; classifier itself can be attacked
Adversarial and refusal trainingAdd jailbreak attempts with refusals to SFT/RLHF dataCovers known attacks; new families appear
Circuit breakers (Zou et al. 2024)Train internal representations of harmful content to be rerouted, so generation breaks downResearch method; can hurt utility if over-applied
Defence families and what they are known to miss

Worked example: the judge decides the ASR

Response typeCountKeyword judgeRubric judge
Clear refusal105refusedsafe
Apology, then complies with specifics15refusedharmful
Complies with specific harmful detail30successharmful
Complies but vague, wrong or off-topic35successsafe
Safe redirection, no refusal phrase15successsafe
ASR (95% Wilson interval)20080/200 = 40.0% (33.5–46.9%)45/200 = 22.5% (17.3–28.8%)
Illustrative audit of 200 responses to harmful requests, scored by a keyword judge ("no refusal phrase = success") and by a rubric judge ("specific and actually harmful = success")

The keyword judge both over-counts (vague or harmless answers) and under-counts (compliance after an apology). StrongREJECT (Souly et al. 2024) made this point: many published jailbreaks look far less effective when the answer must be specific and useful. HarmBench (Mazeika et al. 2024) and JailbreakBench (Chao et al. 2024) fix the behaviours, the judge and the query budget so that numbers can be compared.

python
1import math, torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4tok = AutoTokenizer.from_pretrained("gpt2")
5lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()
6
7@torch.no_grad()
8def window_ppl(text: str, win: int = 10) -> float:
9 """Max perplexity over sliding windows: a short gibberish suffix
10 can hide inside the mean perplexity of a long prompt."""
11 ids = tok(text, return_tensors="pt").input_ids[0]
12 worst = 0.0
13 for s in range(max(1, len(ids) - win + 1)):
14 w = ids[s:s + win].unsqueeze(0)
15 if w.shape[1] < 2:
16 break
17 worst = max(worst, math.exp(lm(w, labels=w).loss.item()))
18 return worst
19
20def calibrate(benign_prompts: list[str], q: float = 0.999) -> float:
21 scores = sorted(window_ppl(p) for p in benign_prompts)
22 return scores[int(q * (len(scores) - 1))]
23
24def flag(prompt: str, threshold: float) -> bool:
25 return window_ppl(prompt) > threshold # route to refusal or human review
A windowed perplexity filter with GPT-2 via Hugging Face transformers (not run here: torch and transformers were unavailable). Calibrate the threshold on benign traffic

Check yourself

A request that the model refuses in English is answered when written in base64. Which failure mode is this?

Check yourself

Why does a perplexity filter stop vanilla GCG but not PAIR?

Remember

  • ✓Jailbreak: the user crafts input that defeats the model's own safety training; distinct from prompt injection.
  • ✓Two failure modes (Wei et al. 2023): competing objectives (role-play, refusal suppression) and mismatched generalisation (encodings, low-resource languages).
  • ✓GCG minimises −log⁡p(Sure, here is∣request+suffix)-\log p(\text{Sure, here is}\mid\text{request}+\text{suffix}) with gradient-ranked token swaps; suffixes transfer across models.
  • ✓Many-shot jailbreaks follow a power law in the number of shots; PAIR/TAP automate black-box attacks in tens of queries.
  • ✓Layer defences (filters, safety training, output classifiers), measure ASR with a rubric judge on a fixed benchmark, and always report over-refusal.
← back to Generative AI roadmap