Definition
Prefix tuning is the parameter-efficient method of Xiang Lisa Li and Percy Liang, "Prefix-Tuning: Optimizing Continuous Prompts for Generation" (Stanford, ACL 2021). The language model stays frozen. A short sequence of continuous, task-specific vectors is prepended at every layer: each layer gets its own learned keys and values for virtual positions, which all real tokens can attend to. For GPT-2 the sequence is . For an encoder–decoder such as BART it is . Only the prefix is trained, about 0.1% of the model's parameter count, and one frozen model serves many tasks by swapping prefixes.
In 2020, adapting a pretrained LM to a task meant full fine-tuning, with one full copy of the model per task, or adapters that insert small trainable layers into every block. GPT-3 had just shown a third option: write an instruction and a few examples in the prompt and change no weights at all. Prompts are discrete, though, and hand-written prompts are brittle. Li and Liang asked what happens if the "prompt" is optimised directly in continuous space, as vectors that need not correspond to any real word.
Optimising only the input embeddings of the virtual tokens is the idea later studied as Prompt tuning (Lester et al. 2021). In Li and Liang's experiments on GPT-2 it was not expressive enough. The ablation they call "embedding-only" was clearly worse than optimising activations at every layer. A virtual token's input embedding influences the model only through the frozen layers above it. A prefix at every layer can steer each layer's attention directly. That is the step from prompt tuning to prefix tuning.
What the prefix does inside attention
- the prefix positions; their activations are free parameters, not computed by the LM
- trainable, ; for a decoder this amounts to a learned key and value of size per layer and position
- the frozen LM parameters
- keys and values of the real tokens, computed by the frozen model exactly as before
In practice this is a learned KV cache. Hugging Face implementations feed the prefix to the model as past_key_values of length , the same structure the KV cache uses for generated tokens. The real tokens' own keys and values are untouched. What changes is what their queries can attend to. He, Zhou, Ma, Berg-Kirkpatrick and Neubig (ICLR 2022) made this exact by splitting the softmax over the two groups of positions:
- the real context; the first term is the frozen model's own attention, unchanged
- attention over the prefix alone: a learned vector that depends on only through the query
- the share of attention mass on the prefix. It grows with and with how well matches the queries
So prefix tuning is a gated additive edit of every attention head's output. The head interpolates between what the frozen model would have said and a learned, query-dependent vector. The real tokens' attention weights are all rescaled by the same factor , so their ranking is unchanged. This also places prefix tuning next to adapters and LoRA in He et al.'s unified view: each adds a learned modification to some hidden representation, and they differ in where it is added and how it is gated.
Reparameterisation
Optimising directly was unstable in the paper: it was sensitive to the learning rate and initialisation, and performance dropped slightly. Li and Liang therefore reparameterise during training, , where has the same number of rows but fewer columns, and a feed-forward network expands each row to the full per-layer size. Once training ends, the MLP is evaluated once, is saved, and and the MLP are thrown away. The MLP adds many training-time parameters. In the Hugging Face PEFT version with hidden size 512 on GPT-2 medium, it is about 25.7M, still only about 7% of the model. The stored artefact is the small prefix alone.
- one key and one value of size for each of layers
- prefix length; 10 worked best for table-to-text and 200 for summarization in the paper
- parameters of GPT-2 medium (24 layers, d = 1,024)
Two more findings from the paper's ablations matter in practice. First, longer is not always better. Performance rose with prefix length up to a threshold, 200 for summarization and 10 for table-to-text, then fell slightly. Second, initialisation matters in low-data settings. Initialising the prefix with the activations of real words, such as "summarize" or "table-to-text", was better than random initialisation when only a few hundred examples were available. Putting the trainable activations between and ("infix", ) was slightly worse than a prefix. The authors explain this by noting that a prefix can influence the activations of both and , while an infix can only influence .
Setup in the paper
- Authors
- Xiang Lisa Li, Percy Liang (Stanford); arXiv January 2021, ACL 2021
- Models
- GPT-2 medium and GPT-2 large for table-to-text; BART-large for summarization
- Tasks
- table-to-text on E2E, WebNLG and DART; abstractive summarization on XSUM
- Trainable share
- about 0.1% of the parameters; GPT-2 medium with stores numbers
- Prefix length
- best around 10 for table-to-text and 200 for summarization; longer was slightly worse
- Results
- comparable to full fine-tuning on table-to-text with full data; better in low-data settings and on topics unseen in training; slightly below fine-tuning on XSUM; better than adapter tuning at a similar parameter budget
- Ablations
- embedding-only (prompt-tuning-like) clearly worse; infix slightly worse than prefix; direct optimisation of P unstable without the MLP
Worked example: one query, one prefix position
Take , a query , two real tokens with keys and and values and , and one learned prefix position with and . Logits are .
| position | key | logit q·k/√2 | exp | weight | value |
|---|---|---|---|---|---|
| prefix | (2, 0) | 1.4142 | 4.1133 | 0.5760 | (−1, 2) |
| token 1 | (1, 0) | 0.7071 | 2.0281 | 0.2840 | (1, 0) |
| token 2 | (0, 1) | 0 | 1 | 0.1400 | (0, 1) |
| sum | 7.1414 | 1 |
Assembling the output
- 1Without the prefix, the head attends to token 1 and to token 2, so .
- 2With it, . The real weights become and , which is the base weights times . Their ratio is still .
- 3Output: .
- 4Check the decomposition: . With a single prefix position, is just .
- 5The head output moved by . Training can change two things: sets the direction of the edit, and sets how much of it each query receives.
1import numpy as np23def attn(q, K, V):4 s = K @ q / np.sqrt(len(q))5 a = np.exp(s - s.max()); a /= a.sum()6 return a, a @ V78rng = np.random.default_rng(0)9d, T, p = 64, 12, 1010q = rng.normal(size=d) # query of the last real token (frozen W_q)11K, V = rng.normal(size=(T, d)), rng.normal(size=(T, d)) # real keys/values (frozen W_k, W_v)12Pk, Pv = rng.normal(size=(p, d)), rng.normal(size=(p, d)) # the prefix: the only trained numbers1314a, out = attn(q, np.vstack([Pk, K]), np.vstack([Pv, V])) # what the model computes15lam = a[:p].sum() # share of attention on the prefix16_, base = attn(q, K, V) # the frozen model without a prefix17_, pre = attn(q, Pk, Pv) # attention over the prefix alone18print(f"lambda = {lam:.4f}") # lambda = 0.565219print(np.allclose(out, (1 - lam) * base + lam * pre)) # True20print(np.allclose(a[p:] / (1 - lam), attn(q, K, V)[0])) # True: rescaled, not reordered2122# ---- with Hugging Face PEFT ----23from transformers import AutoModelForCausalLM24from peft import PrefixTuningConfig, get_peft_model2526model = AutoModelForCausalLM.from_pretrained("gpt2-medium") # 24 layers, d = 1024, frozen27cfg = PrefixTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=10,28 prefix_projection=True, encoder_hidden_size=512) # Li & Liang's MLP29model = get_peft_model(model, cfg)30model.print_trainable_parameters()31# training: 10*1024 (P') + 1024*512+512 + 512*49152+49152 = 25,750,016 trainable32# stored after training: 10 * (2 * 24 * 1024) = 491,520 numbers, passed as past_key_valuesUse when
- +Stores about 0.1% of the parameters per task; the LM is shared and untouched
- +Strong in low-data settings and on topics unseen during training (table-to-text)
- +Different prefixes can be mixed in one batch through the same frozen model
- +Far more expressive than input-only prompt tuning at small and medium model sizes
Avoid when
- −Extra attention cost at every layer and fewer free context positions, permanently
- −Training is unstable without the MLP reparameterisation and sensitive to initialisation
- −Performance is non-monotonic in prefix length, so p must be tuned
- −Generally weaker than LoRA on large models and harder tasks; cannot be merged away
Check yourself
A prefix is added to one attention head. How do the attention weights on the real tokens change?
Check yourself
How many numbers does a length-10 prefix store for GPT-2 medium (24 layers, d = 1,024)?
Remember
- ✓Prefix tuning learns key/value vectors at every layer, and , and keeps the LM frozen. It stores numbers, about 0.1% of GPT-2.
- ✓Each head becomes , a gated, query-dependent edit (He et al. 2022).
- ✓Training goes through an MLP reparameterisation of a smaller matrix for stability. Afterwards only the prefix is kept.
- ✓In the paper it was comparable to fine-tuning on table-to-text, better with little data, slightly worse on XSUM. Best lengths were about 10 for table-to-text and 200 for summarization.
- ✓Prompt tuning is the input-layer-only special case. P-Tuning v2 is deep prefix-style prompting for NLU and matches fine-tuning from 300M to 10B.