Prefix tuning

Prefix tuning (Li & Liang, 2021) freezes the LM and learns p key/value vectors at every layer that all real tokens can attend to. Trained through an MLP reparameterisation, it stores about 0.1% of GPT-2's parameters per task (2·L·p·d) and matches fine-tuning on table-to-text.

14 min read1092 wordsadvanced0 viewsupdated 2026-10-13

Definition

Prefix tuning is the parameter-efficient method of Xiang Lisa Li and Percy Liang, "Prefix-Tuning: Optimizing Continuous Prompts for Generation" (Stanford, ACL 2021). The language model stays frozen. A short sequence of pp continuous, task-specific vectors is prepended at every layer: each layer gets its own learned keys and values for pp virtual positions, which all real tokens can attend to. For GPT-2 the sequence is z=[PREFIX;x;y]z=[\text{PREFIX};x;y]. For an encoder–decoder such as BART it is z=[PREFIX;x;PREFIX′;y]z=[\text{PREFIX};x;\text{PREFIX}';y]. Only the prefix is trained, about 0.1% of the model's parameter count, and one frozen model serves many tasks by swapping prefixes.

In 2020, adapting a pretrained LM to a task meant full fine-tuning, with one full copy of the model per task, or adapters that insert small trainable layers into every block. GPT-3 had just shown a third option: write an instruction and a few examples in the prompt and change no weights at all. Prompts are discrete, though, and hand-written prompts are brittle. Li and Liang asked what happens if the "prompt" is optimised directly in continuous space, as vectors that need not correspond to any real word.

Optimising only the input embeddings of the virtual tokens is the idea later studied as Prompt tuning (Lester et al. 2021). In Li and Liang's experiments on GPT-2 it was not expressive enough. The ablation they call "embedding-only" was clearly worse than optimising activations at every layer. A virtual token's input embedding influences the model only through the frozen layers above it. A prefix at every layer can steer each layer's attention directly. That is the step from prompt tuning to prefix tuning.

What the prefix does inside attention

hi(ℓ)={Pθ(ℓ)[i,:]i∈PidxLMϕ(zi, h<i(≤ℓ))otherwise⟹K(ℓ)=[Pk(ℓ)XWk(ℓ)],  V(ℓ)=[Pv(ℓ)XWv(ℓ)]h_i^{(\ell)}=\begin{cases}P_\theta^{(\ell)}[i,:] & i\in\mathsf P_{\text{idx}}\\[2pt] \mathrm{LM}_\phi\big(z_i,\,h^{(\le\ell)}_{<i}\big) & \text{otherwise}\end{cases}\qquad\Longrightarrow\qquad K^{(\ell)}=\begin{bmatrix}P^{(\ell)}_k\\ XW_k^{(\ell)}\end{bmatrix},\;V^{(\ell)}=\begin{bmatrix}P^{(\ell)}_v\\ XW_v^{(\ell)}\end{bmatrix}
Pidx\mathsf P_{\text{idx}}
the pp prefix positions; their activations are free parameters, not computed by the LM
PθP_\theta
trainable, p×dim⁡(h)p\times\dim(h); for a decoder this amounts to a learned key and value of size dd per layer and position
ϕ\phi
the frozen LM parameters
XWk, XWvXW_k,\ XW_v
keys and values of the real tokens, computed by the frozen model exactly as before

In practice this is a learned KV cache. Hugging Face implementations feed the prefix to the model as past_key_values of length pp, the same structure the KV cache uses for generated tokens. The real tokens' own keys and values are untouched. What changes is what their queries can attend to. He, Zhou, Ma, Berg-Kirkpatrick and Neubig (ICLR 2022) made this exact by splitting the softmax over the two groups of positions:

head(x)=(1−λ(x)) Attn(xWq, CWk, CWv)+λ(x) Attn(xWq, Pk, Pv),λ(x)=∑iexWqPk,i⊤/d∑iexWqPk,i⊤/d+∑jexWq(CWk)j⊤/d\mathrm{head}(x)=(1-\lambda(x))\,\mathrm{Attn}\big(xW_q,\,CW_k,\,CW_v\big)+\lambda(x)\,\mathrm{Attn}\big(xW_q,\,P_k,\,P_v\big),\qquad \lambda(x)=\frac{\sum_i e^{xW_qP_{k,i}^\top/\sqrt{d}}}{\sum_i e^{xW_qP_{k,i}^\top/\sqrt{d}}+\sum_j e^{xW_q(CW_k)_j^\top/\sqrt{d}}}
CC
the real context; the first term is the frozen model's own attention, unchanged
Attn(xWq,Pk,Pv)\mathrm{Attn}(xW_q,P_k,P_v)
attention over the prefix alone: a learned vector that depends on xx only through the query
λ(x)\lambda(x)
the share of attention mass on the prefix. It grows with pp and with how well PkP_k matches the queries

So prefix tuning is a gated additive edit of every attention head's output. The head interpolates between what the frozen model would have said and a learned, query-dependent vector. The real tokens' attention weights are all rescaled by the same factor 1−λ1-\lambda, so their ranking is unchanged. This also places prefix tuning next to adapters and LoRA in He et al.'s unified view: each adds a learned modification to some hidden representation, and they differ in where it is added and how it is gated.

frozen LM (GPT-2 medium, d = 1,024); only the violet blocks are trainednext-token loss on y onlyP_k, P_vp × d eachlayer L = 24: attention + MLPK = [P_k; XW_k], V = [P_v; XW_v]P_k, P_vp × d eachlayer 2: attention + MLPK = [P_k; XW_k], V = [P_v; XW_v]P_k, P_vp × d eachlayer 1: attention + MLPK = [P_k; XW_k], V = [P_v; XW_v]⋮⋮sequence z = [PREFIX; x; y]nameAromitypecaféAromiisano tokens: freeactivationsx: table (input)y: textevery real position can attend to the prefix at every layer;the prefix attends to nothing and is not computed by the LMtraining onlyP′p × k, smallMLPk → 2·L·dP_θ = MLP(P′)p × 2·L·d, split per layerstabilises training; then theMLP is dropped, only P_θ is keptstored per task, p = 10prompt tuningp·d = 10,240prefix tuning2·L·p·d = 491,5200.14% of 354.8Mfull fine-tuning354,823,168P-Tuning v2: the same per-layerprompts, applied to NLU with aclassification head
fig.Prefix tuning on a frozen GPT-2 medium (24 layers, d = 1,024). Every layer gets its own learned prefix keys and values (violet), concatenated in front of the keys and values the frozen model computes from the real tokens: the table x and the text y. The prefix positions have no tokens and attend to nothing; their activations are parameters. The loss is next-token cross-entropy on y only. During training, the prefix is produced by an MLP from a smaller matrix P′ for stability. Afterwards only P_θ is stored: 2·L·p·d = 491,520 numbers for p = 10, 0.14% of the model, against 10,240 for input-only prompt tuning.

Reparameterisation

Optimising PθP_\theta directly was unstable in the paper: it was sensitive to the learning rate and initialisation, and performance dropped slightly. Li and Liang therefore reparameterise during training, Pθ[i,:]=MLPθ(Pθ′[i,:])P_\theta[i,:]=\mathrm{MLP}_\theta(P'_\theta[i,:]), where Pθ′P'_\theta has the same number of rows but fewer columns, and a feed-forward network expands each row to the full per-layer size. Once training ends, the MLP is evaluated once, PθP_\theta is saved, and P′P' and the MLP are thrown away. The MLP adds many training-time parameters. In the Hugging Face PEFT version with hidden size 512 on GPT-2 medium, it is about 25.7M, still only about 7% of the model. The stored artefact is the small prefix alone.

Pθ=MLPθ(Pθ′)∈Rp×2Ld,#stored=2 L p d,2⋅24⋅10⋅1024354,823,168=491,520354,823,168=0.139%P_\theta=\mathrm{MLP}_\theta(P'_\theta)\in\mathbb R^{p\times 2Ld},\qquad \#\text{stored}=2\,L\,p\,d,\qquad \frac{2\cdot24\cdot10\cdot1024}{354{,}823{,}168}=\frac{491{,}520}{354{,}823{,}168}=0.139\%
2Ld2Ld
one key and one value of size dd for each of LL layers
pp
prefix length; 10 worked best for table-to-text and 200 for summarization in the paper
354,823,168354,823,168
parameters of GPT-2 medium (24 layers, d = 1,024)

Two more findings from the paper's ablations matter in practice. First, longer is not always better. Performance rose with prefix length up to a threshold, 200 for summarization and 10 for table-to-text, then fell slightly. Second, initialisation matters in low-data settings. Initialising the prefix with the activations of real words, such as "summarize" or "table-to-text", was better than random initialisation when only a few hundred examples were available. Putting the trainable activations between xx and yy ("infix", [x;INFIX;y][x;\text{INFIX};y]) was slightly worse than a prefix. The authors explain this by noting that a prefix can influence the activations of both xx and yy, while an infix can only influence yy.

Setup in the paper

Authors
Xiang Lisa Li, Percy Liang (Stanford); arXiv January 2021, ACL 2021
Models
GPT-2 medium and GPT-2 large for table-to-text; BART-large for summarization
Tasks
table-to-text on E2E, WebNLG and DART; abstractive summarization on XSUM
Trainable share
about 0.1% of the parameters; GPT-2 medium with p=10p=10 stores 2⋅24⋅10⋅1024=491,5202\cdot24\cdot10\cdot1024=491{,}520 numbers
Prefix length
best around 10 for table-to-text and 200 for summarization; longer was slightly worse
Results
comparable to full fine-tuning on table-to-text with full data; better in low-data settings and on topics unseen in training; slightly below fine-tuning on XSUM; better than adapter tuning at a similar parameter budget
Ablations
embedding-only (prompt-tuning-like) clearly worse; infix slightly worse than prefix; direct optimisation of P unstable without the MLP
query = last real token · out = (1 − λ)·Attn(q, K, V) + λ·Attn(q, P_k, P_v)prefix P_k, P_v (10 of 16 positions, learned)real tokens (frozen K, V)1471013160.108name0.075Aromi0.010type0.072coffee0.031shop0.032areaλ = 0.6721 to the prefix · real tokens share 1 − λ = 0.3279, ratios unchangedhead output, 8 dims: without prefix · prefix only · with prefix○ task direction μ shared by the prefix values · axis ±1.5‖out − base‖ = λ·‖pre − base‖2.2141cos(output, μ) without prefix-0.5666cos(output, μ) with prefix0.9236GPT-2 medium (L = 24, d = 1,024), p = 10: 2·L·p·d = 491,520 = 0.1385% of 354.82Mtraining with the MLP reparameterisation (hidden 512): 25.75M params, MLP then discardedfull fine-tuning would store all 354.82M per task: 722× morethe prefix adds keys and values; it never changes the real tokens' K and V
model
fig.One attention head (d_h = 8) of a frozen model. The last real token's query attends over p learned prefix positions (violet) and six real tokens (blue), and the output is split exactly as He et al. describe: out = (1 − λ)·Attn(q, K, V) + λ·Attn(q, P_k, P_v). The prefix values share a task direction μ (orange circles). Without a prefix (p = 0) the real weights are 0.330, 0.228, 0.031, 0.219, 0.094 and 0.098, and the output points away from μ (cos −0.5666). At the default p = 10 and key gain 1, the prefix takes λ = 0.6721 of the attention. Every real weight is multiplied by the same 0.3279 ("name" drops to 0.108, "Aromi" to 0.075), the output moves by 2.2141 and turns towards μ (cos 0.9236). λ grows with length even when the prefix keys carry no signal: at gain 0, ten prefix positions with logit 0 still take 0.5800. With aligned keys (gain 3) the prefix takes 0.8470 at p = 10 and 0.8978 at p = 16. One prefix position at gain 1 gives λ = 0.1435. The bottom rows count real storage: for GPT-2 medium, p = 10 stores 491,520 numbers (0.1385%, 722× less than a full copy), while training through the MLP reparameterisation uses 25.75M. GPT-2 large at p = 10 stores 921,600 (0.1191%).

Worked example: one query, one prefix position

Take d=2d=2, a query q=(1,0)q=(1,0), two real tokens with keys k1=(1,0)k_1=(1,0) and k2=(0,1)k_2=(0,1) and values v1=(1,0)v_1=(1,0) and v2=(0,1)v_2=(0,1), and one learned prefix position with Pk=(2,0)P_k=(2,0) and Pv=(−1,2)P_v=(-1,2). Logits are q⋅k/2q\cdot k/\sqrt2.

positionkeylogit q·k/√2expweightvalue
prefix(2, 0)1.41424.11330.5760(−1, 2)
token 1(1, 0)0.70712.02810.2840(1, 0)
token 2(0, 1)010.1400(0, 1)
sum7.14141
Attention with the prefix (computed in numpy)

Assembling the output

  1. 1Without the prefix, the head attends 2.0281/3.0281=0.66982.0281/3.0281=0.6698 to token 1 and 0.33020.3302 to token 2, so base=(0.6698,0.3302)\mathrm{base}=(0.6698,0.3302).
  2. 2With it, λ=4.1133/7.1414=0.5760\lambda=4.1133/7.1414=0.5760. The real weights become 0.28400.2840 and 0.14000.1400, which is the base weights times 1−λ=0.42401-\lambda=0.4240. Their ratio is still 2.0281:12.0281:1.
  3. 3Output: 0.576⋅(−1,2)+0.284⋅(1,0)+0.140⋅(0,1)=(−0.292, 1.292)0.576\cdot(-1,2)+0.284\cdot(1,0)+0.140\cdot(0,1)=(-0.292,\,1.292).
  4. 4Check the decomposition: (1−λ)⋅base+λ⋅Pv=0.424⋅(0.6698,0.3302)+0.576⋅(−1,2)=(−0.292,1.292)(1-\lambda)\cdot\mathrm{base}+\lambda\cdot P_v=0.424\cdot(0.6698,0.3302)+0.576\cdot(-1,2)=(-0.292,1.292). With a single prefix position, Attn(q,Pk,Pv)\mathrm{Attn}(q,P_k,P_v) is just PvP_v.
  5. 5The head output moved by λ ∥Pv−base∥=0.576⋅2.3615=1.3601\lambda\,\|P_v-\mathrm{base}\|=0.576\cdot2.3615=1.3601. Training can change two things: PvP_v sets the direction of the edit, and PkP_k sets how much of it each query receives.
python
1import numpy as np
2
3def attn(q, K, V):
4 s = K @ q / np.sqrt(len(q))
5 a = np.exp(s - s.max()); a /= a.sum()
6 return a, a @ V
7
8rng = np.random.default_rng(0)
9d, T, p = 64, 12, 10
10q = rng.normal(size=d) # query of the last real token (frozen W_q)
11K, V = rng.normal(size=(T, d)), rng.normal(size=(T, d)) # real keys/values (frozen W_k, W_v)
12Pk, Pv = rng.normal(size=(p, d)), rng.normal(size=(p, d)) # the prefix: the only trained numbers
13
14a, out = attn(q, np.vstack([Pk, K]), np.vstack([Pv, V])) # what the model computes
15lam = a[:p].sum() # share of attention on the prefix
16_, base = attn(q, K, V) # the frozen model without a prefix
17_, pre = attn(q, Pk, Pv) # attention over the prefix alone
18print(f"lambda = {lam:.4f}") # lambda = 0.5652
19print(np.allclose(out, (1 - lam) * base + lam * pre)) # True
20print(np.allclose(a[p:] / (1 - lam), attn(q, K, V)[0])) # True: rescaled, not reordered
21
22# ---- with Hugging Face PEFT ----
23from transformers import AutoModelForCausalLM
24from peft import PrefixTuningConfig, get_peft_model
25
26model = AutoModelForCausalLM.from_pretrained("gpt2-medium") # 24 layers, d = 1024, frozen
27cfg = PrefixTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=10,
28 prefix_projection=True, encoder_hidden_size=512) # Li & Liang's MLP
29model = get_peft_model(model, cfg)
30model.print_trainable_parameters()
31# training: 10*1024 (P') + 1024*512+512 + 512*49152+49152 = 25,750,016 trainable
32# stored after training: 10 * (2 * 24 * 1024) = 491,520 numbers, passed as past_key_values
The prefix as extra keys and values, checking He et al.'s decomposition in numpy (run; outputs in comments), then prefix tuning GPT-2 medium with PEFT (not run; counts computed by hand)

Use when

  • +Stores about 0.1% of the parameters per task; the LM is shared and untouched
  • +Strong in low-data settings and on topics unseen during training (table-to-text)
  • +Different prefixes can be mixed in one batch through the same frozen model
  • +Far more expressive than input-only prompt tuning at small and medium model sizes

Avoid when

  • −Extra attention cost at every layer and fewer free context positions, permanently
  • −Training is unstable without the MLP reparameterisation and sensitive to initialisation
  • −Performance is non-monotonic in prefix length, so p must be tuned
  • −Generally weaker than LoRA on large models and harder tasks; cannot be merged away

Check yourself

A prefix is added to one attention head. How do the attention weights on the real tokens change?

Check yourself

How many numbers does a length-10 prefix store for GPT-2 medium (24 layers, d = 1,024)?

Remember

  • ✓Prefix tuning learns pp key/value vectors at every layer, K=[Pk;XWk]K=[P_k;XW_k] and V=[Pv;XWv]V=[P_v;XW_v], and keeps the LM frozen. It stores 2Lpd2Lpd numbers, about 0.1% of GPT-2.
  • ✓Each head becomes (1−λ) Attn(q,K,V)+λ Attn(q,Pk,Pv)(1-\lambda)\,\mathrm{Attn}(q,K,V)+\lambda\,\mathrm{Attn}(q,P_k,P_v), a gated, query-dependent edit (He et al. 2022).
  • ✓Training goes through an MLP reparameterisation of a smaller matrix for stability. Afterwards only the prefix is kept.
  • ✓In the paper it was comparable to fine-tuning on table-to-text, better with little data, slightly worse on XSUM. Best lengths were about 10 for table-to-text and 200 for summarization.
  • ✓Prompt tuning is the input-layer-only special case. P-Tuning v2 is deep prefix-style prompting for NLU and matches fine-tuning from 300M to 10B.
← back to Deep Learning roadmap