# Prefix tuning

> Prefix tuning (Li & Liang, 2021) freezes the LM and learns p key/value vectors at every layer that all real tokens can attend to. Trained through an MLP reparameterisation, it stores about 0.1% of GPT-2's parameters per task (2·L·p·d) and matches fine-tuning on table-to-text.

Source: https://www.mlfoundry.in/learn/prefix-tuning  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 14 min read · updated 2026-10-13

**Prefix tuning:** is the parameter-efficient method of Xiang Lisa Li and Percy Liang, "Prefix-Tuning: Optimizing Continuous Prompts for Generation" (Stanford, ACL 2021). The language model stays frozen. A short sequence of $p$ **continuous, task-specific vectors** is prepended at *every* layer: each layer gets its own learned keys and values for $p$ virtual positions, which all real tokens can attend to. For GPT-2 the sequence is $z=[\text{PREFIX};x;y]$. For an encoder–decoder such as BART it is $z=[\text{PREFIX};x;\text{PREFIX}';y]$. Only the prefix is trained, about 0.1% of the model's parameter count, and one frozen model serves many tasks by swapping prefixes.

In 2020, adapting a pretrained LM to a task meant **full fine-tuning**, with one full copy of the model per task, or **adapters** that insert small trainable layers into every block. GPT-3 had just shown a third option: write an instruction and a few examples in the prompt and change no weights at all. Prompts are discrete, though, and hand-written prompts are brittle. Li and Liang asked what happens if the "prompt" is optimised directly in continuous space, as vectors that need not correspond to any real word.

Optimising only the input embeddings of the virtual tokens is the idea later studied as **Prompt tuning** (Lester et al. 2021). In Li and Liang's experiments on GPT-2 it was not expressive enough. The ablation they call "embedding-only" was clearly worse than optimising activations at every layer. A virtual token's input embedding influences the model only through the frozen layers above it. A prefix at every layer can steer each layer's attention directly. That is the step from prompt tuning to prefix tuning.

## What the prefix does inside attention

$$
h_i^{(\ell)}=\begin{cases}P_\theta^{(\ell)}[i,:] & i\in\mathsf P_{\text{idx}}\\[2pt] \mathrm{LM}_\phi\big(z_i,\,h^{(\le\ell)}_{<i}\big) & \text{otherwise}\end{cases}\qquad\Longrightarrow\qquad K^{(\ell)}=\begin{bmatrix}P^{(\ell)}_k\\ XW_k^{(\ell)}\end{bmatrix},\;V^{(\ell)}=\begin{bmatrix}P^{(\ell)}_v\\ XW_v^{(\ell)}\end{bmatrix}
$$

- $\mathsf P_{\text{idx}}$: the $p$ prefix positions; their activations are free parameters, not computed by the LM
- $P_\theta$: trainable, $p\times\dim(h)$; for a decoder this amounts to a learned key and value of size $d$ per layer and position
- $\phi$: the frozen LM parameters
- $XW_k,\ XW_v$: keys and values of the real tokens, computed by the frozen model exactly as before

In practice this is a **learned KV cache**. Hugging Face implementations feed the prefix to the model as `past_key_values` of length $p$, the same structure the **KV cache** uses for generated tokens. The real tokens' own keys and values are untouched. What changes is what their queries can attend to. He, Zhou, Ma, Berg-Kirkpatrick and Neubig (ICLR 2022) made this exact by splitting the softmax over the two groups of positions:

$$
\mathrm{head}(x)=(1-\lambda(x))\,\mathrm{Attn}\big(xW_q,\,CW_k,\,CW_v\big)+\lambda(x)\,\mathrm{Attn}\big(xW_q,\,P_k,\,P_v\big),\qquad \lambda(x)=\frac{\sum_i e^{xW_qP_{k,i}^\top/\sqrt{d}}}{\sum_i e^{xW_qP_{k,i}^\top/\sqrt{d}}+\sum_j e^{xW_q(CW_k)_j^\top/\sqrt{d}}}
$$

- $C$: the real context; the first term is the frozen model's own attention, unchanged
- $\mathrm{Attn}(xW_q,P_k,P_v)$: attention over the prefix alone: a learned vector that depends on $x$ only through the query
- $\lambda(x)$: the share of attention mass on the prefix. It grows with $p$ and with how well $P_k$ matches the queries

So prefix tuning is a **gated additive edit** of every attention head's output. The head interpolates between what the frozen model would have said and a learned, query-dependent vector. The real tokens' attention weights are all rescaled by the same factor $1-\lambda$, so their ranking is unchanged. This also places prefix tuning next to adapters and **LoRA** in He et al.'s unified view: each adds a learned modification to some hidden representation, and they differ in where it is added and how it is gated.

> Figure: Prefix tuning on a frozen GPT-2 medium (24 layers, d = 1,024). Every layer gets its own learned prefix keys and values (violet), concatenated in front of the keys and values the frozen model computes from the real tokens: the table x and the text y. The prefix positions have no tokens and attend to nothing; their activations are parameters. The loss is next-token cross-entropy on y only. During training, the prefix is produced by an MLP from a smaller matrix P′ for stability. Afterwards only P_θ is stored: 2·L·p·d = 491,520 numbers for p = 10, 0.14% of the model, against 10,240 for input-only prompt tuning.

## Reparameterisation

Optimising $P_\theta$ directly was unstable in the paper: it was sensitive to the learning rate and initialisation, and performance dropped slightly. Li and Liang therefore reparameterise during training, $P_\theta[i,:]=\mathrm{MLP}_\theta(P'_\theta[i,:])$, where $P'_\theta$ has the same number of rows but fewer columns, and a feed-forward network expands each row to the full per-layer size. Once training ends, the MLP is evaluated once, $P_\theta$ is saved, and $P'$ and the MLP are thrown away. The MLP adds many training-time parameters. In the Hugging Face PEFT version with hidden size 512 on GPT-2 medium, it is about 25.7M, still only about 7% of the model. The stored artefact is the small prefix alone.

$$
P_\theta=\mathrm{MLP}_\theta(P'_\theta)\in\mathbb R^{p\times 2Ld},\qquad \#\text{stored}=2\,L\,p\,d,\qquad \frac{2\cdot24\cdot10\cdot1024}{354{,}823{,}168}=\frac{491{,}520}{354{,}823{,}168}=0.139\%
$$

- $2Ld$: one key and one value of size $d$ for each of $L$ layers
- $p$: prefix length; 10 worked best for table-to-text and 200 for summarization in the paper
- $354,823,168$: parameters of GPT-2 medium (24 layers, d = 1,024)

Two more findings from the paper's ablations matter in practice. First, **longer is not always better**. Performance rose with prefix length up to a threshold, 200 for summarization and 10 for table-to-text, then fell slightly. Second, **initialisation matters in low-data settings**. Initialising the prefix with the activations of real words, such as "summarize" or "table-to-text", was better than random initialisation when only a few hundred examples were available. Putting the trainable activations between $x$ and $y$ ("infix", $[x;\text{INFIX};y]$) was slightly worse than a prefix. The authors explain this by noting that a prefix can influence the activations of both $x$ and $y$, while an infix can only influence $y$.

**Setup in the paper**

- **Authors:** Xiang Lisa Li, Percy Liang (Stanford); arXiv January 2021, ACL 2021
- **Models:** GPT-2 medium and GPT-2 large for table-to-text; BART-large for summarization
- **Tasks:** table-to-text on E2E, WebNLG and DART; abstractive summarization on XSUM
- **Trainable share:** about 0.1% of the parameters; GPT-2 medium with $p=10$ stores $2\cdot24\cdot10\cdot1024=491{,}520$ numbers
- **Prefix length:** best around 10 for table-to-text and 200 for summarization; longer was slightly worse
- **Results:** comparable to full fine-tuning on table-to-text with full data; better in low-data settings and on topics unseen in training; slightly below fine-tuning on XSUM; better than adapter tuning at a similar parameter budget
- **Ablations:** embedding-only (prompt-tuning-like) clearly worse; infix slightly worse than prefix; direct optimisation of P unstable without the MLP

> Figure: One attention head (d_h = 8) of a frozen model. The last real token's query attends over p learned prefix positions (violet) and six real tokens (blue), and the output is split exactly as He et al. describe: out = (1 − λ)·Attn(q, K, V) + λ·Attn(q, P_k, P_v). The prefix values share a task direction μ (orange circles). Without a prefix (p = 0) the real weights are 0.330, 0.228, 0.031, 0.219, 0.094 and 0.098, and the output points away from μ (cos −0.5666). At the default p = 10 and key gain 1, the prefix takes λ = 0.6721 of the attention. Every real weight is multiplied by the same 0.3279 ("name" drops to 0.108, "Aromi" to 0.075), the output moves by 2.2141 and turns towards μ (cos 0.9236). λ grows with length even when the prefix keys carry no signal: at gain 0, ten prefix positions with logit 0 still take 0.5800. With aligned keys (gain 3) the prefix takes 0.8470 at p = 10 and 0.8978 at p = 16. One prefix position at gain 1 gives λ = 0.1435. The bottom rows count real storage: for GPT-2 medium, p = 10 stores 491,520 numbers (0.1385%, 722× less than a full copy), while training through the MLP reparameterisation uses 25.75M. GPT-2 large at p = 10 stores 921,600 (0.1191%).

## Worked example: one query, one prefix position

Take $d=2$, a query $q=(1,0)$, two real tokens with keys $k_1=(1,0)$ and $k_2=(0,1)$ and values $v_1=(1,0)$ and $v_2=(0,1)$, and one learned prefix position with $P_k=(2,0)$ and $P_v=(-1,2)$. Logits are $q\cdot k/\sqrt2$.

Attention with the prefix (computed in numpy)

| position | key | logit q·k/√2 | exp | weight | value |
| --- | --- | --- | --- | --- | --- |
| prefix | (2, 0) | 1.4142 | 4.1133 | 0.5760 | (−1, 2) |
| token 1 | (1, 0) | 0.7071 | 2.0281 | 0.2840 | (1, 0) |
| token 2 | (0, 1) | 0 | 1 | 0.1400 | (0, 1) |
| sum |  |  | 7.1414 | 1 |  |

**Assembling the output**

1. Without the prefix, the head attends $2.0281/3.0281=0.6698$ to token 1 and $0.3302$ to token 2, so $\mathrm{base}=(0.6698,0.3302)$.
2. With it, $\lambda=4.1133/7.1414=0.5760$. The real weights become $0.2840$ and $0.1400$, which is the base weights times $1-\lambda=0.4240$. Their ratio is still $2.0281:1$.
3. Output: $0.576\cdot(-1,2)+0.284\cdot(1,0)+0.140\cdot(0,1)=(-0.292,\,1.292)$.
4. Check the decomposition: $(1-\lambda)\cdot\mathrm{base}+\lambda\cdot P_v=0.424\cdot(0.6698,0.3302)+0.576\cdot(-1,2)=(-0.292,1.292)$. With a single prefix position, $\mathrm{Attn}(q,P_k,P_v)$ is just $P_v$.
5. The head output moved by $\lambda\,\|P_v-\mathrm{base}\|=0.576\cdot2.3615=1.3601$. Training can change two things: $P_v$ sets the direction of the edit, and $P_k$ sets how much of it each query receives.

```python
import numpy as np

def attn(q, K, V):
    s = K @ q / np.sqrt(len(q))
    a = np.exp(s - s.max()); a /= a.sum()
    return a, a @ V

rng = np.random.default_rng(0)
d, T, p = 64, 12, 10
q = rng.normal(size=d)                                     # query of the last real token (frozen W_q)
K, V = rng.normal(size=(T, d)), rng.normal(size=(T, d))    # real keys/values (frozen W_k, W_v)
Pk, Pv = rng.normal(size=(p, d)), rng.normal(size=(p, d))  # the prefix: the only trained numbers

a, out = attn(q, np.vstack([Pk, K]), np.vstack([Pv, V]))   # what the model computes
lam = a[:p].sum()                                          # share of attention on the prefix
_, base = attn(q, K, V)                                    # the frozen model without a prefix
_, pre = attn(q, Pk, Pv)                                   # attention over the prefix alone
print(f"lambda = {lam:.4f}")                                             # lambda = 0.5652
print(np.allclose(out, (1 - lam) * base + lam * pre))                    # True
print(np.allclose(a[p:] / (1 - lam), attn(q, K, V)[0]))                  # True: rescaled, not reordered

# ---- with Hugging Face PEFT ----
from transformers import AutoModelForCausalLM
from peft import PrefixTuningConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("gpt2-medium")       # 24 layers, d = 1024, frozen
cfg = PrefixTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=10,
                         prefix_projection=True, encoder_hidden_size=512)  # Li & Liang's MLP
model = get_peft_model(model, cfg)
model.print_trainable_parameters()
# training: 10*1024 (P') + 1024*512+512 + 512*49152+49152 = 25,750,016 trainable
# stored after training: 10 * (2 * 24 * 1024) = 491,520 numbers, passed as past_key_values
```

The prefix as extra keys and values, checking He et al.'s decomposition in numpy (run; outputs in comments), then prefix tuning GPT-2 medium with PEFT (not run; counts computed by hand)

> **The prefix costs context and compute:** A prefix of length $p$ adds $p$ keys and values that every token attends to at every layer, and in the original formulation it also occupies $p$ positions of the context window. At $p=200$ for summarization that is real overhead. Unlike **LoRA**, nothing can be merged into the weights, so the cost is paid at inference forever. Prefixes do batch well, though: different requests can carry different prefixes through the same frozen model.

> **A gated edit, not a new instruction:** The decomposition $(1-\lambda)\,\mathrm{Attn}(q,K,V)+\lambda\,\mathrm{Attn}(q,P_k,P_v)$ shows what a prefix can and cannot do. It mixes a learned vector into each head's output, with a weight set by how strongly the query matches the prefix keys. It cannot reweight the real tokens against each other. In the figure, every real weight shrinks by exactly $1-\lambda$.

> **P-Tuning v2 and prompt tuning:** **Prompt tuning** (Lester, Al-Rfou and Constant, 2021) keeps the trainable vectors at the input layer only. It closes the gap to fine-tuning only for very large models, around 10B parameters and above. **P-Tuning v2** (Liu, Ji, Fu, Tam, Du, Yang and Tang, ACL 2022) is "deep prompt tuning", essentially prefix tuning applied to understanding tasks. It uses trainable prompts at every layer, treats the MLP reparameterisation as optional because its benefit depends on the task, and replaces verbalizers with an ordinary classification head. It matched fine-tuning across model sizes from 300M to 10B and on hard sequence-labelling tasks such as NER, with 0.1–3% of the parameters trained.

**Strengths**

- Stores about 0.1% of the parameters per task; the LM is shared and untouched
- Strong in low-data settings and on topics unseen during training (table-to-text)
- Different prefixes can be mixed in one batch through the same frozen model
- Far more expressive than input-only prompt tuning at small and medium model sizes

**Limitations**

- Extra attention cost at every layer and fewer free context positions, permanently
- Training is unstable without the MLP reparameterisation and sensitive to initialisation
- Performance is non-monotonic in prefix length, so p must be tuned
- Generally weaker than LoRA on large models and harder tasks; cannot be merged away

**Question:** A prefix is added to one attention head. How do the attention weights on the real tokens change?

- A. They are recomputed with new keys, so their order can change arbitrarily
- B. They are all multiplied by the same factor 1 − λ, so their ratios are unchanged
- C. They are unchanged; the prefix is added after the softmax
- D. The real tokens stop attending to each other

**Answer:** B. They are all multiplied by the same factor 1 − λ, so their ratios are unchanged — The real logits $q\cdot k_j$ do not change. Adding prefix terms to the softmax denominator scales every real weight by $1-\lambda$. That is why the head output is $(1-\lambda)\cdot$(old output) $+\ \lambda\cdot$(prefix output). In the worked example, $0.6698$ and $0.3302$ become $0.2840$ and $0.1400$.

**Question:** How many numbers does a length-10 prefix store for GPT-2 medium (24 layers, d = 1,024)?

- A. 10,240
- B. 245,760
- C. 491,520
- D. 25,750,016

**Answer:** C. 491,520 — One key and one value of size $d$ per layer and position: $2\cdot L\cdot p\cdot d=2\cdot24\cdot10\cdot1024=491{,}520$, or 0.14% of 354.8M. 10,240 would be input-only prompt tuning, and 25.75M is the training-time count with the MLP reparameterisation.

**Key points**

- Prefix tuning learns $p$ key/value vectors at every layer, $K=[P_k;XW_k]$ and $V=[P_v;XW_v]$, and keeps the LM frozen. It stores $2Lpd$ numbers, about 0.1% of GPT-2.
- Each head becomes $(1-\lambda)\,\mathrm{Attn}(q,K,V)+\lambda\,\mathrm{Attn}(q,P_k,P_v)$, a gated, query-dependent edit (He et al. 2022).
- Training goes through an MLP reparameterisation of a smaller matrix for stability. Afterwards only the prefix is kept.
- In the paper it was comparable to fine-tuning on table-to-text, better with little data, slightly worse on XSUM. Best lengths were about 10 for table-to-text and 200 for summarization.
- Prompt tuning is the input-layer-only special case. P-Tuning v2 is deep prefix-style prompting for NLU and matches fine-tuning from 300M to 10B.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
