Prompt tuning

Prompt tuning (Lester, Al-Rfou, Constant, EMNLP 2021) freezes the whole model and learns only k soft-prompt vectors prepended to the input embeddings (k·d parameters: 409,600 for k = 100 on T5-XXL). The gap to full fine-tuning on SuperGLUE closes as the model grows to 11B.

14 min read1452 wordsadvanced2 viewsupdated 2026-10-13

Definition

Prompt tuning is the parameter-efficient adaptation method of Brian Lester, Rami Al-Rfou and Noah Constant, "The Power of Scale for Parameter-Efficient Prompt Tuning" (EMNLP 2021). Every weight of a pretrained language model stays frozen. For each task you learn a soft prompt: a matrix P∈Rk×dP\in\mathbb R^{k\times d} of kk free vectors that live in the model's dd-dimensional embedding space but are not tied to any vocabulary token. PP is concatenated in front of the embedded input, [P; E[x]][P;\,E[x]], and trained by backpropagation through the frozen network to maximise log⁡p(y∣[P;E[x]])\log p(y\mid[P;E[x]]). A task is then just a file of k⋅dk\cdot d numbers.

GPT-3 (Brown et al., 2020) showed that a big enough model can do a task from a text prompt alone: an instruction plus a few examples, and no weight updates. This prompt design has two problems. Results depend heavily on the exact wording. And on benchmarks like SuperGLUE, few-shot GPT-3 stayed far below fine-tuned models that were much smaller. Model tuning (full fine-tuning) gives the best quality, but every task then needs its own copy of every weight, which is 11B parameters per task for T5-XXL.

Searching for better discrete prompts is the obvious next step. AutoPrompt (Shin et al., 2020) used gradients to pick trigger tokens. But a discrete prompt can only pick points that are rows of the embedding matrix, and the search over VkV^k token sequences is combinatorial. Prompt tuning relaxes the constraint: let the prompt be any point in embedding space and optimise it with gradient descent, exactly like a weight. Two concurrent works took similar routes. Prefix tuning (Li & Liang, 2021) learns vectors at every layer. P-Tuning (Liu et al., "GPT Understands, Too", 2021) learns input-layer prompts produced by a small prompt encoder. Lester et al. stripped the idea down to its simplest form, prompts at the input only with no reparameterisation, and studied how it behaves with scale.

The mechanism

max⁡P ∑(x,y)∈Dlog⁡pθ(y ∣ [P; E[x]]),P∈Rk×d,θ frozen\max_{P}\ \sum_{(x,y)\in\mathcal D}\log p_{\theta}\big(y\ \big|\ [P;\,E[x]]\big),\qquad P\in\mathbb R^{k\times d},\quad \theta\ \text{frozen}
E[x]∈Rn×dE[x]\in\mathbb R^{n\times d}
the frozen embedding rows of the nn input tokens
P=(p1,…,pk)P=(p_1,\dots,p_k)
the soft prompt, kk "virtual tokens"; the only trainable tensor
[P;E[x]][P;E[x]]
concatenation along the sequence axis: the encoder sees k+nk+n positions
yy
the target as text, e.g. the word "positive"; T5 generates it with its decoder, so there is no new classification head
θ\theta
all model weights, including the embedding table; gradients flow through them into PP but they are never updated

From the model's point of view, nothing changes. Attention, feed-forward layers and positional handling treat the kk prompt positions exactly like token positions. They just carry vectors that no token maps to. Every input position can attend to the prompt at every layer, so the prompt can steer the whole computation even though it is injected only once, at the bottom. The gradient is the ordinary backpropagated gradient with respect to the input embeddings at those kk positions:

∇pjL=∂L∂hj(0),h(0)=[P;E[x]]⟹one backward pass, no weight gradients stored\nabla_{p_j}\mathcal L=\frac{\partial\mathcal L}{\partial h^{(0)}_j},\qquad h^{(0)}=[P;E[x]]\quad\Longrightarrow\quad\text{one backward pass, no weight gradients stored}
hj(0)h^{(0)}_j
the layer-0 input at prompt position jj
memory\text{memory}
activations are still stored for the backward pass, but optimiser state exists only for k⋅dk\cdot d numbers
input sequence = [soft prompt ; embedded text], fed to a frozen modelP1P2…Pkthefilmwasgreatsoft prompt P ∈ R^(k×d), trainableE[x]: frozen embedding rowsfrozen pretrained model (all weights fixed)T5 v1.1, LM-adapted · XXL: 24 + 24 layers, d = 4096one copy serves every task in a mixed-task batchdecoder"positive"lossCE on text∂L/∂P only: the model gets no updatewhat is trained per task (sizes for T5-XXL, 11B)Discrete promptreal vocab tokenshand-written or searchedno gradient into prompt0 trained paramsPrompt tuningk free vectors in R^dat the input layer onlyk·d params: 409,600(k = 100, d = 4096)Prefix tuningk vectors at every layerprepended keys/valuesabout 2·L·k·d paramsMLP-reparameterisedModel tuningupdate every weightone full copy per task11B params per taskfor T5-XXL
fig.Prompt tuning in one picture. The soft prompt P (blue) is prepended to the frozen embeddings of the input text and passed through a frozen T5. The label comes out as text from the decoder, and the cross-entropy gradient flows back through the whole network but updates only P. Because the model is shared, examples from different tasks, each with its own prompt, can sit in one batch. The strip below compares what each adaptation method trains per task.

What the paper found

Scale closes the gap. Lester et al. prompt-tuned T5 v1.1 at five sizes, Small to XXL (about 11B parameters), on SuperGLUE. At small sizes, prompt tuning trails model tuning by a wide margin. The gap shrinks as the model grows, and at XXL prompt tuning matches the strong multi-task model-tuning baseline. At that size it tunes less than 0.01% of the parameters. Few-shot prompt design with GPT-3 stayed far below both curves.

The checkpoint matters. Off-the-shelf T5 is pretrained with span corruption. It has only ever produced outputs like "<X> missing span <Y>", with sentinel tokens, and it has never written natural text. A frozen model like that is hard to steer into emitting a fluent label, and prompt tuning on it was poor and unstable. The fix was LM adaptation: continue training T5 for up to 100K steps with a plain prefix language-modelling objective, then freeze it. The authors released these "LM-adapted" T5 checkpoints, and all main results use them.

Initialisation and length matter less with scale. The paper compares three initialisations: random uniform in [−0.5,0.5][-0.5,0.5], embeddings of tokens sampled from the 5,000 most common vocabulary entries, and class-label initialisation. The class-label method starts each prompt vector at the embedding of a label word ("positive", "entailment", …) and fills any remaining slots with sampled vocabulary. Class labels worked best, and random worst, but the differences mostly vanished at XXL. Prompt lengths of 1, 5, 20, 100 and 150 were tested. Going beyond about 20 tokens gave small gains, and the largest model did well even with very short prompts.

Robustness and ensembling. Because the model cannot change, prompt tuning overfits less to the training domain. Prompts trained on SQuAD transferred to out-of-domain question-answering sets better than model tuning on several of them, with a similar effect for paraphrase detection moved between datasets. Several prompts trained for the same task can be run in one batch on one frozen model. This gives a cheap prompt ensemble that beat the average single prompt.

Interpretability is limited. Learned prompt vectors are not near any single token. Their nearest vocabulary neighbours often form tight semantic clusters, and with class-label initialisation the label words tend to remain nearest neighbours of some prompt tokens. Still, the prompt cannot be read back as text.

Setup in Lester et al. (2021)

Backbone
T5 v1.1, LM-adapted (extra 100K steps of prefix LM), frozen. Sizes Small, Base, Large, XL, XXL; dmodel=512,768,1024,2048,4096d_{model}=512,768,1024,2048,4096. XXL: 24 encoder + 24 decoder layers, about 11B parameters
Prompt
default length k=100k=100, class-label initialisation; lengths 1, 5, 20, 100 and 150 in the ablation
Trainable
k⋅dk\cdot d: 409,600 numbers at k=100k=100 on XXL, which is less than 0.01% of the model
Optimisation
Adafactor, learning rate 0.3, batch size 32, 30,000 steps, checkpoint chosen by early stopping on the dev set
Output
labels written as text tokens and generated by the decoder (T5's text-to-text format)
Benchmarks
SuperGLUE (main results); domain transfer from SQuAD to MRQA out-of-domain sets, and between paraphrase datasets

Prompt tuning, prefix tuning, P-Tuning and discrete prompts

methodwhat is learnedwhere it entersper-task size
discrete prompt (GPT-3 style, AutoPrompt)a sequence of real tokensinput, as vocabulary rowsk token ids
Prompt tuning (Lester et al. 2021)k free vectorsinput embeddings onlyk⋅dk\cdot d
P-Tuning (Liu et al. 2021)input vectors produced by a BiLSTM+MLP prompt encoder; can be mixed with anchor tokensinput embeddings onlyk⋅dk\cdot d at inference
Prefix tuning (Li & Liang 2021)k key/value vectors per layer, trained through an MLP reparameterisationevery attention layerabout 2Lkd2Lkd
P-Tuning v2 (Liu et al. 2022)deep prompts at every layer, no reparameterisation neededevery layerabout 2Lkd2Lkd
Where the trainable vectors live (L layers, k prompt positions, model width d)

The key difference from Prefix tuning is depth. A prefix supplies keys and values directly at every layer, so it can shape every attention map. A soft prompt is injected once, and its influence on deeper layers has to pass through the frozen network. That makes prompt tuning about 2L2L times smaller, but also weaker on small models. P-Tuning v2 (Liu et al., ACL 2022) showed that adding prompts at every layer recovers fine-tuning quality on models of a few hundred million parameters and on hard sequence-labelling tasks. In other words, the deep form is what you need when you cannot rely on scale.

Compared with discrete prompting, the soft prompt is a strict relaxation. Any discrete prompt is a point in the soft search space, since each token is a row of EE. Gradient descent can therefore only do at least as well at the global optimum, though it can stop in a worse local one. You give up readability, and you give up the ability to paste the prompt into another model: the vectors only mean something to the exact checkpoint they were trained on.

frozen vocab in R² · trained: 2 × 2 = 4 numbersgreatgoodlovefinebadawfulhatepoormoviefilmplotreviewtheaisnotpositivenegativeP1P2○ init · ● now · ◇ hidden teacher · label wordsexcess loss (nats over the teacher floor)11e-11e-21e-31e-4best discrete promptstep 0300soft prompt, step 300excess 0.0000 · acc 48/48best discrete (all 18² = 324)awful + fine · excess 0.0153 · acc 47/48nearest words to each soft token (distance)P1 (0.15, 0.92): good 0.37, fine 0.49P2 (-0.93, -0.10): plot 0.27, awful 0.29the soft prompt beats every sequence of real wordsfrozen: 18 word vectors, A, B, Q, verbalizer
length k
init
fig.A frozen toy model with a 2-D embedding space and 18 fixed word vectors. The input attends to the k prompt vectors, then a tanh layer and a verbalizer score the label words 'positive' and 'negative' by dot product. The targets come from the same frozen model run with a hidden 2-token teacher prompt (◇) that sits between words. The loss is shown as excess cross-entropy over the teacher's own entropy. At the defaults (k = 2, class-label init, lr 0.5) the prompt starts on 'positive' and 'negative' with excess 0.2022 (36/48 correct). It falls to 0.0122 after 10 steps and 0.0045 after 25, and reaches 0.0000 (48/48) by step 300, landing on the teacher at (0.15, 0.92) and (−0.93, −0.10). The nearest words there are 'good' (distance 0.37) and 'plot' (0.27). Exhaustive search over all 18² = 324 two-word prompts finds 'awful + fine' at best, with excess 0.0153 and 47/48. Random init gets to the same optimum more slowly (0.0107 at step 25). Sampled-vocab init ('film', 'the') sticks in a local minimum at 0.0464, which is worse than the best discrete prompt. With k = 1 no prompt can match the input-dependent teacher: soft 0.0555 vs discrete 'film' 0.0935. With k = 3 the best discrete prompt gets close (0.0057), and the soft one reaches 0.0006.

Worked example: sizes, and one forward pass

modeld_modelk = 1k = 20k = 100k = 150
Small51251210,24051,20076,800
Base76876815,36076,800115,200
Large10241,02420,480102,400153,600
XL20482,04840,960204,800307,200
XXL40964,09681,920409,600614,400
Trainable parameters k·d for the T5 v1.1 sizes in the paper (computed)

At k=100k=100 on XXL the prompt is 409,600 numbers, or 1.64 MB in fp32. One bf16 copy of an 11B model is about 22 GB. That is a ratio of roughly 13,000 to 1 per task, and a fraction 409,600/11⋅109≈0.0037%409{,}600/11\cdot10^9\approx0.0037\% of the parameters.

The toy model of the figure on input x = (0.086, −0.871), class-label init P₁ = e(positive) = (0.6, 0.2), P₂ = e(negative) = (−0.2, −0.25)

  1. 1Attention logits sj=x⊤Qpj=3(x0pj,1−x1pj,0)s_j=x^\top Qp_j=3(x_0p_{j,1}-x_1p_{j,0}): s=(1.620, −0.587)s=(1.620,\,-0.587). Softmax gives α=(0.901, 0.099)\alpha=(0.901,\,0.099), so the input reads c=∑jαjpj=(0.521, 0.155)c=\sum_j\alpha_jp_j=(0.521,\,0.155).
  2. 2Hidden layer: Ax=(1.126, −1.341)Ax=(1.126,\,-1.341) and 2.2c=(1.146, 0.342)2.2c=(1.146,\,0.342), so h=tanh⁡(2.271, −1.000)=(0.979, −0.761)h=\tanh(2.271,\,-1.000)=(0.979,\,-0.761).
  3. 3Verbalizer: w=epos−eneg=(0.8, 0.45)w=e_{pos}-e_{neg}=(0.8,\,0.45), logit z=4 w⊤h=4(0.783−0.343)=1.762z=4\,w^\top h=4(0.783-0.343)=1.762, p(positive)=σ(1.762)=0.853p(\text{positive})=\sigma(1.762)=0.853.
  4. 4The teacher prompt gives p=0.979p=0.979 on this input. Cross-entropy −[0.979log⁡0.853+0.021log⁡0.147]=0.196-[0.979\log0.853+0.021\log0.147]=0.196, against the teacher's entropy floor of 0.1020.102: excess 0.0930.093 on this example.
  5. 5Backward: ∂L/∂z=p−y=−0.125\partial\mathcal L/\partial z=p-y=-0.125. This pushes zz up, the error flows through tanh⁡\tanh and BB into cc, and cc passes it to P1P_1 with weight α1=0.90\alpha_1=0.90. P2P_2 also gets a share through the attention logits. No other number in the model changes.
python
1import torch, torch.nn as nn
2from transformers import AutoTokenizer, T5ForConditionalGeneration
3from transformers.optimization import Adafactor
4
5name = "google/t5-base-lm-adapt" # LM-adapted T5 v1.1, as in the paper
6tok = AutoTokenizer.from_pretrained(name)
7model = T5ForConditionalGeneration.from_pretrained(name)
8model.requires_grad_(False) # freeze every weight, including embeddings
9emb = model.get_input_embeddings() # nn.Embedding(32128, d_model)
10
11class SoftPrompt(nn.Module):
12 def __init__(self, k, emb, init_words, tok):
13 super().__init__()
14 ids = [i for w in init_words for i in tok(w, add_special_tokens=False).input_ids]
15 common = torch.randint(0, 5000, (k,)) # stand-in for "sampled from the 5,000 most common tokens"
16 common[: min(k, len(ids))] = torch.tensor(ids[:k]) # class-label init first
17 self.P = nn.Parameter(emb.weight[common].detach().clone()) # (k, d)
18 def forward(self, input_ids, attention_mask):
19 x = emb(input_ids) # (B, n, d), frozen
20 P = self.P.unsqueeze(0).expand(x.size(0), -1, -1) # (B, k, d), trainable
21 mask = torch.cat([attention_mask.new_ones(x.size(0), P.size(1)), attention_mask], 1)
22 return torch.cat([P, x], 1), mask
23
24prompt = SoftPrompt(k=100, emb=emb, init_words=["positive", "negative"], tok=tok)
25opt = Adafactor(prompt.parameters(), lr=0.3, scale_parameter=False, relative_step=False, warmup_init=False)
26
27batch = tok(["sst2 sentence: the film was great", "sst2 sentence: a dull, lifeless plot"],
28 return_tensors="pt", padding=True)
29labels = tok(["positive", "negative"], return_tensors="pt", padding=True).input_ids
30labels[labels == tok.pad_token_id] = -100
31
32embeds, mask = prompt(batch.input_ids, batch.attention_mask)
33loss = model(inputs_embeds=embeds, attention_mask=mask, labels=labels).loss
34loss.backward() # gradients reach prompt.P only
35opt.step(); opt.zero_grad()
36
37print(sum(p.numel() for p in prompt.parameters())) # 100 * 768 = 76,800 trainable for T5-Base
38# Same thing with the PEFT library:
39# from peft import PromptTuningConfig, PromptTuningInit, TaskType, get_peft_model
40# cfg = PromptTuningConfig(task_type=TaskType.SEQ_2_SEQ_LM, num_virtual_tokens=100,
41# prompt_tuning_init=PromptTuningInit.TEXT,
42# prompt_tuning_init_text="positive negative", tokenizer_name_or_path=name)
43# model = get_peft_model(model, cfg)
Prompt tuning a frozen T5 by hand: soft prompt via inputs_embeds, class-label initialisation, Adafactor at lr 0.3 (PyTorch + Hugging Face; not run here)

Use when

  • +Tiny per-task state: k⋅dk\cdot d numbers, 409,600 at k=100k=100 on T5-XXL
  • +One frozen model serves many tasks, even mixed in one batch
  • +Matches full fine-tuning on SuperGLUE once the model reaches about 11B parameters
  • +Less overfitting to the training domain; cheap prompt ensembles

Avoid when

  • −Clearly worse than fine-tuning on small and mid-size models
  • −Sensitive to the checkpoint (needs LM adaptation for T5) and to initialisation at small scale
  • −Adds kk positions to every sequence, which costs context length and compute
  • −Prompts are not human-readable and do not transfer between models

Check yourself

Which statement best distinguishes prompt tuning from prefix tuning?

Check yourself

In Lester et al., what happened to the gap between prompt tuning and full model tuning on SuperGLUE as T5 grew from Small to XXL?

Remember

  • ✓Prompt tuning learns P∈Rk×dP\in\mathbb R^{k\times d}, prepended to the frozen input embeddings: max⁡Plog⁡pθ(y∣[P;E[x]])\max_P\log p_\theta(y\mid[P;E[x]]) with θ\theta frozen.
  • ✓Lester et al. (EMNLP 2021) used LM-adapted T5 v1.1 with k=100k=100, class-label init, Adafactor at lr 0.3. At XXL (11B, d=4096d=4096) it matched model tuning while training 409,600 numbers.
  • ✓Small models lag clearly. Initialisation (class labels > sampled vocab > random) and prompt length matter less as the model grows.
  • ✓Prefix tuning and P-Tuning v2 put vectors at every layer and work at smaller scale. P-Tuning uses an LSTM prompt encoder at the input. Discrete prompts are the special case where every vector is a vocabulary row.
  • ✓Soft prompts are a relaxation of discrete prompts. They are not readable, they are tied to one checkpoint, and they cost k extra positions per sequence.
← back to Deep Learning roadmap