# Prompt tuning

> Prompt tuning (Lester, Al-Rfou, Constant, EMNLP 2021) freezes the whole model and learns only k soft-prompt vectors prepended to the input embeddings (k·d parameters: 409,600 for k = 100 on T5-XXL). The gap to full fine-tuning on SuperGLUE closes as the model grows to 11B.

Source: https://www.mlfoundry.in/learn/prompt-tuning  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 14 min read · updated 2026-10-13

**Prompt tuning:** is the parameter-efficient adaptation method of Brian Lester, Rami Al-Rfou and Noah Constant, "The Power of Scale for Parameter-Efficient Prompt Tuning" (EMNLP 2021). Every weight of a pretrained language model stays frozen. For each task you learn a **soft prompt**: a matrix $P\in\mathbb R^{k\times d}$ of $k$ free vectors that live in the model's $d$-dimensional embedding space but are not tied to any vocabulary token. $P$ is concatenated in front of the embedded input, $[P;\,E[x]]$, and trained by backpropagation through the frozen network to maximise $\log p(y\mid[P;E[x]])$. A task is then just a file of $k\cdot d$ numbers.

GPT-3 (Brown et al., 2020) showed that a big enough model can do a task from a text prompt alone: an instruction plus a few examples, and no weight updates. This **prompt design** has two problems. Results depend heavily on the exact wording. And on benchmarks like SuperGLUE, few-shot GPT-3 stayed far below fine-tuned models that were much smaller. **Model tuning** (full fine-tuning) gives the best quality, but every task then needs its own copy of every weight, which is 11B parameters per task for T5-XXL.

Searching for better discrete prompts is the obvious next step. AutoPrompt (Shin et al., 2020) used gradients to pick trigger tokens. But a discrete prompt can only pick points that are rows of the embedding matrix, and the search over $V^k$ token sequences is combinatorial. Prompt tuning relaxes the constraint: let the prompt be *any* point in embedding space and optimise it with gradient descent, exactly like a weight. Two concurrent works took similar routes. **Prefix tuning** (Li & Liang, 2021) learns vectors at every layer. P-Tuning (Liu et al., "GPT Understands, Too", 2021) learns input-layer prompts produced by a small prompt encoder. Lester et al. stripped the idea down to its simplest form, prompts at the input only with no reparameterisation, and studied how it behaves with scale.

## The mechanism

$$
\max_{P}\ \sum_{(x,y)\in\mathcal D}\log p_{\theta}\big(y\ \big|\ [P;\,E[x]]\big),\qquad P\in\mathbb R^{k\times d},\quad \theta\ \text{frozen}
$$

- $E[x]\in\mathbb R^{n\times d}$: the frozen embedding rows of the $n$ input tokens
- $P=(p_1,\dots,p_k)$: the soft prompt, $k$ "virtual tokens"; the only trainable tensor
- $[P;E[x]]$: concatenation along the sequence axis: the encoder sees $k+n$ positions
- $y$: the target **as text**, e.g. the word "positive"; T5 generates it with its decoder, so there is no new classification head
- $\theta$: all model weights, including the embedding table; gradients flow through them into $P$ but they are never updated

From the model's point of view, nothing changes. Attention, feed-forward layers and positional handling treat the $k$ prompt positions exactly like token positions. They just carry vectors that no token maps to. Every input position can attend to the prompt at every layer, so the prompt can steer the whole computation even though it is injected only once, at the bottom. The gradient is the ordinary backpropagated gradient with respect to the input embeddings at those $k$ positions:

$$
\nabla_{p_j}\mathcal L=\frac{\partial\mathcal L}{\partial h^{(0)}_j},\qquad h^{(0)}=[P;E[x]]\quad\Longrightarrow\quad\text{one backward pass, no weight gradients stored}
$$

- $h^{(0)}_j$: the layer-0 input at prompt position $j$
- $\text{memory}$: activations are still stored for the backward pass, but optimiser state exists only for $k\cdot d$ numbers

> Figure: Prompt tuning in one picture. The soft prompt P (blue) is prepended to the frozen embeddings of the input text and passed through a frozen T5. The label comes out as text from the decoder, and the cross-entropy gradient flows back through the whole network but updates only P. Because the model is shared, examples from different tasks, each with its own prompt, can sit in one batch. The strip below compares what each adaptation method trains per task.

## What the paper found

**Scale closes the gap.** Lester et al. prompt-tuned T5 v1.1 at five sizes, Small to XXL (about 11B parameters), on SuperGLUE. At small sizes, prompt tuning trails model tuning by a wide margin. The gap shrinks as the model grows, and at XXL prompt tuning matches the strong multi-task model-tuning baseline. At that size it tunes less than 0.01% of the parameters. Few-shot prompt design with GPT-3 stayed far below both curves.

**The checkpoint matters.** Off-the-shelf T5 is pretrained with span corruption. It has only ever produced outputs like "<X> missing span <Y>", with sentinel tokens, and it has never written natural text. A frozen model like that is hard to steer into emitting a fluent label, and prompt tuning on it was poor and unstable. The fix was **LM adaptation**: continue training T5 for up to 100K steps with a plain prefix language-modelling objective, then freeze it. The authors released these "LM-adapted" T5 checkpoints, and all main results use them.

**Initialisation and length matter less with scale.** The paper compares three initialisations: random uniform in $[-0.5,0.5]$, embeddings of tokens sampled from the 5,000 most common vocabulary entries, and **class-label** initialisation. The class-label method starts each prompt vector at the embedding of a label word ("positive", "entailment", …) and fills any remaining slots with sampled vocabulary. Class labels worked best, and random worst, but the differences mostly vanished at XXL. Prompt lengths of 1, 5, 20, 100 and 150 were tested. Going beyond about 20 tokens gave small gains, and the largest model did well even with very short prompts.

**Robustness and ensembling.** Because the model cannot change, prompt tuning overfits less to the training domain. Prompts trained on SQuAD transferred to out-of-domain question-answering sets better than model tuning on several of them, with a similar effect for paraphrase detection moved between datasets. Several prompts trained for the same task can be run in one batch on one frozen model. This gives a cheap **prompt ensemble** that beat the average single prompt.

**Interpretability is limited.** Learned prompt vectors are not near any single token. Their nearest vocabulary neighbours often form tight semantic clusters, and with class-label initialisation the label words tend to remain nearest neighbours of some prompt tokens. Still, the prompt cannot be read back as text.

**Setup in Lester et al. (2021)**

- **Backbone:** T5 v1.1, LM-adapted (extra 100K steps of prefix LM), frozen. Sizes Small, Base, Large, XL, XXL; $d_{model}=512,768,1024,2048,4096$. XXL: 24 encoder + 24 decoder layers, about 11B parameters
- **Prompt:** default length $k=100$, class-label initialisation; lengths 1, 5, 20, 100 and 150 in the ablation
- **Trainable:** $k\cdot d$: 409,600 numbers at $k=100$ on XXL, which is less than 0.01% of the model
- **Optimisation:** Adafactor, learning rate 0.3, batch size 32, 30,000 steps, checkpoint chosen by early stopping on the dev set
- **Output:** labels written as text tokens and generated by the decoder (T5's text-to-text format)
- **Benchmarks:** SuperGLUE (main results); domain transfer from SQuAD to MRQA out-of-domain sets, and between paraphrase datasets

## Prompt tuning, prefix tuning, P-Tuning and discrete prompts

Where the trainable vectors live (L layers, k prompt positions, model width d)

| method | what is learned | where it enters | per-task size |
| --- | --- | --- | --- |
| discrete prompt (GPT-3 style, AutoPrompt) | a sequence of real tokens | input, as vocabulary rows | k token ids |
| **Prompt tuning** (Lester et al. 2021) | k free vectors | input embeddings only | $k\cdot d$ |
| P-Tuning (Liu et al. 2021) | input vectors produced by a BiLSTM+MLP prompt encoder; can be mixed with anchor tokens | input embeddings only | $k\cdot d$ at inference |
| **Prefix tuning** (Li & Liang 2021) | k key/value vectors per layer, trained through an MLP reparameterisation | every attention layer | about $2Lkd$ |
| P-Tuning v2 (Liu et al. 2022) | deep prompts at every layer, no reparameterisation needed | every layer | about $2Lkd$ |

The key difference from **Prefix tuning** is depth. A prefix supplies keys and values directly at every layer, so it can shape every attention map. A soft prompt is injected once, and its influence on deeper layers has to pass through the frozen network. That makes prompt tuning about $2L$ times smaller, but also weaker on small models. P-Tuning v2 (Liu et al., ACL 2022) showed that adding prompts at every layer recovers fine-tuning quality on models of a few hundred million parameters and on hard sequence-labelling tasks. In other words, the deep form is what you need when you cannot rely on scale.

Compared with discrete prompting, the soft prompt is a strict relaxation. Any discrete prompt is a point in the soft search space, since each token is a row of $E$. Gradient descent can therefore only do at least as well at the global optimum, though it can stop in a worse local one. You give up readability, and you give up the ability to paste the prompt into another model: the vectors only mean something to the exact checkpoint they were trained on.

> Figure: A frozen toy model with a 2-D embedding space and 18 fixed word vectors. The input attends to the k prompt vectors, then a tanh layer and a verbalizer score the label words 'positive' and 'negative' by dot product. The targets come from the same frozen model run with a hidden 2-token teacher prompt (◇) that sits between words. The loss is shown as excess cross-entropy over the teacher's own entropy. At the defaults (k = 2, class-label init, lr 0.5) the prompt starts on 'positive' and 'negative' with excess 0.2022 (36/48 correct). It falls to 0.0122 after 10 steps and 0.0045 after 25, and reaches 0.0000 (48/48) by step 300, landing on the teacher at (0.15, 0.92) and (−0.93, −0.10). The nearest words there are 'good' (distance 0.37) and 'plot' (0.27). Exhaustive search over all 18² = 324 two-word prompts finds 'awful + fine' at best, with excess 0.0153 and 47/48. Random init gets to the same optimum more slowly (0.0107 at step 25). Sampled-vocab init ('film', 'the') sticks in a local minimum at 0.0464, which is worse than the best discrete prompt. With k = 1 no prompt can match the input-dependent teacher: soft 0.0555 vs discrete 'film' 0.0935. With k = 3 the best discrete prompt gets close (0.0057), and the soft one reaches 0.0006.

## Worked example: sizes, and one forward pass

Trainable parameters k·d for the T5 v1.1 sizes in the paper (computed)

| model | d_model | k = 1 | k = 20 | k = 100 | k = 150 |
| --- | --- | --- | --- | --- | --- |
| Small | 512 | 512 | 10,240 | 51,200 | 76,800 |
| Base | 768 | 768 | 15,360 | 76,800 | 115,200 |
| Large | 1024 | 1,024 | 20,480 | 102,400 | 153,600 |
| XL | 2048 | 2,048 | 40,960 | 204,800 | 307,200 |
| XXL | 4096 | 4,096 | 81,920 | 409,600 | 614,400 |

At $k=100$ on XXL the prompt is 409,600 numbers, or 1.64 MB in fp32. One bf16 copy of an 11B model is about 22 GB. That is a ratio of roughly 13,000 to 1 per task, and a fraction $409{,}600/11\cdot10^9\approx0.0037\%$ of the parameters.

**The toy model of the figure on input x = (0.086, −0.871), class-label init P₁ = e(positive) = (0.6, 0.2), P₂ = e(negative) = (−0.2, −0.25)**

1. Attention logits $s_j=x^\top Qp_j=3(x_0p_{j,1}-x_1p_{j,0})$: $s=(1.620,\,-0.587)$. Softmax gives $\alpha=(0.901,\,0.099)$, so the input reads $c=\sum_j\alpha_jp_j=(0.521,\,0.155)$.
2. Hidden layer: $Ax=(1.126,\,-1.341)$ and $2.2c=(1.146,\,0.342)$, so $h=\tanh(2.271,\,-1.000)=(0.979,\,-0.761)$.
3. Verbalizer: $w=e_{pos}-e_{neg}=(0.8,\,0.45)$, logit $z=4\,w^\top h=4(0.783-0.343)=1.762$, $p(\text{positive})=\sigma(1.762)=0.853$.
4. The teacher prompt gives $p=0.979$ on this input. Cross-entropy $-[0.979\log0.853+0.021\log0.147]=0.196$, against the teacher's entropy floor of $0.102$: excess $0.093$ on this example.
5. Backward: $\partial\mathcal L/\partial z=p-y=-0.125$. This pushes $z$ up, the error flows through $\tanh$ and $B$ into $c$, and $c$ passes it to $P_1$ with weight $\alpha_1=0.90$. $P_2$ also gets a share through the attention logits. No other number in the model changes.

```python
import torch, torch.nn as nn
from transformers import AutoTokenizer, T5ForConditionalGeneration
from transformers.optimization import Adafactor

name = "google/t5-base-lm-adapt"                  # LM-adapted T5 v1.1, as in the paper
tok = AutoTokenizer.from_pretrained(name)
model = T5ForConditionalGeneration.from_pretrained(name)
model.requires_grad_(False)                       # freeze every weight, including embeddings
emb = model.get_input_embeddings()                # nn.Embedding(32128, d_model)

class SoftPrompt(nn.Module):
    def __init__(self, k, emb, init_words, tok):
        super().__init__()
        ids = [i for w in init_words for i in tok(w, add_special_tokens=False).input_ids]
        common = torch.randint(0, 5000, (k,))     # stand-in for "sampled from the 5,000 most common tokens"
        common[: min(k, len(ids))] = torch.tensor(ids[:k])   # class-label init first
        self.P = nn.Parameter(emb.weight[common].detach().clone())   # (k, d)
    def forward(self, input_ids, attention_mask):
        x = emb(input_ids)                                      # (B, n, d), frozen
        P = self.P.unsqueeze(0).expand(x.size(0), -1, -1)       # (B, k, d), trainable
        mask = torch.cat([attention_mask.new_ones(x.size(0), P.size(1)), attention_mask], 1)
        return torch.cat([P, x], 1), mask

prompt = SoftPrompt(k=100, emb=emb, init_words=["positive", "negative"], tok=tok)
opt = Adafactor(prompt.parameters(), lr=0.3, scale_parameter=False, relative_step=False, warmup_init=False)

batch = tok(["sst2 sentence: the film was great", "sst2 sentence: a dull, lifeless plot"],
            return_tensors="pt", padding=True)
labels = tok(["positive", "negative"], return_tensors="pt", padding=True).input_ids
labels[labels == tok.pad_token_id] = -100

embeds, mask = prompt(batch.input_ids, batch.attention_mask)
loss = model(inputs_embeds=embeds, attention_mask=mask, labels=labels).loss
loss.backward()                                   # gradients reach prompt.P only
opt.step(); opt.zero_grad()

print(sum(p.numel() for p in prompt.parameters()))   # 100 * 768 = 76,800 trainable for T5-Base
# Same thing with the PEFT library:
# from peft import PromptTuningConfig, PromptTuningInit, TaskType, get_peft_model
# cfg = PromptTuningConfig(task_type=TaskType.SEQ_2_SEQ_LM, num_virtual_tokens=100,
#                          prompt_tuning_init=PromptTuningInit.TEXT,
#                          prompt_tuning_init_text="positive negative", tokenizer_name_or_path=name)
# model = get_peft_model(model, cfg)
```

Prompt tuning a frozen T5 by hand: soft prompt via inputs_embeds, class-label initialisation, Adafactor at lr 0.3 (PyTorch + Hugging Face; not run here)

> **Small models and the wrong checkpoint:** Prompt tuning is not a free lunch below a few billion parameters. On T5-Small to Large it trails full fine-tuning clearly, and on a span-corruption checkpoint it can fail outright, because the frozen model has never produced fluent text. If you have to adapt a small model, use **LoRA**, **Prefix tuning** or P-Tuning v2 (prompts at every layer). Expect to need a learning rate far above fine-tuning rates: the paper used 0.3.

> **A soft prompt is a relaxed discrete prompt:** Every token sequence is a point in the soft search space, so the best soft prompt is never worse than the best discrete one. In the figure, the best of all 324 two-word prompts leaves excess 0.0153, and the soft prompt reaches 0. But gradient descent finds local optima, not global ones. The sampled-vocabulary start in the figure stalls at 0.0464, above the discrete optimum. That is why initialisation matters for small models, and why class labels, which already sit in the right part of the space, are the paper's default.

> **Serving many tasks with one model:** Because $\theta$ is shared, a server can hold one copy of the model and a table of prompts. Each request in a batch gets its own $k$ vectors prepended. The cost is $k$ extra positions per sequence: 100 extra tokens of context and attention, at every step. **Prefix tuning** shares this property. **LoRA** can be merged into the weights for zero overhead, but then one copy serves only one task.

**Strengths**

- Tiny per-task state: $k\cdot d$ numbers, 409,600 at $k=100$ on T5-XXL
- One frozen model serves many tasks, even mixed in one batch
- Matches full fine-tuning on SuperGLUE once the model reaches about 11B parameters
- Less overfitting to the training domain; cheap prompt ensembles

**Limitations**

- Clearly worse than fine-tuning on small and mid-size models
- Sensitive to the checkpoint (needs LM adaptation for T5) and to initialisation at small scale
- Adds $k$ positions to every sequence, which costs context length and compute
- Prompts are not human-readable and do not transfer between models

**Question:** Which statement best distinguishes prompt tuning from prefix tuning?

- A. Prompt tuning updates the attention weights; prefix tuning does not
- B. Prompt tuning learns k vectors only at the input embedding layer; prefix tuning learns key/value vectors at every layer
- C. Prompt tuning uses real vocabulary tokens; prefix tuning uses continuous vectors
- D. Prefix tuning works only for encoder–decoder models

**Answer:** B. Prompt tuning learns k vectors only at the input embedding layer; prefix tuning learns key/value vectors at every layer — Both freeze the model and learn continuous vectors. Prompt tuning injects $k\cdot d$ numbers once, at the input. Prefix tuning prepends learned keys and values to every attention layer (about $2Lkd$ numbers) and trains them through an MLP reparameterisation. Prefix tuning is deeper and stronger on small models. Prompt tuning is smaller and relies on scale.

**Question:** In Lester et al., what happened to the gap between prompt tuning and full model tuning on SuperGLUE as T5 grew from Small to XXL?

- A. It stayed constant at about 10 points
- B. It grew, because larger models need more trainable parameters
- C. It shrank, and at XXL (about 11B) prompt tuning matched model tuning
- D. Prompt tuning beat model tuning at every size

**Answer:** C. It shrank, and at XXL (about 11B) prompt tuning matched model tuning — The headline result is the "power of scale": prompt tuning is weak on small models, but the gap closes with size, and at XXL the frozen model plus a 100-token prompt matches the multi-task model-tuning baseline. At that size, initialisation and prompt length matter much less.

**Key points**

- Prompt tuning learns $P\in\mathbb R^{k\times d}$, prepended to the frozen input embeddings: $\max_P\log p_\theta(y\mid[P;E[x]])$ with $\theta$ frozen.
- Lester et al. (EMNLP 2021) used LM-adapted T5 v1.1 with $k=100$, class-label init, Adafactor at lr 0.3. At XXL (11B, $d=4096$) it matched model tuning while training 409,600 numbers.
- Small models lag clearly. Initialisation (class labels > sampled vocab > random) and prompt length matter less as the model grows.
- Prefix tuning and P-Tuning v2 put vectors at every layer and work at smaller scale. P-Tuning uses an LSTM prompt encoder at the input. Discrete prompts are the special case where every vector is a vocabulary row.
- Soft prompts are a relaxation of discrete prompts. They are not readable, they are tied to one checkpoint, and they cost k extra positions per sequence.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
