Parameter-efficient fine-tuning (PEFT)

PEFT freezes a pretrained model and trains a few new or selected weights: adapters, prefixes, soft prompts, IA³ vectors, biases or low-rank LoRA updates. On Llama 2 7B, LoRA r = 8 on W_q, W_v trains 4.19M of 6.74B weights (0.062%), cutting weight + Adam memory from 108 to 13.5 GB.

15 min read1141 wordscore0 viewsupdated 2026-10-13

Definition

Parameter-efficient fine-tuning (PEFT) adapts a pretrained network fθf_\theta to a task by training a small parameter set ϕ\phi with ∣ϕ∣≪∣θ∣|\phi|\ll|\theta|, while θ\theta stays frozen and is shared across tasks. The model becomes fθ,ϕf_{\theta,\phi}. The main families are adapters (Houlsby et al., "Parameter-Efficient Transfer Learning for NLP", ICML 2019), prefix tuning (Li and Liang, ACL 2021), prompt tuning (Lester, Al-Rfou and Constant, EMNLP 2021), BitFit (Ben Zaken, Ravfogel and Goldberg, ACL 2022), LoRA (Hu et al., ICLR 2022) and IA³ (Liu et al., NeurIPS 2022). Lialin, Deshpande and Rumshisky ("Scaling Down to Scale Up", 2023) sort them into additive, selective and reparameterization-based methods.

Full fine-tuning updates every weight. For a 7B model that means a gradient and two Adam moments for 6.7 billion numbers, and a new 13.5 GB checkpoint for every task. Serving ten tasks means storing ten copies of the model. This was already a problem with BERT-large at 340M parameters. With GPT-3 at 175B it stopped being practical for most users.

Houlsby et al. framed the goal in 2019: one frozen model, plus a few new parameters per task. Their adapters came within 0.4% of full fine-tuning on GLUE while adding 3.6% of the parameters per task. Between 2021 and 2022, prefix tuning, prompt tuning, BitFit, LoRA and IA³ pushed the trained fraction down to 0.01–0.1%. He et al. ("Towards a Unified View of Parameter-Efficient Transfer Learning", ICLR 2022) showed that adapters, prefix tuning and LoRA can all be written as a learned modification added to some hidden state. They differ in where that modification is applied and how it is computed.

Where each method plugs in

soft prompt Ptoken embeddings E[x][P ; E[x]] : (p + T) × d× L blocks, frozennormself-attentionW_q+bW_k+bW_v+bW_o+b+ B·A+ B·A⊙ l_k⊙ l_v[P_K;K] [P_V;V]LoRAIA³prefixadapter d→m→d, + skip+ residualnormFFNW_up, W_gate+bW_down+b⊙ l_ff on d_ffadapter d→m→d, + skip+ residualnorm → LM headwhat each method trainsprompt tuningp soft tokens at the input onlyLester et al. 2021prefix tuningp virtual K, V in every layerLi & Liang 2021adaptersd→m→d bottleneck + skip,2 per block, Houlsby et al. 2019IA³learned rescaling of k, v andthe FFN hidden, Liu et al. 2022BitFittrain only the bias vectors bBen Zaken et al. 2022LoRAW + BA, rank r; merges into WHu et al. 2022additiveselectivereparam.hover a family · grey = frozen, shareddashed = the only trained weights
fig.One transformer block. Everything in grey is frozen and shared across tasks. The dashed parts are the only weights a method trains: soft prompts at the input (prompt tuning), virtual keys and values in every layer (prefix tuning), low-rank updates B·A beside W_q and W_v (LoRA), rescaling vectors on k, v and the FFN hidden state (IA³), bottleneck adapters after each sub-layer, and the bias vectors +b (BitFit, only in models that have biases). Hover a family to isolate it.

Additive methods: new modules or new inputs

Adapter(h)=h+Wup σ(Wdownh+bdown)+bup,Wdown∈Rm×d, Wup∈Rd×m, m≪d\text{Adapter}(h)=h+W_{up}\,\sigma(W_{down}h+b_{down})+b_{up},\qquad W_{down}\in\mathbb R^{m\times d},\ W_{up}\in\mathbb R^{d\times m},\ m\ll d
hh
output of an attention or FFN sub-layer, before the residual add
mm
bottleneck size; 2dm+d+m2dm+d+m parameters per adapter, two adapters per block in Houlsby's design
Wup≈0W_{up}\approx0
near-zero initialisation, so the adapter starts as the identity and the pretrained model is unchanged at step 0

Adapters add depth. Every token passes through the extra layers at inference, so they add latency that cannot be merged away. Pfeiffer et al. later showed that one adapter per block, after the FFN, works about as well. That halves the count.

prefix:  Attn(q, [PK;K], [PV;V]) in every layer,prompt:  h(0)=[P; E[x]], P∈Rp×d\text{prefix: }\ \mathrm{Attn}\big(q,\,[P_K;K],\,[P_V;V]\big)\ \text{in every layer},\qquad \text{prompt: }\ h^{(0)}=[P;\,E[x]],\ P\in\mathbb R^{p\times d}
PK,PV∈Rp×dkvP_K,P_V\in\mathbb R^{p\times d_{kv}}
pp trained "virtual tokens" of keys and values per layer, 2pdkvL2pd_{kv}L parameters
PP
pp soft-prompt embeddings at the input only, pdpd parameters
[ ⋅ ; ⋅ ][\,\cdot\,;\,\cdot\,]
concatenation along the sequence axis; real tokens attend to the prefix

Prefix tuning learns continuous key and value vectors that every real token can attend to, in every layer. Optimising PP directly was unstable, so Li and Liang reparameterised it through an MLP during training, P=MLP(P′)P=\mathrm{MLP}(P'), and kept only the resulting PP afterwards. With about 0.1% of the parameters, it matched fine-tuning on table-to-text with GPT-2 and was close on summarisation with BART. See Prefix tuning.

Prompt tuning keeps only the input-level embeddings. That is the smallest possible footprint: 20 tokens on a 4096-wide model is 81,920 numbers. Lester et al. found that the gap to full fine-tuning closes with scale. On SuperGLUE it trailed badly for small T5 models but matched model tuning at T5-XXL (11B). See Prompt tuning.

IA³ rescales activations instead of adding to them. It learns three vectors per block, lk,lv∈Rdkvl_k,l_v\in\mathbb R^{d_{kv}} and lff∈Rdffl_{ff}\in\mathbb R^{d_{ff}}, applied as lk⊙kl_k\odot k, lv⊙vl_v\odot v and lff⊙γ(W1x)l_{ff}\odot\gamma(W_1x). It is initialised to ones. Liu et al. paired it with the T-Few recipe on T0-3B and trained about 0.01% of the parameters, beating in-context learning with much larger models in the few-shot setting.

Selective and reparameterization methods

Selective methods unfreeze a subset of existing weights. BitFit trains only the bias vectors. On BERT-large that is about 0.08% of the parameters, and it was competitive with full fine-tuning on GLUE with small and medium data. The catch for LLMs is that Llama, Mistral and most recent decoders have no biases in their linear layers and use RMSNorm without a bias. On those models BitFit has nothing to train.

h=W0x+ΔWx=W0x+αrBAx,B∈Rdout×r, A∈Rr×din,∣ϕ∣=r (din+dout) per matrixh=W_0x+\Delta W x=W_0x+\tfrac{\alpha}{r}BAx,\qquad B\in\mathbb R^{d_{out}\times r},\ A\in\mathbb R^{r\times d_{in}},\qquad |\phi|=r\,(d_{in}+d_{out})\ \text{per matrix}
W0W_0
frozen pretrained matrix
rr
rank; 1–64 in practice, ≪min⁡(din,dout)\ll\min(d_{in},d_{out})
B=0B=0
at initialisation, so ΔW=0\Delta W=0 at step 0
α/r\alpha/r
scaling, so the effective learning rate changes little when r changes

LoRA reparameterises the update of a weight matrix, not the activations. Because BABA has the same shape as W0W_0, it can be merged after training, W=W0+αrBAW=W_0+\tfrac{\alpha}{r}BA, which leaves zero extra inference latency. On GPT-3 175B, Hu et al. reported 10,000 times fewer trainable parameters than full fine-tuning and 3 times less GPU memory, with checkpoints shrinking from 350 GB to about 35 MB at r=4r=4 on Wq,WvW_q,W_v. Quality was on par with full fine-tuning on WikiSQL, MNLI and SAMSum: slightly lower on one, higher on the other two. The details are in LoRA, and the 4-bit-base variant is in QLoRA. IA³ can be merged too, because a per-channel rescaling of a linear layer's output or input is just a rescaled weight matrix.

Reported configurations

Adapters (2019)
BERT-large, two adapters per block, bottleneck m∈{8,64,256}m\in\{8,64,256\} chosen per task; GLUE within 0.4% of full fine-tuning at 3.6% extra parameters per task
Prefix tuning (2021)
GPT-2 (table-to-text) and BART (summarisation); about 0.1% of parameters; MLP reparameterisation during training
Prompt tuning (2021)
T5 from Small to XXL (11B); prompt lengths 1 to 150 tried, gains flat beyond about 20; matches model tuning on SuperGLUE at XXL
BitFit (2022)
BERT-base and BERT-large; bias terms only, about 0.08% of parameters
LoRA (2022)
RoBERTa, DeBERTa, GPT-2 and GPT-3 175B; Wq,WvW_q,W_v at small rr; GPT-3 at r=4r=4: checkpoint 350 GB → about 35 MB, training memory 1.2 TB → 350 GB
IA³ (2022)
T0-3B with the T-Few recipe; three vectors per block, about 0.01% of parameters; initialised to ones

Memory accounting

Mfull=2N⏟bf16 weights+2N⏟grads+12N⏟fp32 copy, m, v=16N,MPEFT=2N+16NϕM_{full}=\underbrace{2N}_{\text{bf16 weights}}+\underbrace{2N}_{\text{grads}}+\underbrace{12N}_{\text{fp32 copy},\,m,\,v}=16N,\qquad M_{PEFT}=2N+16N_\phi
NN
parameters of the base model
NϕN_\phi
trainable parameters; each needs a weight, a gradient, an fp32 master copy and two Adam moments
12N12N
mixed-precision Adam state, as in the ZeRO paper: 4 bytes each for the master weight, mm and vv
activations\text{activations}
not included; they depend on batch and length, and PEFT still back-propagates through every frozen layer

The bytes per parameter explain most of the saving. Optimizer state is 12 of the 16 bytes, and PEFT removes it for all frozen weights. Once NϕN_\phi is negligible, memory approaches 2N2N, so this accounting caps the saving at 16/2=8×16/2=8\times. To go further you must shrink the frozen weights themselves, which is what QLoRA does with a 4-bit base. Activation memory is a separate cost, and gradient checkpointing is the usual way to reduce it.

Llama 2 7B: L = 32, d = 4096, d_ff = 11008, heads 32 (kv 32), N = 6,738,415,616methodtrainable params (log)count · % of Nweights + grads + AdamGB24 GB80 GB1K1M1B1Tfull FT6.74B · 100.00%108BitFitn/a: no biasesadapters33.82M · 0.50%14.02prefix5.24M · 0.0778%13.56prompt81.9K · 0.0012%13.48IA³614.4K · 0.0091%13.49LoRA4.19M · 0.0622%13.54params: selective · additive · reparameterizationmemory: ■ frozen 16-bit weights · ■ trainable weights + grads · ■ fp32 copy + Adam m, vLoRA: L·r·((d+d) + (d+d_kv)) = 32·8·(8192 + 8192)= 4,194,304 trainable (0.0622% of N, 1607× fewer)2N + 4N_t + 12N_t = 13.48 + 0.02 + 0.05 = 13.54 GB (8.0× less than full FT)checkpoint per task 8.39 MB · 100 tasks 838.86 MB + one shared base (13.48 GB)d_kv = 4096. Activations excluded: PEFT still back-propagates through the frozen model.Prefix counts the final K/V prefixes, not the reparameterization MLP used in training.
model
LoRA targets
fig.Trainable parameters (log scale) and weight + gradient + Adam memory for each method, computed from the model config. At the default (Llama 2 7B, LoRA r = 8 on W_q and W_v, adapter m = 64, prefix and prompt length 20), full fine-tuning trains all 6,738,415,616 weights and needs 107.81 GB. LoRA trains 4,194,304 (0.0622%, 1607× fewer) and needs 13.48 + 0.02 + 0.05 = 13.54 GB, 8.0× less. Prefix tuning trains 5,242,880 and IA³ 614,400. Prompt tuning trains 81,920 (0.0012%), and adapters train 33,820,672 (0.50%) for 14.02 GB. BitFit reads n/a because Llama has no bias vectors. Switch LoRA to all linear layers and it trains 19,988,480 (0.30%, 13.80 GB). At r = 64 on all linear layers it trains 159,907,840 (2.37%, 16.04 GB). On GPT-3 175B, full fine-tuning needs 2,794 GB and LoRA r = 8 trains 37,748,736 (0.0216%) for 349.81 GB. BitFit there trains 12,988,416 biases. On Llama 2 70B, grouped-query attention gives d_kv = 1024, so prefix tuning needs only 3,276,800 parameters. Every method's memory bar stops near 2N: the frozen weights set the floor.

Worked example: Llama 2 7B

Config: L=32L=32, d=4096d=4096, dff=11008d_{ff}=11008, 32 heads with no grouping (so dkv=4096d_{kv}=4096), vocabulary 32,000, SwiGLU MLP and no biases. The parameter count is 2⋅32000⋅4096+32 (4d2+3d dff+2d)+d=6,738,415,6162\cdot32000\cdot4096+32\,(4d^2+3d\,d_{ff}+2d)+d=6{,}738{,}415{,}616.

methodsettingtrainable% of Nmemory
full fine-tuningall weights6,738,415,616100107.81 GB
adaptersm = 64, 2 per block33,820,6720.50214.02 GB
prefix tuningp = 20 per layer5,242,8800.07813.56 GB
LoRAr = 8, W_q and W_v4,194,3040.06213.54 GB
IA³l_k, l_v, l_ff614,4000.00913.49 GB
prompt tuningp = 2081,9200.00113.48 GB
BitFitbiases0 (no biases)0n/a
Trainable parameters and memory (weights + gradients + Adam, no activations), computed in Python

How the numbers come out

  1. 1Adapters: one adapter is 2dm+m+d=2⋅4096⋅64+64+4096=528,4482dm+m+d=2\cdot4096\cdot64+64+4096=528{,}448. Two per block over 32 blocks gives 33,820,67233{,}820{,}672.
  2. 2Prefix: 2 p dkvL=2⋅20⋅4096⋅32=5,242,8802\,p\,d_{kv}L=2\cdot20\cdot4096\cdot32=5{,}242{,}880. Prompt: pd=20⋅4096=81,920pd=20\cdot4096=81{,}920, about 64 times fewer than prefix, because only the input layer gets new vectors.
  3. 3LoRA: each 4096×40964096\times4096 matrix gets r(din+dout)=8⋅8192=65,536r(d_{in}+d_{out})=8\cdot8192=65{,}536. Two matrices over 32 blocks gives 4,194,3044{,}194{,}304.
  4. 4IA³: L(2dkv+dff)=32 (8192+11008)=614,400L(2d_{kv}+d_{ff})=32\,(8192+11008)=614{,}400.
  5. 5Memory: full is 16N=107.8116N=107.81 GB. LoRA is 2N=13.4772N=13.477 GB, plus 16⋅4,194,30416\cdot4{,}194{,}304 bytes =0.067=0.067 GB, for 13.5413.54 GB in total.
python
1import torch
2from transformers import AutoModelForCausalLM
3from peft import (LoraConfig, IA3Config, PrefixTuningConfig, PromptTuningConfig,
4 TaskType, get_peft_model)
5
6BASE = "meta-llama/Llama-2-7b-hf"
7
8configs = {
9 # reparameterization: W0 + (alpha/r)·BA on the query and value projections
10 "lora": LoraConfig(task_type=TaskType.CAUSAL_LM, r=8, lora_alpha=16, lora_dropout=0.05,
11 target_modules=["q_proj", "v_proj"]),
12 # additive rescaling: l_k ⊙ k, l_v ⊙ v, l_ff ⊙ (input of down_proj)
13 "ia3": IA3Config(task_type=TaskType.CAUSAL_LM, target_modules=["k_proj", "v_proj", "down_proj"],
14 feedforward_modules=["down_proj"]),
15 # additive: 20 virtual keys/values in every layer
16 "prefix": PrefixTuningConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=20),
17 # additive: 20 soft-prompt embeddings at the input
18 "prompt": PromptTuningConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=20),
19}
20
21for name, cfg in configs.items():
22 base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
23 model = get_peft_model(base, cfg) # freezes the base, adds the new parameters
24 n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)
25 print(f"{name:7s} {n_train:>10,d} trainable")
26 # lora 4,194,304 ia3 614,400 prefix 5,242,880 prompt 81,920
27
28# training is ordinary: only parameters with requires_grad reach the optimizer
29model = get_peft_model(AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16), configs["lora"])
30model.gradient_checkpointing_enable() # activations are the other memory cost
31model.enable_input_require_grads() # needed with checkpointing and a frozen embedding
32opt = torch.optim.AdamW((p for p in model.parameters() if p.requires_grad), lr=2e-4)
33
34batch = {"input_ids": torch.randint(0, 32000, (2, 512))}
35loss = model(**batch, labels=batch["input_ids"]).loss
36loss.backward(); opt.step(); opt.zero_grad()
37
38model.save_pretrained("llama2-7b-lora") # ~8.4 MB of 16-bit adapter weights, not 13.5 GB
39merged = model.merge_and_unload() # W = W0 + (alpha/r)·BA: no extra inference latency
PEFT with Hugging Face peft: LoRA, IA³, prefix and prompt tuning on Llama 2 7B (not run here; the counts in comments come from the config formulas above)
methodfamilytrainsextra inference cost
adaptersadditivebottleneck MLPs after sub-layersextra sequential layers
prefix tuningadditivep keys and values per layerp extra attention positions
prompt tuningadditivep input embeddingsp tokens of context used
IA³additiverescaling vectors on k, v, FFNnone after merging
BitFitselectivebias vectorsnone
LoRAreparameterizationlow-rank BA per matrixnone after merging
The taxonomy at a glance

Use when

  • +One frozen base serves many tasks; each task is a checkpoint of megabytes
  • +Removes gradients and optimizer state for the base, up to an 8× cut in weight + optimizer memory
  • +Less forgetting of pretrained ability, since the base weights never change
  • +LoRA and IA³ merge into the weights and add no inference latency

Avoid when

  • −Activation memory and per-step compute stay close to full fine-tuning
  • −Can trail full fine-tuning on large distribution shifts, such as continued pretraining or a new language
  • −Prompt tuning is weak below about 10B parameters, and prompts and prefixes use context positions
  • −Adapters add sequential layers and inference latency; BitFit is impossible on bias-free models

Where it's used

Multi-tenant serving

One base model in GPU memory with many small LoRA adapters swapped in per request.

Fine-tuning on one GPU

LoRA on a 7B model fits in about 14 GB of weights and optimizer state. QLoRA brings a 65B model onto a single 48 GB card.

Instruction and preference tuning

Supervised fine-tuning (SFT) and Direct preference optimization (DPO) are often run with LoRA to cut cost and reduce forgetting.

Check yourself

Under the 16-bytes-per-trained-parameter accounting, why can no PEFT method cut the weight + gradient + Adam memory of full fine-tuning by more than about 8×?

Check yourself

You apply BitFit to Llama 2 7B and the trainer reports 0 trainable parameters. What happened?

Remember

  • ✓PEFT freezes θ\theta and trains ϕ\phi with ∣ϕ∣≪∣θ∣|\phi|\ll|\theta|. There are three families: additive (adapters, prefix, prompt, IA³), selective (BitFit) and reparameterization (LoRA).
  • ✓Trainable counts per block: adapter 2(2dm+m+d)2(2dm+m+d), prefix 2pdkv2pd_{kv}, LoRA r(din+dout)r(d_{in}+d_{out}) per matrix, IA³ 2dkv+dff2d_{kv}+d_{ff}. Prompt tuning adds pdpd in total.
  • ✓Memory is 16N16N for full fine-tuning and 2N+16Nϕ2N+16N_\phi for PEFT, so the saving caps near 8×. On Llama 2 7B that is 107.8 GB versus 13.5 GB, before activations.
  • ✓LoRA and IA³ merge into the weights at no inference cost. Adapters add layers, and prefixes and prompts use sequence positions.
  • ✓Prompt tuning only catches up with full fine-tuning at about 10B parameters. BitFit needs biases, which modern decoders lack.
← back to Deep Learning roadmap