# Parameter-efficient fine-tuning (PEFT)

> PEFT freezes a pretrained model and trains a few new or selected weights: adapters, prefixes, soft prompts, IA³ vectors, biases or low-rank LoRA updates. On Llama 2 7B, LoRA r = 8 on W_q, W_v trains 4.19M of 6.74B weights (0.062%), cutting weight + Adam memory from 108 to 13.5 GB.

Source: https://www.mlfoundry.in/learn/parameter-efficient-fine-tuning-peft  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Parameter-efficient fine-tuning (PEFT):** adapts a pretrained network $f_\theta$ to a task by training a small parameter set $\phi$ with $|\phi|\ll|\theta|$, while $\theta$ stays frozen and is shared across tasks. The model becomes $f_{\theta,\phi}$. The main families are **adapters** (Houlsby et al., "Parameter-Efficient Transfer Learning for NLP", ICML 2019), **prefix tuning** (Li and Liang, ACL 2021), **prompt tuning** (Lester, Al-Rfou and Constant, EMNLP 2021), **BitFit** (Ben Zaken, Ravfogel and Goldberg, ACL 2022), **LoRA** (Hu et al., ICLR 2022) and **IA³** (Liu et al., NeurIPS 2022). Lialin, Deshpande and Rumshisky ("Scaling Down to Scale Up", 2023) sort them into **additive**, **selective** and **reparameterization-based** methods.

Full fine-tuning updates every weight. For a 7B model that means a gradient and two Adam moments for 6.7 billion numbers, and a new 13.5 GB checkpoint for every task. Serving ten tasks means storing ten copies of the model. This was already a problem with BERT-large at 340M parameters. With GPT-3 at 175B it stopped being practical for most users.

Houlsby et al. framed the goal in 2019: one frozen model, plus a few new parameters per task. Their adapters came within 0.4% of full fine-tuning on GLUE while adding 3.6% of the parameters per task. Between 2021 and 2022, prefix tuning, prompt tuning, BitFit, LoRA and IA³ pushed the trained fraction down to 0.01–0.1%. He et al. ("Towards a Unified View of Parameter-Efficient Transfer Learning", ICLR 2022) showed that adapters, prefix tuning and LoRA can all be written as a learned modification added to some hidden state. They differ in where that modification is applied and how it is computed.

## Where each method plugs in

> Figure: One transformer block. Everything in grey is frozen and shared across tasks. The dashed parts are the only weights a method trains: soft prompts at the input (prompt tuning), virtual keys and values in every layer (prefix tuning), low-rank updates B·A beside W_q and W_v (LoRA), rescaling vectors on k, v and the FFN hidden state (IA³), bottleneck adapters after each sub-layer, and the bias vectors +b (BitFit, only in models that have biases). Hover a family to isolate it.

## Additive methods: new modules or new inputs

$$
\text{Adapter}(h)=h+W_{up}\,\sigma(W_{down}h+b_{down})+b_{up},\qquad W_{down}\in\mathbb R^{m\times d},\ W_{up}\in\mathbb R^{d\times m},\ m\ll d
$$

- $h$: output of an attention or FFN sub-layer, before the residual add
- $m$: bottleneck size; $2dm+d+m$ parameters per adapter, two adapters per block in Houlsby's design
- $W_{up}\approx0$: near-zero initialisation, so the adapter starts as the identity and the pretrained model is unchanged at step 0

Adapters add depth. Every token passes through the extra layers at inference, so they add latency that cannot be merged away. Pfeiffer et al. later showed that one adapter per block, after the FFN, works about as well. That halves the count.

$$
\text{prefix: }\ \mathrm{Attn}\big(q,\,[P_K;K],\,[P_V;V]\big)\ \text{in every layer},\qquad \text{prompt: }\ h^{(0)}=[P;\,E[x]],\ P\in\mathbb R^{p\times d}
$$

- $P_K,P_V\in\mathbb R^{p\times d_{kv}}$: $p$ trained "virtual tokens" of keys and values per layer, $2pd_{kv}L$ parameters
- $P$: $p$ soft-prompt embeddings at the input only, $pd$ parameters
- $[\,\cdot\,;\,\cdot\,]$: concatenation along the sequence axis; real tokens attend to the prefix

**Prefix tuning** learns continuous key and value vectors that every real token can attend to, in every layer. Optimising $P$ directly was unstable, so Li and Liang reparameterised it through an MLP during training, $P=\mathrm{MLP}(P')$, and kept only the resulting $P$ afterwards. With about 0.1% of the parameters, it matched fine-tuning on table-to-text with GPT-2 and was close on summarisation with BART. See **Prefix tuning**.

**Prompt tuning** keeps only the input-level embeddings. That is the smallest possible footprint: 20 tokens on a 4096-wide model is 81,920 numbers. Lester et al. found that the gap to full fine-tuning closes with scale. On SuperGLUE it trailed badly for small T5 models but matched model tuning at T5-XXL (11B). See **Prompt tuning**.

**IA³** rescales activations instead of adding to them. It learns three vectors per block, $l_k,l_v\in\mathbb R^{d_{kv}}$ and $l_{ff}\in\mathbb R^{d_{ff}}$, applied as $l_k\odot k$, $l_v\odot v$ and $l_{ff}\odot\gamma(W_1x)$. It is initialised to ones. Liu et al. paired it with the T-Few recipe on T0-3B and trained about 0.01% of the parameters, beating in-context learning with much larger models in the few-shot setting.

## Selective and reparameterization methods

**Selective** methods unfreeze a subset of existing weights. BitFit trains only the bias vectors. On BERT-large that is about 0.08% of the parameters, and it was competitive with full fine-tuning on GLUE with small and medium data. The catch for LLMs is that Llama, Mistral and most recent decoders have **no biases** in their linear layers and use RMSNorm without a bias. On those models BitFit has nothing to train.

$$
h=W_0x+\Delta W x=W_0x+\tfrac{\alpha}{r}BAx,\qquad B\in\mathbb R^{d_{out}\times r},\ A\in\mathbb R^{r\times d_{in}},\qquad |\phi|=r\,(d_{in}+d_{out})\ \text{per matrix}
$$

- $W_0$: frozen pretrained matrix
- $r$: rank; 1–64 in practice, $\ll\min(d_{in},d_{out})$
- $B=0$: at initialisation, so $\Delta W=0$ at step 0
- $\alpha/r$: scaling, so the effective learning rate changes little when r changes

**LoRA** reparameterises the update of a weight matrix, not the activations. Because $BA$ has the same shape as $W_0$, it can be **merged** after training, $W=W_0+\tfrac{\alpha}{r}BA$, which leaves zero extra inference latency. On GPT-3 175B, Hu et al. reported 10,000 times fewer trainable parameters than full fine-tuning and 3 times less GPU memory, with checkpoints shrinking from 350 GB to about 35 MB at $r=4$ on $W_q,W_v$. Quality was on par with full fine-tuning on WikiSQL, MNLI and SAMSum: slightly lower on one, higher on the other two. The details are in **LoRA**, and the 4-bit-base variant is in **QLoRA**. IA³ can be merged too, because a per-channel rescaling of a linear layer's output or input is just a rescaled weight matrix.

**Reported configurations**

- **Adapters (2019):** BERT-large, two adapters per block, bottleneck $m\in\{8,64,256\}$ chosen per task; GLUE within 0.4% of full fine-tuning at 3.6% extra parameters per task
- **Prefix tuning (2021):** GPT-2 (table-to-text) and BART (summarisation); about 0.1% of parameters; MLP reparameterisation during training
- **Prompt tuning (2021):** T5 from Small to XXL (11B); prompt lengths 1 to 150 tried, gains flat beyond about 20; matches model tuning on SuperGLUE at XXL
- **BitFit (2022):** BERT-base and BERT-large; bias terms only, about 0.08% of parameters
- **LoRA (2022):** RoBERTa, DeBERTa, GPT-2 and GPT-3 175B; $W_q,W_v$ at small $r$; GPT-3 at $r=4$: checkpoint 350 GB → about 35 MB, training memory 1.2 TB → 350 GB
- **IA³ (2022):** T0-3B with the T-Few recipe; three vectors per block, about 0.01% of parameters; initialised to ones

## Memory accounting

$$
M_{full}=\underbrace{2N}_{\text{bf16 weights}}+\underbrace{2N}_{\text{grads}}+\underbrace{12N}_{\text{fp32 copy},\,m,\,v}=16N,\qquad M_{PEFT}=2N+16N_\phi
$$

- $N$: parameters of the base model
- $N_\phi$: trainable parameters; each needs a weight, a gradient, an fp32 master copy and two Adam moments
- $12N$: mixed-precision Adam state, as in the ZeRO paper: 4 bytes each for the master weight, $m$ and $v$
- $\text{activations}$: not included; they depend on batch and length, and PEFT still back-propagates through every frozen layer

The bytes per parameter explain most of the saving. Optimizer state is 12 of the 16 bytes, and PEFT removes it for all frozen weights. Once $N_\phi$ is negligible, memory approaches $2N$, so this accounting caps the saving at $16/2=8\times$. To go further you must shrink the frozen weights themselves, which is what **QLoRA** does with a 4-bit base. Activation memory is a separate cost, and gradient checkpointing is the usual way to reduce it.

> Figure: Trainable parameters (log scale) and weight + gradient + Adam memory for each method, computed from the model config. At the default (Llama 2 7B, LoRA r = 8 on W_q and W_v, adapter m = 64, prefix and prompt length 20), full fine-tuning trains all 6,738,415,616 weights and needs 107.81 GB. LoRA trains 4,194,304 (0.0622%, 1607× fewer) and needs 13.48 + 0.02 + 0.05 = 13.54 GB, 8.0× less. Prefix tuning trains 5,242,880 and IA³ 614,400. Prompt tuning trains 81,920 (0.0012%), and adapters train 33,820,672 (0.50%) for 14.02 GB. BitFit reads n/a because Llama has no bias vectors. Switch LoRA to all linear layers and it trains 19,988,480 (0.30%, 13.80 GB). At r = 64 on all linear layers it trains 159,907,840 (2.37%, 16.04 GB). On GPT-3 175B, full fine-tuning needs 2,794 GB and LoRA r = 8 trains 37,748,736 (0.0216%) for 349.81 GB. BitFit there trains 12,988,416 biases. On Llama 2 70B, grouped-query attention gives d_kv = 1024, so prefix tuning needs only 3,276,800 parameters. Every method's memory bar stops near 2N: the frozen weights set the floor.

## Worked example: Llama 2 7B

Config: $L=32$, $d=4096$, $d_{ff}=11008$, 32 heads with no grouping (so $d_{kv}=4096$), vocabulary 32,000, SwiGLU MLP and no biases. The parameter count is $2\cdot32000\cdot4096+32\,(4d^2+3d\,d_{ff}+2d)+d=6{,}738{,}415{,}616$.

Trainable parameters and memory (weights + gradients + Adam, no activations), computed in Python

| method | setting | trainable | % of N | memory |
| --- | --- | --- | --- | --- |
| full fine-tuning | all weights | 6,738,415,616 | 100 | 107.81 GB |
| adapters | m = 64, 2 per block | 33,820,672 | 0.502 | 14.02 GB |
| prefix tuning | p = 20 per layer | 5,242,880 | 0.078 | 13.56 GB |
| LoRA | r = 8, W_q and W_v | 4,194,304 | 0.062 | 13.54 GB |
| IA³ | l_k, l_v, l_ff | 614,400 | 0.009 | 13.49 GB |
| prompt tuning | p = 20 | 81,920 | 0.001 | 13.48 GB |
| BitFit | biases | 0 (no biases) | 0 | n/a |

**How the numbers come out**

1. **Adapters**: one adapter is $2dm+m+d=2\cdot4096\cdot64+64+4096=528{,}448$. Two per block over 32 blocks gives $33{,}820{,}672$.
2. **Prefix**: $2\,p\,d_{kv}L=2\cdot20\cdot4096\cdot32=5{,}242{,}880$. **Prompt**: $pd=20\cdot4096=81{,}920$, about 64 times fewer than prefix, because only the input layer gets new vectors.
3. **LoRA**: each $4096\times4096$ matrix gets $r(d_{in}+d_{out})=8\cdot8192=65{,}536$. Two matrices over 32 blocks gives $4{,}194{,}304$.
4. **IA³**: $L(2d_{kv}+d_{ff})=32\,(8192+11008)=614{,}400$.
5. **Memory**: full is $16N=107.81$ GB. LoRA is $2N=13.477$ GB, plus $16\cdot4{,}194{,}304$ bytes $=0.067$ GB, for $13.54$ GB in total.

```python
import torch
from transformers import AutoModelForCausalLM
from peft import (LoraConfig, IA3Config, PrefixTuningConfig, PromptTuningConfig,
                  TaskType, get_peft_model)

BASE = "meta-llama/Llama-2-7b-hf"

configs = {
    # reparameterization: W0 + (alpha/r)·BA on the query and value projections
    "lora":   LoraConfig(task_type=TaskType.CAUSAL_LM, r=8, lora_alpha=16, lora_dropout=0.05,
                         target_modules=["q_proj", "v_proj"]),
    # additive rescaling: l_k ⊙ k, l_v ⊙ v, l_ff ⊙ (input of down_proj)
    "ia3":    IA3Config(task_type=TaskType.CAUSAL_LM, target_modules=["k_proj", "v_proj", "down_proj"],
                        feedforward_modules=["down_proj"]),
    # additive: 20 virtual keys/values in every layer
    "prefix": PrefixTuningConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=20),
    # additive: 20 soft-prompt embeddings at the input
    "prompt": PromptTuningConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=20),
}

for name, cfg in configs.items():
    base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
    model = get_peft_model(base, cfg)            # freezes the base, adds the new parameters
    n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f"{name:7s} {n_train:>10,d} trainable")
    # lora     4,194,304   ia3 614,400   prefix 5,242,880   prompt 81,920

# training is ordinary: only parameters with requires_grad reach the optimizer
model = get_peft_model(AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16), configs["lora"])
model.gradient_checkpointing_enable()            # activations are the other memory cost
model.enable_input_require_grads()               # needed with checkpointing and a frozen embedding
opt = torch.optim.AdamW((p for p in model.parameters() if p.requires_grad), lr=2e-4)

batch = {"input_ids": torch.randint(0, 32000, (2, 512))}
loss = model(**batch, labels=batch["input_ids"]).loss
loss.backward(); opt.step(); opt.zero_grad()

model.save_pretrained("llama2-7b-lora")          # ~8.4 MB of 16-bit adapter weights, not 13.5 GB
merged = model.merge_and_unload()                # W = W0 + (alpha/r)·BA: no extra inference latency
```

PEFT with Hugging Face peft: LoRA, IA³, prefix and prompt tuning on Llama 2 7B (not run here; the counts in comments come from the config formulas above)

The taxonomy at a glance

| method | family | trains | extra inference cost |
| --- | --- | --- | --- |
| adapters | additive | bottleneck MLPs after sub-layers | extra sequential layers |
| prefix tuning | additive | p keys and values per layer | p extra attention positions |
| prompt tuning | additive | p input embeddings | p tokens of context used |
| IA³ | additive | rescaling vectors on k, v, FFN | none after merging |
| BitFit | selective | bias vectors | none |
| LoRA | reparameterization | low-rank BA per matrix | none after merging |

> **Optimizer state is the real cost:** Mixed-precision Adam stores 16 bytes per trained parameter but only 2 bytes per frozen one. That 16 : 2 ratio, not the parameter count, sets the size of the saving. Going from 4.2M to 0.6M trainable parameters on Llama 2 7B changes memory from 13.54 to 13.49 GB. Below about 1% trainable, picking a method by its parameter count is pointless for memory. Choose by quality, latency and how easily it merges.

> **PEFT does not shrink activations:** Gradients still flow back through every frozen layer to reach the adapters, so activation memory is about what full fine-tuning needs. At long contexts it can exceed the weights. Gradient checkpointing, shorter sequences or smaller micro-batches are the levers. Prompt and prefix tuning also lengthen the sequence by $p$ positions.

> **Check the architecture before choosing:** BitFit needs biases. Prefix size depends on $d_{kv}$, so grouped-query models such as Llama 2 70B get prefixes 8 times smaller than full multi-head attention would. LoRA's target list matters more than its rank: on Llama 2 7B, adding the MLP and $W_k,W_o$ matrices raises $r=8$ from 4.2M to 20.0M trainable parameters, and later work generally found that adapting all linear layers helps quality.

**Strengths**

- One frozen base serves many tasks; each task is a checkpoint of megabytes
- Removes gradients and optimizer state for the base, up to an 8× cut in weight + optimizer memory
- Less forgetting of pretrained ability, since the base weights never change
- LoRA and IA³ merge into the weights and add no inference latency

**Limitations**

- Activation memory and per-step compute stay close to full fine-tuning
- Can trail full fine-tuning on large distribution shifts, such as continued pretraining or a new language
- Prompt tuning is weak below about 10B parameters, and prompts and prefixes use context positions
- Adapters add sequential layers and inference latency; BitFit is impossible on bias-free models

- **Multi-tenant serving:** One base model in GPU memory with many small LoRA adapters swapped in per request.
- **Fine-tuning on one GPU:** LoRA on a 7B model fits in about 14 GB of weights and optimizer state. **QLoRA** brings a 65B model onto a single 48 GB card.
- **Instruction and preference tuning:** **Supervised fine-tuning (SFT)** and **Direct preference optimization (DPO)** are often run with LoRA to cut cost and reduce forgetting.

**Question:** Under the 16-bytes-per-trained-parameter accounting, why can no PEFT method cut the weight + gradient + Adam memory of full fine-tuning by more than about 8×?

- A. Adam needs at least 8 steps of history
- B. The frozen base weights still take 2 bytes each, and full fine-tuning takes 16
- C. LoRA matrices are stored in fp32
- D. Gradients of the frozen weights are still computed and stored

**Answer:** B. The frozen base weights still take 2 bytes each, and full fine-tuning takes 16 — $M_{PEFT}=2N+16N_\phi\to2N$ as $N_\phi\to0$, and $M_{full}=16N$. The floor is the frozen copy of the model. **QLoRA** lowers it by storing the base in 4 bits. Frozen weights get no stored gradient: autograd computes activation gradients through them, but not weight gradients.

**Question:** You apply BitFit to Llama 2 7B and the trainer reports 0 trainable parameters. What happened?

- A. BitFit only works on encoders
- B. The biases were frozen by default
- C. Llama's linear layers have no biases and RMSNorm has no shift, so there are no bias vectors to train
- D. The learning rate was too low

**Answer:** C. Llama's linear layers have no biases and RMSNorm has no shift, so there are no bias vectors to train — BitFit selects existing bias vectors. BERT and GPT-3 have them in every projection and LayerNorm (271,360 in RoBERTa-large, 12,988,416 in GPT-3 175B). Llama-family models dropped them, so a selective bias-only method has nothing to select.

**Key points**

- PEFT freezes $\theta$ and trains $\phi$ with $|\phi|\ll|\theta|$. There are three families: additive (adapters, prefix, prompt, IA³), selective (BitFit) and reparameterization (LoRA).
- Trainable counts per block: adapter $2(2dm+m+d)$, prefix $2pd_{kv}$, LoRA $r(d_{in}+d_{out})$ per matrix, IA³ $2d_{kv}+d_{ff}$. Prompt tuning adds $pd$ in total.
- Memory is $16N$ for full fine-tuning and $2N+16N_\phi$ for PEFT, so the saving caps near 8×. On Llama 2 7B that is 107.8 GB versus 13.5 GB, before activations.
- LoRA and IA³ merge into the weights at no inference cost. Adapters add layers, and prefixes and prompts use sequence positions.
- Prompt tuning only catches up with full fine-tuning at about 10B parameters. BitFit needs biases, which modern decoders lack.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
