Definition
Parameter-efficient fine-tuning (PEFT) adapts a pretrained network to a task by training a small parameter set with , while stays frozen and is shared across tasks. The model becomes . The main families are adapters (Houlsby et al., "Parameter-Efficient Transfer Learning for NLP", ICML 2019), prefix tuning (Li and Liang, ACL 2021), prompt tuning (Lester, Al-Rfou and Constant, EMNLP 2021), BitFit (Ben Zaken, Ravfogel and Goldberg, ACL 2022), LoRA (Hu et al., ICLR 2022) and IA³ (Liu et al., NeurIPS 2022). Lialin, Deshpande and Rumshisky ("Scaling Down to Scale Up", 2023) sort them into additive, selective and reparameterization-based methods.
Full fine-tuning updates every weight. For a 7B model that means a gradient and two Adam moments for 6.7 billion numbers, and a new 13.5 GB checkpoint for every task. Serving ten tasks means storing ten copies of the model. This was already a problem with BERT-large at 340M parameters. With GPT-3 at 175B it stopped being practical for most users.
Houlsby et al. framed the goal in 2019: one frozen model, plus a few new parameters per task. Their adapters came within 0.4% of full fine-tuning on GLUE while adding 3.6% of the parameters per task. Between 2021 and 2022, prefix tuning, prompt tuning, BitFit, LoRA and IA³ pushed the trained fraction down to 0.01–0.1%. He et al. ("Towards a Unified View of Parameter-Efficient Transfer Learning", ICLR 2022) showed that adapters, prefix tuning and LoRA can all be written as a learned modification added to some hidden state. They differ in where that modification is applied and how it is computed.
Where each method plugs in
Additive methods: new modules or new inputs
- output of an attention or FFN sub-layer, before the residual add
- bottleneck size; parameters per adapter, two adapters per block in Houlsby's design
- near-zero initialisation, so the adapter starts as the identity and the pretrained model is unchanged at step 0
Adapters add depth. Every token passes through the extra layers at inference, so they add latency that cannot be merged away. Pfeiffer et al. later showed that one adapter per block, after the FFN, works about as well. That halves the count.
- trained "virtual tokens" of keys and values per layer, parameters
- soft-prompt embeddings at the input only, parameters
- concatenation along the sequence axis; real tokens attend to the prefix
Prefix tuning learns continuous key and value vectors that every real token can attend to, in every layer. Optimising directly was unstable, so Li and Liang reparameterised it through an MLP during training, , and kept only the resulting afterwards. With about 0.1% of the parameters, it matched fine-tuning on table-to-text with GPT-2 and was close on summarisation with BART. See Prefix tuning.
Prompt tuning keeps only the input-level embeddings. That is the smallest possible footprint: 20 tokens on a 4096-wide model is 81,920 numbers. Lester et al. found that the gap to full fine-tuning closes with scale. On SuperGLUE it trailed badly for small T5 models but matched model tuning at T5-XXL (11B). See Prompt tuning.
IA³ rescales activations instead of adding to them. It learns three vectors per block, and , applied as , and . It is initialised to ones. Liu et al. paired it with the T-Few recipe on T0-3B and trained about 0.01% of the parameters, beating in-context learning with much larger models in the few-shot setting.
Selective and reparameterization methods
Selective methods unfreeze a subset of existing weights. BitFit trains only the bias vectors. On BERT-large that is about 0.08% of the parameters, and it was competitive with full fine-tuning on GLUE with small and medium data. The catch for LLMs is that Llama, Mistral and most recent decoders have no biases in their linear layers and use RMSNorm without a bias. On those models BitFit has nothing to train.
- frozen pretrained matrix
- rank; 1–64 in practice,
- at initialisation, so at step 0
- scaling, so the effective learning rate changes little when r changes
LoRA reparameterises the update of a weight matrix, not the activations. Because has the same shape as , it can be merged after training, , which leaves zero extra inference latency. On GPT-3 175B, Hu et al. reported 10,000 times fewer trainable parameters than full fine-tuning and 3 times less GPU memory, with checkpoints shrinking from 350 GB to about 35 MB at on . Quality was on par with full fine-tuning on WikiSQL, MNLI and SAMSum: slightly lower on one, higher on the other two. The details are in LoRA, and the 4-bit-base variant is in QLoRA. IA³ can be merged too, because a per-channel rescaling of a linear layer's output or input is just a rescaled weight matrix.
Reported configurations
- Adapters (2019)
- BERT-large, two adapters per block, bottleneck chosen per task; GLUE within 0.4% of full fine-tuning at 3.6% extra parameters per task
- Prefix tuning (2021)
- GPT-2 (table-to-text) and BART (summarisation); about 0.1% of parameters; MLP reparameterisation during training
- Prompt tuning (2021)
- T5 from Small to XXL (11B); prompt lengths 1 to 150 tried, gains flat beyond about 20; matches model tuning on SuperGLUE at XXL
- BitFit (2022)
- BERT-base and BERT-large; bias terms only, about 0.08% of parameters
- LoRA (2022)
- RoBERTa, DeBERTa, GPT-2 and GPT-3 175B; at small ; GPT-3 at : checkpoint 350 GB → about 35 MB, training memory 1.2 TB → 350 GB
- IA³ (2022)
- T0-3B with the T-Few recipe; three vectors per block, about 0.01% of parameters; initialised to ones
Memory accounting
- parameters of the base model
- trainable parameters; each needs a weight, a gradient, an fp32 master copy and two Adam moments
- mixed-precision Adam state, as in the ZeRO paper: 4 bytes each for the master weight, and
- not included; they depend on batch and length, and PEFT still back-propagates through every frozen layer
The bytes per parameter explain most of the saving. Optimizer state is 12 of the 16 bytes, and PEFT removes it for all frozen weights. Once is negligible, memory approaches , so this accounting caps the saving at . To go further you must shrink the frozen weights themselves, which is what QLoRA does with a 4-bit base. Activation memory is a separate cost, and gradient checkpointing is the usual way to reduce it.
Worked example: Llama 2 7B
Config: , , , 32 heads with no grouping (so ), vocabulary 32,000, SwiGLU MLP and no biases. The parameter count is .
| method | setting | trainable | % of N | memory |
|---|---|---|---|---|
| full fine-tuning | all weights | 6,738,415,616 | 100 | 107.81 GB |
| adapters | m = 64, 2 per block | 33,820,672 | 0.502 | 14.02 GB |
| prefix tuning | p = 20 per layer | 5,242,880 | 0.078 | 13.56 GB |
| LoRA | r = 8, W_q and W_v | 4,194,304 | 0.062 | 13.54 GB |
| IA³ | l_k, l_v, l_ff | 614,400 | 0.009 | 13.49 GB |
| prompt tuning | p = 20 | 81,920 | 0.001 | 13.48 GB |
| BitFit | biases | 0 (no biases) | 0 | n/a |
How the numbers come out
- 1Adapters: one adapter is . Two per block over 32 blocks gives .
- 2Prefix: . Prompt: , about 64 times fewer than prefix, because only the input layer gets new vectors.
- 3LoRA: each matrix gets . Two matrices over 32 blocks gives .
- 4IA³: .
- 5Memory: full is GB. LoRA is GB, plus bytes GB, for GB in total.
1import torch2from transformers import AutoModelForCausalLM3from peft import (LoraConfig, IA3Config, PrefixTuningConfig, PromptTuningConfig,4 TaskType, get_peft_model)56BASE = "meta-llama/Llama-2-7b-hf"78configs = {9 # reparameterization: W0 + (alpha/r)·BA on the query and value projections10 "lora": LoraConfig(task_type=TaskType.CAUSAL_LM, r=8, lora_alpha=16, lora_dropout=0.05,11 target_modules=["q_proj", "v_proj"]),12 # additive rescaling: l_k ⊙ k, l_v ⊙ v, l_ff ⊙ (input of down_proj)13 "ia3": IA3Config(task_type=TaskType.CAUSAL_LM, target_modules=["k_proj", "v_proj", "down_proj"],14 feedforward_modules=["down_proj"]),15 # additive: 20 virtual keys/values in every layer16 "prefix": PrefixTuningConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=20),17 # additive: 20 soft-prompt embeddings at the input18 "prompt": PromptTuningConfig(task_type=TaskType.CAUSAL_LM, num_virtual_tokens=20),19}2021for name, cfg in configs.items():22 base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)23 model = get_peft_model(base, cfg) # freezes the base, adds the new parameters24 n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)25 print(f"{name:7s} {n_train:>10,d} trainable")26 # lora 4,194,304 ia3 614,400 prefix 5,242,880 prompt 81,9202728# training is ordinary: only parameters with requires_grad reach the optimizer29model = get_peft_model(AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16), configs["lora"])30model.gradient_checkpointing_enable() # activations are the other memory cost31model.enable_input_require_grads() # needed with checkpointing and a frozen embedding32opt = torch.optim.AdamW((p for p in model.parameters() if p.requires_grad), lr=2e-4)3334batch = {"input_ids": torch.randint(0, 32000, (2, 512))}35loss = model(**batch, labels=batch["input_ids"]).loss36loss.backward(); opt.step(); opt.zero_grad()3738model.save_pretrained("llama2-7b-lora") # ~8.4 MB of 16-bit adapter weights, not 13.5 GB39merged = model.merge_and_unload() # W = W0 + (alpha/r)·BA: no extra inference latency| method | family | trains | extra inference cost |
|---|---|---|---|
| adapters | additive | bottleneck MLPs after sub-layers | extra sequential layers |
| prefix tuning | additive | p keys and values per layer | p extra attention positions |
| prompt tuning | additive | p input embeddings | p tokens of context used |
| IA³ | additive | rescaling vectors on k, v, FFN | none after merging |
| BitFit | selective | bias vectors | none |
| LoRA | reparameterization | low-rank BA per matrix | none after merging |
Use when
- +One frozen base serves many tasks; each task is a checkpoint of megabytes
- +Removes gradients and optimizer state for the base, up to an 8× cut in weight + optimizer memory
- +Less forgetting of pretrained ability, since the base weights never change
- +LoRA and IA³ merge into the weights and add no inference latency
Avoid when
- −Activation memory and per-step compute stay close to full fine-tuning
- −Can trail full fine-tuning on large distribution shifts, such as continued pretraining or a new language
- −Prompt tuning is weak below about 10B parameters, and prompts and prefixes use context positions
- −Adapters add sequential layers and inference latency; BitFit is impossible on bias-free models
Where it's used
Multi-tenant serving
One base model in GPU memory with many small LoRA adapters swapped in per request.
Fine-tuning on one GPU
LoRA on a 7B model fits in about 14 GB of weights and optimizer state. QLoRA brings a 65B model onto a single 48 GB card.
Instruction and preference tuning
Supervised fine-tuning (SFT) and Direct preference optimization (DPO) are often run with LoRA to cut cost and reduce forgetting.
Check yourself
Under the 16-bytes-per-trained-parameter accounting, why can no PEFT method cut the weight + gradient + Adam memory of full fine-tuning by more than about 8×?
Check yourself
You apply BitFit to Llama 2 7B and the trainer reports 0 trainable parameters. What happened?
Remember
- ✓PEFT freezes and trains with . There are three families: additive (adapters, prefix, prompt, IA³), selective (BitFit) and reparameterization (LoRA).
- ✓Trainable counts per block: adapter , prefix , LoRA per matrix, IA³ . Prompt tuning adds in total.
- ✓Memory is for full fine-tuning and for PEFT, so the saving caps near 8×. On Llama 2 7B that is 107.8 GB versus 13.5 GB, before activations.
- ✓LoRA and IA³ merge into the weights at no inference cost. Adapters add layers, and prefixes and prompts use sequence positions.
- ✓Prompt tuning only catches up with full fine-tuning at about 10B parameters. BitFit needs biases, which modern decoders lack.