Definition
LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning method of Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, "LoRA: Low-Rank Adaptation of Large Language Models" (Microsoft, arXiv 2021, ICLR 2022). It keeps a pretrained weight frozen and learns the update as a product of two thin matrices, with , and rank . The forward pass becomes . starts at zero, so training begins exactly at the pretrained model. After training, is added into and the model runs at its original speed.
Full fine-tuning of GPT-3 175B means storing a new 350 GB checkpoint per task and keeping Adam state for every one of its 175B parameters. Parameter-efficient fine-tuning (PEFT) methods before LoRA each paid a price elsewhere. Adapter layers (Houlsby et al. 2019) insert small bottleneck MLPs in series with each block, so every token at inference passes through extra sequential layers. Hu et al. measured noticeable latency increases for online, batch-size-1 serving. Prefix tuning and Prompt tuning spend part of the context window on learned vectors and were hard to optimise in their experiments. LoRA adds its matrices in parallel with existing weights, which removes the latency problem once they are merged.
The motivating hypothesis comes from work on intrinsic dimensionality. Li et al. (2018) trained networks inside random low-dimensional subspaces of their parameter space and found that surprisingly few dimensions suffice. Aghajanyan, Zettlemoyer and Gupta (2021) showed the same for fine-tuning: optimising only 200 parameters, randomly projected into the full space of RoBERTa-large, reached 90% of full fine-tuning performance on MRPC. Pretrained models seem to need only a small change to adapt. Hu et al. made the stronger, more specific bet that the weight update of each matrix has low rank, and parametrised it that way.
The mechanism
- pretrained weight, frozen: no gradient and no optimizer state
- down-projection to dimensions; random Gaussian init in the paper (Kaiming-uniform in the reference code and PEFT)
- up-projection back to ; initialised to zero, so at step 0
- fixed scale. The authors set to the first they tried and did not tune it: with Adam, changing acts like changing the learning rate
- the rank; 1–8 in the paper's GPT-3 experiments, 8–64 in common practice
The trainable count per adapted matrix is instead of . For square projections that is a fraction . GPT-3 has , so gives of each adapted matrix. Over and in all 96 layers that is parameters. The paper reports what this buys on GPT-3 175B: training VRAM falls from 1.2 TB to 350 GB, the per-task checkpoint from 350 GB to about 35 MB, and training throughput rises by 25%, because most weights need neither gradients nor Adam state.
Why zero-initialise B, and what α/r does
With the adapted model is the pretrained model at step 0, which is the right place to start fine-tuning. The two factors cannot both be zero. Then every gradient would vanish: and , where . With random and , the first step moves only . From the second step on, both factors learn. The worked example below shows this.
- gradient arriving at the layer output; the same signal full fine-tuning would use for
- sees the input through the random projection
- zero at initialisation, so is frozen for the first step
The factor keeps the update's size roughly constant when you change , so a learning rate tuned at one rank transfers to another. Kalajdzievski (2023, rsLoRA) argued that the right normaliser is . With , the update of a rank- adapter shrinks as grows and learning slows at high rank. With , larger ranks keep improving.
Which matrices, and how much rank
The paper restricts LoRA to the attention projections and freezes the MLPs. It then asks two questions on GPT-3 175B, each under a fixed budget of about 18M trainable parameters. The first is which projections to adapt. The answer is to spread the budget: alone is clearly worst, and adapting several matrices at a low rank beats one matrix at a high rank.
| adapted | rank r | WikiSQL | MultiNLI |
|---|---|---|---|
| W_q | 8 | 70.4 | 91.0 |
| W_k | 8 | 70.0 | 90.8 |
| W_v | 8 | 73.0 | 91.0 |
| W_o | 8 | 73.2 | 91.3 |
| W_q, W_k | 4 | 71.4 | 91.3 |
| W_q, W_v | 4 | 73.7 | 91.3 |
| W_q, W_k, W_v, W_o | 2 | 73.7 | 91.7 |
The second question is how much rank is needed. Surprisingly little. With and adapted, already scores 73.4 on WikiSQL and 91.7 on MultiNLI, and going up to brings no gain. The authors then compared the subspaces learned at and . The top singular directions overlap strongly, and the extra directions at look mostly like noise. They also found that correlates with but amplifies directions that are not emphasised in , by a factor of about 21.5 for in one layer. Their reading is that fine-tuning boosts task-specific features the pretrained model learned but did not use much. Later practice, for example QLoRA, adapts every linear layer including the MLPs, which helps when the task is far from pretraining.
From the paper
- Authors
- Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen (Microsoft), arXiv June 2021, ICLR 2022; reference code
loralib - GPT-3 175B
- 4.7M trainable params: WikiSQL 73.4, MNLI-m 91.7; 37.7M: 74.0, 91.6; full fine-tuning (175,255.8M): 73.8, 89.5
- Savings (GPT-3)
- on : VRAM 1.2 TB → 350 GB, checkpoint 350 GB → 35 MB (~10,000×), 25% faster training
- RoBERTa / DeBERTa
- GLUE average: RoBERTa-base 87.2 with 0.3M params vs 86.4 full FT; RoBERTa-large 89.0 (0.8M) vs 88.9; DeBERTa XXL 91.3 (4.7M) vs 91.1 (1.5B)
- GPT-2 medium
- E2E NLG BLEU 70.4 with 0.35M trainable params vs 68.2 for full fine-tuning
- Init and scale
- random Gaussian, ; output scaled by with fixed to the first tried
- Latency
- none after merging ; switch tasks by subtracting one and adding another
Worked example: one LoRA step, then merge
Take , , , so the scale is . Use squared loss , plain SGD with learning rate , input and target .
| matrix | value |
|---|---|
| W₀ (frozen) | [[1, 0, 2], [0, 1, −1], [1, 1, 0]] |
| A (1 × 3, random) | [0.5, −0.5, 1] |
| B (3 × 1) | [0, 0, 0]ᵀ |
| trainable | r(d + k) = 6 numbers, vs dk = 9 for ΔW |
Forward, backward, update, merge
- 1Forward: and . Since , , exactly the pretrained output. The loss is .
- 2Backward: . . , so does not move on the first step.
- 3Update: . New output . The loss drops from to .
- 4Second step: now , and , so both factors learn from here on.
- 5Merge (after step 1): . Then , identical to the two-path output, at the cost of one ordinary matmul.
1import math, torch, torch.nn as nn23class LoRALinear(nn.Module):4 """y = W0 x + (alpha/r) * B(A(dropout(x))), with W0 frozen."""5 def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.05):6 super().__init__()7 self.base, self.r, self.scale = base, r, alpha / r8 base.weight.requires_grad_(False)9 if base.bias is not None:10 base.bias.requires_grad_(False)11 self.A = nn.Parameter(torch.empty(r, base.in_features))12 self.B = nn.Parameter(torch.zeros(base.out_features, r)) # B = 0: start at W0 exactly13 nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) # paper: Gaussian; loralib/PEFT: this14 self.drop, self.merged = nn.Dropout(dropout), False1516 def forward(self, x):17 y = self.base(x)18 if not self.merged:19 y = y + (self.drop(x) @ self.A.T @ self.B.T) * self.scale20 return y2122 @torch.no_grad()23 def merge(self): # deploy: one matmul, zero added latency24 if not self.merged:25 self.base.weight += self.scale * (self.B @ self.A)26 self.merged = True2728 @torch.no_grad()29 def unmerge(self): # back to W0, e.g. to swap in another task's B, A30 if self.merged:31 self.base.weight -= self.scale * (self.B @ self.A)32 self.merged = False3334def add_lora(model, names=("q_proj", "v_proj"), r=8, alpha=16):35 for p in model.parameters():36 p.requires_grad_(False)37 for parent in list(model.modules()):38 for n, child in list(parent.named_children()):39 if n in names and isinstance(child, nn.Linear):40 setattr(parent, n, LoRALinear(child, r, alpha))41 return model4243# Same idea with PEFT. GPT-2 fuses q, k, v into one Conv1D "c_attn" (768 -> 2304):44from transformers import AutoModelForCausalLM45from peft import LoraConfig, get_peft_model4647model = AutoModelForCausalLM.from_pretrained("gpt2")48cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05, target_modules=["c_attn"],49 fan_in_fan_out=True, task_type="CAUSAL_LM")50model = get_peft_model(model, cfg)51model.print_trainable_parameters()52# 12 layers * 8 * (768 + 2304) = 294,912 trainable of 124,734,720 total (0.2364%)5354# ... train as usual, then fold the adapters in for serving:55model = model.merge_and_unload() # a plain GPT-2 again, with W0 + (alpha/r) BAUse when
- +Trains well under 1% of the parameters and needs no gradients or Adam state for the base
- +Zero added inference latency after merging; same architecture and kernels
- +Per-task checkpoints of megabytes; one base can serve many adapters
- +Matches or beats full fine-tuning on GLUE, E2E and GPT-3 tasks in the paper
Avoid when
- −The update is capped at rank r per matrix; tasks far from pretraining (new languages, long code training) can need more
- −Forward and backward still run the full model: activation memory and FLOPs barely fall
- −Batching requests for different adapters requires unmerged adapters and special kernels
- −Adds hyperparameters (r, α, target modules, dropout) whose best values vary by model
Where it's used
Instruction and chat fine-tuning
The default way to run Supervised fine-tuning (SFT) of open models such as Llama on one GPU, usually with QLoRA.
Multi-tenant serving
One frozen base in memory with hundreds of small task or customer adapters swapped in per request.
Diffusion models
Style and subject LoRAs for Stable Diffusion apply the same low-rank update to the attention projections of the UNet.
Check yourself
LoRA initialises A randomly and B to zero. What happens at the first optimisation step?
Check yourself
A 4096 × 4096 projection is adapted with r = 16. How many trainable parameters does LoRA add for that matrix, and what fraction of the matrix is that?
Remember
- ✓LoRA: with frozen, zero-initialised and random, so training starts at the pretrained model.
- ✓Each adapted matrix trains numbers instead of . On GPT-3 175B, on is 18.9M parameters, about 10,000× fewer, with a 35 MB checkpoint.
- ✓Fine-tuning updates are empirically low rank: r = 1 to 8 matches full fine-tuning on GPT-3, and spreading rank over several matrices beats a high rank on one.
- ✓Merging removes all inference overhead, unlike serial adapter layers.
- ✓Variants change the scale (rsLoRA, ), the learning rates (LoRA+) or the parametrisation (DoRA: magnitude times direction), and all still merge into one matrix.