LoRA

LoRA (Hu et al., 2021) freezes a pretrained weight W₀ and trains a low-rank update: h = W₀x + (α/r)·BAx, B zero-initialised, r ≪ d. Rank 4 on GPT-3's W_q and W_v trains 18.9M parameters, about 10,000× fewer than full fine-tuning, and BA merges into W₀ so inference costs nothing extra.

15 min read1137 wordscore2 viewsupdated 2026-10-13

Definition

LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning method of Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, "LoRA: Low-Rank Adaptation of Large Language Models" (Microsoft, arXiv 2021, ICLR 2022). It keeps a pretrained weight W0∈Rd×kW_0\in\mathbb R^{d\times k} frozen and learns the update as a product of two thin matrices, ΔW=BA\Delta W=BA with B∈Rd×rB\in\mathbb R^{d\times r}, A∈Rr×kA\in\mathbb R^{r\times k} and rank r≪min⁡(d,k)r\ll\min(d,k). The forward pass becomes h=W0x+αrBAxh=W_0x+\tfrac{\alpha}{r}BAx. BB starts at zero, so training begins exactly at the pretrained model. After training, BABA is added into W0W_0 and the model runs at its original speed.

Full fine-tuning of GPT-3 175B means storing a new 350 GB checkpoint per task and keeping Adam state for every one of its 175B parameters. Parameter-efficient fine-tuning (PEFT) methods before LoRA each paid a price elsewhere. Adapter layers (Houlsby et al. 2019) insert small bottleneck MLPs in series with each block, so every token at inference passes through extra sequential layers. Hu et al. measured noticeable latency increases for online, batch-size-1 serving. Prefix tuning and Prompt tuning spend part of the context window on learned vectors and were hard to optimise in their experiments. LoRA adds its matrices in parallel with existing weights, which removes the latency problem once they are merged.

The motivating hypothesis comes from work on intrinsic dimensionality. Li et al. (2018) trained networks inside random low-dimensional subspaces of their parameter space and found that surprisingly few dimensions suffice. Aghajanyan, Zettlemoyer and Gupta (2021) showed the same for fine-tuning: optimising only 200 parameters, randomly projected into the full space of RoBERTa-large, reached 90% of full fine-tuning performance on MRPC. Pretrained models seem to need only a small change to adapt. Hu et al. made the stronger, more specific bet that the weight update of each matrix has low rank, and parametrised it that way.

The mechanism

h=W0x+ΔW x=W0x+αr B (A x),B∈Rd×r,  A∈Rr×k,  r≪min⁡(d,k)h=W_0x+\Delta W\,x=W_0x+\frac{\alpha}{r}\,B\,(A\,x),\qquad B\in\mathbb R^{d\times r},\;A\in\mathbb R^{r\times k},\;r\ll\min(d,k)
W0W_0
pretrained weight, frozen: no gradient and no optimizer state
AA
down-projection to rr dimensions; random Gaussian init in the paper (Kaiming-uniform in the reference code and PEFT)
BB
up-projection back to dd; initialised to zero, so ΔW=BA=0\Delta W=BA=0 at step 0
α/r\alpha/r
fixed scale. The authors set α\alpha to the first rr they tried and did not tune it: with Adam, changing α\alpha acts like changing the learning rate
rr
the rank; 1–8 in the paper's GPT-3 experiments, 8–64 in common practice
training: W₀ frozen, only A and B receive gradientsh = W₀x + (α/r)·B(Ax)+W₀d × d, pretrainedfrozen (no grad,no Adam state)B (d × r)init 0A (r × d)N(0, σ²)r ≪ d× α / rx (d), one token's hidden stateΔW = BA = 0 at step 0: training starts at thepretrained model exactlyattention (per layer)MLP (SwiGLU; GPT has 2)W_qW_kW_vW_oW_upW_gateW_downHu et al. on GPT-3: W_q and W_vQLoRA and later: all linearone GPT-3 W_q, d = 12,288, r = 4frozen d² = 150,994,944trainable 2dr = 98,304 (0.065%)× 2 matrices × 96 layers = 18.9Mserving many tasksW₀ ± (α/r)BA swaps a task in place;a 35 MB file per task vs 350 GB
view
fig.LoRA in one linear layer. The input feeds the frozen W₀ and, in parallel, a down-projection A (r × d) and an up-projection B (d × r, zero-initialised). Their product is scaled by α/r and added to the output. Toggle to the merged view: at deployment BA is folded into W₀ once, the matrix keeps its shape, and the forward pass is the original single matmul. On the right are the matrices it can target. Hu et al. adapted only W_q and W_v of GPT-3; at d = 12,288 and r = 4 one matrix trains 98,304 numbers instead of 150,994,944.

The trainable count per adapted matrix is r(d+k)r(d+k) instead of dkdk. For square d×dd\times d projections that is a fraction 2r/d2r/d. GPT-3 has d=12288d=12288, so r=4r=4 gives 2⋅4/12288=0.065%2\cdot4/12288=0.065\% of each adapted matrix. Over WqW_q and WvW_v in all 96 layers that is 96⋅2⋅2⋅12288⋅4=18,874,36896\cdot2\cdot2\cdot12288\cdot4=18{,}874{,}368 parameters. The paper reports what this buys on GPT-3 175B: training VRAM falls from 1.2 TB to 350 GB, the per-task checkpoint from 350 GB to about 35 MB, and training throughput rises by 25%, because most weights need neither gradients nor Adam state.

Why zero-initialise B, and what α/r does

With B=0B=0 the adapted model is the pretrained model at step 0, which is the right place to start fine-tuning. The two factors cannot both be zero. Then every gradient would vanish: ∂L/∂B=αr g (Ax)⊤\partial\mathcal L/\partial B=\tfrac{\alpha}{r}\,g\,(Ax)^\top and ∂L/∂A=αr B⊤g x⊤\partial\mathcal L/\partial A=\tfrac{\alpha}{r}\,B^\top g\,x^\top, where g=∂L/∂hg=\partial\mathcal L/\partial h. With AA random and B=0B=0, the first step moves only BB. From the second step on, both factors learn. The worked example below shows this.

∂L∂B=αr g (Ax)⊤,∂L∂A=αr B⊤g x⊤,g=∂L∂h\frac{\partial\mathcal L}{\partial B}=\frac{\alpha}{r}\,g\,(Ax)^\top,\qquad \frac{\partial\mathcal L}{\partial A}=\frac{\alpha}{r}\,B^\top g\,x^\top,\qquad g=\frac{\partial\mathcal L}{\partial h}
gg
gradient arriving at the layer output; the same signal full fine-tuning would use for ∂L/∂W=g x⊤\partial\mathcal L/\partial W=g\,x^\top
(Ax)⊤(Ax)^\top
BB sees the input through the random projection AA
B⊤gB^\top g
zero at initialisation, so AA is frozen for the first step

The α/r\alpha/r factor keeps the update's size roughly constant when you change rr, so a learning rate tuned at one rank transfers to another. Kalajdzievski (2023, rsLoRA) argued that the right normaliser is α/r\alpha/\sqrt r. With α/r\alpha/r, the update of a rank-rr adapter shrinks as rr grows and learning slows at high rank. With α/r\alpha/\sqrt r, larger ranks keep improving.

Which matrices, and how much rank

The paper restricts LoRA to the attention projections and freezes the MLPs. It then asks two questions on GPT-3 175B, each under a fixed budget of about 18M trainable parameters. The first is which projections to adapt. The answer is to spread the budget: WqW_q alone is clearly worst, and adapting several matrices at a low rank beats one matrix at a high rank.

adaptedrank rWikiSQLMultiNLI
W_q870.491.0
W_k870.090.8
W_v873.091.0
W_o873.291.3
W_q, W_k471.491.3
W_q, W_v473.791.3
W_q, W_k, W_v, W_o273.791.7
Hu et al., Table 5: GPT-3 175B with an 18M-parameter LoRA budget (validation accuracy, %)

The second question is how much rank is needed. Surprisingly little. With WqW_q and WvW_v adapted, r=1r=1 already scores 73.4 on WikiSQL and 91.7 on MultiNLI, and going up to r=64r=64 brings no gain. The authors then compared the subspaces learned at r=8r=8 and r=64r=64. The top singular directions overlap strongly, and the extra directions at r=64r=64 look mostly like noise. They also found that ΔW\Delta W correlates with W0W_0 but amplifies directions that are not emphasised in W0W_0, by a factor of about 21.5 for r=4r=4 in one layer. Their reading is that fine-tuning boosts task-specific features the pretrained model learned but did not use much. Later practice, for example QLoRA, adapts every linear layer including the MLPs, which helps when the task is far from pretraining.

From the paper

Authors
Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen (Microsoft), arXiv June 2021, ICLR 2022; reference code loralib
GPT-3 175B
4.7M trainable params: WikiSQL 73.4, MNLI-m 91.7; 37.7M: 74.0, 91.6; full fine-tuning (175,255.8M): 73.8, 89.5
Savings (GPT-3)
r=4r=4 on Wq,WvW_q,W_v: VRAM 1.2 TB → 350 GB, checkpoint 350 GB → 35 MB (~10,000×), 25% faster training
RoBERTa / DeBERTa
GLUE average: RoBERTa-base 87.2 with 0.3M params vs 86.4 full FT; RoBERTa-large 89.0 (0.8M) vs 88.9; DeBERTa XXL 91.3 (4.7M) vs 91.1 (1.5B)
GPT-2 medium
E2E NLG BLEU 70.4 with 0.35M trainable params vs 68.2 for full fine-tuning
Init and scale
AA random Gaussian, B=0B=0; output scaled by α/r\alpha/r with α\alpha fixed to the first rr tried
Latency
none after merging W=W0+αrBAW=W_0+\tfrac{\alpha}{r}BA; switch tasks by subtracting one BABA and adding another
ΔW ≈ B·A with B = U_r S_r^½ (32×4), A = S_r^½ V_rᵀ (4×32): the best rank-4 fitΔW (low intrinsic rank)B·A, rank 4residual ΔW − B·Aenergy captured90.46%‖ΔW − BA‖_F / ‖ΔW‖_F0.3089params, full ΔW1,024params, B and A256 (25.0%)singular values s_i / s₁ (bars) · cumulative energy0.51r = 418162432GPT-3 175BL 96 · d 12,288adapt W_q, W_v: B, A18.87Mshare of model0.0108%fewer than full FT9,272×each adapted d_in × d_out matrix trains r·(d_in + d_out) numbers instead of d_in·d_outa few directions carry the update: rank 4 keeps 90.5% of ‖ΔW‖² with 25.0% of the parameterscolour: positive · negative, same scale in all three panels
update
real model
fig.A 32 × 32 update ΔW with a few strong, smooth directions plus noise (rows and columns ordered so the structure is visible), its best rank-r approximation B·A from the SVD, and the residual. By Eckart–Young, no rank-r factorisation does better, so this is the ceiling for any LoRA of that rank. At the default r = 4, B·A keeps 90.46% of ‖ΔW‖²_F (relative error 0.3089) with 256 of 1,024 parameters. Rank 1 keeps 55.39%, rank 8 keeps 95.81% and rank 16 keeps 98.95%. Switch to a random Gaussian ΔW and the spectrum is flat: rank 4 keeps only 36.59%, rank 8 keeps 61.57%, and rank 16 keeps 89.52%. Low rank works only when the update is low rank, which is what Hu et al. found for real fine-tuning. The right column counts the trainable parameters for real models. On GPT-3 175B with W_q and W_v, r = 4 gives 18.87M (0.0108% of the model, 9,272× fewer), r = 1 gives 4.72M and r = 8 gives 37.75M. Llama 2 7B with all seven linear layers at r = 8 trains 19.99M (0.2966%), and at r = 64 it trains 159.91M (2.3731%).

Worked example: one LoRA step, then merge

Take d=k=3d=k=3, r=1r=1, α=2\alpha=2, so the scale is α/r=2\alpha/r=2. Use squared loss L=12∥h−y∥2\mathcal L=\tfrac12\|h-y\|^2, plain SGD with learning rate 0.10.1, input x=(1,2,−1)x=(1,2,-1) and target y=(0,3,2)y=(0,3,2).

matrixvalue
W₀ (frozen)[[1, 0, 2], [0, 1, −1], [1, 1, 0]]
A (1 × 3, random)[0.5, −0.5, 1]
B (3 × 1)[0, 0, 0]ᵀ
trainabler(d + k) = 6 numbers, vs dk = 9 for ΔW
Weights at step 0 (all values checked in numpy)

Forward, backward, update, merge

  1. 1Forward: W0x=(−1,3,3)W_0x=(-1,3,3) and Ax=0.5−1−1=−1.5Ax=0.5-1-1=-1.5. Since B=0B=0, h=(−1,3,3)h=(-1,3,3), exactly the pretrained output. The loss is 12(1+0+1)=1.0\tfrac12(1+0+1)=1.0.
  2. 2Backward: g=h−y=(−1,0,1)g=h-y=(-1,0,1). ∂L/∂B=2⋅g⋅(−1.5)=(3,0,−3)\partial\mathcal L/\partial B=2\cdot g\cdot(-1.5)=(3,0,-3). ∂L/∂A=2⋅B⊤g⋅x⊤=0\partial\mathcal L/\partial A=2\cdot B^\top g\cdot x^\top=0, so AA does not move on the first step.
  3. 3Update: B←0−0.1⋅(3,0,−3)=(−0.3,0,0.3)B\leftarrow0-0.1\cdot(3,0,-3)=(-0.3,0,0.3). New output h=W0x+2⋅B⋅(−1.5)=(−1,3,3)+(0.9,0,−0.9)=(−0.1,3,2.1)h=W_0x+2\cdot B\cdot(-1.5)=(-1,3,3)+(0.9,0,-0.9)=(-0.1,3,2.1). The loss drops from 1.01.0 to 0.010.01.
  4. 4Second step: now B≠0B\neq0, and ∂L/∂A=2 (B⊤g) x⊤=(0.12,0.24,−0.12)\partial\mathcal L/\partial A=2\,(B^\top g)\,x^\top=(0.12,0.24,-0.12), so both factors learn from here on.
  5. 5Merge (after step 1): W=W0+2 BA=[[0.7,0.3,1.4],[0,1,−1],[1.3,0.7,0.6]]W=W_0+2\,BA=[[0.7,0.3,1.4],[0,1,-1],[1.3,0.7,0.6]]. Then Wx=(−0.1,3,2.1)Wx=(-0.1,3,2.1), identical to the two-path output, at the cost of one ordinary matmul.
python
1import math, torch, torch.nn as nn
2
3class LoRALinear(nn.Module):
4 """y = W0 x + (alpha/r) * B(A(dropout(x))), with W0 frozen."""
5 def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.05):
6 super().__init__()
7 self.base, self.r, self.scale = base, r, alpha / r
8 base.weight.requires_grad_(False)
9 if base.bias is not None:
10 base.bias.requires_grad_(False)
11 self.A = nn.Parameter(torch.empty(r, base.in_features))
12 self.B = nn.Parameter(torch.zeros(base.out_features, r)) # B = 0: start at W0 exactly
13 nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) # paper: Gaussian; loralib/PEFT: this
14 self.drop, self.merged = nn.Dropout(dropout), False
15
16 def forward(self, x):
17 y = self.base(x)
18 if not self.merged:
19 y = y + (self.drop(x) @ self.A.T @ self.B.T) * self.scale
20 return y
21
22 @torch.no_grad()
23 def merge(self): # deploy: one matmul, zero added latency
24 if not self.merged:
25 self.base.weight += self.scale * (self.B @ self.A)
26 self.merged = True
27
28 @torch.no_grad()
29 def unmerge(self): # back to W0, e.g. to swap in another task's B, A
30 if self.merged:
31 self.base.weight -= self.scale * (self.B @ self.A)
32 self.merged = False
33
34def add_lora(model, names=("q_proj", "v_proj"), r=8, alpha=16):
35 for p in model.parameters():
36 p.requires_grad_(False)
37 for parent in list(model.modules()):
38 for n, child in list(parent.named_children()):
39 if n in names and isinstance(child, nn.Linear):
40 setattr(parent, n, LoRALinear(child, r, alpha))
41 return model
42
43# Same idea with PEFT. GPT-2 fuses q, k, v into one Conv1D "c_attn" (768 -> 2304):
44from transformers import AutoModelForCausalLM
45from peft import LoraConfig, get_peft_model
46
47model = AutoModelForCausalLM.from_pretrained("gpt2")
48cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05, target_modules=["c_attn"],
49 fan_in_fan_out=True, task_type="CAUSAL_LM")
50model = get_peft_model(model, cfg)
51model.print_trainable_parameters()
52# 12 layers * 8 * (768 + 2304) = 294,912 trainable of 124,734,720 total (0.2364%)
53
54# ... train as usual, then fold the adapters in for serving:
55model = model.merge_and_unload() # a plain GPT-2 again, with W0 + (alpha/r) BA
A LoRA linear layer from scratch with merge/unmerge, and the same thing through Hugging Face PEFT (not run here; the parameter counts in the comments were computed by hand and checked in Python)

Use when

  • +Trains well under 1% of the parameters and needs no gradients or Adam state for the base
  • +Zero added inference latency after merging; same architecture and kernels
  • +Per-task checkpoints of megabytes; one base can serve many adapters
  • +Matches or beats full fine-tuning on GLUE, E2E and GPT-3 tasks in the paper

Avoid when

  • −The update is capped at rank r per matrix; tasks far from pretraining (new languages, long code training) can need more
  • −Forward and backward still run the full model: activation memory and FLOPs barely fall
  • −Batching requests for different adapters requires unmerged adapters and special kernels
  • −Adds hyperparameters (r, α, target modules, dropout) whose best values vary by model

Where it's used

Instruction and chat fine-tuning

The default way to run Supervised fine-tuning (SFT) of open models such as Llama on one GPU, usually with QLoRA.

Multi-tenant serving

One frozen base in memory with hundreds of small task or customer adapters swapped in per request.

Diffusion models

Style and subject LoRAs for Stable Diffusion apply the same low-rank update to the attention projections of the UNet.

Check yourself

LoRA initialises A randomly and B to zero. What happens at the first optimisation step?

Check yourself

A 4096 × 4096 projection is adapted with r = 16. How many trainable parameters does LoRA add for that matrix, and what fraction of the matrix is that?

Remember

  • ✓LoRA: h=W0x+αrBAxh=W_0x+\tfrac{\alpha}{r}BAx with W0W_0 frozen, B∈Rd×rB\in\mathbb R^{d\times r} zero-initialised and AA random, so training starts at the pretrained model.
  • ✓Each adapted matrix trains r(d+k)r(d+k) numbers instead of dkdk. On GPT-3 175B, r=4r=4 on Wq,WvW_q,W_v is 18.9M parameters, about 10,000× fewer, with a 35 MB checkpoint.
  • ✓Fine-tuning updates are empirically low rank: r = 1 to 8 matches full fine-tuning on GPT-3, and spreading rank over several matrices beats a high rank on one.
  • ✓Merging W=W0+αrBAW=W_0+\tfrac{\alpha}{r}BA removes all inference overhead, unlike serial adapter layers.
  • ✓Variants change the scale (rsLoRA, α/r\alpha/\sqrt r), the learning rates (LoRA+) or the parametrisation (DoRA: magnitude times direction), and all still merge into one matrix.
← back to Deep Learning roadmap