# LoRA

> LoRA (Hu et al., 2021) freezes a pretrained weight W₀ and trains a low-rank update: h = W₀x + (α/r)·BAx, B zero-initialised, r ≪ d. Rank 4 on GPT-3's W_q and W_v trains 18.9M parameters, about 10,000× fewer than full fine-tuning, and BA merges into W₀ so inference costs nothing extra.

Source: https://www.mlfoundry.in/learn/lora  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**LoRA:** (Low-Rank Adaptation) is the parameter-efficient fine-tuning method of Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, "LoRA: Low-Rank Adaptation of Large Language Models" (Microsoft, arXiv 2021, ICLR 2022). It keeps a pretrained weight $W_0\in\mathbb R^{d\times k}$ frozen and learns the update as a product of two thin matrices, $\Delta W=BA$ with $B\in\mathbb R^{d\times r}$, $A\in\mathbb R^{r\times k}$ and rank $r\ll\min(d,k)$. The forward pass becomes $h=W_0x+\tfrac{\alpha}{r}BAx$. $B$ starts at zero, so training begins exactly at the pretrained model. After training, $BA$ is added into $W_0$ and the model runs at its original speed.

Full fine-tuning of GPT-3 175B means storing a new 350 GB checkpoint per task and keeping Adam state for every one of its 175B parameters. **Parameter-efficient fine-tuning (PEFT)** methods before LoRA each paid a price elsewhere. Adapter layers (Houlsby et al. 2019) insert small bottleneck MLPs *in series* with each block, so every token at inference passes through extra sequential layers. Hu et al. measured noticeable latency increases for online, batch-size-1 serving. **Prefix tuning** and **Prompt tuning** spend part of the context window on learned vectors and were hard to optimise in their experiments. LoRA adds its matrices *in parallel* with existing weights, which removes the latency problem once they are merged.

The motivating hypothesis comes from work on **intrinsic dimensionality**. Li et al. (2018) trained networks inside random low-dimensional subspaces of their parameter space and found that surprisingly few dimensions suffice. Aghajanyan, Zettlemoyer and Gupta (2021) showed the same for fine-tuning: optimising only 200 parameters, randomly projected into the full space of RoBERTa-large, reached 90% of full fine-tuning performance on MRPC. Pretrained models seem to need only a small change to adapt. Hu et al. made the stronger, more specific bet that the *weight update* of each matrix has low rank, and parametrised it that way.

## The mechanism

$$
h=W_0x+\Delta W\,x=W_0x+\frac{\alpha}{r}\,B\,(A\,x),\qquad B\in\mathbb R^{d\times r},\;A\in\mathbb R^{r\times k},\;r\ll\min(d,k)
$$

- $W_0$: pretrained weight, frozen: no gradient and no optimizer state
- $A$: down-projection to $r$ dimensions; random Gaussian init in the paper (Kaiming-uniform in the reference code and PEFT)
- $B$: up-projection back to $d$; **initialised to zero**, so $\Delta W=BA=0$ at step 0
- $\alpha/r$: fixed scale. The authors set $\alpha$ to the first $r$ they tried and did not tune it: with Adam, changing $\alpha$ acts like changing the learning rate
- $r$: the rank; 1–8 in the paper's GPT-3 experiments, 8–64 in common practice

> Figure: LoRA in one linear layer. The input feeds the frozen W₀ and, in parallel, a down-projection A (r × d) and an up-projection B (d × r, zero-initialised). Their product is scaled by α/r and added to the output. Toggle to the merged view: at deployment BA is folded into W₀ once, the matrix keeps its shape, and the forward pass is the original single matmul. On the right are the matrices it can target. Hu et al. adapted only W_q and W_v of GPT-3; at d = 12,288 and r = 4 one matrix trains 98,304 numbers instead of 150,994,944.

The trainable count per adapted matrix is $r(d+k)$ instead of $dk$. For square $d\times d$ projections that is a fraction $2r/d$. GPT-3 has $d=12288$, so $r=4$ gives $2\cdot4/12288=0.065\%$ of each adapted matrix. Over $W_q$ and $W_v$ in all 96 layers that is $96\cdot2\cdot2\cdot12288\cdot4=18{,}874{,}368$ parameters. The paper reports what this buys on GPT-3 175B: training VRAM falls from 1.2 TB to 350 GB, the per-task checkpoint from 350 GB to about 35 MB, and training throughput rises by 25%, because most weights need neither gradients nor Adam state.

## Why zero-initialise B, and what α/r does

With $B=0$ the adapted model is the pretrained model at step 0, which is the right place to start fine-tuning. The two factors cannot both be zero. Then every gradient would vanish: $\partial\mathcal L/\partial B=\tfrac{\alpha}{r}\,g\,(Ax)^\top$ and $\partial\mathcal L/\partial A=\tfrac{\alpha}{r}\,B^\top g\,x^\top$, where $g=\partial\mathcal L/\partial h$. With $A$ random and $B=0$, the first step moves only $B$. From the second step on, both factors learn. The worked example below shows this.

$$
\frac{\partial\mathcal L}{\partial B}=\frac{\alpha}{r}\,g\,(Ax)^\top,\qquad \frac{\partial\mathcal L}{\partial A}=\frac{\alpha}{r}\,B^\top g\,x^\top,\qquad g=\frac{\partial\mathcal L}{\partial h}
$$

- $g$: gradient arriving at the layer output; the same signal full fine-tuning would use for $\partial\mathcal L/\partial W=g\,x^\top$
- $(Ax)^\top$: $B$ sees the input through the random projection $A$
- $B^\top g$: zero at initialisation, so $A$ is frozen for the first step

The $\alpha/r$ factor keeps the update's size roughly constant when you change $r$, so a learning rate tuned at one rank transfers to another. Kalajdzievski (2023, **rsLoRA**) argued that the right normaliser is $\alpha/\sqrt r$. With $\alpha/r$, the update of a rank-$r$ adapter shrinks as $r$ grows and learning slows at high rank. With $\alpha/\sqrt r$, larger ranks keep improving.

## Which matrices, and how much rank

The paper restricts LoRA to the attention projections and freezes the MLPs. It then asks two questions on GPT-3 175B, each under a fixed budget of about 18M trainable parameters. The first is which projections to adapt. The answer is to spread the budget: $W_q$ alone is clearly worst, and adapting several matrices at a low rank beats one matrix at a high rank.

Hu et al., Table 5: GPT-3 175B with an 18M-parameter LoRA budget (validation accuracy, %)

| adapted | rank r | WikiSQL | MultiNLI |
| --- | --- | --- | --- |
| W_q | 8 | 70.4 | 91.0 |
| W_k | 8 | 70.0 | 90.8 |
| W_v | 8 | 73.0 | 91.0 |
| W_o | 8 | 73.2 | 91.3 |
| W_q, W_k | 4 | 71.4 | 91.3 |
| W_q, W_v | 4 | 73.7 | 91.3 |
| W_q, W_k, W_v, W_o | 2 | 73.7 | 91.7 |

The second question is how much rank is needed. Surprisingly little. With $W_q$ and $W_v$ adapted, $r=1$ already scores 73.4 on WikiSQL and 91.7 on MultiNLI, and going up to $r=64$ brings no gain. The authors then compared the subspaces learned at $r=8$ and $r=64$. The top singular directions overlap strongly, and the extra directions at $r=64$ look mostly like noise. They also found that $\Delta W$ correlates with $W_0$ but amplifies directions that are *not* emphasised in $W_0$, by a factor of about 21.5 for $r=4$ in one layer. Their reading is that fine-tuning boosts task-specific features the pretrained model learned but did not use much. Later practice, for example **QLoRA**, adapts every linear layer including the MLPs, which helps when the task is far from pretraining.

**From the paper**

- **Authors:** Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen (Microsoft), arXiv June 2021, ICLR 2022; reference code `loralib`
- **GPT-3 175B:** 4.7M trainable params: WikiSQL 73.4, MNLI-m 91.7; 37.7M: 74.0, 91.6; full fine-tuning (175,255.8M): 73.8, 89.5
- **Savings (GPT-3):** $r=4$ on $W_q,W_v$: VRAM 1.2 TB → 350 GB, checkpoint 350 GB → 35 MB (~10,000×), 25% faster training
- **RoBERTa / DeBERTa:** GLUE average: RoBERTa-base 87.2 with 0.3M params vs 86.4 full FT; RoBERTa-large 89.0 (0.8M) vs 88.9; DeBERTa XXL 91.3 (4.7M) vs 91.1 (1.5B)
- **GPT-2 medium:** E2E NLG BLEU 70.4 with 0.35M trainable params vs 68.2 for full fine-tuning
- **Init and scale:** $A$ random Gaussian, $B=0$; output scaled by $\alpha/r$ with $\alpha$ fixed to the first $r$ tried
- **Latency:** none after merging $W=W_0+\tfrac{\alpha}{r}BA$; switch tasks by subtracting one $BA$ and adding another

> Figure: A 32 × 32 update ΔW with a few strong, smooth directions plus noise (rows and columns ordered so the structure is visible), its best rank-r approximation B·A from the SVD, and the residual. By Eckart–Young, no rank-r factorisation does better, so this is the ceiling for any LoRA of that rank. At the default r = 4, B·A keeps 90.46% of ‖ΔW‖²_F (relative error 0.3089) with 256 of 1,024 parameters. Rank 1 keeps 55.39%, rank 8 keeps 95.81% and rank 16 keeps 98.95%. Switch to a random Gaussian ΔW and the spectrum is flat: rank 4 keeps only 36.59%, rank 8 keeps 61.57%, and rank 16 keeps 89.52%. Low rank works only when the update is low rank, which is what Hu et al. found for real fine-tuning. The right column counts the trainable parameters for real models. On GPT-3 175B with W_q and W_v, r = 4 gives 18.87M (0.0108% of the model, 9,272× fewer), r = 1 gives 4.72M and r = 8 gives 37.75M. Llama 2 7B with all seven linear layers at r = 8 trains 19.99M (0.2966%), and at r = 64 it trains 159.91M (2.3731%).

## Worked example: one LoRA step, then merge

Take $d=k=3$, $r=1$, $\alpha=2$, so the scale is $\alpha/r=2$. Use squared loss $\mathcal L=\tfrac12\|h-y\|^2$, plain SGD with learning rate $0.1$, input $x=(1,2,-1)$ and target $y=(0,3,2)$.

Weights at step 0 (all values checked in numpy)

| matrix | value |
| --- | --- |
| W₀ (frozen) | [[1, 0, 2], [0, 1, −1], [1, 1, 0]] |
| A (1 × 3, random) | [0.5, −0.5, 1] |
| B (3 × 1) | [0, 0, 0]ᵀ |
| trainable | r(d + k) = 6 numbers, vs dk = 9 for ΔW |

**Forward, backward, update, merge**

1. Forward: $W_0x=(-1,3,3)$ and $Ax=0.5-1-1=-1.5$. Since $B=0$, $h=(-1,3,3)$, exactly the pretrained output. The loss is $\tfrac12(1+0+1)=1.0$.
2. Backward: $g=h-y=(-1,0,1)$. $\partial\mathcal L/\partial B=2\cdot g\cdot(-1.5)=(3,0,-3)$. $\partial\mathcal L/\partial A=2\cdot B^\top g\cdot x^\top=0$, so $A$ does not move on the first step.
3. Update: $B\leftarrow0-0.1\cdot(3,0,-3)=(-0.3,0,0.3)$. New output $h=W_0x+2\cdot B\cdot(-1.5)=(-1,3,3)+(0.9,0,-0.9)=(-0.1,3,2.1)$. The loss drops from $1.0$ to $0.01$.
4. Second step: now $B\neq0$, and $\partial\mathcal L/\partial A=2\,(B^\top g)\,x^\top=(0.12,0.24,-0.12)$, so both factors learn from here on.
5. Merge (after step 1): $W=W_0+2\,BA=[[0.7,0.3,1.4],[0,1,-1],[1.3,0.7,0.6]]$. Then $Wx=(-0.1,3,2.1)$, identical to the two-path output, at the cost of one ordinary matmul.

```python
import math, torch, torch.nn as nn

class LoRALinear(nn.Module):
    """y = W0 x + (alpha/r) * B(A(dropout(x))), with W0 frozen."""
    def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.05):
        super().__init__()
        self.base, self.r, self.scale = base, r, alpha / r
        base.weight.requires_grad_(False)
        if base.bias is not None:
            base.bias.requires_grad_(False)
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))     # B = 0: start at W0 exactly
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))            # paper: Gaussian; loralib/PEFT: this
        self.drop, self.merged = nn.Dropout(dropout), False

    def forward(self, x):
        y = self.base(x)
        if not self.merged:
            y = y + (self.drop(x) @ self.A.T @ self.B.T) * self.scale
        return y

    @torch.no_grad()
    def merge(self):                     # deploy: one matmul, zero added latency
        if not self.merged:
            self.base.weight += self.scale * (self.B @ self.A)
            self.merged = True

    @torch.no_grad()
    def unmerge(self):                   # back to W0, e.g. to swap in another task's B, A
        if self.merged:
            self.base.weight -= self.scale * (self.B @ self.A)
            self.merged = False

def add_lora(model, names=("q_proj", "v_proj"), r=8, alpha=16):
    for p in model.parameters():
        p.requires_grad_(False)
    for parent in list(model.modules()):
        for n, child in list(parent.named_children()):
            if n in names and isinstance(child, nn.Linear):
                setattr(parent, n, LoRALinear(child, r, alpha))
    return model

# Same idea with PEFT. GPT-2 fuses q, k, v into one Conv1D "c_attn" (768 -> 2304):
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("gpt2")
cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05, target_modules=["c_attn"],
                 fan_in_fan_out=True, task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)
model.print_trainable_parameters()
# 12 layers * 8 * (768 + 2304) = 294,912 trainable of 124,734,720 total (0.2364%)

# ... train as usual, then fold the adapters in for serving:
model = model.merge_and_unload()         # a plain GPT-2 again, with W0 + (alpha/r) BA
```

A LoRA linear layer from scratch with merge/unmerge, and the same thing through Hugging Face PEFT (not run here; the parameter counts in the comments were computed by hand and checked in Python)

> **Merging and quantized bases do not mix freely:** Merging assumes $W_0$ and $BA$ live in the same precision. If the base is stored in 4 bits (**QLoRA**), $W_0+\tfrac{\alpha}{r}BA$ is a 16-bit matrix. Re-quantizing it moves every weight to a 4-bit level and discards part of the fine-tune. You either serve the base in 16 bits with the adapter merged, or keep the adapter unmerged on top of the 4-bit base and pay a small extra matmul per layer.

> **LoRA is a constraint on the update, not on the model:** $W_0+BA$ has full rank: the model keeps everything it knew. Only the *change* is low-rank. The figure makes the limit concrete. If the ideal update had a flat spectrum, like the random $\Delta W$, rank 4 could capture only 36.59% of it. Fine-tuning updates appear to be very far from that, which is why ranks of 1–8 match full fine-tuning on GPT-3.

> **Variants worth knowing:** **DoRA** (Liu et al., ICML 2024) splits each weight into a magnitude vector and a direction, $W'=m\odot\frac{W_0+BA}{\|W_0+BA\|_c}$ with column norms $\|\cdot\|_c$. LoRA updates the direction, and $m$ is trained directly. **rsLoRA** (Kalajdzievski 2023) scales by $\alpha/\sqrt r$ so that high ranks keep learning. **LoRA+** (Hayou et al. 2024) gives $B$ a larger learning rate than $A$. All of them still merge into a plain weight matrix.

**Strengths**

- Trains well under 1% of the parameters and needs no gradients or Adam state for the base
- Zero added inference latency after merging; same architecture and kernels
- Per-task checkpoints of megabytes; one base can serve many adapters
- Matches or beats full fine-tuning on GLUE, E2E and GPT-3 tasks in the paper

**Limitations**

- The update is capped at rank r per matrix; tasks far from pretraining (new languages, long code training) can need more
- Forward and backward still run the full model: activation memory and FLOPs barely fall
- Batching requests for different adapters requires unmerged adapters and special kernels
- Adds hyperparameters (r, α, target modules, dropout) whose best values vary by model

- **Instruction and chat fine-tuning:** The default way to run **Supervised fine-tuning (SFT)** of open models such as **Llama** on one GPU, usually with **QLoRA**.
- **Multi-tenant serving:** One frozen base in memory with hundreds of small task or customer adapters swapped in per request.
- **Diffusion models:** Style and subject LoRAs for Stable Diffusion apply the same low-rank update to the attention projections of the UNet.

**Question:** LoRA initialises A randomly and B to zero. What happens at the first optimisation step?

- A. Both A and B receive zero gradient, so training never starts
- B. Only B changes; A's gradient is zero because it is multiplied by Bᵀg = 0
- C. Only A changes, because B is zero
- D. Both change equally because of the α/r scale

**Answer:** B. Only B changes; A's gradient is zero because it is multiplied by Bᵀg = 0 — $\partial\mathcal L/\partial B=\tfrac{\alpha}{r}g(Ax)^\top$ is non-zero because $A$ is random, while $\partial\mathcal L/\partial A=\tfrac{\alpha}{r}B^\top g\,x^\top=0$. After one step $B\neq0$ and both learn. If both were zero, both gradients would vanish forever.

**Question:** A 4096 × 4096 projection is adapted with r = 16. How many trainable parameters does LoRA add for that matrix, and what fraction of the matrix is that?

- A. 65,536 (0.39%)
- B. 131,072 (0.78%)
- C. 262,144 (1.56%)
- D. 16,777,216 (100%)

**Answer:** B. 131,072 (0.78%) — $r(d+k)=16\cdot(4096+4096)=131{,}072$, and $131{,}072/4096^2=2r/d=32/4096=0.78\%$.

**Key points**

- LoRA: $h=W_0x+\tfrac{\alpha}{r}BAx$ with $W_0$ frozen, $B\in\mathbb R^{d\times r}$ zero-initialised and $A$ random, so training starts at the pretrained model.
- Each adapted matrix trains $r(d+k)$ numbers instead of $dk$. On GPT-3 175B, $r=4$ on $W_q,W_v$ is 18.9M parameters, about 10,000× fewer, with a 35 MB checkpoint.
- Fine-tuning updates are empirically low rank: r = 1 to 8 matches full fine-tuning on GPT-3, and spreading rank over several matrices beats a high rank on one.
- Merging $W=W_0+\tfrac{\alpha}{r}BA$ removes all inference overhead, unlike serial adapter layers.
- Variants change the scale (rsLoRA, $\alpha/\sqrt r$), the learning rates (LoRA+) or the parametrisation (DoRA: magnitude times direction), and all still merge into one matrix.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
