# QLoRA

> QLoRA (Dettmers et al., 2023) trains BF16 LoRA adapters on a frozen 4-bit base. NF4 levels at N(0,1) quantiles, double-quantized block scales (0.5 → 0.127 bits/param) and paged optimizers cut 65B fine-tuning from >780 GB to <48 GB, matching 16-bit results.

Source: https://www.mlfoundry.in/learn/qlora  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**QLoRA:** is the fine-tuning method of Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs" (University of Washington, arXiv May 2023, NeurIPS 2023). It stores the frozen pretrained model in a 4-bit format, dequantizes each weight matrix to BF16 on the fly for the matmul, and backpropagates through it into **LoRA** adapters kept in BF16. Three ideas make this work without losing quality: the **4-bit NormalFloat (NF4)** data type, **double quantization** of the per-block scales, and **paged optimizers** that spill optimizer state to CPU RAM during memory spikes. With them, a 65B model fine-tunes on one 48 GB GPU.

**LoRA** removed the gradients and Adam state of the base model, but the base weights themselves still had to sit in GPU memory in 16 bits. For LLaMA-65B that is 130 GB before a single activation is stored. The paper puts regular 16-bit full fine-tuning of the same model at more than 780 GB. **Quantization** to 4 bits had been shown to work for *inference*: Dettmers and Zettlemoyer (2022) found that 4-bit weights give the best accuracy for a fixed number of model bits. Before QLoRA, nobody had shown that you could also *train* through a 4-bit model and match 16-bit fine-tuning.

The design separates a **storage** data type from a **computation** data type. Weights are stored in NF4 and multiplied in BF16. Gradients flow through the dequantized weights to the activations and on into the adapters, but the 4-bit weights are never updated. Quantization error therefore enters only the forward pass of a fixed base, and the adapters can learn to compensate for it.

## The forward pass

$$
\mathbf Y^{\text{BF16}}=\mathbf X^{\text{BF16}}\,\text{doubleDequant}\big(c_1^{\text{FP32}},c_2^{\text{k-bit}},\mathbf W^{\text{NF4}}\big)+\mathbf X^{\text{BF16}}\,\mathbf L_1^{\text{BF16}}\,\mathbf L_2^{\text{BF16}}
$$

- $\mathbf W^{\text{NF4}}$: frozen base weight, 4-bit codes in blocks of 64
- $c_2^{\text{k-bit}}$: one absmax scale per block of 64, itself quantized to FP8
- $c_1^{\text{FP32}}$: second-level scale, one per 256 first-level scales
- $\text{doubleDequant}$: $\text{dequant}(\text{dequant}(c_1,c_2),\mathbf W)$: first recover the block scales, then the weights
- $\mathbf L_1\in\mathbb R^{h\times r},\ \mathbf L_2\in\mathbb R^{r\times o}$: the LoRA factors (the paper's $L_1,L_2$ are LoRA's $A^\top,B^\top$), the only trainable tensors

> Figure: One QLoRA linear layer. The frozen base sits in GPU memory as 4-bit NF4 codes, an FP8 absmax for every 64 weights and an FP32 scale for every 256 of those. It is dequantized to BF16 just in time for the matmul X·W. The BF16 adapter path X·L₁·L₂ runs in parallel and is the only thing that learns, applied to every linear layer with r = 64 in the paper. AdamW state for the adapters lives in paged memory that NVIDIA unified memory can evict to CPU RAM during a spike. Bottom: the paper's memory figures for a 65B model.

## NF4: levels at the quantiles of a Gaussian

A 4-bit code has 16 values. Plain INT4 spaces them evenly between $-\text{absmax}$ and $+\text{absmax}$. Pretrained weights, however, are roughly zero-centred normal. Most of them crowd near zero, where evenly spaced levels are coarse, and few reach the tails, where levels go unused. **Quantile quantization** puts the levels where the probability mass is, so each code is used about equally often. For a known distribution this is information-theoretically optimal, because every code carries close to its full 4 bits.

$$
q_i=\tfrac12\Big(Q_X\big(\tfrac{i}{2^k+1}\big)+Q_X\big(\tfrac{i+1}{2^k+1}\big)\Big),\qquad Q_X=\Phi^{-1}\ \text{for}\ X\sim\mathcal N(0,1),\qquad x_{\text{code}}=\arg\min_i\Big|q_i-\frac{w}{\max_{b}|w|}\Big|
$$

- $Q_X$: quantile function of the standard normal; the levels are fixed once, independent of the model
- $\max_b|w|$: absmax of the block of 64 weights, which rescales every block into $[-1,1]$ so one table of levels fits all
- $\text{zero}$: to represent 0 exactly, NF4 uses 8 positive and 7 negative quantiles plus a 0. The table is asymmetric: the smallest positive level is 0.0796, the smallest negative level is −0.0911

The released levels come from bitsandbytes' construction: take quantiles of $\mathcal N(0,1)$ at probabilities evenly spaced from $0.5$ to $\delta=0.9677083=1-\tfrac12(\tfrac1{32}+\tfrac1{30})$ (8 positive and 7 negative), add an exact zero, and divide by the largest value. Cutting off at $\delta$ rather than $1$ avoids the infinite quantile. Recomputed in Python, the 16 levels are $-1,\,-0.6962,\,-0.5251,\,-0.3949,\,-0.2844,\,-0.1848,\,-0.0911,\,0,\,0.0796,\,0.1609,\,0.2461,\,0.3379,\,0.4407,\,0.5626,\,0.7230,\,1$, matching the published table to about $10^{-7}$.

## Double quantization and paged optimizers

Each block of 64 weights needs its own absmax, stored as FP32: $32/64=0.5$ extra bits per parameter, or 4 GB for a 65B model. **Double quantization** quantizes those scales as well. Subtract their mean (they are all positive), then store them in FP8 with one FP32 scale per 256 of them. The cost drops to $8/64+32/(64\cdot256)=0.127$ bits per parameter, a saving of $0.373$ bits, or about 3 GB on 65B, with no measurable loss.

**Paged optimizers** handle the rest. With gradient checkpointing, a long sequence in a mini-batch can briefly push memory over the limit and crash the run. QLoRA allocates the optimizer state in NVIDIA **unified memory**, which pages it out to CPU RAM automatically when the GPU runs out and pages it back for the optimizer step. The adapters' AdamW state is small, so the paging rarely triggers, but it turns occasional out-of-memory crashes into brief slowdowns.

**From the paper**

- **Authors:** Dettmers, Pagnoni, Holtzman, Zettlemoyer (University of Washington), arXiv May 2023, NeurIPS 2023; released with bitsandbytes 4-bit kernels and Hugging Face PEFT integration
- **Quantization:** NF4 storage, BF16 compute; block size 64; double quantization with FP8 second level, block size 256
- **LoRA setup:** adapters on **all** linear layers of every block, $r=64$, $\alpha=16$; dropout 0.1 up to 13B, 0.05 for 33B and 65B; learning rate $2\cdot10^{-4}$ (7B, 13B) and $10^{-4}$ (33B, 65B)
- **Memory:** 65B fine-tuning: more than 780 GB for 16-bit full fine-tuning, less than 48 GB for QLoRA
- **Quality:** NF4 + DQ matched 16-bit full fine-tuning and 16-bit LoRA on GLUE and Super-NaturalInstructions, and on 5-shot MMLU for LLaMA 7B–65B fine-tuned on FLAN v2 and Alpaca; NF4 beat FP4 and Int4
- **Guanaco:** LLaMA + QLoRA on the OASST1 dataset; Guanaco 65B reached 99.3% of ChatGPT's score on the Vicuna benchmark (GPT-4 as judge) after 24 hours of fine-tuning on a single GPU; 33B reached 97.8%
- **Scale of study:** more than 1,000 fine-tuned models; a 9k-example dataset (OASST1) beat a 450k-example FLAN v2 subset for chatbot quality, so data quality mattered more than size

> Figure: 4,096 weights from N(0, 0.02²), split into blocks. Each block is divided by its absmax and every value is rounded to the nearest 4-bit level: NF4 (blue) or the 15 evenly spaced INT4 levels k/7 (orange). The NF4 levels crowd the centre, where the histogram is tall. At the default block size of 64, the RMSE is 0.0931σ for NF4 and 0.1124σ for INT4, 1.207× worse. The code-usage bars show why: NF4's 16 codes carry 3.865 bits of entropy, while INT4's 15 codes carry 3.470 of a possible 3.91, because its tail codes are almost never used. Blocks of 16 shrink both errors (0.0831σ vs 0.0876σ) but cost 2 bits per parameter in FP32 scales. One block for the whole tensor (4,096) raises them to 0.1121σ vs 0.1596σ. Add a single 12σ outlier and blocking becomes essential: at 64 it costs little (0.0982σ vs 0.1300σ), at 4,096 it squeezes everything else into a few codes (0.3025σ vs 0.4951σ, NF4 entropy 2.091 bits). Double quantization cuts the scales from 0.5000 to 0.1270 bits per parameter at block 64, and for LLaMA-65B the stored weights shrink from 36.56 GB to 33.53 GB.

## Worked example: one block in NF4 and INT4

Take a block of 8 weights. The absmax is $c=0.045$. Divide by it, snap to the nearest level, and multiply back. All values below were computed in numpy.

w → w/c → nearest level → dequantized value (c = 0.045)

| w | w / c | NF4 code: level | NF4 ŵ (error) | INT4 k/7 | INT4 ŵ (error) |
| --- | --- | --- | --- | --- | --- |
| 0.031 | 0.6889 | 14: 0.7230 | 0.03253 (+0.00153) | 5/7 | 0.03214 (+0.00114) |
| −0.012 | −0.2667 | 4: −0.2844 | −0.01280 (−0.00080) | −2/7 | −0.01286 (−0.00086) |
| 0.004 | 0.0889 | 8: 0.0796 | 0.00358 (−0.00042) | 1/7 | 0.00643 (+0.00243) |
| −0.045 | −1 | 0: −1 | −0.04500 (0) | −7/7 | −0.04500 (0) |
| 0.020 | 0.4444 | 12: 0.4407 | 0.01983 (−0.00017) | 3/7 | 0.01929 (−0.00071) |
| 0.009 | 0.2000 | 9: 0.1609 | 0.00724 (−0.00176) | 1/7 | 0.00643 (−0.00257) |
| −0.027 | −0.6000 | 2: −0.5251 | −0.02363 (+0.00337) | −4/7 | −0.02571 (+0.00129) |
| 0.015 | 0.3333 | 11: 0.3379 | 0.01521 (+0.00021) | 2/7 | 0.01286 (−0.00214) |

**Reading the block, then the 65B budget**

1. Small weights are where NF4 wins. $0.004$ lands $0.00042$ from its NF4 level but $0.00243$ from INT4's nearest level $1/7$, since INT4 has nothing between $0$ and $0.143$. INT4 wins a few larger weights, such as $-0.027$, where NF4's levels are sparser.
2. Over the block, the RMSE is $1.487\cdot10^{-3}$ for NF4 and $1.632\cdot10^{-3}$ for INT4, and the mean absolute error is $1.032\cdot10^{-3}$ vs $1.393\cdot10^{-3}$.
3. Storage: 8 codes × 4 bits + one 32-bit scale = 64 bits, or 8 bits per weight for this tiny block. At the paper's block of 64 the overhead is $0.5$ bits, and $0.127$ with double quantization.
4. 65B base, rounded to $65\cdot10^9$ parameters: $65\cdot10^9\times4/8=32.5$ GB of codes plus $65\cdot10^9\times0.127/8=1.03$ GB of scales.
5. Adapters on all 7 linear layers of LLaMA-65B (80 layers, $d=8192$, $d_{ff}=22016$) at $r=64$: $80\cdot64\cdot(4\cdot2\cdot8192+3\cdot(8192+22016))=799{,}539{,}200$ parameters. In BF16 with BF16 gradients that is $1.60+1.60$ GB, and 32-bit AdamW adds $8$ bytes each, $6.40$ GB.
6. Total before activations: $32.5+1.03+3.20+6.40\approx43.1$ GB. That leaves about 5 GB of a 48 GB card for checkpointed activations, which is why paging matters. This estimate is ours; the paper reports only the under-48 GB total.

```python
import numpy as np
from statistics import NormalDist

def nf4_levels(offset=0.9677083):            # offset = 1 - (1/32 + 1/30) / 2
    ppf = NormalDist().inv_cdf
    pos = [ppf(p) for p in np.linspace(offset, 0.5, 9)[:-1]]    # 8 positive quantiles
    neg = [-ppf(p) for p in np.linspace(offset, 0.5, 8)[:-1]]   # 7 negative quantiles
    v = np.sort(np.array(pos + [0.0] + neg))                     # plus an exact zero
    return v / np.abs(v).max()

NF4 = nf4_levels()
INT4 = np.arange(-7, 8) / 7

def quantize(w, levels, B=64):
    """Block-wise absmax quantization: 4-bit codes plus one FP32 scale per block."""
    blocks = w.reshape(-1, B)
    c = np.abs(blocks).max(axis=1)
    x = blocks / c[:, None]                                       # every block now spans [-1, 1]
    edges = (levels[1:] + levels[:-1]) / 2                        # nearest level via midpoints
    return np.searchsorted(edges, x).astype(np.uint8), c

def dequantize(codes, c, levels):
    return (levels[codes] * c[:, None]).ravel()

def double_quantize(c, B2=256):
    """Quantize the FP32 scales themselves (paper: FP8, here int8 for brevity), one FP32 per 256."""
    mean = c.mean()                                               # scales are positive: centre them
    x = (c - mean).reshape(-1, B2)
    s = np.abs(x).max(axis=1) / 127
    return np.round(x / s[:, None]).astype(np.int8), s, mean

def double_dequantize(q, s, mean):
    return (q * s[:, None]).ravel() + mean

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, size=4096 * 1024).astype(np.float32)     # one 4096 x 1024 weight
print(np.round(NF4, 4))
# [-1. -0.6962 -0.5251 -0.3949 -0.2844 -0.1848 -0.091 0. 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.723 1.]

for name, L in [("NF4", NF4), ("INT4", INT4)]:
    codes, c = quantize(w, L)
    err = dequantize(codes, c, L) - w
    print(f"{name}: RMSE/sigma = {np.sqrt((err ** 2).mean()) / 0.02:.4f}")   # NF4 0.0920, INT4 0.1077

codes, c = quantize(w, NF4)
q, s, mean = double_quantize(c)
err = dequantize(codes, double_dequantize(q, s, mean), NF4) - w
print(f"NF4 + DQ: RMSE/sigma = {np.sqrt((err ** 2).mean()) / 0.02:.4f}")     # 0.0920: DQ costs nothing visible
print(f"scale bits/param: {32 / 64:.3f} -> {8 / 64 + 32 / (64 * 256):.3f}")  # 0.500 -> 0.127
```

NF4 from scratch in numpy: build the 16 levels from Gaussian quantiles, quantize block-wise, and double-quantize the scales (run; outputs in comments)

```python
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # storage type: NormalFloat4
    bnb_4bit_use_double_quant=True,         # quantize the block scales too
    bnb_4bit_compute_dtype=torch.bfloat16,  # computation type
)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf",
                                             quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)  # gradient checkpointing, norms in fp32

cfg = LoraConfig(r=64, lora_alpha=16, lora_dropout=0.1, task_type="CAUSAL_LM",
                 target_modules=["q_proj", "k_proj", "v_proj", "o_proj",       # every linear layer,
                                 "gate_proj", "up_proj", "down_proj"])          # as in the paper
model = get_peft_model(model, cfg)
# trainable: 32 layers * 64 * (4*2*4096 + 3*(4096 + 11008)) = 159,907,840

args = TrainingArguments(
    output_dir="qlora-out",
    optim="paged_adamw_32bit",              # paged optimizer state
    learning_rate=2e-4, lr_scheduler_type="constant", max_grad_norm=0.3,
    per_device_train_batch_size=4, gradient_accumulation_steps=4,
    gradient_checkpointing=True, bf16=True,
)
```

The standard QLoRA recipe with transformers, bitsandbytes and PEFT (not run here; the trainable count was computed by hand)

> **Do not merge into the 4-bit weights:** After training, $W_0+\tfrac{\alpha}{r}BA$ is a BF16 matrix. Re-quantizing it to NF4 snaps every entry back onto the 16 levels, and the fine-tuning signal, which is small compared with the gap between levels, can largely disappear. Either dequantize the base to 16 bits, merge, and serve in 16 bits (or re-quantize carefully and re-evaluate), or keep the adapter unmerged on top of the 4-bit base.

> **Why train through a lossy model at all:** The adapters see the *quantized* model's activations and gradients, so they learn to correct for the quantization error as well as to fit the task. This is why NF4 + LoRA can match 16-bit fine-tuning even though a 4-bit model on its own is slightly worse than the 16-bit original.

> **Adapters everywhere:** The paper found that the number of adapted layers mattered far more than the rank. LoRA on the attention projections alone did not match full fine-tuning. LoRA on every linear layer did, and once all layers were adapted the choice of $r$ made little difference. That is why $r=64$ on all seven Llama projections became the default recipe.

**Strengths**

- Fine-tunes a 65B model on one 48 GB GPU and 7B–13B models on consumer cards
- Matches 16-bit full fine-tuning and 16-bit LoRA on the paper's benchmarks
- NF4 levels are fixed and model-independent; quantizing a checkpoint needs no calibration data
- Paged optimizers prevent out-of-memory crashes on long-sequence spikes

**Limitations**

- Slower per step than 16-bit LoRA: every forward and backward dequantizes each weight
- Merged models cannot stay in NF4 without re-quantization error
- NF4 assumes roughly Gaussian blocks; outlier-heavy weights rely on small blocks
- Activation memory is unchanged, so long contexts still need checkpointing and paging

**Question:** Why does NF4 give lower error than evenly spaced INT4 on pretrained weights at the same 4 bits?

- A. NF4 uses 5 bits internally
- B. Its levels are quantiles of N(0,1), so they are densest near zero where most normalised weights fall and each code is used about equally often
- C. NF4 does not need a per-block scale
- D. NF4 rounds stochastically

**Answer:** B. Its levels are quantiles of N(0,1), so they are densest near zero where most normalised weights fall and each code is used about equally often — Quantile levels equalise the probability mass per code. In the figure at block 64, NF4's codes carry 3.865 bits of entropy versus 3.470 for INT4, and the RMSE is 0.0931σ versus 0.1124σ. Both use one absmax per block.

**Question:** With block size 64, how many bits per parameter do the quantization constants cost with and without double quantization (FP8 scales, one FP32 per 256 blocks)?

- A. 0.5 and 0.25
- B. 0.5 and 0.127
- C. 4 and 0.5
- D. 0.125 and 0.0625

**Answer:** B. 0.5 and 0.127 — Without it: one FP32 per 64 weights, $32/64=0.5$. With it: $8/64+32/(64\cdot256)=0.125+0.00195\approx0.127$. On a 65B model the difference is about 3 GB.

**Key points**

- QLoRA: $\mathbf Y=\mathbf X\,\text{doubleDequant}(c_1,c_2,\mathbf W^{\text{NF4}})+\mathbf X\mathbf L_1\mathbf L_2$. The base is stored in 4 bits and computed in BF16, and only the BF16 LoRA factors train.
- NF4 puts 16 levels at quantiles of N(0,1), with an exact zero, 7 negative and 8 positive levels. After block-wise absmax scaling, each code is used about equally often.
- Double quantization stores the per-64 absmax scales in FP8 with an FP32 scale per 256 blocks, cutting their cost from 0.5 to 0.127 bits per parameter.
- Paged optimizers keep optimizer state in unified memory, which spills to CPU RAM instead of crashing on memory spikes.
- 65B fine-tuning drops from over 780 GB to under 48 GB with no loss against 16-bit. Adapting every linear layer matters more than the rank.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
