QLoRA

QLoRA (Dettmers et al., 2023) trains BF16 LoRA adapters on a frozen 4-bit base. NF4 levels at N(0,1) quantiles, double-quantized block scales (0.5 → 0.127 bits/param) and paged optimizers cut 65B fine-tuning from >780 GB to <48 GB, matching 16-bit results.

15 min read1069 wordsadvanced0 viewsupdated 2026-10-13

Definition

QLoRA is the fine-tuning method of Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs" (University of Washington, arXiv May 2023, NeurIPS 2023). It stores the frozen pretrained model in a 4-bit format, dequantizes each weight matrix to BF16 on the fly for the matmul, and backpropagates through it into LoRA adapters kept in BF16. Three ideas make this work without losing quality: the 4-bit NormalFloat (NF4) data type, double quantization of the per-block scales, and paged optimizers that spill optimizer state to CPU RAM during memory spikes. With them, a 65B model fine-tunes on one 48 GB GPU.

LoRA removed the gradients and Adam state of the base model, but the base weights themselves still had to sit in GPU memory in 16 bits. For LLaMA-65B that is 130 GB before a single activation is stored. The paper puts regular 16-bit full fine-tuning of the same model at more than 780 GB. Quantization to 4 bits had been shown to work for inference: Dettmers and Zettlemoyer (2022) found that 4-bit weights give the best accuracy for a fixed number of model bits. Before QLoRA, nobody had shown that you could also train through a 4-bit model and match 16-bit fine-tuning.

The design separates a storage data type from a computation data type. Weights are stored in NF4 and multiplied in BF16. Gradients flow through the dequantized weights to the activations and on into the adapters, but the 4-bit weights are never updated. Quantization error therefore enters only the forward pass of a fixed base, and the adapters can learn to compensate for it.

The forward pass

YBF16=XBF16 doubleDequant(c1FP32,c2k-bit,WNF4)+XBF16 L1BF16 L2BF16\mathbf Y^{\text{BF16}}=\mathbf X^{\text{BF16}}\,\text{doubleDequant}\big(c_1^{\text{FP32}},c_2^{\text{k-bit}},\mathbf W^{\text{NF4}}\big)+\mathbf X^{\text{BF16}}\,\mathbf L_1^{\text{BF16}}\,\mathbf L_2^{\text{BF16}}
WNF4\mathbf W^{\text{NF4}}
frozen base weight, 4-bit codes in blocks of 64
c2k-bitc_2^{\text{k-bit}}
one absmax scale per block of 64, itself quantized to FP8
c1FP32c_1^{\text{FP32}}
second-level scale, one per 256 first-level scales
doubleDequant\text{doubleDequant}
dequant(dequant(c1,c2),W)\text{dequant}(\text{dequant}(c_1,c_2),\mathbf W): first recover the block scales, then the weights
L1∈Rh×r, L2∈Rr×o\mathbf L_1\in\mathbb R^{h\times r},\ \mathbf L_2\in\mathbb R^{r\times o}
the LoRA factors (the paper's L1,L2L_1,L_2 are LoRA's A⊤,B⊤A^\top,B^\top), the only trainable tensors
Y = X · doubleDequant(c₁, c₂, W) + X · L₁ · L₂, per linear layerfrozen base, in GPU memoryW: NF4 codes4 bits per weightc₂: FP8 absmax1 per block of 64c₁: FP32 scale1 per 256 blocksdoubleDequant→ BF16, on the flyX · WBF16 matmul+Y (BF16)X (BF16)activationsX · L₁ · L₂LoRA, trainableBF16, r = 64,every linear layer∂L/∂X passes through W;W itself gets no gradientpaged optimizerGPU: AdamW statesfor L₁, L₂ only (32-bit)CPU RAMpages evicted on a spikeunifiedmemoryfine-tuning a 65B modelfull FT, 16-bit> 780 GB (paper)16-bit weights130 GB before any training stateQLoRA< 48 GB total: one 48 GB GPUQLoRA weights: 65·10⁹ × 4 bits = 32.5 GB, constants 1.03 GB
fig.One QLoRA linear layer. The frozen base sits in GPU memory as 4-bit NF4 codes, an FP8 absmax for every 64 weights and an FP32 scale for every 256 of those. It is dequantized to BF16 just in time for the matmul X·W. The BF16 adapter path X·L₁·L₂ runs in parallel and is the only thing that learns, applied to every linear layer with r = 64 in the paper. AdamW state for the adapters lives in paged memory that NVIDIA unified memory can evict to CPU RAM during a spike. Bottom: the paper's memory figures for a 65B model.

NF4: levels at the quantiles of a Gaussian

A 4-bit code has 16 values. Plain INT4 spaces them evenly between −absmax-\text{absmax} and +absmax+\text{absmax}. Pretrained weights, however, are roughly zero-centred normal. Most of them crowd near zero, where evenly spaced levels are coarse, and few reach the tails, where levels go unused. Quantile quantization puts the levels where the probability mass is, so each code is used about equally often. For a known distribution this is information-theoretically optimal, because every code carries close to its full 4 bits.

qi=12(QX(i2k+1)+QX(i+12k+1)),QX=Φ−1 for X∼N(0,1),xcode=arg⁡min⁡i∣qi−wmax⁡b∣w∣∣q_i=\tfrac12\Big(Q_X\big(\tfrac{i}{2^k+1}\big)+Q_X\big(\tfrac{i+1}{2^k+1}\big)\Big),\qquad Q_X=\Phi^{-1}\ \text{for}\ X\sim\mathcal N(0,1),\qquad x_{\text{code}}=\arg\min_i\Big|q_i-\frac{w}{\max_{b}|w|}\Big|
QXQ_X
quantile function of the standard normal; the levels are fixed once, independent of the model
max⁡b∣w∣\max_b|w|
absmax of the block of 64 weights, which rescales every block into [−1,1][-1,1] so one table of levels fits all
zero\text{zero}
to represent 0 exactly, NF4 uses 8 positive and 7 negative quantiles plus a 0. The table is asymmetric: the smallest positive level is 0.0796, the smallest negative level is −0.0911

The released levels come from bitsandbytes' construction: take quantiles of N(0,1)\mathcal N(0,1) at probabilities evenly spaced from 0.50.5 to δ=0.9677083=1−12(132+130)\delta=0.9677083=1-\tfrac12(\tfrac1{32}+\tfrac1{30}) (8 positive and 7 negative), add an exact zero, and divide by the largest value. Cutting off at δ\delta rather than 11 avoids the infinite quantile. Recomputed in Python, the 16 levels are −1, −0.6962, −0.5251, −0.3949, −0.2844, −0.1848, −0.0911, 0, 0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1-1,\,-0.6962,\,-0.5251,\,-0.3949,\,-0.2844,\,-0.1848,\,-0.0911,\,0,\,0.0796,\,0.1609,\,0.2461,\,0.3379,\,0.4407,\,0.5626,\,0.7230,\,1, matching the published table to about 10−710^{-7}.

Double quantization and paged optimizers

Each block of 64 weights needs its own absmax, stored as FP32: 32/64=0.532/64=0.5 extra bits per parameter, or 4 GB for a 65B model. Double quantization quantizes those scales as well. Subtract their mean (they are all positive), then store them in FP8 with one FP32 scale per 256 of them. The cost drops to 8/64+32/(64⋅256)=0.1278/64+32/(64\cdot256)=0.127 bits per parameter, a saving of 0.3730.373 bits, or about 3 GB on 65B, with no measurable loss.

Paged optimizers handle the rest. With gradient checkpointing, a long sequence in a mini-batch can briefly push memory over the limit and crash the run. QLoRA allocates the optimizer state in NVIDIA unified memory, which pages it out to CPU RAM automatically when the GPU runs out and pages it back for the optimizer step. The adapters' AdamW state is small, so the paging rarely triggers, but it turns occasional out-of-memory crashes into brief slowdowns.

From the paper

Authors
Dettmers, Pagnoni, Holtzman, Zettlemoyer (University of Washington), arXiv May 2023, NeurIPS 2023; released with bitsandbytes 4-bit kernels and Hugging Face PEFT integration
Quantization
NF4 storage, BF16 compute; block size 64; double quantization with FP8 second level, block size 256
LoRA setup
adapters on all linear layers of every block, r=64r=64, α=16\alpha=16; dropout 0.1 up to 13B, 0.05 for 33B and 65B; learning rate 2⋅10−42\cdot10^{-4} (7B, 13B) and 10−410^{-4} (33B, 65B)
Memory
65B fine-tuning: more than 780 GB for 16-bit full fine-tuning, less than 48 GB for QLoRA
Quality
NF4 + DQ matched 16-bit full fine-tuning and 16-bit LoRA on GLUE and Super-NaturalInstructions, and on 5-shot MMLU for LLaMA 7B–65B fine-tuned on FLAN v2 and Alpaca; NF4 beat FP4 and Int4
Guanaco
LLaMA + QLoRA on the OASST1 dataset; Guanaco 65B reached 99.3% of ChatGPT's score on the Vicuna benchmark (GPT-4 as judge) after 24 hours of fine-tuning on a single GPU; 33B reached 97.8%
Scale of study
more than 1,000 fine-tuned models; a 9k-example dataset (OASST1) beat a 450k-example FLAN v2 subset for chatbot quality, so data quality mattered more than size
4,096 weights ~ N(0, 0.02²), blocks of 64: w / absmax → nearest 4-bit level-1-0.500.51w / absmax, all blocks pooledNF4INT4NF4: 16 quantiles of N(0,1), exact 0, 7 negative and 8 positive · INT4: k/7, k = −7…7NF4 code usage · entropy 3.865 of 4.00 bitsINT4 code usage · entropy 3.470 of 3.91 bitsRMSE / σ: NF4 0.0931 · INT4 0.1124 · INT4 error is 1.207× NF4'sconstants: FP8 absmax per 64 + FP32 per 64·256 → 0.1270 bits/param (plain FP32: 0.5000)LLaMA-65B: 4-bit weights 32.50 GB + constants 1.03 GB = 33.53 GB (BF16: 130 GB)quantile levels sit densest where the weights are; uniform levels waste codes on the tailssmaller blocks: less error, more constants; the paper uses 64 and double quantization
options
fig.4,096 weights from N(0, 0.02²), split into blocks. Each block is divided by its absmax and every value is rounded to the nearest 4-bit level: NF4 (blue) or the 15 evenly spaced INT4 levels k/7 (orange). The NF4 levels crowd the centre, where the histogram is tall. At the default block size of 64, the RMSE is 0.0931σ for NF4 and 0.1124σ for INT4, 1.207× worse. The code-usage bars show why: NF4's 16 codes carry 3.865 bits of entropy, while INT4's 15 codes carry 3.470 of a possible 3.91, because its tail codes are almost never used. Blocks of 16 shrink both errors (0.0831σ vs 0.0876σ) but cost 2 bits per parameter in FP32 scales. One block for the whole tensor (4,096) raises them to 0.1121σ vs 0.1596σ. Add a single 12σ outlier and blocking becomes essential: at 64 it costs little (0.0982σ vs 0.1300σ), at 4,096 it squeezes everything else into a few codes (0.3025σ vs 0.4951σ, NF4 entropy 2.091 bits). Double quantization cuts the scales from 0.5000 to 0.1270 bits per parameter at block 64, and for LLaMA-65B the stored weights shrink from 36.56 GB to 33.53 GB.

Worked example: one block in NF4 and INT4

Take a block of 8 weights. The absmax is c=0.045c=0.045. Divide by it, snap to the nearest level, and multiply back. All values below were computed in numpy.

ww / cNF4 code: levelNF4 ŵ (error)INT4 k/7INT4 ŵ (error)
0.0310.688914: 0.72300.03253 (+0.00153)5/70.03214 (+0.00114)
−0.012−0.26674: −0.2844−0.01280 (−0.00080)−2/7−0.01286 (−0.00086)
0.0040.08898: 0.07960.00358 (−0.00042)1/70.00643 (+0.00243)
−0.045−10: −1−0.04500 (0)−7/7−0.04500 (0)
0.0200.444412: 0.44070.01983 (−0.00017)3/70.01929 (−0.00071)
0.0090.20009: 0.16090.00724 (−0.00176)1/70.00643 (−0.00257)
−0.027−0.60002: −0.5251−0.02363 (+0.00337)−4/7−0.02571 (+0.00129)
0.0150.333311: 0.33790.01521 (+0.00021)2/70.01286 (−0.00214)
w → w/c → nearest level → dequantized value (c = 0.045)

Reading the block, then the 65B budget

  1. 1Small weights are where NF4 wins. 0.0040.004 lands 0.000420.00042 from its NF4 level but 0.002430.00243 from INT4's nearest level 1/71/7, since INT4 has nothing between 00 and 0.1430.143. INT4 wins a few larger weights, such as −0.027-0.027, where NF4's levels are sparser.
  2. 2Over the block, the RMSE is 1.487⋅10−31.487\cdot10^{-3} for NF4 and 1.632⋅10−31.632\cdot10^{-3} for INT4, and the mean absolute error is 1.032⋅10−31.032\cdot10^{-3} vs 1.393⋅10−31.393\cdot10^{-3}.
  3. 3Storage: 8 codes × 4 bits + one 32-bit scale = 64 bits, or 8 bits per weight for this tiny block. At the paper's block of 64 the overhead is 0.50.5 bits, and 0.1270.127 with double quantization.
  4. 465B base, rounded to 65⋅10965\cdot10^9 parameters: 65⋅109×4/8=32.565\cdot10^9\times4/8=32.5 GB of codes plus 65⋅109×0.127/8=1.0365\cdot10^9\times0.127/8=1.03 GB of scales.
  5. 5Adapters on all 7 linear layers of LLaMA-65B (80 layers, d=8192d=8192, dff=22016d_{ff}=22016) at r=64r=64: 80⋅64⋅(4⋅2⋅8192+3⋅(8192+22016))=799,539,20080\cdot64\cdot(4\cdot2\cdot8192+3\cdot(8192+22016))=799{,}539{,}200 parameters. In BF16 with BF16 gradients that is 1.60+1.601.60+1.60 GB, and 32-bit AdamW adds 88 bytes each, 6.406.40 GB.
  6. 6Total before activations: 32.5+1.03+3.20+6.40≈43.132.5+1.03+3.20+6.40\approx43.1 GB. That leaves about 5 GB of a 48 GB card for checkpointed activations, which is why paging matters. This estimate is ours; the paper reports only the under-48 GB total.
python
1import numpy as np
2from statistics import NormalDist
3
4def nf4_levels(offset=0.9677083): # offset = 1 - (1/32 + 1/30) / 2
5 ppf = NormalDist().inv_cdf
6 pos = [ppf(p) for p in np.linspace(offset, 0.5, 9)[:-1]] # 8 positive quantiles
7 neg = [-ppf(p) for p in np.linspace(offset, 0.5, 8)[:-1]] # 7 negative quantiles
8 v = np.sort(np.array(pos + [0.0] + neg)) # plus an exact zero
9 return v / np.abs(v).max()
10
11NF4 = nf4_levels()
12INT4 = np.arange(-7, 8) / 7
13
14def quantize(w, levels, B=64):
15 """Block-wise absmax quantization: 4-bit codes plus one FP32 scale per block."""
16 blocks = w.reshape(-1, B)
17 c = np.abs(blocks).max(axis=1)
18 x = blocks / c[:, None] # every block now spans [-1, 1]
19 edges = (levels[1:] + levels[:-1]) / 2 # nearest level via midpoints
20 return np.searchsorted(edges, x).astype(np.uint8), c
21
22def dequantize(codes, c, levels):
23 return (levels[codes] * c[:, None]).ravel()
24
25def double_quantize(c, B2=256):
26 """Quantize the FP32 scales themselves (paper: FP8, here int8 for brevity), one FP32 per 256."""
27 mean = c.mean() # scales are positive: centre them
28 x = (c - mean).reshape(-1, B2)
29 s = np.abs(x).max(axis=1) / 127
30 return np.round(x / s[:, None]).astype(np.int8), s, mean
31
32def double_dequantize(q, s, mean):
33 return (q * s[:, None]).ravel() + mean
34
35rng = np.random.default_rng(0)
36w = rng.normal(0, 0.02, size=4096 * 1024).astype(np.float32) # one 4096 x 1024 weight
37print(np.round(NF4, 4))
38# [-1. -0.6962 -0.5251 -0.3949 -0.2844 -0.1848 -0.091 0. 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.723 1.]
39
40for name, L in [("NF4", NF4), ("INT4", INT4)]:
41 codes, c = quantize(w, L)
42 err = dequantize(codes, c, L) - w
43 print(f"{name}: RMSE/sigma = {np.sqrt((err ** 2).mean()) / 0.02:.4f}") # NF4 0.0920, INT4 0.1077
44
45codes, c = quantize(w, NF4)
46q, s, mean = double_quantize(c)
47err = dequantize(codes, double_dequantize(q, s, mean), NF4) - w
48print(f"NF4 + DQ: RMSE/sigma = {np.sqrt((err ** 2).mean()) / 0.02:.4f}") # 0.0920: DQ costs nothing visible
49print(f"scale bits/param: {32 / 64:.3f} -> {8 / 64 + 32 / (64 * 256):.3f}") # 0.500 -> 0.127
NF4 from scratch in numpy: build the 16 levels from Gaussian quantiles, quantize block-wise, and double-quantize the scales (run; outputs in comments)
python
1import torch
2from transformers import AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments
3from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
4
5bnb = BitsAndBytesConfig(
6 load_in_4bit=True,
7 bnb_4bit_quant_type="nf4", # storage type: NormalFloat4
8 bnb_4bit_use_double_quant=True, # quantize the block scales too
9 bnb_4bit_compute_dtype=torch.bfloat16, # computation type
10)
11model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf",
12 quantization_config=bnb, device_map="auto")
13model = prepare_model_for_kbit_training(model) # gradient checkpointing, norms in fp32
14
15cfg = LoraConfig(r=64, lora_alpha=16, lora_dropout=0.1, task_type="CAUSAL_LM",
16 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", # every linear layer,
17 "gate_proj", "up_proj", "down_proj"]) # as in the paper
18model = get_peft_model(model, cfg)
19# trainable: 32 layers * 64 * (4*2*4096 + 3*(4096 + 11008)) = 159,907,840
20
21args = TrainingArguments(
22 output_dir="qlora-out",
23 optim="paged_adamw_32bit", # paged optimizer state
24 learning_rate=2e-4, lr_scheduler_type="constant", max_grad_norm=0.3,
25 per_device_train_batch_size=4, gradient_accumulation_steps=4,
26 gradient_checkpointing=True, bf16=True,
27)
The standard QLoRA recipe with transformers, bitsandbytes and PEFT (not run here; the trainable count was computed by hand)

Use when

  • +Fine-tunes a 65B model on one 48 GB GPU and 7B–13B models on consumer cards
  • +Matches 16-bit full fine-tuning and 16-bit LoRA on the paper's benchmarks
  • +NF4 levels are fixed and model-independent; quantizing a checkpoint needs no calibration data
  • +Paged optimizers prevent out-of-memory crashes on long-sequence spikes

Avoid when

  • −Slower per step than 16-bit LoRA: every forward and backward dequantizes each weight
  • −Merged models cannot stay in NF4 without re-quantization error
  • −NF4 assumes roughly Gaussian blocks; outlier-heavy weights rely on small blocks
  • −Activation memory is unchanged, so long contexts still need checkpointing and paging

Check yourself

Why does NF4 give lower error than evenly spaced INT4 on pretrained weights at the same 4 bits?

Check yourself

With block size 64, how many bits per parameter do the quantization constants cost with and without double quantization (FP8 scales, one FP32 per 256 blocks)?

Remember

  • ✓QLoRA: Y=X doubleDequant(c1,c2,WNF4)+XL1L2\mathbf Y=\mathbf X\,\text{doubleDequant}(c_1,c_2,\mathbf W^{\text{NF4}})+\mathbf X\mathbf L_1\mathbf L_2. The base is stored in 4 bits and computed in BF16, and only the BF16 LoRA factors train.
  • ✓NF4 puts 16 levels at quantiles of N(0,1), with an exact zero, 7 negative and 8 positive levels. After block-wise absmax scaling, each code is used about equally often.
  • ✓Double quantization stores the per-64 absmax scales in FP8 with an FP32 scale per 256 blocks, cutting their cost from 0.5 to 0.127 bits per parameter.
  • ✓Paged optimizers keep optimizer state in unified memory, which spills to CPU RAM instead of crashing on memory spikes.
  • ✓65B fine-tuning drops from over 780 GB to under 48 GB with no loss against 16-bit. Adapting every linear layer matters more than the rank.
← back to Deep Learning roadmap