Definition
QLoRA is the fine-tuning method of Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs" (University of Washington, arXiv May 2023, NeurIPS 2023). It stores the frozen pretrained model in a 4-bit format, dequantizes each weight matrix to BF16 on the fly for the matmul, and backpropagates through it into LoRA adapters kept in BF16. Three ideas make this work without losing quality: the 4-bit NormalFloat (NF4) data type, double quantization of the per-block scales, and paged optimizers that spill optimizer state to CPU RAM during memory spikes. With them, a 65B model fine-tunes on one 48 GB GPU.
LoRA removed the gradients and Adam state of the base model, but the base weights themselves still had to sit in GPU memory in 16 bits. For LLaMA-65B that is 130 GB before a single activation is stored. The paper puts regular 16-bit full fine-tuning of the same model at more than 780 GB. Quantization to 4 bits had been shown to work for inference: Dettmers and Zettlemoyer (2022) found that 4-bit weights give the best accuracy for a fixed number of model bits. Before QLoRA, nobody had shown that you could also train through a 4-bit model and match 16-bit fine-tuning.
The design separates a storage data type from a computation data type. Weights are stored in NF4 and multiplied in BF16. Gradients flow through the dequantized weights to the activations and on into the adapters, but the 4-bit weights are never updated. Quantization error therefore enters only the forward pass of a fixed base, and the adapters can learn to compensate for it.
The forward pass
- frozen base weight, 4-bit codes in blocks of 64
- one absmax scale per block of 64, itself quantized to FP8
- second-level scale, one per 256 first-level scales
- : first recover the block scales, then the weights
- the LoRA factors (the paper's are LoRA's ), the only trainable tensors
NF4: levels at the quantiles of a Gaussian
A 4-bit code has 16 values. Plain INT4 spaces them evenly between and . Pretrained weights, however, are roughly zero-centred normal. Most of them crowd near zero, where evenly spaced levels are coarse, and few reach the tails, where levels go unused. Quantile quantization puts the levels where the probability mass is, so each code is used about equally often. For a known distribution this is information-theoretically optimal, because every code carries close to its full 4 bits.
- quantile function of the standard normal; the levels are fixed once, independent of the model
- absmax of the block of 64 weights, which rescales every block into so one table of levels fits all
- to represent 0 exactly, NF4 uses 8 positive and 7 negative quantiles plus a 0. The table is asymmetric: the smallest positive level is 0.0796, the smallest negative level is −0.0911
The released levels come from bitsandbytes' construction: take quantiles of at probabilities evenly spaced from to (8 positive and 7 negative), add an exact zero, and divide by the largest value. Cutting off at rather than avoids the infinite quantile. Recomputed in Python, the 16 levels are , matching the published table to about .
Double quantization and paged optimizers
Each block of 64 weights needs its own absmax, stored as FP32: extra bits per parameter, or 4 GB for a 65B model. Double quantization quantizes those scales as well. Subtract their mean (they are all positive), then store them in FP8 with one FP32 scale per 256 of them. The cost drops to bits per parameter, a saving of bits, or about 3 GB on 65B, with no measurable loss.
Paged optimizers handle the rest. With gradient checkpointing, a long sequence in a mini-batch can briefly push memory over the limit and crash the run. QLoRA allocates the optimizer state in NVIDIA unified memory, which pages it out to CPU RAM automatically when the GPU runs out and pages it back for the optimizer step. The adapters' AdamW state is small, so the paging rarely triggers, but it turns occasional out-of-memory crashes into brief slowdowns.
From the paper
- Authors
- Dettmers, Pagnoni, Holtzman, Zettlemoyer (University of Washington), arXiv May 2023, NeurIPS 2023; released with bitsandbytes 4-bit kernels and Hugging Face PEFT integration
- Quantization
- NF4 storage, BF16 compute; block size 64; double quantization with FP8 second level, block size 256
- LoRA setup
- adapters on all linear layers of every block, , ; dropout 0.1 up to 13B, 0.05 for 33B and 65B; learning rate (7B, 13B) and (33B, 65B)
- Memory
- 65B fine-tuning: more than 780 GB for 16-bit full fine-tuning, less than 48 GB for QLoRA
- Quality
- NF4 + DQ matched 16-bit full fine-tuning and 16-bit LoRA on GLUE and Super-NaturalInstructions, and on 5-shot MMLU for LLaMA 7B–65B fine-tuned on FLAN v2 and Alpaca; NF4 beat FP4 and Int4
- Guanaco
- LLaMA + QLoRA on the OASST1 dataset; Guanaco 65B reached 99.3% of ChatGPT's score on the Vicuna benchmark (GPT-4 as judge) after 24 hours of fine-tuning on a single GPU; 33B reached 97.8%
- Scale of study
- more than 1,000 fine-tuned models; a 9k-example dataset (OASST1) beat a 450k-example FLAN v2 subset for chatbot quality, so data quality mattered more than size
Worked example: one block in NF4 and INT4
Take a block of 8 weights. The absmax is . Divide by it, snap to the nearest level, and multiply back. All values below were computed in numpy.
| w | w / c | NF4 code: level | NF4 ŵ (error) | INT4 k/7 | INT4 ŵ (error) |
|---|---|---|---|---|---|
| 0.031 | 0.6889 | 14: 0.7230 | 0.03253 (+0.00153) | 5/7 | 0.03214 (+0.00114) |
| −0.012 | −0.2667 | 4: −0.2844 | −0.01280 (−0.00080) | −2/7 | −0.01286 (−0.00086) |
| 0.004 | 0.0889 | 8: 0.0796 | 0.00358 (−0.00042) | 1/7 | 0.00643 (+0.00243) |
| −0.045 | −1 | 0: −1 | −0.04500 (0) | −7/7 | −0.04500 (0) |
| 0.020 | 0.4444 | 12: 0.4407 | 0.01983 (−0.00017) | 3/7 | 0.01929 (−0.00071) |
| 0.009 | 0.2000 | 9: 0.1609 | 0.00724 (−0.00176) | 1/7 | 0.00643 (−0.00257) |
| −0.027 | −0.6000 | 2: −0.5251 | −0.02363 (+0.00337) | −4/7 | −0.02571 (+0.00129) |
| 0.015 | 0.3333 | 11: 0.3379 | 0.01521 (+0.00021) | 2/7 | 0.01286 (−0.00214) |
Reading the block, then the 65B budget
- 1Small weights are where NF4 wins. lands from its NF4 level but from INT4's nearest level , since INT4 has nothing between and . INT4 wins a few larger weights, such as , where NF4's levels are sparser.
- 2Over the block, the RMSE is for NF4 and for INT4, and the mean absolute error is vs .
- 3Storage: 8 codes × 4 bits + one 32-bit scale = 64 bits, or 8 bits per weight for this tiny block. At the paper's block of 64 the overhead is bits, and with double quantization.
- 465B base, rounded to parameters: GB of codes plus GB of scales.
- 5Adapters on all 7 linear layers of LLaMA-65B (80 layers, , ) at : parameters. In BF16 with BF16 gradients that is GB, and 32-bit AdamW adds bytes each, GB.
- 6Total before activations: GB. That leaves about 5 GB of a 48 GB card for checkpointed activations, which is why paging matters. This estimate is ours; the paper reports only the under-48 GB total.
1import numpy as np2from statistics import NormalDist34def nf4_levels(offset=0.9677083): # offset = 1 - (1/32 + 1/30) / 25 ppf = NormalDist().inv_cdf6 pos = [ppf(p) for p in np.linspace(offset, 0.5, 9)[:-1]] # 8 positive quantiles7 neg = [-ppf(p) for p in np.linspace(offset, 0.5, 8)[:-1]] # 7 negative quantiles8 v = np.sort(np.array(pos + [0.0] + neg)) # plus an exact zero9 return v / np.abs(v).max()1011NF4 = nf4_levels()12INT4 = np.arange(-7, 8) / 71314def quantize(w, levels, B=64):15 """Block-wise absmax quantization: 4-bit codes plus one FP32 scale per block."""16 blocks = w.reshape(-1, B)17 c = np.abs(blocks).max(axis=1)18 x = blocks / c[:, None] # every block now spans [-1, 1]19 edges = (levels[1:] + levels[:-1]) / 2 # nearest level via midpoints20 return np.searchsorted(edges, x).astype(np.uint8), c2122def dequantize(codes, c, levels):23 return (levels[codes] * c[:, None]).ravel()2425def double_quantize(c, B2=256):26 """Quantize the FP32 scales themselves (paper: FP8, here int8 for brevity), one FP32 per 256."""27 mean = c.mean() # scales are positive: centre them28 x = (c - mean).reshape(-1, B2)29 s = np.abs(x).max(axis=1) / 12730 return np.round(x / s[:, None]).astype(np.int8), s, mean3132def double_dequantize(q, s, mean):33 return (q * s[:, None]).ravel() + mean3435rng = np.random.default_rng(0)36w = rng.normal(0, 0.02, size=4096 * 1024).astype(np.float32) # one 4096 x 1024 weight37print(np.round(NF4, 4))38# [-1. -0.6962 -0.5251 -0.3949 -0.2844 -0.1848 -0.091 0. 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.723 1.]3940for name, L in [("NF4", NF4), ("INT4", INT4)]:41 codes, c = quantize(w, L)42 err = dequantize(codes, c, L) - w43 print(f"{name}: RMSE/sigma = {np.sqrt((err ** 2).mean()) / 0.02:.4f}") # NF4 0.0920, INT4 0.10774445codes, c = quantize(w, NF4)46q, s, mean = double_quantize(c)47err = dequantize(codes, double_dequantize(q, s, mean), NF4) - w48print(f"NF4 + DQ: RMSE/sigma = {np.sqrt((err ** 2).mean()) / 0.02:.4f}") # 0.0920: DQ costs nothing visible49print(f"scale bits/param: {32 / 64:.3f} -> {8 / 64 + 32 / (64 * 256):.3f}") # 0.500 -> 0.1271import torch2from transformers import AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments3from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training45bnb = BitsAndBytesConfig(6 load_in_4bit=True,7 bnb_4bit_quant_type="nf4", # storage type: NormalFloat48 bnb_4bit_use_double_quant=True, # quantize the block scales too9 bnb_4bit_compute_dtype=torch.bfloat16, # computation type10)11model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf",12 quantization_config=bnb, device_map="auto")13model = prepare_model_for_kbit_training(model) # gradient checkpointing, norms in fp321415cfg = LoraConfig(r=64, lora_alpha=16, lora_dropout=0.1, task_type="CAUSAL_LM",16 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", # every linear layer,17 "gate_proj", "up_proj", "down_proj"]) # as in the paper18model = get_peft_model(model, cfg)19# trainable: 32 layers * 64 * (4*2*4096 + 3*(4096 + 11008)) = 159,907,8402021args = TrainingArguments(22 output_dir="qlora-out",23 optim="paged_adamw_32bit", # paged optimizer state24 learning_rate=2e-4, lr_scheduler_type="constant", max_grad_norm=0.3,25 per_device_train_batch_size=4, gradient_accumulation_steps=4,26 gradient_checkpointing=True, bf16=True,27)Use when
- +Fine-tunes a 65B model on one 48 GB GPU and 7B–13B models on consumer cards
- +Matches 16-bit full fine-tuning and 16-bit LoRA on the paper's benchmarks
- +NF4 levels are fixed and model-independent; quantizing a checkpoint needs no calibration data
- +Paged optimizers prevent out-of-memory crashes on long-sequence spikes
Avoid when
- −Slower per step than 16-bit LoRA: every forward and backward dequantizes each weight
- −Merged models cannot stay in NF4 without re-quantization error
- −NF4 assumes roughly Gaussian blocks; outlier-heavy weights rely on small blocks
- −Activation memory is unchanged, so long contexts still need checkpointing and paging
Check yourself
Why does NF4 give lower error than evenly spaced INT4 on pretrained weights at the same 4 bits?
Check yourself
With block size 64, how many bits per parameter do the quantization constants cost with and without double quantization (FP8 scales, one FP32 per 256 blocks)?
Remember
- ✓QLoRA: . The base is stored in 4 bits and computed in BF16, and only the BF16 LoRA factors train.
- ✓NF4 puts 16 levels at quantiles of N(0,1), with an exact zero, 7 negative and 8 positive levels. After block-wise absmax scaling, each code is used about equally often.
- ✓Double quantization stores the per-64 absmax scales in FP8 with an FP32 scale per 256 blocks, cutting their cost from 0.5 to 0.127 bits per parameter.
- ✓Paged optimizers keep optimizer state in unified memory, which spills to CPU RAM instead of crashing on memory spikes.
- ✓65B fine-tuning drops from over 780 GB to under 48 GB with no loss against 16-bit. Adapting every linear layer matters more than the rank.