Definition
Quantization maps real-valued tensors onto a small set of representable values, typically 8-, 4- or 3-bit integers or 8-bit floats. Each value is stored as an integer code plus a shared scale and, optionally, a zero-point , so that . The affine scheme with integer-only arithmetic was laid out by Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" (Google, CVPR 2018). For LLMs, the key methods are LLM.int8() (Dettmers, Lewis, Belkada, Zettlemoyer, NeurIPS 2022), GPTQ (Frantar, Ashkboos, Hoefler, Alistarh, ICLR 2023), SmoothQuant (Xiao, Lin, Seznec, Wu, Demouth, Han, ICML 2023) and AWQ (Lin et al., MLSys 2024).
An LLM is mostly weight matrices. A 7B-parameter model in FP16 needs 13.5 GB just for its weights. A 70B model needs 140 GB, more than one 80 GB GPU. The memory is a problem, and so is the time. Generating one token at batch size 1 reads every weight once and does only two FLOPs per weight read. This decode step is limited by memory bandwidth, so time per token is roughly proportional to bytes of weights. Halving the bytes nearly halves the latency.
CNNs had been quantized to INT8 for years by the time LLMs arrived. LLMs brought two new problems. First, models are too expensive to retrain, so methods must work post-training from a small calibration set. Second, beyond a few billion parameters, transformer activations develop outlier features: a handful of hidden dimensions with magnitudes far above the rest. Naive INT8 quantization of such activations collapses accuracy. Most of the LLM quantization literature is about handling these outliers.
Affine and symmetric quantization
- bit width: 8, 4, 3 or 2
- scale (step size), stored in FP16 per block
- zero-point: the integer code that represents real exactly, so zero padding and ReLU zeros cost no error
- round to nearest (RTN); the error per weight is at most unless clipped
- 127 for INT8, 7 for INT4: the range is restricted to be symmetric around zero
- with , the integer product needs no zero-point correction terms
Because the step is uniform, the error is governed by the range of the block. One weight that is 10 times larger than the rest stretches tenfold. Every ordinary weight then lands on one of a few levels near zero. The signal-to-quantization-noise ratio of uniform quantization grows by about 6 dB per bit, and a single outlier can cost several bits.
Granularity: per-tensor, per-channel, per-group
The block that shares one is a design choice. Per-tensor uses one scale for the whole matrix. It is cheap but hostage to the largest weight anywhere. Per-channel gives each output row of its own scale. It costs nothing in an INT8 GEMM, because a per-row scale factors out of each dot product. Per-group splits each row into groups of consecutive input weights, usually , sometimes 64 or 32. This is the standard for 4-bit weight-only formats. The cost is storage: an FP16 scale per 128 weights adds bits per weight, and an INT4 zero-point adds another .
Activations change at every step and cannot be calibrated once and for all. Per-token scaling computes one scale per row of on the fly. Per-tensor static scaling fixes a scale from calibration data. Since , a per-token scale on and a per-channel scale on both factor out of each output element: . Per-input-channel scaling of , the direction where the outliers live, does not factor out this way. That is the root of the outlier problem.
PTQ and QAT
Post-training quantization (PTQ) quantizes a trained model with no gradient updates, using at most a few hundred calibration sequences to set scales or correct errors. Everything below, from RTN to GPTQ, AWQ and SmoothQuant, is PTQ. Quantization-aware training (QAT) inserts fake quantization in the forward pass and backpropagates through the rounding with the straight-through estimator, treating . QAT recovers more accuracy at very low bit widths but needs training compute and data. LLM-QAT (Liu et al., 2023) distills from the full-precision model on text the model generates itself. BitNet b1.58 (Ma et al., 2024) goes further and trains from scratch with ternary weights . Low-bit fine-tuning of a frozen quantized base is the territory of QLoRA.
Outliers and LLM.int8()
Dettmers et al. found that from about 6.7B parameters, a small number of hidden dimensions carry very large values in most layers and most tokens. Clipping them destroys performance. Quantizing per-tensor with them in range wastes the grid on everything else. LLM.int8() splits the matrix multiply. Feature dimensions where any activation exceeds a threshold of are multiplied in FP16. The rest, more than 99.9% of the values, go through INT8 vector-wise quantization: a scale per row of and per column of .
- columns of with some ; a handful of dimensions in large models
- absmax scales per row of X and per column of W (vector-wise)
The result was zero-degradation INT8 inference up to 175B parameters. The cost is speed: gathering outlier columns and running two GEMMs made it slower than FP16 at small scales. The method's main contribution was to diagnose the outlier problem. The next methods find faster ways around it.
GPTQ: correct the error you just made
RTN rounds each weight independently. But the layer's loss is about outputs, and output errors from different weights can cancel. GPTQ minimises the layer-wise reconstruction error on calibration inputs, one row of at a time, all rows in parallel. Its Hessian is the same for every row: . Following Optimal Brain Quantization (Frantar and Alistarh, 2022), it quantizes one input column and then updates the not-yet-quantized columns to absorb the error:
- the set of columns not yet quantized; j is the next one
- inverse Hessian restricted to ; GPTQ gets all of them at once from one Cholesky factor of
- dampening, 1% of the mean diagonal, for numerical stability
Three engineering choices made this scale. All rows quantize their columns in the same fixed order, so is shared. Updates are applied lazily in blocks of 128 columns to stay compute-bound. The Cholesky form avoids accumulating numerical error. GPTQ quantized OPT-175B and BLOOM-176B to 3–4 bits in about four GPU hours, with small perplexity increases at 4 bits, and fit OPT-175B on a single 80 GB A100.
AWQ and SmoothQuant: rescale before you round
Both methods use the same identity: for any positive per-input-channel vector , . The product is unchanged in exact arithmetic, but the quantization difficulty moves between and . The on is folded into the preceding LayerNorm or linear layer, so it costs nothing at inference.
- divides outlier activation channels down and multiplies the matching weight columns up, so both become INT8-friendly (W8A8)
- migration strength; 0.5 splits the difficulty evenly, and models with harsher outliers use larger values
- weight-only: scales up the weight channels that see large activations, so their relative rounding error shrinks; is grid-searched per layer on calibration data
AWQ's observation is that a small fraction of weights matter far more than the rest, about 1% of input channels, and they are found by activation magnitude, not weight magnitude. Keeping them in FP16 recovers most of the accuracy, but mixed precision is awkward for kernels. Scaling those channels by before rounding protects them almost as well while keeping one uniform INT4 format. AWQ needs no backpropagation or reconstruction, so it overfits the calibration set less than GPTQ. SmoothQuant made W8A8 work for OPT-175B, BLOOM-176B, GLM-130B and MT-NLG 530B, with up to 1.56× speedup and 2× memory reduction.
Numbers from the papers and formats
- LLM.int8()
- outlier threshold ; outlier features emerge sharply around 6.7B parameters; more than 99.9% of values multiplied in INT8; no degradation up to 175B
- GPTQ
- 128 random 2048-token segments from C4 for calibration; 3–4 bit weights, group size 128 in common use; OPT-175B in about 4 GPU hours; block updates of 128 columns
- AWQ
- INT4 weights, group size 128; per-channel scale with a grid search over ; no backprop or reconstruction
- SmoothQuant
- W8A8 with per-token or per-tensor activation scales; default; up to 1.56× speedup and 2× memory reduction
- FP8
- Micikevicius et al. 2022: E4M3 (max 448, used for weights and activations) and E5M2 (max 57,344, used for gradients); native on NVIDIA H100 tensor cores
- Block formats
- OCP microscaling MXFP4: 4-bit E2M1 elements in blocks of 32 sharing an 8-bit power-of-two scale; llama.cpp GGUF k-quants use super-blocks of 256 weights
- Memory, 7B (6.74B params)
- FP16 13.48 GB · INT8 6.74 GB · INT4 g128 (4.125 bit) 3.48 GB · INT3 g128 2.63 GB
Weight-only versus W8A8
Weight-only schemes (W4A16, W3A16; GPTQ, AWQ, GGUF) keep activations in FP16. The kernel loads packed INT4 weights, dequantizes them in registers, and runs an FP16 matrix multiply. The arithmetic is unchanged, and the gain comes from moving a quarter of the bytes. That suits decoding at small batch sizes, which is memory-bound. Weight-and-activation schemes (W8A8 INT8 via SmoothQuant, or FP8) feed integer or FP8 tensor cores directly, which roughly doubles peak throughput over FP16. That helps compute-bound work: prefill of long prompts and large-batch serving. The two can be combined, for example W4A8.
Quantizing the KV cache
For long contexts and large batches the KV cache can exceed the weights in size, since it grows with . Storing it in FP8 or INT8 halves that memory and the bandwidth spent on attention. More aggressive schemes exploit its structure. KIVI (Liu et al., ICML 2024) observed that keys have outlier channels, so it quantizes them per-channel. Values have no such channels and are quantized per-token. With this split, a 2-bit KV cache keeps quality close to FP16. The most recent tokens stay in full precision until a group fills.
Worked example: one outlier in eight weights
Quantize to symmetric INT4 (codes ) in three ways.
| w | per-tensor, s = 0.3429 | groups of 4 | groups of 4 + 2.40 in FP16 |
|---|---|---|---|
| 0.12 | 0 | 0.1429 | 0.1429 |
| −0.50 | −0.3429 | −0.5000 | −0.5000 |
| 0.33 | 0.3429 | 0.3571 | 0.3571 |
| 0.05 | 0 | 0.0714 | 0.0714 |
| 2.40 | 2.4000 | 2.4000 | 2.40 (FP16) |
| −0.08 | 0 | 0 | −0.0771 |
| 0.21 | 0.3429 | 0.3429 | 0.1929 |
| −0.27 | −0.3429 | −0.3429 | −0.2700 |
| MSE | 0.00889 | 0.00388 | 0.00025 |
| SQNR | 19.45 dB | 23.04 dB | 34.91 dB |
Step by step
- 1Per-tensor: . The ratios are , which round to . Only 4 of the 15 levels are used, and three weights become exactly 0.
- 2Groups of 4: the first group's range is , so and the codes are . The error there drops to at most . The second group still contains 2.40 and keeps . MSE falls from to .
- 3Pull 2.40 out into an FP16 side list: the second group's range becomes , , codes . MSE falls to , 36 times lower than per-tensor, and SQNR rises from 19.45 to 34.91 dB.
- 4Storage per weight for these 8 weights: per-tensor bits; two groups bits; plus one outlier at 32 bits (value and index), bits. On a real layer with groups of 128 the scale overhead is only bit, and outliers are a tiny fraction of weights.
- 5This is the LLM.int8() idea at the scale of one vector. GPTQ and AWQ reach the same goal without a side list: GPTQ compensates the error with other weights, and AWQ rescales channels before rounding.
1import numpy as np23rng = np.random.default_rng(0)4d_in, d_out, n = 512, 256, 40965# calibration activations: correlated features (they live near a 64-dim subspace),6# and a few input features are 20x larger, like LLM outlier features7X = rng.normal(size=(n, 64)) @ rng.normal(size=(64, d_in)) + 0.3 * rng.normal(size=(n, d_in))8X[:, rng.choice(d_in, 8, replace=False)] *= 20.09W = rng.normal(0, 0.02, (d_out, d_in))1011def quantize(w, bits, scale):12 """Symmetric round-to-nearest with a given scale (broadcast)."""13 qmax = 2 ** (bits - 1) - 114 return np.clip(np.round(w / scale), -qmax, qmax) * scale1516def group_scale(w, bits): # one FP16 scale per row of this block17 return np.abs(w).max(axis=1, keepdims=True) / (2 ** (bits - 1) - 1)1819def rtn(W, bits=4, group=128):20 Q = np.empty_like(W)21 for g in range(0, W.shape[1], group):22 blk = W[:, g:g + group]23 Q[:, g:g + group] = quantize(blk, bits, group_scale(blk, bits))24 return Q2526def gptq(W, X, bits=4, group=128, damp=0.01):27 """GPTQ (Frantar et al. 2023): quantize column by column, push each column's error onto the28 not-yet-quantized columns using the inverse Hessian of the layer loss ||XW^T - XQ^T||^2."""29 W = W.copy()30 H = 2 * X.T @ X # Hessian of the layer-wise squared error31 H += damp * np.mean(np.diag(H)) * np.eye(H.shape[0])32 U = np.linalg.cholesky(np.linalg.inv(H)).T # upper Cholesky factor of H^-133 Q = np.zeros_like(W)34 for j in range(W.shape[1]):35 if j % group == 0: # scales from the (already updated) block36 s = group_scale(W[:, j:j + group], bits)37 Q[:, j] = quantize(W[:, j:j + 1], bits, s)[:, 0]38 err = (W[:, j] - Q[:, j]) / U[j, j]39 W[:, j:] -= np.outer(err, U[j, j:]) # compensate on the remaining columns40 return Q4142def out_err(Q):43 return np.linalg.norm(X @ (W - Q).T) / np.linalg.norm(X @ W.T)4445for bits in (4, 3):46 print(f"{bits}-bit, g=128: RTN output error {out_err(rtn(W, bits)):.4f}, "47 f"GPTQ {out_err(gptq(W, X, bits)):.4f}")48# 4-bit, g=128: RTN output error 0.1188, GPTQ 0.024449# 3-bit, g=128: RTN output error 0.2769, GPTQ 0.07335051# memory of a 7B model's weights (6.74e9 parameters), FP16 scale per group of 12852for name, b in [("FP16", 16), ("INT8", 8), ("INT4 g128", 4 + 16 / 128), ("INT3 g128", 3 + 16 / 128)]:53 print(f"{name:10s} {b:6.3f} bits/weight {6.74e9 * b / 8 / 1e9:6.2f} GB")54# FP16 13.48 GB, INT8 6.74 GB, INT4 g128 3.48 GB, INT3 g128 2.63 GBGPTQ cuts the output error about fivefold at 4 bits here, because the correlated inputs let later columns cancel earlier rounding errors. With independent input features, is diagonal and GPTQ reduces to RTN. In practice you rarely write this loop. Hugging Face transformers loads GPTQ, AWQ, bitsandbytes (LLM.int8() and NF4) and FP8 checkpoints through a quantization_config. vLLM, TensorRT-LLM and llama.cpp provide the fused low-bit kernels.
Use when
- +4× less weight memory than FP16 at 4 bits; a 70B model fits in about 36 GB
- +Faster memory-bound decoding (weight-only), or faster compute-bound prefill on INT8/FP8 tensor cores (W8A8)
- +PTQ needs only a few hundred calibration sequences and minutes to hours of GPU time
- +Combines with KV-cache quantization, Speculative decoding and Pruning
Avoid when
- −Accuracy drops grow quickly below 4 bits and for smaller models
- −Activation outliers make naive INT8 activation quantization fail beyond a few billion parameters
- −Speedups depend on fused kernels for each format and GPU; unpacking can make small batches slower than expected
- −Calibration-based methods can overfit their calibration domain
Where it's used
Local and edge inference
GGUF k-quants in llama.cpp and Ollama run 7–70B models on laptops and phones.
Single-GPU serving
GPTQ/AWQ INT4 checkpoints let a 70B model run on one or two GPUs instead of four or more.
High-throughput serving
FP8 or INT8 W8A8 plus an FP8 KV cache on H100-class GPUs.
Low-memory fine-tuning
QLoRA trains adapters on top of a frozen 4-bit (NF4) base.
Check yourself
A weight row has 127 values within ±0.05 and one value of 1.27. With symmetric INT8 per-channel quantization, what is the step size, and what happens to the small weights?
Check yourself
SmoothQuant computes Y = (X·diag(s)⁻¹)·(diag(s)·Wᵀ) with s_j > 1 on outlier channels. What does this achieve?
Remember
- ✓ with . Error per weight is at most , so the block range sets the error, and each extra bit gives about 6 dB.
- ✓Granularity trades accuracy for metadata: per-tensor, then per-channel, then per-group (g = 128 adds 0.125 bits per weight).
- ✓From about 6.7B parameters, a few activation features become huge outliers. LLM.int8() splits them off into FP16; SmoothQuant moves them into the weights.
- ✓GPTQ uses to push each column's rounding error onto unquantized columns. AWQ scales the weight channels that see large activations before rounding.
- ✓Weight-only INT4 speeds memory-bound decoding. W8A8 and FP8 (E4M3/E5M2) speed compute-bound prefill. A 7B model shrinks from 13.48 GB to about 3.5 GB at 4 bits.