Quantization

Quantization stores LLM weights (and optionally activations and the KV cache) as low-bit integers or FP8 with ŵ = s·(q − z) per tensor, channel or group. GPTQ, AWQ and SmoothQuant handle outliers, so a 7B model shrinks from 13.5 GB in FP16 to about 3.5 GB at 4 bits.

16 min read1819 wordscore0 viewsupdated 2026-10-13

Definition

Quantization maps real-valued tensors onto a small set of representable values, typically 8-, 4- or 3-bit integers or 8-bit floats. Each value is stored as an integer code qq plus a shared scale ss and, optionally, a zero-point zz, so that w^=s (q−z)\hat w=s\,(q-z). The affine scheme with integer-only arithmetic was laid out by Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" (Google, CVPR 2018). For LLMs, the key methods are LLM.int8() (Dettmers, Lewis, Belkada, Zettlemoyer, NeurIPS 2022), GPTQ (Frantar, Ashkboos, Hoefler, Alistarh, ICLR 2023), SmoothQuant (Xiao, Lin, Seznec, Wu, Demouth, Han, ICML 2023) and AWQ (Lin et al., MLSys 2024).

An LLM is mostly weight matrices. A 7B-parameter model in FP16 needs 13.5 GB just for its weights. A 70B model needs 140 GB, more than one 80 GB GPU. The memory is a problem, and so is the time. Generating one token at batch size 1 reads every weight once and does only two FLOPs per weight read. This decode step is limited by memory bandwidth, so time per token is roughly proportional to bytes of weights. Halving the bytes nearly halves the latency.

CNNs had been quantized to INT8 for years by the time LLMs arrived. LLMs brought two new problems. First, models are too expensive to retrain, so methods must work post-training from a small calibration set. Second, beyond a few billion parameters, transformer activations develop outlier features: a handful of hidden dimensions with magnitudes far above the rest. Naive INT8 quantization of such activations collapses accuracy. Most of the LLM quantization literature is about handling these outliers.

Affine and symmetric quantization

q=clamp ⁣(⌊ws⌉+z,  0,  2b−1),w^=s (q−z),s=wmax⁡−wmin⁡2b−1,z=⌊−wmin⁡s⌉q=\mathrm{clamp}\!\Big(\Big\lfloor\frac{w}{s}\Big\rceil+z,\;0,\;2^b-1\Big),\qquad \hat w=s\,(q-z),\qquad s=\frac{w_{\max}-w_{\min}}{2^b-1},\qquad z=\Big\lfloor-\frac{w_{\min}}{s}\Big\rceil
bb
bit width: 8, 4, 3 or 2
ss
scale (step size), stored in FP16 per block
zz
zero-point: the integer code that represents real 00 exactly, so zero padding and ReLU zeros cost no error
⌊⋅⌉\lfloor\cdot\rceil
round to nearest (RTN); the error per weight is at most s/2s/2 unless clipped
symmetric: z=0,s=max⁡∣w∣2b−1−1,q∈{−(2b−1−1),…,2b−1−1}\text{symmetric: } z=0,\quad s=\frac{\max|w|}{2^{b-1}-1},\quad q\in\{-(2^{b-1}-1),\dots,2^{b-1}-1\}
2b−1−12^{b-1}-1
127 for INT8, 7 for INT4: the range is restricted to be symmetric around zero
why\text{why}
with z=0z=0, the integer product ∑qwqx\sum q_w q_x needs no zero-point correction terms

Because the step is uniform, the error is governed by the range of the block. One weight that is 10 times larger than the rest stretches ss tenfold. Every ordinary weight then lands on one of a few levels near zero. The signal-to-quantization-noise ratio of uniform quantization grows by about 6 dB per bit, and a single outlier can cost several bits.

1 · affine (asymmetric) 4-bit quantization of one block: q = clamp(round(w/s) + z, 0, 15), ŵ = s·(q − z)real wcode q0123456789101112131415-1.20-0.500.000.832.40s = (max − min)/(2^4 − 1) = 3.6/15 = 0.24 · z = round(1.2/0.24) = 5 · symmetric: z = 0, s = max|w|/7e.g. 0.83/0.24 = 3.46 → 3, q = 3 + z = 8, ŵ = 0.24·(8 − 5) = 0.72;w = 0 maps exactly to q = z2 · granularity: which weights share one (s, z)per-tensor1 scale per matrixper-channel1 per output rowper-group g = 128+16/128 = 0.125 bit/wfiner blocks: smaller ranges, so smaller steps and less error, but more scales to store3 · two ways to run a quantized linear layer Y = X·WᵀW4A16GPTQ, AWQINT4 W + FP16 s4.125 bit/weightdequantizein registersFP16 GEMMX stays FP16Y FP16memory-boundW8A8SmoothQuantX → INT8per-token s_xW INT8per-channel s_wINT8 GEMMINT32 accumulate× s_x · s_wrescale to FP16Y FP16compute-bound
fig.1. An affine 4-bit grid for a block whose values span [−1.20, 2.40]. The scale is 3.6/15 = 0.24, and the zero-point z = 5 makes real 0 an exact code. The value 0.83 rounds to code 8 and is reconstructed as 0.24·(8 − 5) = 0.72. 2. Granularity: the same matrix can share one scale, one per output channel, or one per group of 128 consecutive weights, which adds 16/128 = 0.125 bits per weight for the FP16 scales. 3. The two deployment paths. Weight-only W4A16 dequantizes inside the GEMM kernel and helps memory-bound decoding. W8A8 runs an INT8 GEMM with INT32 accumulation and rescales by s_x·s_w, which helps compute-bound prefill and large batches.

Granularity: per-tensor, per-channel, per-group

The block that shares one (s,z)(s,z) is a design choice. Per-tensor uses one scale for the whole matrix. It is cheap but hostage to the largest weight anywhere. Per-channel gives each output row of WW its own scale. It costs nothing in an INT8 GEMM, because a per-row scale factors out of each dot product. Per-group splits each row into groups of gg consecutive input weights, usually g=128g=128, sometimes 64 or 32. This is the standard for 4-bit weight-only formats. The cost is storage: an FP16 scale per 128 weights adds 16/128=0.12516/128=0.125 bits per weight, and an INT4 zero-point adds another 4/1284/128.

Activations change at every step and cannot be calibrated once and for all. Per-token scaling computes one scale per row of XX on the fly. Per-tensor static scaling fixes a scale from calibration data. Since Y=XW⊤Y=XW^\top, a per-token scale on XX and a per-channel scale on WW both factor out of each output element: yij≈siX sjW∑kqikXqjkWy_{ij}\approx s^X_i\,s^W_j\sum_k q^X_{ik}q^W_{jk}. Per-input-channel scaling of XX, the direction where the outliers live, does not factor out this way. That is the root of the outlier problem.

PTQ and QAT

Post-training quantization (PTQ) quantizes a trained model with no gradient updates, using at most a few hundred calibration sequences to set scales or correct errors. Everything below, from RTN to GPTQ, AWQ and SmoothQuant, is PTQ. Quantization-aware training (QAT) inserts fake quantization w↦s⌊w/s⌉w\mapsto s\lfloor w/s\rceil in the forward pass and backpropagates through the rounding with the straight-through estimator, treating ∂⌊x⌉/∂x≈1\partial\lfloor x\rceil/\partial x\approx1. QAT recovers more accuracy at very low bit widths but needs training compute and data. LLM-QAT (Liu et al., 2023) distills from the full-precision model on text the model generates itself. BitNet b1.58 (Ma et al., 2024) goes further and trains from scratch with ternary weights {−1,0,1}\{-1,0,1\}. Low-bit fine-tuning of a frozen quantized base is the territory of QLoRA.

Outliers and LLM.int8()

Dettmers et al. found that from about 6.7B parameters, a small number of hidden dimensions carry very large values in most layers and most tokens. Clipping them destroys performance. Quantizing XX per-tensor with them in range wastes the grid on everything else. LLM.int8() splits the matrix multiply. Feature dimensions where any activation exceeds a threshold of 6.06.0 are multiplied in FP16. The rest, more than 99.9% of the values, go through INT8 vector-wise quantization: a scale per row of XX and per column of WW.

XW=X:,O WO,:⏟FP16, outlier dims O+sX ⁣⊙(Q:,OˉX QOˉ,:W)⊙sW⏟INT8, everything elseXW=\underbrace{X_{:,O}\,W_{O,:}}_{\text{FP16, outlier dims }O}+\underbrace{s^X\!\odot\big(Q^X_{:,\bar O}\,Q^W_{\bar O,:}\big)\odot s^W}_{\text{INT8, everything else}}
OO
columns of XX with some ∣x∣≥6|x|\ge6; a handful of dimensions in large models
sX, sWs^X,\ s^W
absmax scales per row of X and per column of W (vector-wise)

The result was zero-degradation INT8 inference up to 175B parameters. The cost is speed: gathering outlier columns and running two GEMMs made it slower than FP16 at small scales. The method's main contribution was to diagnose the outlier problem. The next methods find faster ways around it.

GPTQ: correct the error you just made

RTN rounds each weight independently. But the layer's loss is about outputs, and output errors from different weights can cancel. GPTQ minimises the layer-wise reconstruction error ∥XW⊤−XW^⊤∥2\|XW^\top-X\hat W^\top\|^2 on calibration inputs, one row of WW at a time, all rows in parallel. Its Hessian is the same for every row: H=2X⊤XH=2X^\top X. Following Optimal Brain Quantization (Frantar and Alistarh, 2022), it quantizes one input column jj and then updates the not-yet-quantized columns to absorb the error:

δ:,F=−w:,j−quant(w:,j)[HF−1]jj (HF−1)j,:,H=2X⊤X+λI\delta_{:,F}=-\frac{w_{:,j}-\mathrm{quant}(w_{:,j})}{[H_F^{-1}]_{jj}}\,(H_F^{-1})_{j,:},\qquad H=2X^\top X+\lambda I
FF
the set of columns not yet quantized; j is the next one
HF−1H_F^{-1}
inverse Hessian restricted to FF; GPTQ gets all of them at once from one Cholesky factor of H−1H^{-1}
λ\lambda
dampening, 1% of the mean diagonal, for numerical stability

Three engineering choices made this scale. All rows quantize their columns in the same fixed order, so H−1H^{-1} is shared. Updates are applied lazily in blocks of 128 columns to stay compute-bound. The Cholesky form avoids accumulating numerical error. GPTQ quantized OPT-175B and BLOOM-176B to 3–4 bits in about four GPU hours, with small perplexity increases at 4 bits, and fit OPT-175B on a single 80 GB A100.

AWQ and SmoothQuant: rescale before you round

Both methods use the same identity: for any positive per-input-channel vector ss,   XW⊤=(X diag(s)−1)(W diag(s))⊤\;XW^\top=(X\,\mathrm{diag}(s)^{-1})(W\,\mathrm{diag}(s))^\top. The product is unchanged in exact arithmetic, but the quantization difficulty moves between XX and WW. The diag(s)−1\mathrm{diag}(s)^{-1} on XX is folded into the preceding LayerNorm or linear layer, so it costs nothing at inference.

SmoothQuant: sj=max⁡∣X:,j∣αmax⁡∣W:,j∣1−α, α=0.5AWQ: sj=∣X:,j∣‾ α,  α∗=arg⁡min⁡α∈[0,1]∥Q(Wdiag(s))(diag(s)−1X⊤)−WX⊤∥\text{SmoothQuant: } s_j=\frac{\max|X_{:,j}|^{\alpha}}{\max|W_{:,j}|^{1-\alpha}},\ \alpha=0.5\qquad\qquad \text{AWQ: } s_j=\overline{|X_{:,j}|}^{\,\alpha},\ \ \alpha^\ast=\arg\min_{\alpha\in[0,1]}\big\|Q(W\mathrm{diag}(s))(\mathrm{diag}(s)^{-1}X^\top)-WX^\top\big\|
SmoothQuant\text{SmoothQuant}
divides outlier activation channels down and multiplies the matching weight columns up, so both become INT8-friendly (W8A8)
α\alpha
migration strength; 0.5 splits the difficulty evenly, and models with harsher outliers use larger values
AWQ\text{AWQ}
weight-only: scales up the weight channels that see large activations, so their relative rounding error shrinks; α\alpha is grid-searched per layer on calibration data

AWQ's observation is that a small fraction of weights matter far more than the rest, about 1% of input channels, and they are found by activation magnitude, not weight magnitude. Keeping them in FP16 recovers most of the accuracy, but mixed precision is awkward for kernels. Scaling those channels by s>1s>1 before rounding protects them almost as well while keeping one uniform INT4 format. AWQ needs no backpropagation or reconstruction, so it overfits the calibration set less than GPTQ. SmoothQuant made W8A8 work for OPT-175B, BLOOM-176B, GLM-130B and MT-NLG 530B, with up to 1.56× speedup and 2× memory reduction.

Numbers from the papers and formats

LLM.int8()
outlier threshold 6.06.0; outlier features emerge sharply around 6.7B parameters; more than 99.9% of values multiplied in INT8; no degradation up to 175B
GPTQ
128 random 2048-token segments from C4 for calibration; 3–4 bit weights, group size 128 in common use; OPT-175B in about 4 GPU hours; block updates of 128 columns
AWQ
INT4 weights, group size 128; per-channel scale s=∣x∣‾αs=\overline{|x|}^{\alpha} with a grid search over α∈[0,1]\alpha\in[0,1]; no backprop or reconstruction
SmoothQuant
W8A8 with per-token or per-tensor activation scales; α=0.5\alpha=0.5 default; up to 1.56× speedup and 2× memory reduction
FP8
Micikevicius et al. 2022: E4M3 (max 448, used for weights and activations) and E5M2 (max 57,344, used for gradients); native on NVIDIA H100 tensor cores
Block formats
OCP microscaling MXFP4: 4-bit E2M1 elements in blocks of 32 sharing an 8-bit power-of-two scale; llama.cpp GGUF k-quants use super-blocks of 256 weights
Memory, 7B (6.74B params)
FP16 13.48 GB · INT8 6.74 GB · INT4 g128 (4.125 bit) 3.48 GB · INT3 g128 2.63 GB

Weight-only versus W8A8

Weight-only schemes (W4A16, W3A16; GPTQ, AWQ, GGUF) keep activations in FP16. The kernel loads packed INT4 weights, dequantizes them in registers, and runs an FP16 matrix multiply. The arithmetic is unchanged, and the gain comes from moving a quarter of the bytes. That suits decoding at small batch sizes, which is memory-bound. Weight-and-activation schemes (W8A8 INT8 via SmoothQuant, or FP8) feed integer or FP8 tensor cores directly, which roughly doubles peak throughput over FP16. That helps compute-bound work: prefill of long prompts and large-batch serving. The two can be combined, for example W4A8.

Quantizing the KV cache

For long contexts and large batches the KV cache can exceed the weights in size, since it grows with batch×context×layers\text{batch}\times\text{context}\times\text{layers}. Storing it in FP8 or INT8 halves that memory and the bandwidth spent on attention. More aggressive schemes exploit its structure. KIVI (Liu et al., ICML 2024) observed that keys have outlier channels, so it quantizes them per-channel. Values have no such channels and are quantized per-token. With this split, a 2-bit KV cache keeps quality close to FP16. The most recent tokens stay in full precision until a group fills.

symmetric: s = max|w| / (2^(b−1) − 1), ŵ = s·round(w/s)W|ŵ−w|8 output channels × 64 inputs · boxes = blocks sharing one scale (1)rounding error (saturates at 0.5)row 2: ● w · — ŵ · grid 15 levels per block7.50weight SQNR10.24 dBMSE0.09398output error ‖ΔW·X‖/‖W·X‖32.94%scale, row 2 block 01.0714bits / weight here4.031Llama-2-7B weights (FP16 13.48 GB)3.37 GBround-to-nearest · 16-bit scale per block
granularity
scheme
fig.An 8 × 64 weight matrix with a different spread per output row and three outliers (7.5 in row 2, −6.0 in row 5, 5.2 in row 6), quantized by round-to-nearest. The output error is measured on 32 random input vectors. Default: 4-bit symmetric per-tensor. The outlier sets s = 7.5/7 = 1.0714, so row 2's ordinary weights land on a handful of levels near zero. Weight SQNR is 10.24 dB and output error 32.94%, at 4.031 bits per weight (Llama-2-7B: 3.37 GB). Per-channel gives 15.21 dB and 18.48%. Group 16 gives 18.93 dB and 11.83%, but at 5.000 bits per weight (4.21 GB), because a 16-bit scale per 16 weights is expensive. Group 32 gives 17.22 dB at 4.500 bits. Removing the outliers lifts per-tensor INT4 to 14.72 dB. Keeping the three |w| > 4 in FP16 instead gives 15.89 dB at 4.219 bits. Affine group 16 reaches 22.73 dB (8.13% output error) at 5.250 bits. 8-bit per-tensor is 35.03 dB (1.93%), and 2-bit per-tensor collapses to 1.11 dB (92.23%). Each extra bit adds about 6 dB.

Worked example: one outlier in eight weights

Quantize w=(0.12,−0.50,0.33,0.05,2.40,−0.08,0.21,−0.27)w=(0.12,-0.50,0.33,0.05,\mathbf{2.40},-0.08,0.21,-0.27) to symmetric INT4 (codes −7…7-7\ldots7) in three ways.

wper-tensor, s = 0.3429groups of 4groups of 4 + 2.40 in FP16
0.1200.14290.1429
−0.50−0.3429−0.5000−0.5000
0.330.34290.35710.3571
0.0500.07140.0714
2.402.40002.40002.40 (FP16)
−0.0800−0.0771
0.210.34290.34290.1929
−0.27−0.3429−0.3429−0.2700
MSE0.008890.003880.00025
SQNR19.45 dB23.04 dB34.91 dB
Reconstructed weights ŵ for each scheme (computed in Python)

Step by step

  1. 1Per-tensor: s=2.40/7=0.3429s=2.40/7=0.3429. The ratios w/sw/s are 0.35,−1.46,0.96,0.15,7,−0.23,0.61,−0.790.35,-1.46,0.96,0.15,7,-0.23,0.61,-0.79, which round to 0,−1,1,0,7,0,1,−10,-1,1,0,7,0,1,-1. Only 4 of the 15 levels are used, and three weights become exactly 0.
  2. 2Groups of 4: the first group's range is 0.500.50, so s1=0.50/7=0.0714s_1=0.50/7=0.0714 and the codes are 2,−7,5,12,-7,5,1. The error there drops to at most 0.0230.023. The second group still contains 2.40 and keeps s2=0.3429s_2=0.3429. MSE falls from 0.008890.00889 to 0.003880.00388.
  3. 3Pull 2.40 out into an FP16 side list: the second group's range becomes 0.270.27, s2=0.0386s_2=0.0386, codes −2,5,−7-2,5,-7. MSE falls to 0.000250.00025, 36 times lower than per-tensor, and SQNR rises from 19.45 to 34.91 dB.
  4. 4Storage per weight for these 8 weights: per-tensor 4+16/8=64+16/8=6 bits; two groups 4+32/8=84+32/8=8 bits; plus one outlier at 32 bits (value and index), 1212 bits. On a real layer with groups of 128 the scale overhead is only 0.1250.125 bit, and outliers are a tiny fraction of weights.
  5. 5This is the LLM.int8() idea at the scale of one vector. GPTQ and AWQ reach the same goal without a side list: GPTQ compensates the error with other weights, and AWQ rescales channels before rounding.
python
1import numpy as np
2
3rng = np.random.default_rng(0)
4d_in, d_out, n = 512, 256, 4096
5# calibration activations: correlated features (they live near a 64-dim subspace),
6# and a few input features are 20x larger, like LLM outlier features
7X = rng.normal(size=(n, 64)) @ rng.normal(size=(64, d_in)) + 0.3 * rng.normal(size=(n, d_in))
8X[:, rng.choice(d_in, 8, replace=False)] *= 20.0
9W = rng.normal(0, 0.02, (d_out, d_in))
10
11def quantize(w, bits, scale):
12 """Symmetric round-to-nearest with a given scale (broadcast)."""
13 qmax = 2 ** (bits - 1) - 1
14 return np.clip(np.round(w / scale), -qmax, qmax) * scale
15
16def group_scale(w, bits): # one FP16 scale per row of this block
17 return np.abs(w).max(axis=1, keepdims=True) / (2 ** (bits - 1) - 1)
18
19def rtn(W, bits=4, group=128):
20 Q = np.empty_like(W)
21 for g in range(0, W.shape[1], group):
22 blk = W[:, g:g + group]
23 Q[:, g:g + group] = quantize(blk, bits, group_scale(blk, bits))
24 return Q
25
26def gptq(W, X, bits=4, group=128, damp=0.01):
27 """GPTQ (Frantar et al. 2023): quantize column by column, push each column's error onto the
28 not-yet-quantized columns using the inverse Hessian of the layer loss ||XW^T - XQ^T||^2."""
29 W = W.copy()
30 H = 2 * X.T @ X # Hessian of the layer-wise squared error
31 H += damp * np.mean(np.diag(H)) * np.eye(H.shape[0])
32 U = np.linalg.cholesky(np.linalg.inv(H)).T # upper Cholesky factor of H^-1
33 Q = np.zeros_like(W)
34 for j in range(W.shape[1]):
35 if j % group == 0: # scales from the (already updated) block
36 s = group_scale(W[:, j:j + group], bits)
37 Q[:, j] = quantize(W[:, j:j + 1], bits, s)[:, 0]
38 err = (W[:, j] - Q[:, j]) / U[j, j]
39 W[:, j:] -= np.outer(err, U[j, j:]) # compensate on the remaining columns
40 return Q
41
42def out_err(Q):
43 return np.linalg.norm(X @ (W - Q).T) / np.linalg.norm(X @ W.T)
44
45for bits in (4, 3):
46 print(f"{bits}-bit, g=128: RTN output error {out_err(rtn(W, bits)):.4f}, "
47 f"GPTQ {out_err(gptq(W, X, bits)):.4f}")
48# 4-bit, g=128: RTN output error 0.1188, GPTQ 0.0244
49# 3-bit, g=128: RTN output error 0.2769, GPTQ 0.0733
50
51# memory of a 7B model's weights (6.74e9 parameters), FP16 scale per group of 128
52for name, b in [("FP16", 16), ("INT8", 8), ("INT4 g128", 4 + 16 / 128), ("INT3 g128", 3 + 16 / 128)]:
53 print(f"{name:10s} {b:6.3f} bits/weight {6.74e9 * b / 8 / 1e9:6.2f} GB")
54# FP16 13.48 GB, INT8 6.74 GB, INT4 g128 3.48 GB, INT3 g128 2.63 GB
RTN vs GPTQ at 4 and 3 bits with group size 128 on a toy layer with correlated, outlier-heavy activations, plus 7B memory math (numpy; run, printed values from that run)

GPTQ cuts the output error about fivefold at 4 bits here, because the correlated inputs let later columns cancel earlier rounding errors. With independent input features, HH is diagonal and GPTQ reduces to RTN. In practice you rarely write this loop. Hugging Face transformers loads GPTQ, AWQ, bitsandbytes (LLM.int8() and NF4) and FP8 checkpoints through a quantization_config. vLLM, TensorRT-LLM and llama.cpp provide the fused low-bit kernels.

Use when

  • +4× less weight memory than FP16 at 4 bits; a 70B model fits in about 36 GB
  • +Faster memory-bound decoding (weight-only), or faster compute-bound prefill on INT8/FP8 tensor cores (W8A8)
  • +PTQ needs only a few hundred calibration sequences and minutes to hours of GPU time
  • +Combines with KV-cache quantization, Speculative decoding and Pruning

Avoid when

  • −Accuracy drops grow quickly below 4 bits and for smaller models
  • −Activation outliers make naive INT8 activation quantization fail beyond a few billion parameters
  • −Speedups depend on fused kernels for each format and GPU; unpacking can make small batches slower than expected
  • −Calibration-based methods can overfit their calibration domain

Where it's used

Local and edge inference

GGUF k-quants in llama.cpp and Ollama run 7–70B models on laptops and phones.

Single-GPU serving

GPTQ/AWQ INT4 checkpoints let a 70B model run on one or two GPUs instead of four or more.

High-throughput serving

FP8 or INT8 W8A8 plus an FP8 KV cache on H100-class GPUs.

Low-memory fine-tuning

QLoRA trains adapters on top of a frozen 4-bit (NF4) base.

Check yourself

A weight row has 127 values within ±0.05 and one value of 1.27. With symmetric INT8 per-channel quantization, what is the step size, and what happens to the small weights?

Check yourself

SmoothQuant computes Y = (X·diag(s)⁻¹)·(diag(s)·Wᵀ) with s_j > 1 on outlier channels. What does this achieve?

Remember

  • ✓w^=s(q−z)\hat w=s(q-z) with s=range/(2b−1)s=\text{range}/(2^b-1). Error per weight is at most s/2s/2, so the block range sets the error, and each extra bit gives about 6 dB.
  • ✓Granularity trades accuracy for metadata: per-tensor, then per-channel, then per-group (g = 128 adds 0.125 bits per weight).
  • ✓From about 6.7B parameters, a few activation features become huge outliers. LLM.int8() splits them off into FP16; SmoothQuant moves them into the weights.
  • ✓GPTQ uses H=2X⊤XH=2X^\top X to push each column's rounding error onto unquantized columns. AWQ scales the weight channels that see large activations before rounding.
  • ✓Weight-only INT4 speeds memory-bound decoding. W8A8 and FP8 (E4M3/E5M2) speed compute-bound prefill. A 7B model shrinks from 13.48 GB to about 3.5 GB at 4 bits.
← back to Deep Learning roadmap