# Quantization

> Quantization stores LLM weights (and optionally activations and the KV cache) as low-bit integers or FP8 with ŵ = s·(q − z) per tensor, channel or group. GPTQ, AWQ and SmoothQuant handle outliers, so a 7B model shrinks from 13.5 GB in FP16 to about 3.5 GB at 4 bits.

Source: https://www.mlfoundry.in/learn/quantization  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 16 min read · updated 2026-10-13

**Quantization:** maps real-valued tensors onto a small set of representable values, typically 8-, 4- or 3-bit integers or 8-bit floats. Each value is stored as an integer code $q$ plus a shared **scale** $s$ and, optionally, a **zero-point** $z$, so that $\hat w=s\,(q-z)$. The affine scheme with integer-only arithmetic was laid out by Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" (Google, CVPR 2018). For LLMs, the key methods are **LLM.int8()** (Dettmers, Lewis, Belkada, Zettlemoyer, NeurIPS 2022), **GPTQ** (Frantar, Ashkboos, Hoefler, Alistarh, ICLR 2023), **SmoothQuant** (Xiao, Lin, Seznec, Wu, Demouth, Han, ICML 2023) and **AWQ** (Lin et al., MLSys 2024).

An LLM is mostly weight matrices. A 7B-parameter model in FP16 needs 13.5 GB just for its weights. A 70B model needs 140 GB, more than one 80 GB GPU. The memory is a problem, and so is the time. Generating one token at batch size 1 reads every weight once and does only two FLOPs per weight read. This decode step is limited by **memory bandwidth**, so time per token is roughly proportional to bytes of weights. Halving the bytes nearly halves the latency.

CNNs had been quantized to INT8 for years by the time LLMs arrived. LLMs brought two new problems. First, models are too expensive to retrain, so methods must work **post-training** from a small calibration set. Second, beyond a few billion parameters, transformer activations develop **outlier features**: a handful of hidden dimensions with magnitudes far above the rest. Naive INT8 quantization of such activations collapses accuracy. Most of the LLM quantization literature is about handling these outliers.

## Affine and symmetric quantization

$$
q=\mathrm{clamp}\!\Big(\Big\lfloor\frac{w}{s}\Big\rceil+z,\;0,\;2^b-1\Big),\qquad \hat w=s\,(q-z),\qquad s=\frac{w_{\max}-w_{\min}}{2^b-1},\qquad z=\Big\lfloor-\frac{w_{\min}}{s}\Big\rceil
$$

- $b$: bit width: 8, 4, 3 or 2
- $s$: scale (step size), stored in FP16 per block
- $z$: zero-point: the integer code that represents real $0$ exactly, so zero padding and ReLU zeros cost no error
- $\lfloor\cdot\rceil$: round to nearest (**RTN**); the error per weight is at most $s/2$ unless clipped

$$
\text{symmetric: } z=0,\quad s=\frac{\max|w|}{2^{b-1}-1},\quad q\in\{-(2^{b-1}-1),\dots,2^{b-1}-1\}
$$

- $2^{b-1}-1$: 127 for INT8, 7 for INT4: the range is restricted to be symmetric around zero
- $\text{why}$: with $z=0$, the integer product $\sum q_w q_x$ needs no zero-point correction terms

Because the step is uniform, the error is governed by the range of the block. One weight that is 10 times larger than the rest stretches $s$ tenfold. Every ordinary weight then lands on one of a few levels near zero. The signal-to-quantization-noise ratio of uniform quantization grows by about 6 dB per bit, and a single outlier can cost several bits.

> Figure: 1. An affine 4-bit grid for a block whose values span [−1.20, 2.40]. The scale is 3.6/15 = 0.24, and the zero-point z = 5 makes real 0 an exact code. The value 0.83 rounds to code 8 and is reconstructed as 0.24·(8 − 5) = 0.72. 2. Granularity: the same matrix can share one scale, one per output channel, or one per group of 128 consecutive weights, which adds 16/128 = 0.125 bits per weight for the FP16 scales. 3. The two deployment paths. Weight-only W4A16 dequantizes inside the GEMM kernel and helps memory-bound decoding. W8A8 runs an INT8 GEMM with INT32 accumulation and rescales by s_x·s_w, which helps compute-bound prefill and large batches.

## Granularity: per-tensor, per-channel, per-group

The block that shares one $(s,z)$ is a design choice. **Per-tensor** uses one scale for the whole matrix. It is cheap but hostage to the largest weight anywhere. **Per-channel** gives each output row of $W$ its own scale. It costs nothing in an INT8 GEMM, because a per-row scale factors out of each dot product. **Per-group** splits each row into groups of $g$ consecutive input weights, usually $g=128$, sometimes 64 or 32. This is the standard for 4-bit weight-only formats. The cost is storage: an FP16 scale per 128 weights adds $16/128=0.125$ bits per weight, and an INT4 zero-point adds another $4/128$.

Activations change at every step and cannot be calibrated once and for all. **Per-token** scaling computes one scale per row of $X$ on the fly. **Per-tensor static** scaling fixes a scale from calibration data. Since $Y=XW^\top$, a per-token scale on $X$ and a per-channel scale on $W$ both factor out of each output element: $y_{ij}\approx s^X_i\,s^W_j\sum_k q^X_{ik}q^W_{jk}$. Per-input-channel scaling of $X$, the direction where the outliers live, does not factor out this way. That is the root of the outlier problem.

## PTQ and QAT

**Post-training quantization (PTQ)** quantizes a trained model with no gradient updates, using at most a few hundred calibration sequences to set scales or correct errors. Everything below, from RTN to GPTQ, AWQ and SmoothQuant, is PTQ. **Quantization-aware training (QAT)** inserts fake quantization $w\mapsto s\lfloor w/s\rceil$ in the forward pass and backpropagates through the rounding with the **straight-through estimator**, treating $\partial\lfloor x\rceil/\partial x\approx1$. QAT recovers more accuracy at very low bit widths but needs training compute and data. LLM-QAT (Liu et al., 2023) distills from the full-precision model on text the model generates itself. BitNet b1.58 (Ma et al., 2024) goes further and trains from scratch with ternary weights $\{-1,0,1\}$. Low-bit fine-tuning of a frozen quantized base is the territory of **QLoRA**.

## Outliers and LLM.int8()

Dettmers et al. found that from about 6.7B parameters, a small number of hidden dimensions carry very large values in most layers and most tokens. Clipping them destroys performance. Quantizing $X$ per-tensor with them in range wastes the grid on everything else. **LLM.int8()** splits the matrix multiply. Feature dimensions where any activation exceeds a threshold of $6.0$ are multiplied in FP16. The rest, more than 99.9% of the values, go through INT8 **vector-wise** quantization: a scale per row of $X$ and per column of $W$.

$$
XW=\underbrace{X_{:,O}\,W_{O,:}}_{\text{FP16, outlier dims }O}+\underbrace{s^X\!\odot\big(Q^X_{:,\bar O}\,Q^W_{\bar O,:}\big)\odot s^W}_{\text{INT8, everything else}}
$$

- $O$: columns of $X$ with some $|x|\ge6$; a handful of dimensions in large models
- $s^X,\ s^W$: absmax scales per row of X and per column of W (vector-wise)

The result was zero-degradation INT8 inference up to 175B parameters. The cost is speed: gathering outlier columns and running two GEMMs made it slower than FP16 at small scales. The method's main contribution was to diagnose the outlier problem. The next methods find faster ways around it.

## GPTQ: correct the error you just made

RTN rounds each weight independently. But the layer's loss is about outputs, and output errors from different weights can cancel. GPTQ minimises the layer-wise reconstruction error $\|XW^\top-X\hat W^\top\|^2$ on calibration inputs, one row of $W$ at a time, all rows in parallel. Its Hessian is the same for every row: $H=2X^\top X$. Following Optimal Brain Quantization (Frantar and Alistarh, 2022), it quantizes one input column $j$ and then updates the not-yet-quantized columns to absorb the error:

$$
\delta_{:,F}=-\frac{w_{:,j}-\mathrm{quant}(w_{:,j})}{[H_F^{-1}]_{jj}}\,(H_F^{-1})_{j,:},\qquad H=2X^\top X+\lambda I
$$

- $F$: the set of columns not yet quantized; j is the next one
- $H_F^{-1}$: inverse Hessian restricted to $F$; GPTQ gets all of them at once from one Cholesky factor of $H^{-1}$
- $\lambda$: dampening, 1% of the mean diagonal, for numerical stability

Three engineering choices made this scale. All rows quantize their columns in the **same fixed order**, so $H^{-1}$ is shared. Updates are applied **lazily** in blocks of 128 columns to stay compute-bound. The **Cholesky** form avoids accumulating numerical error. GPTQ quantized OPT-175B and BLOOM-176B to 3–4 bits in about four GPU hours, with small perplexity increases at 4 bits, and fit OPT-175B on a single 80 GB A100.

## AWQ and SmoothQuant: rescale before you round

Both methods use the same identity: for any positive per-input-channel vector $s$, $\;XW^\top=(X\,\mathrm{diag}(s)^{-1})(W\,\mathrm{diag}(s))^\top$. The product is unchanged in exact arithmetic, but the quantization difficulty moves between $X$ and $W$. The $\mathrm{diag}(s)^{-1}$ on $X$ is folded into the preceding LayerNorm or linear layer, so it costs nothing at inference.

$$
\text{SmoothQuant: } s_j=\frac{\max|X_{:,j}|^{\alpha}}{\max|W_{:,j}|^{1-\alpha}},\ \alpha=0.5\qquad\qquad \text{AWQ: } s_j=\overline{|X_{:,j}|}^{\,\alpha},\ \ \alpha^\ast=\arg\min_{\alpha\in[0,1]}\big\|Q(W\mathrm{diag}(s))(\mathrm{diag}(s)^{-1}X^\top)-WX^\top\big\|
$$

- $\text{SmoothQuant}$: divides outlier activation channels down and multiplies the matching weight columns up, so **both** become INT8-friendly (W8A8)
- $\alpha$: migration strength; 0.5 splits the difficulty evenly, and models with harsher outliers use larger values
- $\text{AWQ}$: weight-only: scales up the weight channels that see large activations, so their *relative* rounding error shrinks; $\alpha$ is grid-searched per layer on calibration data

AWQ's observation is that a small fraction of weights matter far more than the rest, about 1% of input channels, and they are found by **activation** magnitude, not weight magnitude. Keeping them in FP16 recovers most of the accuracy, but mixed precision is awkward for kernels. Scaling those channels by $s>1$ before rounding protects them almost as well while keeping one uniform INT4 format. AWQ needs no backpropagation or reconstruction, so it overfits the calibration set less than GPTQ. SmoothQuant made W8A8 work for OPT-175B, BLOOM-176B, GLM-130B and MT-NLG 530B, with up to 1.56× speedup and 2× memory reduction.

**Numbers from the papers and formats**

- **LLM.int8():** outlier threshold $6.0$; outlier features emerge sharply around 6.7B parameters; more than 99.9% of values multiplied in INT8; no degradation up to 175B
- **GPTQ:** 128 random 2048-token segments from C4 for calibration; 3–4 bit weights, group size 128 in common use; OPT-175B in about 4 GPU hours; block updates of 128 columns
- **AWQ:** INT4 weights, group size 128; per-channel scale $s=\overline{|x|}^{\alpha}$ with a grid search over $\alpha\in[0,1]$; no backprop or reconstruction
- **SmoothQuant:** W8A8 with per-token or per-tensor activation scales; $\alpha=0.5$ default; up to 1.56× speedup and 2× memory reduction
- **FP8:** Micikevicius et al. 2022: **E4M3** (max 448, used for weights and activations) and **E5M2** (max 57,344, used for gradients); native on NVIDIA H100 tensor cores
- **Block formats:** OCP microscaling MXFP4: 4-bit E2M1 elements in blocks of 32 sharing an 8-bit power-of-two scale; llama.cpp GGUF k-quants use super-blocks of 256 weights
- **Memory, 7B (6.74B params):** FP16 13.48 GB · INT8 6.74 GB · INT4 g128 (4.125 bit) 3.48 GB · INT3 g128 2.63 GB

## Weight-only versus W8A8

**Weight-only** schemes (W4A16, W3A16; GPTQ, AWQ, GGUF) keep activations in FP16. The kernel loads packed INT4 weights, dequantizes them in registers, and runs an FP16 matrix multiply. The arithmetic is unchanged, and the gain comes from moving a quarter of the bytes. That suits **decoding** at small batch sizes, which is memory-bound. **Weight-and-activation** schemes (W8A8 INT8 via SmoothQuant, or FP8) feed integer or FP8 tensor cores directly, which roughly doubles peak throughput over FP16. That helps **compute-bound** work: prefill of long prompts and large-batch serving. The two can be combined, for example W4A8.

## Quantizing the KV cache

For long contexts and large batches the **KV cache** can exceed the weights in size, since it grows with $\text{batch}\times\text{context}\times\text{layers}$. Storing it in FP8 or INT8 halves that memory and the bandwidth spent on attention. More aggressive schemes exploit its structure. KIVI (Liu et al., ICML 2024) observed that **keys** have outlier *channels*, so it quantizes them per-channel. **Values** have no such channels and are quantized per-token. With this split, a 2-bit KV cache keeps quality close to FP16. The most recent tokens stay in full precision until a group fills.

> Figure: An 8 × 64 weight matrix with a different spread per output row and three outliers (7.5 in row 2, −6.0 in row 5, 5.2 in row 6), quantized by round-to-nearest. The output error is measured on 32 random input vectors. Default: 4-bit symmetric per-tensor. The outlier sets s = 7.5/7 = 1.0714, so row 2's ordinary weights land on a handful of levels near zero. Weight SQNR is 10.24 dB and output error 32.94%, at 4.031 bits per weight (Llama-2-7B: 3.37 GB). Per-channel gives 15.21 dB and 18.48%. Group 16 gives 18.93 dB and 11.83%, but at 5.000 bits per weight (4.21 GB), because a 16-bit scale per 16 weights is expensive. Group 32 gives 17.22 dB at 4.500 bits. Removing the outliers lifts per-tensor INT4 to 14.72 dB. Keeping the three |w| > 4 in FP16 instead gives 15.89 dB at 4.219 bits. Affine group 16 reaches 22.73 dB (8.13% output error) at 5.250 bits. 8-bit per-tensor is 35.03 dB (1.93%), and 2-bit per-tensor collapses to 1.11 dB (92.23%). Each extra bit adds about 6 dB.

## Worked example: one outlier in eight weights

Quantize $w=(0.12,-0.50,0.33,0.05,\mathbf{2.40},-0.08,0.21,-0.27)$ to symmetric INT4 (codes $-7\ldots7$) in three ways.

Reconstructed weights ŵ for each scheme (computed in Python)

| w | per-tensor, s = 0.3429 | groups of 4 | groups of 4 + 2.40 in FP16 |
| --- | --- | --- | --- |
| 0.12 | 0 | 0.1429 | 0.1429 |
| −0.50 | −0.3429 | −0.5000 | −0.5000 |
| 0.33 | 0.3429 | 0.3571 | 0.3571 |
| 0.05 | 0 | 0.0714 | 0.0714 |
| 2.40 | 2.4000 | 2.4000 | 2.40 (FP16) |
| −0.08 | 0 | 0 | −0.0771 |
| 0.21 | 0.3429 | 0.3429 | 0.1929 |
| −0.27 | −0.3429 | −0.3429 | −0.2700 |
| MSE | 0.00889 | 0.00388 | 0.00025 |
| SQNR | 19.45 dB | 23.04 dB | 34.91 dB |

**Step by step**

1. Per-tensor: $s=2.40/7=0.3429$. The ratios $w/s$ are $0.35,-1.46,0.96,0.15,7,-0.23,0.61,-0.79$, which round to $0,-1,1,0,7,0,1,-1$. Only 4 of the 15 levels are used, and three weights become exactly 0.
2. Groups of 4: the first group's range is $0.50$, so $s_1=0.50/7=0.0714$ and the codes are $2,-7,5,1$. The error there drops to at most $0.023$. The second group still contains 2.40 and keeps $s_2=0.3429$. MSE falls from $0.00889$ to $0.00388$.
3. Pull 2.40 out into an FP16 side list: the second group's range becomes $0.27$, $s_2=0.0386$, codes $-2,5,-7$. MSE falls to $0.00025$, 36 times lower than per-tensor, and SQNR rises from 19.45 to 34.91 dB.
4. Storage per weight for these 8 weights: per-tensor $4+16/8=6$ bits; two groups $4+32/8=8$ bits; plus one outlier at 32 bits (value and index), $12$ bits. On a real layer with groups of 128 the scale overhead is only $0.125$ bit, and outliers are a tiny fraction of weights.
5. This is the LLM.int8() idea at the scale of one vector. GPTQ and AWQ reach the same goal without a side list: GPTQ compensates the error with other weights, and AWQ rescales channels before rounding.

```python
import numpy as np

rng = np.random.default_rng(0)
d_in, d_out, n = 512, 256, 4096
# calibration activations: correlated features (they live near a 64-dim subspace),
# and a few input features are 20x larger, like LLM outlier features
X = rng.normal(size=(n, 64)) @ rng.normal(size=(64, d_in)) + 0.3 * rng.normal(size=(n, d_in))
X[:, rng.choice(d_in, 8, replace=False)] *= 20.0
W = rng.normal(0, 0.02, (d_out, d_in))

def quantize(w, bits, scale):
    """Symmetric round-to-nearest with a given scale (broadcast)."""
    qmax = 2 ** (bits - 1) - 1
    return np.clip(np.round(w / scale), -qmax, qmax) * scale

def group_scale(w, bits):                       # one FP16 scale per row of this block
    return np.abs(w).max(axis=1, keepdims=True) / (2 ** (bits - 1) - 1)

def rtn(W, bits=4, group=128):
    Q = np.empty_like(W)
    for g in range(0, W.shape[1], group):
        blk = W[:, g:g + group]
        Q[:, g:g + group] = quantize(blk, bits, group_scale(blk, bits))
    return Q

def gptq(W, X, bits=4, group=128, damp=0.01):
    """GPTQ (Frantar et al. 2023): quantize column by column, push each column's error onto the
    not-yet-quantized columns using the inverse Hessian of the layer loss ||XW^T - XQ^T||^2."""
    W = W.copy()
    H = 2 * X.T @ X                             # Hessian of the layer-wise squared error
    H += damp * np.mean(np.diag(H)) * np.eye(H.shape[0])
    U = np.linalg.cholesky(np.linalg.inv(H)).T  # upper Cholesky factor of H^-1
    Q = np.zeros_like(W)
    for j in range(W.shape[1]):
        if j % group == 0:                      # scales from the (already updated) block
            s = group_scale(W[:, j:j + group], bits)
        Q[:, j] = quantize(W[:, j:j + 1], bits, s)[:, 0]
        err = (W[:, j] - Q[:, j]) / U[j, j]
        W[:, j:] -= np.outer(err, U[j, j:])     # compensate on the remaining columns
    return Q

def out_err(Q):
    return np.linalg.norm(X @ (W - Q).T) / np.linalg.norm(X @ W.T)

for bits in (4, 3):
    print(f"{bits}-bit, g=128: RTN output error {out_err(rtn(W, bits)):.4f}, "
          f"GPTQ {out_err(gptq(W, X, bits)):.4f}")
# 4-bit, g=128: RTN output error 0.1188, GPTQ 0.0244
# 3-bit, g=128: RTN output error 0.2769, GPTQ 0.0733

# memory of a 7B model's weights (6.74e9 parameters), FP16 scale per group of 128
for name, b in [("FP16", 16), ("INT8", 8), ("INT4 g128", 4 + 16 / 128), ("INT3 g128", 3 + 16 / 128)]:
    print(f"{name:10s} {b:6.3f} bits/weight  {6.74e9 * b / 8 / 1e9:6.2f} GB")
# FP16 13.48 GB, INT8 6.74 GB, INT4 g128 3.48 GB, INT3 g128 2.63 GB
```

RTN vs GPTQ at 4 and 3 bits with group size 128 on a toy layer with correlated, outlier-heavy activations, plus 7B memory math (numpy; run, printed values from that run)

GPTQ cuts the output error about fivefold at 4 bits here, because the correlated inputs let later columns cancel earlier rounding errors. With independent input features, $H$ is diagonal and GPTQ reduces to RTN. In practice you rarely write this loop. Hugging Face `transformers` loads GPTQ, AWQ, bitsandbytes (LLM.int8() and NF4) and FP8 checkpoints through a `quantization_config`. vLLM, TensorRT-LLM and llama.cpp provide the fused low-bit kernels.

> **Perplexity is not the whole story:** 4-bit weight-only models usually lose little perplexity on WikiText or C4. Losses show up first on long-context retrieval, multilingual text, maths and code, and with smaller models, which have less redundancy. Calibration data matters too: GPTQ reconstructs on its calibration set and can overfit to its domain. Evaluate the quantized model on your own tasks (see **LLM evaluation and benchmarking**), not just perplexity.

> **Quantization error is a range problem:** For uniform grids, the error is set by $s=\text{range}/(2^b-1)$, so every method attacks the range. Finer groups shrink it. Mixed precision removes outliers from it. SmoothQuant and AWQ rescale channels to flatten it. Rotation methods such as QuaRot and SpinQuant multiply by orthogonal matrices so that no coordinate stands out. In the figure, each of these moves raises the SQNR by several dB, about as much as one extra bit.

> **Bits per weight include the metadata:** "4-bit" models are rarely exactly 4 bits. Group size 128 with an FP16 scale is 4.125 bits; adding an INT4 zero-point makes it about 4.16. Embeddings and the LM head are often kept at higher precision. In the toy figure, group 16 costs a full extra bit per weight. Real formats therefore use $g\ge32$, or quantize the scales themselves (QLoRA's double quantization, k-quants' 6-bit sub-block scales).

**Strengths**

- 4× less weight memory than FP16 at 4 bits; a 70B model fits in about 36 GB
- Faster memory-bound decoding (weight-only), or faster compute-bound prefill on INT8/FP8 tensor cores (W8A8)
- PTQ needs only a few hundred calibration sequences and minutes to hours of GPU time
- Combines with KV-cache quantization, **Speculative decoding** and **Pruning**

**Limitations**

- Accuracy drops grow quickly below 4 bits and for smaller models
- Activation outliers make naive INT8 activation quantization fail beyond a few billion parameters
- Speedups depend on fused kernels for each format and GPU; unpacking can make small batches slower than expected
- Calibration-based methods can overfit their calibration domain

- **Local and edge inference:** GGUF k-quants in llama.cpp and Ollama run 7–70B models on laptops and phones.
- **Single-GPU serving:** GPTQ/AWQ INT4 checkpoints let a 70B model run on one or two GPUs instead of four or more.
- **High-throughput serving:** FP8 or INT8 W8A8 plus an FP8 KV cache on H100-class GPUs.
- **Low-memory fine-tuning:** **QLoRA** trains adapters on top of a frozen 4-bit (NF4) base.

**Question:** A weight row has 127 values within ±0.05 and one value of 1.27. With symmetric INT8 per-channel quantization, what is the step size, and what happens to the small weights?

- A. s = 0.05/127; all weights keep full precision
- B. s = 1.27/127 = 0.01; the small weights use only about 11 of 255 levels (codes −5…5)
- C. s = 1.27/255; the outlier is clipped
- D. s = 0.01; the outlier is rounded to zero

**Answer:** B. s = 1.27/127 = 0.01; the small weights use only about 11 of 255 levels (codes −5…5) — $s=\max|w|/127=0.01$. The small weights span $\pm0.05=\pm5s$, so they use codes $-5\ldots5$: 11 levels out of 255, about 3.5 bits of effective precision. This is why per-group scales, outlier extraction or channel rescaling matter.

**Question:** SmoothQuant computes Y = (X·diag(s)⁻¹)·(diag(s)·Wᵀ) with s_j > 1 on outlier channels. What does this achieve?

- A. It changes Y, trading accuracy for speed
- B. It removes the need to quantize W
- C. It shrinks the outlier activation channels and enlarges the matching weights, so both X and W become easy to quantize to INT8 while Y is mathematically unchanged
- D. It converts the model to FP8

**Answer:** C. It shrinks the outlier activation channels and enlarges the matching weights, so both X and W become easy to quantize to INT8 while Y is mathematically unchanged — The scaling cancels exactly in full precision. It moves quantization difficulty from activations, which have large outlier channels, to weights, which are smooth and can absorb some range. With $\alpha=0.5$ the per-channel maxima are balanced as $\sqrt{\max|X_j|\,\max|W_j|}$ on both sides.

**Key points**

- $\hat w=s(q-z)$ with $s=\text{range}/(2^b-1)$. Error per weight is at most $s/2$, so the block range sets the error, and each extra bit gives about 6 dB.
- Granularity trades accuracy for metadata: per-tensor, then per-channel, then per-group (g = 128 adds 0.125 bits per weight).
- From about 6.7B parameters, a few activation features become huge outliers. LLM.int8() splits them off into FP16; SmoothQuant moves them into the weights.
- GPTQ uses $H=2X^\top X$ to push each column's rounding error onto unquantized columns. AWQ scales the weight channels that see large activations before rounding.
- Weight-only INT4 speeds memory-bound decoding. W8A8 and FP8 (E4M3/E5M2) speed compute-bound prefill. A 7B model shrinks from 13.48 GB to about 3.5 GB at 4 bits.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
