Pruning

Pruning zeros or removes LLM weights: unstructured, 2:4 (two of every four, run on NVIDIA sparse tensor cores) or whole heads, neurons and layers. SparseGPT (OBS updates) and Wanda (|W|·‖X‖) reach 50% sparsity in one shot; structured pruning plus distillation (Minitron) builds smaller dense models.

15 min read1621 wordscore0 viewsupdated 2026-10-13

Definition

Pruning removes parameters from a trained network: it sets individual weights to zero (unstructured), zeros them in a fixed hardware-friendly pattern such as 2:4 (semi-structured), or deletes whole attention heads, MLP neurons, embedding channels or layers (structured). The classic criteria are Optimal Brain Damage (LeCun, Denker, Solla, 1989) and Optimal Brain Surgeon (Hassibi and Stork, 1993), followed by magnitude pruning (Han, Pool, Tran, Dally, NeurIPS 2015). For LLMs, the one-shot methods are SparseGPT (Frantar and Alistarh, ICML 2023) and Wanda (Sun, Liu, Bair, Kolter, ICLR 2024). Structured pruning followed by distillation builds smaller dense models, as in Sheared-LLaMA (Xia et al., ICLR 2024) and Minitron (NVIDIA, 2024).

Trained networks are heavily over-parameterised. Han et al. pruned AlexNet's parameters by 9× and VGG-16's by 13× with no loss of ImageNet accuracy. Their recipe was to train, remove the smallest-magnitude weights, retrain the rest, and repeat. Frankle and Carbin's lottery ticket hypothesis ("The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", ICLR 2019) sharpened the point. Inside a dense network there are sparse subnetworks that, reset to their original initialisation and trained alone, match the full network's accuracy. For larger networks, the weights had to be rewound to an early training iteration rather than all the way to initialisation (Frankle et al., 2020).

Neither recipe works for LLMs as is. Iterative prune-and-retrain means repeated training runs at a cost comparable to pre-training. And the simple magnitude criterion that worked for CNNs fails badly on large transformers. LLM pruning therefore splits two ways. One-shot methods prune a trained model with a small calibration set and no retraining, accepting moderate sparsity (50–60%). Structured pruning with distillation removes whole components and then retrains on billions of tokens, a few percent of the original pre-training budget, to recover quality.

1 · sparsity patterns on the same 8 × 8 weights (filled = kept, stronger colour = larger score)unstructured, 50%2:4 semi-structured, 50%structured: rows + columnszeros anywherebest accuracydense GPUs seelittle speedup2 of every 4in a row keptsparse tensorcores (A100+)drop heads,neurons,channels,layersstores 2 values + 2-bit indicesper 4: 9 bits/weight in FP16needs high sparsity andsparse kernels to pay off→ a smaller dense matrix(6 × 5 here), fast anywhere2 · two pipelinesdense LLM+ calibrationset Xone-shot (SparseGPT, Wanda): minutes to hours, no trainingscore weights|W|·‖X_j‖ / OBSmask per rowor per 2:4 groupupdate kept WSparseGPT onlysparse LLM50–60% zerosstructured + distillation (Minitron, Sheared-LLaMA): billions of tokensscore unitsheads, neuronscut width/depthto a target shapedistillteacher = densesmall densee.g. 15B → 8BWanda score: S_ij = |W_ij| · ‖X_j‖₂, compared within each output row; no weight updateSparseGPT: prune column by column, update the remaining weights with H⁻¹, H = X Xᵀ (OBS)magnitude: S_ij = |W_ij| ignores that a few LLM input features are far larger than the rest
fig.Top: the three sparsity patterns at 50%. Unstructured zeros can sit anywhere and give the best accuracy, but dense GPU kernels gain little from them. 2:4 keeps exactly two of every four consecutive weights in a row. Ampere-and-later NVIDIA sparse tensor cores store only the two values plus 2-bit indices, (2·16 + 2·2)/4 = 9 bits per FP16 weight, and skip the zeros. Structured pruning deletes whole rows and columns (here 2 rows and 3 columns, leaving a dense 6 × 5 matrix), which speeds up any hardware. Bottom: one-shot pruning (score, mask, optionally update the kept weights) and structured pruning followed by distillation from the dense teacher.

What a pruning criterion estimates

Removing weight wqw_q changes the loss. A second-order Taylor expansion around a trained minimum, where the gradient is about zero, gives δL≈12δw⊤H δw\delta L\approx\frac12\delta w^\top H\,\delta w. OBD assumes a diagonal Hessian, so the cost of zeroing wqw_q is 12Hqqwq2\frac12H_{qq}w_q^2. OBS lets the remaining weights move to compensate, with the constraint δwq=−wq\delta w_q=-w_q:

saliencyq=wq22 [H−1]qq,δw=−wq[H−1]qq H−1eq\text{saliency}_q=\frac{w_q^2}{2\,[H^{-1}]_{qq}},\qquad \delta w=-\frac{w_q}{[H^{-1}]_{qq}}\,H^{-1}e_q
HH
Hessian of the loss; for one linear layer with squared reconstruction error it is H=2XX⊤H=2XX^\top over calibration inputs
[H−1]qq[H^{-1}]_{qq}
diagonal entry of the inverse Hessian; a small value means the weight is hard to compensate for
δw\delta w
optimal update of all other weights once w_q is removed
eqe_q
unit vector selecting weight q

Magnitude pruning is the crudest special case. It sets H∝IH\propto I, so the score is ∣wq∣|w_q|. That is reasonable when every input has the same scale. In LLMs that assumption fails: from a few billion parameters on, a handful of hidden features carry activations far larger than the rest (the outlier features described in Quantization). A small weight that multiplies such a feature can contribute more to the output than a large weight on a quiet feature.

Wanda: weights times activations

Sij=∣Wij∣⋅∥Xj∥2,prune the lowest Sij within each output row iS_{ij}=|W_{ij}|\cdot\|X_j\|_2,\qquad \text{prune the lowest }S_{ij}\text{ within each output row }i
WijW_{ij}
weight from input feature jj to output ii
∥Xj∥2\|X_j\|_2
ℓ2\ell_2 norm of input feature jj over all calibration tokens (one forward pass)
per row\text{per row}
compare weights feeding the same output; this per-output grouping mattered for LLMs in the paper's ablations

Wanda is OBD for a single linear layer. With H=2XX⊤H=2XX^\top restricted to its diagonal, Hjj=2∥Xj∥2H_{jj}=2\|X_j\|^2, and OBD's cost of zeroing WijW_{ij} is ∥Xj∥2Wij2=Sij2\|X_j\|^2W_{ij}^2=S_{ij}^2. Wanda makes no weight update and needs no Hessian inverse. Its cost is one forward pass to collect ∥Xj∥\|X_j\|, then sorting. On LLaMA-7B at 50% unstructured sparsity, the paper reports WikiText perplexity of 7.26 for Wanda, 7.22 for SparseGPT and 17.29 for magnitude pruning, against 5.68 for the dense model.

SparseGPT: OBS at 175B scale

Exact OBS needs a separate H−1H^{-1} update after every removed weight in every row, which is far too slow for billions of weights. SparseGPT reuses GPTQ's trick (see Quantization). It processes the columns of WW left to right in the same order for every row, so all rows share one sequence of inverse Hessians, read off a single Cholesky factor of H−1H^{-1}. For each block of columns it picks the mask by OBS saliency w2/[H−1]jj2w^2/[H^{-1}]_{jj}^2, zeros the chosen weights, and pushes their error onto the not-yet-processed columns. The same loop supports 2:4 and 4:8 patterns, and quantization of the surviving weights.

SparseGPT pruned OPT-175B and BLOOM-176B to 50–60% unstructured sparsity in under 4.5 hours on a single GPU, with only a small perplexity increase. More than 100 billion weights could be ignored at inference. Larger models proved easier to prune: the relative perplexity gap at fixed sparsity shrinks with scale.

Numbers from the papers

Magnitude (Han et al. 2015)
AlexNet 9× and VGG-16 13× fewer parameters with no accuracy loss, using iterative prune + retrain
SparseGPT
OPT-175B, BLOOM-176B: 50–60% unstructured sparsity, under 4.5 GPU-hours; 2:4 and 4:8 supported; calibration on 128 segments of 2048 tokens from C4
Wanda
LLaMA-7B at 50%: WikiText ppl 7.26 (SparseGPT 7.22, magnitude 17.29, dense 5.68); no weight update; one forward pass
2:4 sparsity
NVIDIA Ampere (A100) and later Sparse Tensor Cores: up to 2× math throughput for the GEMM; 50% exactly; values plus 2-bit metadata per kept weight
Sheared-LLaMA
LLaMA2-7B pruned to 1.3B and 2.7B with targeted structured pruning (learned masks to a target shape), then continued pre-training on 50B tokens with dynamic batch loading
Minitron
Nemotron-4 15B pruned in width and depth to 8B and 4B, then retrained by distillation; up to 40× fewer training tokens than training each model from scratch
prune W (12 × 24) to 50.0% sparsity · unstructured · compared per output row‖X_j‖ per inputmagnitude |W|kept on outlier inputs: 20 / 36output error ‖ΔW·X‖/‖W·X‖ = 0.221‖X_j‖ per inputWanda |W|·‖X‖kept on outlier inputs: 34 / 36output error ‖ΔW·X‖/‖W·X‖ = 0.110output error vs sparsity (unstructured)0.00.20.40.60%12.5%25%37.5%50%62.5%75%87.5%▮ 3 outlier input features with ~6× larger activations · empty cells = prunederror ratio magnitude / Wanda = 2.01refit = least-squares update of the kept weights on X, what SparseGPT approximates
pattern
fig.A 12 × 24 linear layer whose inputs include three outlier features (columns 3, 11 and 18, about 6× larger, with norms 88.8, 62.0 and 56.7 against about 7–16 elsewhere). The left mask uses magnitude scores and the right uses Wanda scores, both compared within each output row. The output error is ‖WX − ŴX‖/‖WX‖ on the 64 calibration inputs. At the default 50% unstructured sparsity, magnitude keeps only 20 of the 36 weights on outlier inputs and its error is 0.221. Wanda keeps 34 of 36 and its error is 0.110, half as large. Switch to 2:4: magnitude 0.317 against Wanda 0.131. Structured pruning at 50% removes whole input columns. Magnitude keeps only one of the three outlier columns (0.436), while Wanda keeps all three (0.166). At 87.5% unstructured sparsity the errors are 0.631 and 0.285. “Refit kept weights” solves the least-squares problem that SparseGPT approximates: at 50% unstructured the errors fall to 0.147 and 0.068, and at 2:4 to 0.196 and 0.083. A better mask and a weight update help independently.

Unstructured, 2:4 and structured

Unstructured sparsity keeps the most accuracy at a given count of zeros, but GPUs do dense math. A 50% unstructured matrix in FP16 still needs index storage, and sparse kernels beat dense ones only at very high sparsity. It saves memory with compressed formats and helps CPU runtimes, but rarely speeds up GPU inference at 50%.

2:4 semi-structured sparsity is the compromise that hardware supports. Within every group of 4 consecutive weights along the input dimension, exactly 2 are zero. NVIDIA's Sparse Tensor Cores (Ampere and later) store the two values plus a 2-bit index each and skip the zero products. That gives up to 2× math throughput for the GEMM and (2⋅16+2⋅2)/4=9(2\cdot16+2\cdot2)/4=9 bits per FP16 weight. End-to-end LLM speedups are smaller, since attention and memory traffic are unchanged. The pattern constrains the mask more than unstructured 50% does, and accuracy drops more, as in the figure.

Structured pruning deletes whole units: attention heads, MLP neurons, embedding channels or entire transformer blocks. The result is a smaller dense model that is fast on any hardware with no special kernels. But removing whole units costs far more accuracy than zeroing scattered weights. One-shot structured pruning of LLMs degrades quality quickly, so it is paired with retraining.

Width and depth pruning with distillation

Sheared-LLaMA prunes to a target architecture: a chosen number of layers, heads, hidden size and MLP width matching an existing small model's shape. It learns masks over these units with Lagrange-multiplier constraints that enforce the target shape. Then it continues pre-training, with dynamic batch loading that reweights data domains by how far each lags behind its reference loss. From LLaMA2-7B it produced 1.3B and 2.7B models that outperformed open models of similar size trained from scratch, at a fraction of their compute.

Minitron ranks heads, MLP neurons, embedding channels and layers by activation-based importance on a small calibration set. It prunes the model to the target size and retrains it with Knowledge distillation, matching the dense teacher's output distribution. From Nemotron-4 15B, NVIDIA obtained 8B and 4B models using up to 40× fewer training tokens per model than training from scratch. They found width pruning generally better than depth pruning at these sizes, and distillation better than plain retraining. The same recipe produced Llama-3.1-Minitron 4B from Llama 3.1 8B.

Depth pruning on its own is surprisingly forgiving. Several 2024 studies (ShortGPT; Gromov et al., "The Unreasonable Ineffectiveness of the Deeper Layers") removed blocks whose input and output representations are most similar. A substantial fraction of the deeper layers could go with modest loss on many benchmarks after a short healing fine-tune. Removing layers also cuts the KV cache proportionally, which width pruning of MLP neurons does not.

Worked example: one row, 2:4

One output row with weights w=(0.9,−0.6,0.4,0.1)w=(0.9,-0.6,0.4,0.1) on four input features with calibration norms ∥Xj∥=(1,2,1,20)\|X_j\|=(1,2,1,20). The fourth feature is an outlier. Prune 2 of the 4 weights, as 2:4 requires, and assume the features are uncorrelated, so the squared output error is ∑prunedwj2∥Xj∥2\sum_{\text{pruned}}w_j^2\|X_j\|^2.

jw_j‖X_j‖magnitude |w_j|Wanda |w_j|·‖X_j‖cost w_j²‖X_j‖²
10.910.9 (keep)0.9 (prune)0.81
2−0.620.6 (keep)1.2 (keep)1.44
30.410.4 (prune)0.4 (prune)0.16
40.1200.1 (prune)2.0 (keep)4.00
total‖y‖² = 6.41
Scores and the cost of removing each weight (computed in Python)

Which two weights to drop

  1. 1Magnitude drops the two smallest weights, 0.40.4 and 0.10.1. The tiny 0.10.1 sits on the outlier feature, so its removal costs 0.12⋅202=4.000.1^2\cdot20^2=4.00 on its own. Total cost 0.16+4.00=4.160.16+4.00=4.16, relative output error 4.16/6.41=0.806\sqrt{4.16/6.41}=0.806.
  2. 2Wanda's scores are 0.9,1.2,0.4,2.00.9,1.2,0.4,2.0. It drops j=3j=3 and, surprisingly, the largest weight 0.90.9, because that weight multiplies a quiet feature. Cost 0.16+0.81=0.970.16+0.81=0.97, relative error 0.97/6.41=0.389\sqrt{0.97/6.41}=0.389.
  3. 3The output error falls by half. Under the uncorrelated assumption, Wanda's score squared is exactly the removal cost, so Wanda is optimal here. With correlated features, OBS/SparseGPT also uses the off-diagonal Hessian terms and moves the kept weights to compensate.
  4. 4Storage after 2:4 in FP16: 2 values and 2 two-bit indices, 3636 bits for 4 weights, 9 bits per weight instead of 16 (56%).
python
1import numpy as np
2
3rng = np.random.default_rng(0)
4d_in, d_out, n = 512, 256, 4096
5# calibration inputs: correlated features, 8 of them with 20x larger activations (outlier features)
6X = rng.normal(size=(n, 64)) @ rng.normal(size=(64, d_in)) / 8 + rng.normal(size=(n, d_in))
7X[:, rng.choice(d_in, 8, replace=False)] *= 20.0
8W = rng.normal(0, 0.02, (d_out, d_in)) # y = x @ W.T
9
10def mask_per_row(score, sparsity): # drop the lowest scores in each output row
11 k = int(score.shape[1] * sparsity)
12 drop = np.argsort(score, axis=1)[:, :k]
13 m = np.ones_like(score, dtype=bool)
14 np.put_along_axis(m, drop, False, axis=1)
15 return m
16
17def mask_2_4(score): # keep the 2 largest of every 4 consecutive inputs
18 s = score.reshape(score.shape[0], -1, 4)
19 keep = np.argsort(s, axis=2)[:, :, 2:]
20 m = np.zeros_like(s, dtype=bool)
21 np.put_along_axis(m, keep, True, axis=2)
22 return m.reshape(score.shape)
23
24def magnitude(W, X, sparsity=0.5, nm=False):
25 S = np.abs(W)
26 return W * (mask_2_4(S) if nm else mask_per_row(S, sparsity))
27
28def wanda(W, X, sparsity=0.5, nm=False): # Sun et al. 2024: |W_ij| * ||X_j||_2
29 S = np.abs(W) * np.linalg.norm(X, axis=0)
30 return W * (mask_2_4(S) if nm else mask_per_row(S, sparsity))
31
32def sparsegpt(W, X, sparsity=0.5, nm=False, block=128, damp=0.01):
33 """Frantar & Alistarh 2023: choose the mask block by block with the OBS saliency
34 w^2 / [H^-1]_jj^2, and push each pruned weight's error onto the later columns."""
35 W = W.copy()
36 H = X.T @ X
37 H += damp * np.mean(np.diag(H)) * np.eye(d_in)
38 U = np.linalg.cholesky(np.linalg.inv(H)).T # upper Cholesky factor of H^-1
39 for b in range(0, d_in, block):
40 Wb = W[:, b:b + block]
41 sal = Wb ** 2 / np.diag(U)[b:b + block] ** 2
42 keep = mask_2_4(sal) if nm else mask_per_row(sal, sparsity)
43 for j in range(b, min(b + block, d_in)):
44 q = W[:, j] * keep[:, j - b]
45 err = (W[:, j] - q) / U[j, j]
46 W[:, j:] -= np.outer(err, U[j, j:]) # OBS update of the not-yet-pruned columns
47 W[:, j] = q
48 return W
49
50def out_err(Wp):
51 return np.linalg.norm(X @ (W - Wp).T) / np.linalg.norm(X @ W.T)
52
53for name, f in [("magnitude", magnitude), ("Wanda", wanda), ("SparseGPT", sparsegpt)]:
54 print(f"{name:10s} 50% unstructured: {out_err(f(W, X)):.3f} 2:4: {out_err(f(W, X, nm=True)):.3f}")
55# magnitude 50% unstructured: 0.267 2:4: 0.376
56# Wanda 50% unstructured: 0.101 2:4: 0.139
57# SparseGPT 50% unstructured: 0.084 2:4: 0.114
Magnitude, Wanda and a compact SparseGPT at 50% unstructured and 2:4 on a toy layer with outlier features (numpy; run, printed values from that run)

The ordering matches the papers: the activation-aware mask does most of the work, and the OBS update adds a further gain. In PyTorch, torch.nn.utils.prune applies masks for experiments. torch.sparse.to_sparse_semi_structured converts a 2:4-pruned FP16 weight into the compressed format that runs on sparse tensor cores. A real LLM is pruned layer by layer: run the calibration batch through the already-pruned earlier layers, collect each linear layer's inputs, and prune it.

Use when

  • +One-shot methods need only a calibration set: Wanda one forward pass, SparseGPT hours for 175B models
  • +2:4 sparsity runs on NVIDIA sparse tensor cores with up to 2× GEMM throughput
  • +Structured pruning with distillation builds a family of smaller dense models at a fraction of pre-training cost
  • +Combines with quantization (SparseGPT can prune and quantize in one pass)

Avoid when

  • −Unstructured sparsity rarely speeds up GPU inference
  • −Quality drops quickly beyond 50–60% one-shot sparsity, especially on reasoning tasks
  • −Structured pruning needs substantial retraining (billions of tokens) to recover
  • −2:4 is fixed at exactly 50% and costs more accuracy than unstructured 50%

Where it's used

Small-model families

Minitron-style width/depth pruning plus distillation produces 4–8B models from a larger trained one.

Sparse inference

2:4 sparse weights with sparse tensor-core GEMMs, often combined with INT8 or FP8 quantization.

Faster drafts

A pruned or layer-skipped copy of the target can serve as the draft in Speculative decoding.

Check yourself

Two weights in the same row: w₁ = 0.8 on a feature with ‖X₁‖ = 1, and w₂ = 0.05 on a feature with ‖X₂‖ = 30. Which does Wanda prune first, and why?

Check yourself

Why does a 2:4 sparse FP16 weight matrix take 9 bits per weight rather than 8?

Remember

  • ✓OBS saliency wq2/(2[H−1]qq)w_q^2/(2[H^{-1}]_{qq}) with the compensating update δw=−wq[H−1]qqH−1eq\delta w=-\frac{w_q}{[H^{-1}]_{qq}}H^{-1}e_q. Magnitude pruning is the special case H∝IH\propto I.
  • ✓Wanda scores ∣Wij∣⋅∥Xj∥2|W_{ij}|\cdot\|X_j\|_2 per output row with no update. On LLaMA-7B at 50% it gives perplexity 7.26, against 7.22 for SparseGPT and 17.29 for magnitude.
  • ✓SparseGPT runs OBS column by column with a shared Cholesky factor of H⁻¹. It prunes 175B models to 50–60% in under 4.5 GPU-hours and also does 2:4.
  • ✓Unstructured sparsity rarely speeds up GPUs. 2:4 runs on sparse tensor cores (9 bits per FP16 weight). Structured pruning yields smaller dense models.
  • ✓Width and depth pruning followed by distillation (Minitron: 15B to 8B and 4B with up to 40× fewer tokens; Sheared-LLaMA: 7B to 1.3B and 2.7B) builds small models cheaply.
← back to Deep Learning roadmap