Definition
Pruning removes parameters from a trained network: it sets individual weights to zero (unstructured), zeros them in a fixed hardware-friendly pattern such as 2:4 (semi-structured), or deletes whole attention heads, MLP neurons, embedding channels or layers (structured). The classic criteria are Optimal Brain Damage (LeCun, Denker, Solla, 1989) and Optimal Brain Surgeon (Hassibi and Stork, 1993), followed by magnitude pruning (Han, Pool, Tran, Dally, NeurIPS 2015). For LLMs, the one-shot methods are SparseGPT (Frantar and Alistarh, ICML 2023) and Wanda (Sun, Liu, Bair, Kolter, ICLR 2024). Structured pruning followed by distillation builds smaller dense models, as in Sheared-LLaMA (Xia et al., ICLR 2024) and Minitron (NVIDIA, 2024).
Trained networks are heavily over-parameterised. Han et al. pruned AlexNet's parameters by 9× and VGG-16's by 13× with no loss of ImageNet accuracy. Their recipe was to train, remove the smallest-magnitude weights, retrain the rest, and repeat. Frankle and Carbin's lottery ticket hypothesis ("The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", ICLR 2019) sharpened the point. Inside a dense network there are sparse subnetworks that, reset to their original initialisation and trained alone, match the full network's accuracy. For larger networks, the weights had to be rewound to an early training iteration rather than all the way to initialisation (Frankle et al., 2020).
Neither recipe works for LLMs as is. Iterative prune-and-retrain means repeated training runs at a cost comparable to pre-training. And the simple magnitude criterion that worked for CNNs fails badly on large transformers. LLM pruning therefore splits two ways. One-shot methods prune a trained model with a small calibration set and no retraining, accepting moderate sparsity (50–60%). Structured pruning with distillation removes whole components and then retrains on billions of tokens, a few percent of the original pre-training budget, to recover quality.
What a pruning criterion estimates
Removing weight changes the loss. A second-order Taylor expansion around a trained minimum, where the gradient is about zero, gives . OBD assumes a diagonal Hessian, so the cost of zeroing is . OBS lets the remaining weights move to compensate, with the constraint :
- Hessian of the loss; for one linear layer with squared reconstruction error it is over calibration inputs
- diagonal entry of the inverse Hessian; a small value means the weight is hard to compensate for
- optimal update of all other weights once w_q is removed
- unit vector selecting weight q
Magnitude pruning is the crudest special case. It sets , so the score is . That is reasonable when every input has the same scale. In LLMs that assumption fails: from a few billion parameters on, a handful of hidden features carry activations far larger than the rest (the outlier features described in Quantization). A small weight that multiplies such a feature can contribute more to the output than a large weight on a quiet feature.
Wanda: weights times activations
- weight from input feature to output
- norm of input feature over all calibration tokens (one forward pass)
- compare weights feeding the same output; this per-output grouping mattered for LLMs in the paper's ablations
Wanda is OBD for a single linear layer. With restricted to its diagonal, , and OBD's cost of zeroing is . Wanda makes no weight update and needs no Hessian inverse. Its cost is one forward pass to collect , then sorting. On LLaMA-7B at 50% unstructured sparsity, the paper reports WikiText perplexity of 7.26 for Wanda, 7.22 for SparseGPT and 17.29 for magnitude pruning, against 5.68 for the dense model.
SparseGPT: OBS at 175B scale
Exact OBS needs a separate update after every removed weight in every row, which is far too slow for billions of weights. SparseGPT reuses GPTQ's trick (see Quantization). It processes the columns of left to right in the same order for every row, so all rows share one sequence of inverse Hessians, read off a single Cholesky factor of . For each block of columns it picks the mask by OBS saliency , zeros the chosen weights, and pushes their error onto the not-yet-processed columns. The same loop supports 2:4 and 4:8 patterns, and quantization of the surviving weights.
SparseGPT pruned OPT-175B and BLOOM-176B to 50–60% unstructured sparsity in under 4.5 hours on a single GPU, with only a small perplexity increase. More than 100 billion weights could be ignored at inference. Larger models proved easier to prune: the relative perplexity gap at fixed sparsity shrinks with scale.
Numbers from the papers
- Magnitude (Han et al. 2015)
- AlexNet 9× and VGG-16 13× fewer parameters with no accuracy loss, using iterative prune + retrain
- SparseGPT
- OPT-175B, BLOOM-176B: 50–60% unstructured sparsity, under 4.5 GPU-hours; 2:4 and 4:8 supported; calibration on 128 segments of 2048 tokens from C4
- Wanda
- LLaMA-7B at 50%: WikiText ppl 7.26 (SparseGPT 7.22, magnitude 17.29, dense 5.68); no weight update; one forward pass
- 2:4 sparsity
- NVIDIA Ampere (A100) and later Sparse Tensor Cores: up to 2× math throughput for the GEMM; 50% exactly; values plus 2-bit metadata per kept weight
- Sheared-LLaMA
- LLaMA2-7B pruned to 1.3B and 2.7B with targeted structured pruning (learned masks to a target shape), then continued pre-training on 50B tokens with dynamic batch loading
- Minitron
- Nemotron-4 15B pruned in width and depth to 8B and 4B, then retrained by distillation; up to 40× fewer training tokens than training each model from scratch
Unstructured, 2:4 and structured
Unstructured sparsity keeps the most accuracy at a given count of zeros, but GPUs do dense math. A 50% unstructured matrix in FP16 still needs index storage, and sparse kernels beat dense ones only at very high sparsity. It saves memory with compressed formats and helps CPU runtimes, but rarely speeds up GPU inference at 50%.
2:4 semi-structured sparsity is the compromise that hardware supports. Within every group of 4 consecutive weights along the input dimension, exactly 2 are zero. NVIDIA's Sparse Tensor Cores (Ampere and later) store the two values plus a 2-bit index each and skip the zero products. That gives up to 2× math throughput for the GEMM and bits per FP16 weight. End-to-end LLM speedups are smaller, since attention and memory traffic are unchanged. The pattern constrains the mask more than unstructured 50% does, and accuracy drops more, as in the figure.
Structured pruning deletes whole units: attention heads, MLP neurons, embedding channels or entire transformer blocks. The result is a smaller dense model that is fast on any hardware with no special kernels. But removing whole units costs far more accuracy than zeroing scattered weights. One-shot structured pruning of LLMs degrades quality quickly, so it is paired with retraining.
Width and depth pruning with distillation
Sheared-LLaMA prunes to a target architecture: a chosen number of layers, heads, hidden size and MLP width matching an existing small model's shape. It learns masks over these units with Lagrange-multiplier constraints that enforce the target shape. Then it continues pre-training, with dynamic batch loading that reweights data domains by how far each lags behind its reference loss. From LLaMA2-7B it produced 1.3B and 2.7B models that outperformed open models of similar size trained from scratch, at a fraction of their compute.
Minitron ranks heads, MLP neurons, embedding channels and layers by activation-based importance on a small calibration set. It prunes the model to the target size and retrains it with Knowledge distillation, matching the dense teacher's output distribution. From Nemotron-4 15B, NVIDIA obtained 8B and 4B models using up to 40× fewer training tokens per model than training from scratch. They found width pruning generally better than depth pruning at these sizes, and distillation better than plain retraining. The same recipe produced Llama-3.1-Minitron 4B from Llama 3.1 8B.
Depth pruning on its own is surprisingly forgiving. Several 2024 studies (ShortGPT; Gromov et al., "The Unreasonable Ineffectiveness of the Deeper Layers") removed blocks whose input and output representations are most similar. A substantial fraction of the deeper layers could go with modest loss on many benchmarks after a short healing fine-tune. Removing layers also cuts the KV cache proportionally, which width pruning of MLP neurons does not.
Worked example: one row, 2:4
One output row with weights on four input features with calibration norms . The fourth feature is an outlier. Prune 2 of the 4 weights, as 2:4 requires, and assume the features are uncorrelated, so the squared output error is .
| j | w_j | ‖X_j‖ | magnitude |w_j| | Wanda |w_j|·‖X_j‖ | cost w_j²‖X_j‖² |
|---|---|---|---|---|---|
| 1 | 0.9 | 1 | 0.9 (keep) | 0.9 (prune) | 0.81 |
| 2 | −0.6 | 2 | 0.6 (keep) | 1.2 (keep) | 1.44 |
| 3 | 0.4 | 1 | 0.4 (prune) | 0.4 (prune) | 0.16 |
| 4 | 0.1 | 20 | 0.1 (prune) | 2.0 (keep) | 4.00 |
| total | ‖y‖² = 6.41 |
Which two weights to drop
- 1Magnitude drops the two smallest weights, and . The tiny sits on the outlier feature, so its removal costs on its own. Total cost , relative output error .
- 2Wanda's scores are . It drops and, surprisingly, the largest weight , because that weight multiplies a quiet feature. Cost , relative error .
- 3The output error falls by half. Under the uncorrelated assumption, Wanda's score squared is exactly the removal cost, so Wanda is optimal here. With correlated features, OBS/SparseGPT also uses the off-diagonal Hessian terms and moves the kept weights to compensate.
- 4Storage after 2:4 in FP16: 2 values and 2 two-bit indices, bits for 4 weights, 9 bits per weight instead of 16 (56%).
1import numpy as np23rng = np.random.default_rng(0)4d_in, d_out, n = 512, 256, 40965# calibration inputs: correlated features, 8 of them with 20x larger activations (outlier features)6X = rng.normal(size=(n, 64)) @ rng.normal(size=(64, d_in)) / 8 + rng.normal(size=(n, d_in))7X[:, rng.choice(d_in, 8, replace=False)] *= 20.08W = rng.normal(0, 0.02, (d_out, d_in)) # y = x @ W.T910def mask_per_row(score, sparsity): # drop the lowest scores in each output row11 k = int(score.shape[1] * sparsity)12 drop = np.argsort(score, axis=1)[:, :k]13 m = np.ones_like(score, dtype=bool)14 np.put_along_axis(m, drop, False, axis=1)15 return m1617def mask_2_4(score): # keep the 2 largest of every 4 consecutive inputs18 s = score.reshape(score.shape[0], -1, 4)19 keep = np.argsort(s, axis=2)[:, :, 2:]20 m = np.zeros_like(s, dtype=bool)21 np.put_along_axis(m, keep, True, axis=2)22 return m.reshape(score.shape)2324def magnitude(W, X, sparsity=0.5, nm=False):25 S = np.abs(W)26 return W * (mask_2_4(S) if nm else mask_per_row(S, sparsity))2728def wanda(W, X, sparsity=0.5, nm=False): # Sun et al. 2024: |W_ij| * ||X_j||_229 S = np.abs(W) * np.linalg.norm(X, axis=0)30 return W * (mask_2_4(S) if nm else mask_per_row(S, sparsity))3132def sparsegpt(W, X, sparsity=0.5, nm=False, block=128, damp=0.01):33 """Frantar & Alistarh 2023: choose the mask block by block with the OBS saliency34 w^2 / [H^-1]_jj^2, and push each pruned weight's error onto the later columns."""35 W = W.copy()36 H = X.T @ X37 H += damp * np.mean(np.diag(H)) * np.eye(d_in)38 U = np.linalg.cholesky(np.linalg.inv(H)).T # upper Cholesky factor of H^-139 for b in range(0, d_in, block):40 Wb = W[:, b:b + block]41 sal = Wb ** 2 / np.diag(U)[b:b + block] ** 242 keep = mask_2_4(sal) if nm else mask_per_row(sal, sparsity)43 for j in range(b, min(b + block, d_in)):44 q = W[:, j] * keep[:, j - b]45 err = (W[:, j] - q) / U[j, j]46 W[:, j:] -= np.outer(err, U[j, j:]) # OBS update of the not-yet-pruned columns47 W[:, j] = q48 return W4950def out_err(Wp):51 return np.linalg.norm(X @ (W - Wp).T) / np.linalg.norm(X @ W.T)5253for name, f in [("magnitude", magnitude), ("Wanda", wanda), ("SparseGPT", sparsegpt)]:54 print(f"{name:10s} 50% unstructured: {out_err(f(W, X)):.3f} 2:4: {out_err(f(W, X, nm=True)):.3f}")55# magnitude 50% unstructured: 0.267 2:4: 0.37656# Wanda 50% unstructured: 0.101 2:4: 0.13957# SparseGPT 50% unstructured: 0.084 2:4: 0.114The ordering matches the papers: the activation-aware mask does most of the work, and the OBS update adds a further gain. In PyTorch, torch.nn.utils.prune applies masks for experiments. torch.sparse.to_sparse_semi_structured converts a 2:4-pruned FP16 weight into the compressed format that runs on sparse tensor cores. A real LLM is pruned layer by layer: run the calibration batch through the already-pruned earlier layers, collect each linear layer's inputs, and prune it.
Use when
- +One-shot methods need only a calibration set: Wanda one forward pass, SparseGPT hours for 175B models
- +2:4 sparsity runs on NVIDIA sparse tensor cores with up to 2× GEMM throughput
- +Structured pruning with distillation builds a family of smaller dense models at a fraction of pre-training cost
- +Combines with quantization (SparseGPT can prune and quantize in one pass)
Avoid when
- −Unstructured sparsity rarely speeds up GPU inference
- −Quality drops quickly beyond 50–60% one-shot sparsity, especially on reasoning tasks
- −Structured pruning needs substantial retraining (billions of tokens) to recover
- −2:4 is fixed at exactly 50% and costs more accuracy than unstructured 50%
Where it's used
Small-model families
Minitron-style width/depth pruning plus distillation produces 4–8B models from a larger trained one.
Sparse inference
2:4 sparse weights with sparse tensor-core GEMMs, often combined with INT8 or FP8 quantization.
Faster drafts
A pruned or layer-skipped copy of the target can serve as the draft in Speculative decoding.
Check yourself
Two weights in the same row: w₁ = 0.8 on a feature with ‖X₁‖ = 1, and w₂ = 0.05 on a feature with ‖X₂‖ = 30. Which does Wanda prune first, and why?
Check yourself
Why does a 2:4 sparse FP16 weight matrix take 9 bits per weight rather than 8?
Remember
- ✓OBS saliency with the compensating update . Magnitude pruning is the special case .
- ✓Wanda scores per output row with no update. On LLaMA-7B at 50% it gives perplexity 7.26, against 7.22 for SparseGPT and 17.29 for magnitude.
- ✓SparseGPT runs OBS column by column with a shared Cholesky factor of H⁻¹. It prunes 175B models to 50–60% in under 4.5 GPU-hours and also does 2:4.
- ✓Unstructured sparsity rarely speeds up GPUs. 2:4 runs on sparse tensor cores (9 bits per FP16 weight). Structured pruning yields smaller dense models.
- ✓Width and depth pruning followed by distillation (Minitron: 15B to 8B and 4B with up to 40× fewer tokens; Sheared-LLaMA: 7B to 1.3B and 2.7B) builds small models cheaply.