# Pruning

> Pruning zeros or removes LLM weights: unstructured, 2:4 (two of every four, run on NVIDIA sparse tensor cores) or whole heads, neurons and layers. SparseGPT (OBS updates) and Wanda (|W|·‖X‖) reach 50% sparsity in one shot; structured pruning plus distillation (Minitron) builds smaller dense models.

Source: https://www.mlfoundry.in/learn/pruning  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Pruning:** removes parameters from a trained network: it sets individual weights to zero (**unstructured**), zeros them in a fixed hardware-friendly pattern such as **2:4** (**semi-structured**), or deletes whole attention heads, MLP neurons, embedding channels or layers (**structured**). The classic criteria are Optimal Brain Damage (LeCun, Denker, Solla, 1989) and Optimal Brain Surgeon (Hassibi and Stork, 1993), followed by magnitude pruning (Han, Pool, Tran, Dally, NeurIPS 2015). For LLMs, the one-shot methods are **SparseGPT** (Frantar and Alistarh, ICML 2023) and **Wanda** (Sun, Liu, Bair, Kolter, ICLR 2024). Structured pruning followed by distillation builds smaller dense models, as in **Sheared-LLaMA** (Xia et al., ICLR 2024) and **Minitron** (NVIDIA, 2024).

Trained networks are heavily over-parameterised. Han et al. pruned AlexNet's parameters by 9× and VGG-16's by 13× with no loss of ImageNet accuracy. Their recipe was to train, remove the smallest-magnitude weights, retrain the rest, and repeat. Frankle and Carbin's **lottery ticket hypothesis** ("The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", ICLR 2019) sharpened the point. Inside a dense network there are sparse subnetworks that, reset to their *original initialisation* and trained alone, match the full network's accuracy. For larger networks, the weights had to be rewound to an early training iteration rather than all the way to initialisation (Frankle et al., 2020).

Neither recipe works for LLMs as is. Iterative prune-and-retrain means repeated training runs at a cost comparable to pre-training. And the simple magnitude criterion that worked for CNNs fails badly on large transformers. LLM pruning therefore splits two ways. **One-shot** methods prune a trained model with a small calibration set and no retraining, accepting moderate sparsity (50–60%). **Structured pruning with distillation** removes whole components and then retrains on billions of tokens, a few percent of the original pre-training budget, to recover quality.

> Figure: Top: the three sparsity patterns at 50%. Unstructured zeros can sit anywhere and give the best accuracy, but dense GPU kernels gain little from them. 2:4 keeps exactly two of every four consecutive weights in a row. Ampere-and-later NVIDIA sparse tensor cores store only the two values plus 2-bit indices, (2·16 + 2·2)/4 = 9 bits per FP16 weight, and skip the zeros. Structured pruning deletes whole rows and columns (here 2 rows and 3 columns, leaving a dense 6 × 5 matrix), which speeds up any hardware. Bottom: one-shot pruning (score, mask, optionally update the kept weights) and structured pruning followed by distillation from the dense teacher.

## What a pruning criterion estimates

Removing weight $w_q$ changes the loss. A second-order Taylor expansion around a trained minimum, where the gradient is about zero, gives $\delta L\approx\frac12\delta w^\top H\,\delta w$. OBD assumes a diagonal Hessian, so the cost of zeroing $w_q$ is $\frac12H_{qq}w_q^2$. OBS lets the remaining weights move to compensate, with the constraint $\delta w_q=-w_q$:

$$
\text{saliency}_q=\frac{w_q^2}{2\,[H^{-1}]_{qq}},\qquad \delta w=-\frac{w_q}{[H^{-1}]_{qq}}\,H^{-1}e_q
$$

- $H$: Hessian of the loss; for one linear layer with squared reconstruction error it is $H=2XX^\top$ over calibration inputs
- $[H^{-1}]_{qq}$: diagonal entry of the inverse Hessian; a small value means the weight is hard to compensate for
- $\delta w$: optimal update of all other weights once w_q is removed
- $e_q$: unit vector selecting weight q

Magnitude pruning is the crudest special case. It sets $H\propto I$, so the score is $|w_q|$. That is reasonable when every input has the same scale. In LLMs that assumption fails: from a few billion parameters on, a handful of hidden features carry activations far larger than the rest (the outlier features described in **Quantization**). A small weight that multiplies such a feature can contribute more to the output than a large weight on a quiet feature.

## Wanda: weights times activations

$$
S_{ij}=|W_{ij}|\cdot\|X_j\|_2,\qquad \text{prune the lowest }S_{ij}\text{ within each output row }i
$$

- $W_{ij}$: weight from input feature $j$ to output $i$
- $\|X_j\|_2$: $\ell_2$ norm of input feature $j$ over all calibration tokens (one forward pass)
- $\text{per row}$: compare weights feeding the same output; this per-output grouping mattered for LLMs in the paper's ablations

Wanda is OBD for a single linear layer. With $H=2XX^\top$ restricted to its diagonal, $H_{jj}=2\|X_j\|^2$, and OBD's cost of zeroing $W_{ij}$ is $\|X_j\|^2W_{ij}^2=S_{ij}^2$. Wanda makes **no weight update** and needs no Hessian inverse. Its cost is one forward pass to collect $\|X_j\|$, then sorting. On LLaMA-7B at 50% unstructured sparsity, the paper reports WikiText perplexity of 7.26 for Wanda, 7.22 for SparseGPT and 17.29 for magnitude pruning, against 5.68 for the dense model.

## SparseGPT: OBS at 175B scale

Exact OBS needs a separate $H^{-1}$ update after every removed weight in every row, which is far too slow for billions of weights. SparseGPT reuses GPTQ's trick (see **Quantization**). It processes the columns of $W$ left to right in the same order for every row, so all rows share one sequence of inverse Hessians, read off a single Cholesky factor of $H^{-1}$. For each block of columns it picks the mask by OBS saliency $w^2/[H^{-1}]_{jj}^2$, zeros the chosen weights, and pushes their error onto the not-yet-processed columns. The same loop supports 2:4 and 4:8 patterns, and quantization of the surviving weights.

SparseGPT pruned OPT-175B and BLOOM-176B to 50–60% unstructured sparsity in under 4.5 hours on a single GPU, with only a small perplexity increase. More than 100 billion weights could be ignored at inference. Larger models proved *easier* to prune: the relative perplexity gap at fixed sparsity shrinks with scale.

**Numbers from the papers**

- **Magnitude (Han et al. 2015):** AlexNet 9× and VGG-16 13× fewer parameters with no accuracy loss, using iterative prune + retrain
- **SparseGPT:** OPT-175B, BLOOM-176B: 50–60% unstructured sparsity, under 4.5 GPU-hours; 2:4 and 4:8 supported; calibration on 128 segments of 2048 tokens from C4
- **Wanda:** LLaMA-7B at 50%: WikiText ppl 7.26 (SparseGPT 7.22, magnitude 17.29, dense 5.68); no weight update; one forward pass
- **2:4 sparsity:** NVIDIA Ampere (A100) and later Sparse Tensor Cores: up to 2× math throughput for the GEMM; 50% exactly; values plus 2-bit metadata per kept weight
- **Sheared-LLaMA:** LLaMA2-7B pruned to 1.3B and 2.7B with targeted structured pruning (learned masks to a target shape), then continued pre-training on 50B tokens with dynamic batch loading
- **Minitron:** Nemotron-4 15B pruned in width and depth to 8B and 4B, then retrained by distillation; up to 40× fewer training tokens than training each model from scratch

> Figure: A 12 × 24 linear layer whose inputs include three outlier features (columns 3, 11 and 18, about 6× larger, with norms 88.8, 62.0 and 56.7 against about 7–16 elsewhere). The left mask uses magnitude scores and the right uses Wanda scores, both compared within each output row. The output error is ‖WX − ŴX‖/‖WX‖ on the 64 calibration inputs. At the default 50% unstructured sparsity, magnitude keeps only 20 of the 36 weights on outlier inputs and its error is 0.221. Wanda keeps 34 of 36 and its error is 0.110, half as large. Switch to 2:4: magnitude 0.317 against Wanda 0.131. Structured pruning at 50% removes whole input columns. Magnitude keeps only one of the three outlier columns (0.436), while Wanda keeps all three (0.166). At 87.5% unstructured sparsity the errors are 0.631 and 0.285. “Refit kept weights” solves the least-squares problem that SparseGPT approximates: at 50% unstructured the errors fall to 0.147 and 0.068, and at 2:4 to 0.196 and 0.083. A better mask and a weight update help independently.

## Unstructured, 2:4 and structured

**Unstructured** sparsity keeps the most accuracy at a given count of zeros, but GPUs do dense math. A 50% unstructured matrix in FP16 still needs index storage, and sparse kernels beat dense ones only at very high sparsity. It saves memory with compressed formats and helps CPU runtimes, but rarely speeds up GPU inference at 50%.

**2:4 semi-structured** sparsity is the compromise that hardware supports. Within every group of 4 consecutive weights along the input dimension, exactly 2 are zero. NVIDIA's Sparse Tensor Cores (Ampere and later) store the two values plus a 2-bit index each and skip the zero products. That gives up to 2× math throughput for the GEMM and $(2\cdot16+2\cdot2)/4=9$ bits per FP16 weight. End-to-end LLM speedups are smaller, since attention and memory traffic are unchanged. The pattern constrains the mask more than unstructured 50% does, and accuracy drops more, as in the figure.

**Structured** pruning deletes whole units: attention heads, MLP neurons, embedding channels or entire transformer blocks. The result is a smaller *dense* model that is fast on any hardware with no special kernels. But removing whole units costs far more accuracy than zeroing scattered weights. One-shot structured pruning of LLMs degrades quality quickly, so it is paired with retraining.

## Width and depth pruning with distillation

**Sheared-LLaMA** prunes to a *target architecture*: a chosen number of layers, heads, hidden size and MLP width matching an existing small model's shape. It learns masks over these units with Lagrange-multiplier constraints that enforce the target shape. Then it continues pre-training, with **dynamic batch loading** that reweights data domains by how far each lags behind its reference loss. From LLaMA2-7B it produced 1.3B and 2.7B models that outperformed open models of similar size trained from scratch, at a fraction of their compute.

**Minitron** ranks heads, MLP neurons, embedding channels and layers by activation-based importance on a small calibration set. It prunes the model to the target size and retrains it with **Knowledge distillation**, matching the dense teacher's output distribution. From Nemotron-4 15B, NVIDIA obtained 8B and 4B models using up to 40× fewer training tokens per model than training from scratch. They found width pruning generally better than depth pruning at these sizes, and distillation better than plain retraining. The same recipe produced Llama-3.1-Minitron 4B from Llama 3.1 8B.

Depth pruning on its own is surprisingly forgiving. Several 2024 studies (ShortGPT; Gromov et al., "The Unreasonable Ineffectiveness of the Deeper Layers") removed blocks whose input and output representations are most similar. A substantial fraction of the deeper layers could go with modest loss on many benchmarks after a short healing fine-tune. Removing layers also cuts the **KV cache** proportionally, which width pruning of MLP neurons does not.

## Worked example: one row, 2:4

One output row with weights $w=(0.9,-0.6,0.4,0.1)$ on four input features with calibration norms $\|X_j\|=(1,2,1,20)$. The fourth feature is an outlier. Prune 2 of the 4 weights, as 2:4 requires, and assume the features are uncorrelated, so the squared output error is $\sum_{\text{pruned}}w_j^2\|X_j\|^2$.

Scores and the cost of removing each weight (computed in Python)

| j | w_j | ‖X_j‖ | magnitude \|w_j\| | Wanda \|w_j\|·‖X_j‖ | cost w_j²‖X_j‖² |
| --- | --- | --- | --- | --- | --- |
| 1 | 0.9 | 1 | 0.9 (keep) | 0.9 (prune) | 0.81 |
| 2 | −0.6 | 2 | 0.6 (keep) | 1.2 (keep) | 1.44 |
| 3 | 0.4 | 1 | 0.4 (prune) | 0.4 (prune) | 0.16 |
| 4 | 0.1 | 20 | 0.1 (prune) | 2.0 (keep) | 4.00 |
| total |  |  |  |  | ‖y‖² = 6.41 |

**Which two weights to drop**

1. Magnitude drops the two smallest weights, $0.4$ and $0.1$. The tiny $0.1$ sits on the outlier feature, so its removal costs $0.1^2\cdot20^2=4.00$ on its own. Total cost $0.16+4.00=4.16$, relative output error $\sqrt{4.16/6.41}=0.806$.
2. Wanda's scores are $0.9,1.2,0.4,2.0$. It drops $j=3$ and, surprisingly, the *largest* weight $0.9$, because that weight multiplies a quiet feature. Cost $0.16+0.81=0.97$, relative error $\sqrt{0.97/6.41}=0.389$.
3. The output error falls by half. Under the uncorrelated assumption, Wanda's score squared is exactly the removal cost, so Wanda is optimal here. With correlated features, OBS/SparseGPT also uses the off-diagonal Hessian terms and moves the kept weights to compensate.
4. Storage after 2:4 in FP16: 2 values and 2 two-bit indices, $36$ bits for 4 weights, 9 bits per weight instead of 16 (56%).

```python
import numpy as np

rng = np.random.default_rng(0)
d_in, d_out, n = 512, 256, 4096
# calibration inputs: correlated features, 8 of them with 20x larger activations (outlier features)
X = rng.normal(size=(n, 64)) @ rng.normal(size=(64, d_in)) / 8 + rng.normal(size=(n, d_in))
X[:, rng.choice(d_in, 8, replace=False)] *= 20.0
W = rng.normal(0, 0.02, (d_out, d_in))           # y = x @ W.T

def mask_per_row(score, sparsity):               # drop the lowest scores in each output row
    k = int(score.shape[1] * sparsity)
    drop = np.argsort(score, axis=1)[:, :k]
    m = np.ones_like(score, dtype=bool)
    np.put_along_axis(m, drop, False, axis=1)
    return m

def mask_2_4(score):                             # keep the 2 largest of every 4 consecutive inputs
    s = score.reshape(score.shape[0], -1, 4)
    keep = np.argsort(s, axis=2)[:, :, 2:]
    m = np.zeros_like(s, dtype=bool)
    np.put_along_axis(m, keep, True, axis=2)
    return m.reshape(score.shape)

def magnitude(W, X, sparsity=0.5, nm=False):
    S = np.abs(W)
    return W * (mask_2_4(S) if nm else mask_per_row(S, sparsity))

def wanda(W, X, sparsity=0.5, nm=False):          # Sun et al. 2024: |W_ij| * ||X_j||_2
    S = np.abs(W) * np.linalg.norm(X, axis=0)
    return W * (mask_2_4(S) if nm else mask_per_row(S, sparsity))

def sparsegpt(W, X, sparsity=0.5, nm=False, block=128, damp=0.01):
    """Frantar & Alistarh 2023: choose the mask block by block with the OBS saliency
    w^2 / [H^-1]_jj^2, and push each pruned weight's error onto the later columns."""
    W = W.copy()
    H = X.T @ X
    H += damp * np.mean(np.diag(H)) * np.eye(d_in)
    U = np.linalg.cholesky(np.linalg.inv(H)).T   # upper Cholesky factor of H^-1
    for b in range(0, d_in, block):
        Wb = W[:, b:b + block]
        sal = Wb ** 2 / np.diag(U)[b:b + block] ** 2
        keep = mask_2_4(sal) if nm else mask_per_row(sal, sparsity)
        for j in range(b, min(b + block, d_in)):
            q = W[:, j] * keep[:, j - b]
            err = (W[:, j] - q) / U[j, j]
            W[:, j:] -= np.outer(err, U[j, j:])   # OBS update of the not-yet-pruned columns
            W[:, j] = q
    return W

def out_err(Wp):
    return np.linalg.norm(X @ (W - Wp).T) / np.linalg.norm(X @ W.T)

for name, f in [("magnitude", magnitude), ("Wanda", wanda), ("SparseGPT", sparsegpt)]:
    print(f"{name:10s} 50% unstructured: {out_err(f(W, X)):.3f}   2:4: {out_err(f(W, X, nm=True)):.3f}")
# magnitude  50% unstructured: 0.267   2:4: 0.376
# Wanda      50% unstructured: 0.101   2:4: 0.139
# SparseGPT  50% unstructured: 0.084   2:4: 0.114
```

Magnitude, Wanda and a compact SparseGPT at 50% unstructured and 2:4 on a toy layer with outlier features (numpy; run, printed values from that run)

The ordering matches the papers: the activation-aware mask does most of the work, and the OBS update adds a further gain. In PyTorch, `torch.nn.utils.prune` applies masks for experiments. `torch.sparse.to_sparse_semi_structured` converts a 2:4-pruned FP16 weight into the compressed format that runs on sparse tensor cores. A real LLM is pruned layer by layer: run the calibration batch through the already-pruned earlier layers, collect each linear layer's inputs, and prune it.

> **Sparsity is not speed:** A model with 50% unstructured zeros has exactly the same dense GEMMs unless the runtime uses sparse kernels, and on GPUs those rarely win at 50%. Even 2:4, with up to 2× faster GEMMs, gives a smaller end-to-end gain, because attention, the KV cache and memory traffic are untouched. Measure latency, not parameter counts. For memory savings at moderate accuracy cost, **Quantization** is usually the better first step, and the two combine.

> **Score by what the weight does to the output:** A weight's importance is its effect on the output, $|W_{ij}|\,\|X_j\|$ to first order. The weight alone is not enough. This is the same lesson as AWQ in **Quantization**: salient weights are found through activations. In the figure, magnitude pruning throws away the small weights on the three loud input features and doubles the output error.

> **Perplexity hides damage:** One-shot pruned LLMs at 50% often look fine on perplexity yet lose more on knowledge-intensive and reasoning tasks, and the loss grows with sparsity. Results also differ across model families. Check task benchmarks (**LLM evaluation and benchmarking**), and treat one-shot sparsity above 50–60% with suspicion unless followed by retraining.

**Strengths**

- One-shot methods need only a calibration set: Wanda one forward pass, SparseGPT hours for 175B models
- 2:4 sparsity runs on NVIDIA sparse tensor cores with up to 2× GEMM throughput
- Structured pruning with distillation builds a family of smaller dense models at a fraction of pre-training cost
- Combines with quantization (SparseGPT can prune and quantize in one pass)

**Limitations**

- Unstructured sparsity rarely speeds up GPU inference
- Quality drops quickly beyond 50–60% one-shot sparsity, especially on reasoning tasks
- Structured pruning needs substantial retraining (billions of tokens) to recover
- 2:4 is fixed at exactly 50% and costs more accuracy than unstructured 50%

- **Small-model families:** Minitron-style width/depth pruning plus distillation produces 4–8B models from a larger trained one.
- **Sparse inference:** 2:4 sparse weights with sparse tensor-core GEMMs, often combined with INT8 or FP8 quantization.
- **Faster drafts:** A pruned or layer-skipped copy of the target can serve as the draft in **Speculative decoding**.

**Question:** Two weights in the same row: w₁ = 0.8 on a feature with ‖X₁‖ = 1, and w₂ = 0.05 on a feature with ‖X₂‖ = 30. Which does Wanda prune first, and why?

- A. w₂, because it has the smaller magnitude
- B. w₁, because its score 0.8 is below w₂'s score 1.5: removing it changes the output less
- C. Neither; Wanda only prunes whole rows
- D. Both equally, because Wanda normalises the weights

**Answer:** B. w₁, because its score 0.8 is below w₂'s score 1.5: removing it changes the output less — Wanda's scores are $0.8\cdot1=0.8$ and $0.05\cdot30=1.5$. For uncorrelated features the output change from removing a weight is $w^2\|X\|^2$: $0.64$ against $2.25$. The small weight on the loud feature matters more.

**Question:** Why does a 2:4 sparse FP16 weight matrix take 9 bits per weight rather than 8?

- A. Sparse tensor cores need FP32 values
- B. Each kept value needs a 2-bit index giving its position within its group of 4: (2·16 + 2·2)/4 = 9
- C. A per-row scale is stored
- D. Zeros are still stored, at 1 bit each

**Answer:** B. Each kept value needs a 2-bit index giving its position within its group of 4: (2·16 + 2·2)/4 = 9 — Half the values are dropped (16·2/4 = 8 bits per weight), but the hardware must know which two of the four positions each kept value came from. That costs 2 bits per kept value, 1 bit per weight on average.

**Key points**

- OBS saliency $w_q^2/(2[H^{-1}]_{qq})$ with the compensating update $\delta w=-\frac{w_q}{[H^{-1}]_{qq}}H^{-1}e_q$. Magnitude pruning is the special case $H\propto I$.
- Wanda scores $|W_{ij}|\cdot\|X_j\|_2$ per output row with no update. On LLaMA-7B at 50% it gives perplexity 7.26, against 7.22 for SparseGPT and 17.29 for magnitude.
- SparseGPT runs OBS column by column with a shared Cholesky factor of H⁻¹. It prunes 175B models to 50–60% in under 4.5 GPU-hours and also does 2:4.
- Unstructured sparsity rarely speeds up GPUs. 2:4 runs on sparse tensor cores (9 bits per FP16 weight). Structured pruning yields smaller dense models.
- Width and depth pruning followed by distillation (Minitron: 15B to 8B and 4B with up to 40× fewer tokens; Sheared-LLaMA: 7B to 1.3B and 2.7B) builds small models cheaply.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
