# Sparse autoencoder

> Give the autoencoder more hidden units than inputs but let only a few fire per example, so each unit learns one reusable feature.

Source: https://www.mlfoundry.in/learn/sparse-autoencoder  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 10 min read · updated 2026-10-11

**Sparse autoencoder:** is an autoencoder whose hidden code $h=f(x)$ is pushed to have **few active units per input**, by adding a sparsity penalty $\Omega(h)$ to the reconstruction loss. The hidden layer can then be **overcomplete** (more units than inputs) without collapsing into a copy of the input, and each unit tends to learn one recognisable part of the data.

A plain autoencoder (see the Autoencoder article) avoids learning the identity map by squeezing the data through a bottleneck narrower than the input. That is a blunt constraint. Real data is often built from many possible parts, of which only a few are present at once: an image patch contains one or two edges, a sentence touches a handful of concepts. A code that has a slot for **every** part, but uses only a few slots at a time, fits that structure better than a narrow code where every unit is busy on every input.

Sparsity replaces the bottleneck. Width stops limiting capacity; instead, a penalty makes each unit fire rarely. A unit that fires rarely has to fire for something specific, so it becomes a detector for one pattern.

The idea comes from neuroscience. Olshausen and Field (1996) showed that asking for a sparse linear code of natural image patches yields localized, oriented, Gabor-like filters, much like receptive fields in visual cortex. Andrew Ng's CS294A notes (2011) made the neural-network version, with a KL penalty, the standard teaching example. Since 2023 the same tool has become central to **interpretability** of large language models.

## The objective

Encoder $h=\varphi(W_1x+b_1)$ with $m$ hidden units, $m$ possibly larger than the input dimension $d$; decoder $\hat x=W_2h+b_2$. Over a training set of $N$ examples:

$$
J(W,b)=\frac{1}{2N}\sum_{n=1}^{N}\big\|\hat x^{(n)}-x^{(n)}\big\|^2+\frac{\lambda}{2}\|W\|^2+\beta\,\Omega(h)
$$

- $\Omega(h)$: sparsity penalty on the hidden activations (two common choices below)
- $\beta$: how strongly sparsity is traded against reconstruction
- $\lambda$: weight decay; stops the network from making activations tiny and decoder weights huge to dodge the penalty

**L1 penalty.** With ReLU units, $\Omega=\frac1N\sum_n\sum_j|h^{(n)}_j|$. Its gradient is a constant push toward 0 for every active unit, so units that are not needed land exactly at zero, just as L1 zeroes coefficients in the lasso.

**KL penalty (Ng's notes).** With sigmoid units, $h_j\in(0,1)$. Let $\hat\rho_j=\frac1N\sum_n h_j(x^{(n)})$ be the **average activation** of unit $j$. Pick a small target $\rho$, say $0.05$, and penalize the divergence between a Bernoulli($\rho$) and a Bernoulli($\hat\rho_j$):

$$
\Omega=\sum_{j=1}^{m}\mathrm{KL}(\rho\,\|\,\hat\rho_j)=\sum_{j=1}^{m}\Big[\rho\log\frac{\rho}{\hat\rho_j}+(1-\rho)\log\frac{1-\rho}{1-\hat\rho_j}\Big]
$$

- $\hat\rho_j$: mean activation of hidden unit j over the training set (or the mini-batch)
- $\rho$: sparsity target: unit $j$ should be "on" for about a fraction $\rho$ of inputs
- $\mathrm{KL}$: 0 when $\hat\rho_j=\rho$, and grows without bound as $\hat\rho_j\to0$ or $1$

The KL penalty constrains each unit's **average**, not each example. A unit can satisfy it by being strongly on for 5% of inputs (a detector) or weakly on, at 0.05, for all of them (useless). Reconstruction pressure usually prefers the detector, but weight decay matters: without it the encoder can shrink every activation and the decoder scale them back up.

## The KL term in backprop

Since $\partial\,\mathrm{KL}(\rho\|\hat\rho_j)/\partial\hat\rho_j=-\rho/\hat\rho_j+(1-\rho)/(1-\hat\rho_j)$ and $\partial\hat\rho_j/\partial h^{(n)}_j=1/N$, the penalty simply adds one term to each hidden unit's backpropagated error, before the sigmoid slope:

$$
\delta^{(2)}_j=\Big(\big(W_2^\top\delta^{(3)}\big)_j+\beta\Big(-\frac{\rho}{\hat\rho_j}+\frac{1-\rho}{1-\hat\rho_j}\Big)\Big)\,h_j(1-h_j)
$$

- $\delta^{(3)}$: output error $\hat x-x$ (linear decoder, squared error)
- $\beta(\cdots)$: positive when $\hat\rho_j>\rho$ (pushes the unit down), negative when $\hat\rho_j<\rho$ (pushes it up)

The term is the same for every example, but it needs $\hat\rho_j$, which depends on all of them. Ng's notes run a forward pass over the whole set first to get $\hat\rho$; with mini-batches one uses the batch mean or a running average. (With the $\frac1N$-averaged reconstruction loss written above, the per-example term is divided by $N$ too.)

> Figure: A real 16→24→16 autoencoder trained live (full-batch Adam, 3000 steps) on 120 seeded 4×4 images, each made of one or two of 8 bars. With β = 0, all 24 sigmoid units fire on every image and no atom looks like a bar. At the default KL setting (ρ = 0.02, β = 0.3) about 2.2 units fire per image and 8 atoms match the 8 bars. Push β to 3 and the units sit at ρ̂ ≈ 0.02–0.03 but stop matching bars, while reconstruction error doubles. Switch to L1 with β = 0.1 for exact zeros: 1.5 units per image, 13 units dead.

## Worked example

**One unit, target ρ = 0.05**

1. The unit fires too often: $\hat\rho_j=0.2$. Penalty $0.05\log\frac{0.05}{0.2}+0.95\log\frac{0.95}{0.8}=-0.0693+0.1633=0.0939$.
2. Its gradient factor is $-\frac{0.05}{0.2}+\frac{0.95}{0.8}=-0.25+1.1875=0.9375>0$, so gradient descent lowers its activations.
3. Now it is nearly silent, $\hat\rho_j=0.01$: penalty $0.0413$, factor $-\frac{0.05}{0.01}+\frac{0.95}{0.99}=-4.04$. The negative sign raises activity, and the push is much stronger than in the first case. The KL is asymmetric: it punishes dead units harder than busy ones.
4. At $\hat\rho_j=\rho$ both the penalty and the gradient are 0.

The figure's toy problem after 3000 Adam steps (120 images with 1.56 bars on average; seed fixed). "Firing" = average number of units with h > 0.1 per image; "bars found" = distinct bars matched by a live unit's decoder atom with cosine > 0.9.

| Penalty | β | ½‖x̂−x‖² | Units firing / image | Dead units | Bars found |
| --- | --- | --- | --- | --- | --- |
| KL, ρ = 0.02 | 0 | 0.0060 | 23.97 | 0 | 0/8 |
| KL, ρ = 0.02 | 0.1 | 0.0332 | 3.59 | 0 | 4/8 |
| KL, ρ = 0.02 | 0.3 | 0.0542 | 2.20 | 0 | 8/8 |
| KL, ρ = 0.02 | 3 | 0.1138 | 1.93 | 0 | 0/8 |
| L1, ReLU | 0 | 0.0319 | 7.11 | 10 | 0/8 |
| L1, ReLU | 0.1 | 0.1038 | 1.52 | 13 | 8/8 |
| L1, ReLU | 0.3 | 0.4002 | 1.32 | 16 | 3/8 |

Without a penalty the overcomplete layer reconstructs best but explains each image with nearly every unit. There is a sweet spot where the code uses roughly as many units as there are bars (1.56 on average) and each atom is a bar. Beyond it, reconstruction gets worse and the features get worse with it. The L1 rows show the cost of the penalty's constant pull on active units: their activations shrink, so reconstruction is worse than KL at similar sparsity.

**Training a sparse autoencoder**

1. Choose an overcomplete width $m$ (from $2d$ up to many times $d$), a penalty (KL with target $\rho$, L1, or top-$k$) and its weight $\beta$.
2. Forward a batch: $h=\varphi(W_1x+b_1)$, $\hat x=W_2h+b_2$. For KL, compute each unit's mean activation $\hat\rho_j$.
3. Backpropagate the reconstruction error, adding the sparsity term to each hidden unit's error ($\beta\,\mathrm{sign}(h_j)$ for L1, the KL factor above for KL).
4. Update, keeping decoder atoms bounded (weight decay, or renormalise each column of $W_2$ to unit length).
5. Monitor three numbers: reconstruction error, average active units per input ($L_0$), and **dead units** that never fire. Resample dead units if there are many.

## k-sparse (top-k) autoencoders

Instead of penalizing activity, Makhzani and Frey (2013) **enforce** it: keep the $k$ largest pre-activations, set the rest to 0, and backpropagate only through the survivors. Sparsity is exactly $k$ by construction, there is no $\beta$ to tune, and the surviving activations are not shrunk. OpenAI's 2024 work on scaling sparse autoencoders to GPT-4 used top-$k$ for these reasons.

## Modern use: features inside language models

A transformer layer with 512 neurons seems to represent far more than 512 concepts. In **superposition** (Elhage et al., *Toy Models of Superposition*, 2022), features are stored as almost-orthogonal directions, more of them than there are dimensions. That works because any one input uses only a few features. Individual neurons are then **polysemantic**: each responds to a mix of unrelated features.

A sparse autoencoder trained on the model's activations is exactly the tool to undo this. It is **dictionary learning**: each decoder column $W_{2,:j}$ is one feature direction, and the sparse code says which few features are present. Anthropic's *Towards Monosemanticity* (Bricken et al., 2023) trained SAEs on the 512-neuron MLP layer of a one-layer transformer, with dictionaries of up to 131,072 features, many of them human-interpretable (DNA sequences, Arabic script, base64). *Scaling Monosemanticity* (2024) extended this to Claude 3 Sonnet, with up to 34 million features. Cunningham et al. (2023) reported similar findings independently.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class SAE(nn.Module):
    def __init__(self, d, m):                       # m >> d: overcomplete dictionary
        super().__init__()
        self.enc = nn.Linear(d, m)
        self.dec = nn.Linear(m, d)

    def forward(self, x):
        h = F.relu(self.enc(x - self.dec.bias))     # sparse code, exact zeros
        return self.dec(h), h

def loss_fn(sae, x, beta=5.0):
    x_hat, h = sae(x)
    recon = (x_hat - x).pow(2).sum(-1).mean()
    sparsity = (h * sae.dec.weight.norm(dim=0)).sum(-1).mean()   # Σ_j h_j ‖W_dec[:, j]‖
    return recon + beta * sparsity

def kl_penalty(h, rho=0.05):                         # Ng-style alternative, sigmoid h
    rho_hat = h.mean(0).clamp(1e-6, 1 - 1e-6)        # mean activation per unit over the batch
    return (rho * torch.log(rho / rho_hat) + (1 - rho) * torch.log((1 - rho) / (1 - rho_hat))).sum()
```

A dictionary-learning SAE on model activations, in the style used for interpretability (ReLU code, L1 weighted by decoder norm)

Weighting each $h_j$ by its decoder column's norm stops the optimizer from cheating, that is, from shrinking $h_j$ while growing the column. Normalizing the columns to unit length at every step achieves the same.

> **Dead units and shrinkage:** ReLU units whose pre-activation is negative on every input get no gradient and never recover. Dead units were 10–16 of 24 in the L1 runs above, and in large SAEs they can be a big share of the dictionary. Practitioners resample them, or use top-$k$ and auxiliary losses. Separately, an L1 penalty systematically shrinks the activations that remain. Top-$k$ and JumpReLU SAEs (DeepMind, 2024) were designed to avoid this.

> **Why sparsity yields parts:** If at most a few units may explain an input, the cheapest code assigns one unit to each recurring, independent part, so that a new input is a sum of a few parts. On natural image patches those parts are edges. On LLM activations they are concepts. The figure's 4×4 bars are the smallest version of the same effect.

**Strengths**

- Overcomplete codes without a bottleneck: capacity comes from width, structure from sparsity.
- Features are individually meaningful (edges, bars, concepts), which makes them useful for interpretability and as inputs to simple classifiers.
- The decoder is linear, so each feature is a direction you can inspect, ablate or steer with.

**Limitations**

- Another trade-off knob ($\beta$, $\rho$ or $k$) that interacts with learning rate and width; too much sparsity hurts both reconstruction and features.
- Dead units, feature splitting and shrinkage need dedicated fixes at scale.
- Sparse features are a hypothesis about the data. An SAE always finds some dictionary, and its interpretability must be checked, not assumed.

**Question:** With a KL sparsity penalty and target $\rho=0.05$, a hidden unit currently has mean activation $\hat\rho_j=0.3$. What does the extra backprop term do?

- A. Adds a positive amount to that unit's error, so gradient descent lowers its activations
- B. Adds a negative amount, raising its activations
- C. Nothing, because the KL penalty only acts on the decoder
- D. Sets the unit's output to zero for 95% of inputs

**Answer:** A. Adds a positive amount to that unit's error, so gradient descent lowers its activations — The factor is $-\rho/\hat\rho_j+(1-\rho)/(1-\hat\rho_j)=-0.167+1.357=1.19>0$. A positive error on $z_j$ means descent decreases $z_j$, so the unit fires less. The term is negative only when $\hat\rho_j<\rho$.

**Question:** Why can a sparse autoencoder use more hidden units than inputs without learning the identity map?

- A. Because the decoder is linear
- B. Because the sparsity penalty makes using many units at once costly, so the code cannot simply copy every input coordinate
- C. Because weight decay forces the weights to zero
- D. It can't: overcomplete autoencoders always learn the identity

**Answer:** B. Because the sparsity penalty makes using many units at once costly, so the code cannot simply copy every input coordinate — Copying a dense input into a dense code would make every unit active on every example, which the penalty charges heavily (24 units firing per image at β = 0 in the table). The cheaper solution is to reuse a few feature detectors per input.

**Key points**

- Sparsity, not width, limits capacity: the hidden layer may be overcomplete, but only a few units fire per input.
- KL penalty: $\sum_j\mathrm{KL}(\rho\|\hat\rho_j)$ on mean activations; backprop adds $\beta(-\rho/\hat\rho_j+(1-\rho)/(1-\hat\rho_j))$ before $\sigma'$.
- L1 on ReLU codes gives exact zeros; top-k fixes the count directly and avoids shrinkage.
- Learned atoms are parts of the data: Gabor-like edges on images, bars in the figure, concepts in LLM activations.
- Interpretability SAEs are dictionary learning on activations, motivated by superposition (Towards Monosemanticity, 2023).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
