Sparse autoencoder

Give the autoencoder more hidden units than inputs but let only a few fire per example, so each unit learns one reusable feature.

10 min read1186 wordscore0 viewsupdated 2026-10-11

Definition

Sparse autoencoder is an autoencoder whose hidden code h=f(x)h=f(x) is pushed to have few active units per input, by adding a sparsity penalty Ω(h)\Omega(h) to the reconstruction loss. The hidden layer can then be overcomplete (more units than inputs) without collapsing into a copy of the input, and each unit tends to learn one recognisable part of the data.

A plain autoencoder (see the Autoencoder article) avoids learning the identity map by squeezing the data through a bottleneck narrower than the input. That is a blunt constraint. Real data is often built from many possible parts, of which only a few are present at once: an image patch contains one or two edges, a sentence touches a handful of concepts. A code that has a slot for every part, but uses only a few slots at a time, fits that structure better than a narrow code where every unit is busy on every input.

Sparsity replaces the bottleneck. Width stops limiting capacity; instead, a penalty makes each unit fire rarely. A unit that fires rarely has to fire for something specific, so it becomes a detector for one pattern.

The idea comes from neuroscience. Olshausen and Field (1996) showed that asking for a sparse linear code of natural image patches yields localized, oriented, Gabor-like filters, much like receptive fields in visual cortex. Andrew Ng's CS294A notes (2011) made the neural-network version, with a KL penalty, the standard teaching example. Since 2023 the same tool has become central to interpretability of large language models.

The objective

Encoder h=φ(W1x+b1)h=\varphi(W_1x+b_1) with mm hidden units, mm possibly larger than the input dimension dd; decoder x^=W2h+b2\hat x=W_2h+b_2. Over a training set of NN examples:

J(W,b)=12N∑n=1N∥x^(n)−x(n)∥2+λ2∥W∥2+β Ω(h)J(W,b)=\frac{1}{2N}\sum_{n=1}^{N}\big\|\hat x^{(n)}-x^{(n)}\big\|^2+\frac{\lambda}{2}\|W\|^2+\beta\,\Omega(h)
Ω(h)\Omega(h)
sparsity penalty on the hidden activations (two common choices below)
β\beta
how strongly sparsity is traded against reconstruction
λ\lambda
weight decay; stops the network from making activations tiny and decoder weights huge to dodge the penalty

L1 penalty. With ReLU units, Ω=1N∑n∑j∣hj(n)∣\Omega=\frac1N\sum_n\sum_j|h^{(n)}_j|. Its gradient is a constant push toward 0 for every active unit, so units that are not needed land exactly at zero, just as L1 zeroes coefficients in the lasso.

KL penalty (Ng's notes). With sigmoid units, hj∈(0,1)h_j\in(0,1). Let ρ^j=1N∑nhj(x(n))\hat\rho_j=\frac1N\sum_n h_j(x^{(n)}) be the average activation of unit jj. Pick a small target ρ\rho, say 0.050.05, and penalize the divergence between a Bernoulli(ρ\rho) and a Bernoulli(ρ^j\hat\rho_j):

Ω=∑j=1mKL(ρ ∥ ρ^j)=∑j=1m[ρlog⁡ρρ^j+(1−ρ)log⁡1−ρ1−ρ^j]\Omega=\sum_{j=1}^{m}\mathrm{KL}(\rho\,\|\,\hat\rho_j)=\sum_{j=1}^{m}\Big[\rho\log\frac{\rho}{\hat\rho_j}+(1-\rho)\log\frac{1-\rho}{1-\hat\rho_j}\Big]
ρ^j\hat\rho_j
mean activation of hidden unit j over the training set (or the mini-batch)
ρ\rho
sparsity target: unit jj should be "on" for about a fraction ρ\rho of inputs
KL\mathrm{KL}
0 when ρ^j=ρ\hat\rho_j=\rho, and grows without bound as ρ^j→0\hat\rho_j\to0 or 11

The KL penalty constrains each unit's average, not each example. A unit can satisfy it by being strongly on for 5% of inputs (a detector) or weakly on, at 0.05, for all of them (useless). Reconstruction pressure usually prefers the detector, but weight decay matters: without it the encoder can shrink every activation and the decoder scale them back up.

The KL term in backprop

Since ∂ KL(ρ∥ρ^j)/∂ρ^j=−ρ/ρ^j+(1−ρ)/(1−ρ^j)\partial\,\mathrm{KL}(\rho\|\hat\rho_j)/\partial\hat\rho_j=-\rho/\hat\rho_j+(1-\rho)/(1-\hat\rho_j) and ∂ρ^j/∂hj(n)=1/N\partial\hat\rho_j/\partial h^{(n)}_j=1/N, the penalty simply adds one term to each hidden unit's backpropagated error, before the sigmoid slope:

δj(2)=((W2⊤δ(3))j+β(−ρρ^j+1−ρ1−ρ^j)) hj(1−hj)\delta^{(2)}_j=\Big(\big(W_2^\top\delta^{(3)}\big)_j+\beta\Big(-\frac{\rho}{\hat\rho_j}+\frac{1-\rho}{1-\hat\rho_j}\Big)\Big)\,h_j(1-h_j)
δ(3)\delta^{(3)}
output error x^−x\hat x-x (linear decoder, squared error)
β(⋯ )\beta(\cdots)
positive when ρ^j>ρ\hat\rho_j>\rho (pushes the unit down), negative when ρ^j<ρ\hat\rho_j<\rho (pushes it up)

The term is the same for every example, but it needs ρ^j\hat\rho_j, which depends on all of them. Ng's notes run a forward pass over the whole set first to get ρ^\hat\rho; with mini-batches one uses the batch mean or a running average. (With the 1N\frac1N-averaged reconstruction loss written above, the per-example term is divided by NN too.)

decoder atoms W₂[:, j], 24 > 16 pixels123456789101112131415161718192021222324input #1output x̂firing 24/24‖x̂−x‖² 21.540col 1 + col 3KL(ρ‖ρ̂ⱼ) per unit (dots) · ρ = 0.020ρ0.000.150.30ρ̂ⱼ = mean activation over 120 images →code h = σ(W₁x + b₁) for input #1 · dashed = firing threshold 0.11.00123456789101112131415161718192021222324atoms: ■ + weight ■ − weight · □ unit fires on this inputgreen number: atom matches one bar (cos > 0.9) · dim: dead unit (never fires)
penalty
Adam step 0/3000
recon ½‖x̂−x‖² 10.8330penalty 14.395avg units firing 23.94dead 0bars found 0/8
fig.A real 16→24→16 autoencoder trained live (full-batch Adam, 3000 steps) on 120 seeded 4×4 images, each made of one or two of 8 bars. With β = 0, all 24 sigmoid units fire on every image and no atom looks like a bar. At the default KL setting (ρ = 0.02, β = 0.3) about 2.2 units fire per image and 8 atoms match the 8 bars. Push β to 3 and the units sit at ρ̂ ≈ 0.02–0.03 but stop matching bars, while reconstruction error doubles. Switch to L1 with β = 0.1 for exact zeros: 1.5 units per image, 13 units dead.

Worked example

One unit, target ρ = 0.05

  1. 1The unit fires too often: ρ^j=0.2\hat\rho_j=0.2. Penalty 0.05log⁡0.050.2+0.95log⁡0.950.8=−0.0693+0.1633=0.09390.05\log\frac{0.05}{0.2}+0.95\log\frac{0.95}{0.8}=-0.0693+0.1633=0.0939.
  2. 2Its gradient factor is −0.050.2+0.950.8=−0.25+1.1875=0.9375>0-\frac{0.05}{0.2}+\frac{0.95}{0.8}=-0.25+1.1875=0.9375>0, so gradient descent lowers its activations.
  3. 3Now it is nearly silent, ρ^j=0.01\hat\rho_j=0.01: penalty 0.04130.0413, factor −0.050.01+0.950.99=−4.04-\frac{0.05}{0.01}+\frac{0.95}{0.99}=-4.04. The negative sign raises activity, and the push is much stronger than in the first case. The KL is asymmetric: it punishes dead units harder than busy ones.
  4. 4At ρ^j=ρ\hat\rho_j=\rho both the penalty and the gradient are 0.
Penaltyβ½‖x̂−x‖²Units firing / imageDead unitsBars found
KL, ρ = 0.0200.006023.9700/8
KL, ρ = 0.020.10.03323.5904/8
KL, ρ = 0.020.30.05422.2008/8
KL, ρ = 0.0230.11381.9300/8
L1, ReLU00.03197.11100/8
L1, ReLU0.10.10381.52138/8
L1, ReLU0.30.40021.32163/8
The figure's toy problem after 3000 Adam steps (120 images with 1.56 bars on average; seed fixed). "Firing" = average number of units with h > 0.1 per image; "bars found" = distinct bars matched by a live unit's decoder atom with cosine > 0.9.

Without a penalty the overcomplete layer reconstructs best but explains each image with nearly every unit. There is a sweet spot where the code uses roughly as many units as there are bars (1.56 on average) and each atom is a bar. Beyond it, reconstruction gets worse and the features get worse with it. The L1 rows show the cost of the penalty's constant pull on active units: their activations shrink, so reconstruction is worse than KL at similar sparsity.

Training a sparse autoencoder

  1. 1Choose an overcomplete width mm (from 2d2d up to many times dd), a penalty (KL with target ρ\rho, L1, or top-kk) and its weight β\beta.
  2. 2Forward a batch: h=φ(W1x+b1)h=\varphi(W_1x+b_1), x^=W2h+b2\hat x=W_2h+b_2. For KL, compute each unit's mean activation ρ^j\hat\rho_j.
  3. 3Backpropagate the reconstruction error, adding the sparsity term to each hidden unit's error (β sign(hj)\beta\,\mathrm{sign}(h_j) for L1, the KL factor above for KL).
  4. 4Update, keeping decoder atoms bounded (weight decay, or renormalise each column of W2W_2 to unit length).
  5. 5Monitor three numbers: reconstruction error, average active units per input (L0L_0), and dead units that never fire. Resample dead units if there are many.

k-sparse (top-k) autoencoders

Instead of penalizing activity, Makhzani and Frey (2013) enforce it: keep the kk largest pre-activations, set the rest to 0, and backpropagate only through the survivors. Sparsity is exactly kk by construction, there is no β\beta to tune, and the surviving activations are not shrunk. OpenAI's 2024 work on scaling sparse autoencoders to GPT-4 used top-kk for these reasons.

Modern use: features inside language models

A transformer layer with 512 neurons seems to represent far more than 512 concepts. In superposition (Elhage et al., Toy Models of Superposition, 2022), features are stored as almost-orthogonal directions, more of them than there are dimensions. That works because any one input uses only a few features. Individual neurons are then polysemantic: each responds to a mix of unrelated features.

A sparse autoencoder trained on the model's activations is exactly the tool to undo this. It is dictionary learning: each decoder column W2,:jW_{2,:j} is one feature direction, and the sparse code says which few features are present. Anthropic's Towards Monosemanticity (Bricken et al., 2023) trained SAEs on the 512-neuron MLP layer of a one-layer transformer, with dictionaries of up to 131,072 features, many of them human-interpretable (DNA sequences, Arabic script, base64). Scaling Monosemanticity (2024) extended this to Claude 3 Sonnet, with up to 34 million features. Cunningham et al. (2023) reported similar findings independently.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class SAE(nn.Module):
4 def __init__(self, d, m): # m >> d: overcomplete dictionary
5 super().__init__()
6 self.enc = nn.Linear(d, m)
7 self.dec = nn.Linear(m, d)
8
9 def forward(self, x):
10 h = F.relu(self.enc(x - self.dec.bias)) # sparse code, exact zeros
11 return self.dec(h), h
12
13def loss_fn(sae, x, beta=5.0):
14 x_hat, h = sae(x)
15 recon = (x_hat - x).pow(2).sum(-1).mean()
16 sparsity = (h * sae.dec.weight.norm(dim=0)).sum(-1).mean() # Σ_j h_j ‖W_dec[:, j]‖
17 return recon + beta * sparsity
18
19def kl_penalty(h, rho=0.05): # Ng-style alternative, sigmoid h
20 rho_hat = h.mean(0).clamp(1e-6, 1 - 1e-6) # mean activation per unit over the batch
21 return (rho * torch.log(rho / rho_hat) + (1 - rho) * torch.log((1 - rho) / (1 - rho_hat))).sum()
A dictionary-learning SAE on model activations, in the style used for interpretability (ReLU code, L1 weighted by decoder norm)

Weighting each hjh_j by its decoder column's norm stops the optimizer from cheating, that is, from shrinking hjh_j while growing the column. Normalizing the columns to unit length at every step achieves the same.

Use when

  • +Overcomplete codes without a bottleneck: capacity comes from width, structure from sparsity.
  • +Features are individually meaningful (edges, bars, concepts), which makes them useful for interpretability and as inputs to simple classifiers.
  • +The decoder is linear, so each feature is a direction you can inspect, ablate or steer with.

Avoid when

  • −Another trade-off knob (β\beta, ρ\rho or kk) that interacts with learning rate and width; too much sparsity hurts both reconstruction and features.
  • −Dead units, feature splitting and shrinkage need dedicated fixes at scale.
  • −Sparse features are a hypothesis about the data. An SAE always finds some dictionary, and its interpretability must be checked, not assumed.

Check yourself

With a KL sparsity penalty and target ρ=0.05\rho=0.05, a hidden unit currently has mean activation ρ^j=0.3\hat\rho_j=0.3. What does the extra backprop term do?

Check yourself

Why can a sparse autoencoder use more hidden units than inputs without learning the identity map?

Remember

  • ✓Sparsity, not width, limits capacity: the hidden layer may be overcomplete, but only a few units fire per input.
  • ✓KL penalty: ∑jKL(ρ∥ρ^j)\sum_j\mathrm{KL}(\rho\|\hat\rho_j) on mean activations; backprop adds β(−ρ/ρ^j+(1−ρ)/(1−ρ^j))\beta(-\rho/\hat\rho_j+(1-\rho)/(1-\hat\rho_j)) before σ′\sigma'.
  • ✓L1 on ReLU codes gives exact zeros; top-k fixes the count directly and avoids shrinkage.
  • ✓Learned atoms are parts of the data: Gabor-like edges on images, bars in the figure, concepts in LLM activations.
  • ✓Interpretability SAEs are dictionary learning on activations, motivated by superposition (Towards Monosemanticity, 2023).
← back to Deep Learning roadmap