Definition
Sparse autoencoder is an autoencoder whose hidden code is pushed to have few active units per input, by adding a sparsity penalty to the reconstruction loss. The hidden layer can then be overcomplete (more units than inputs) without collapsing into a copy of the input, and each unit tends to learn one recognisable part of the data.
A plain autoencoder (see the Autoencoder article) avoids learning the identity map by squeezing the data through a bottleneck narrower than the input. That is a blunt constraint. Real data is often built from many possible parts, of which only a few are present at once: an image patch contains one or two edges, a sentence touches a handful of concepts. A code that has a slot for every part, but uses only a few slots at a time, fits that structure better than a narrow code where every unit is busy on every input.
Sparsity replaces the bottleneck. Width stops limiting capacity; instead, a penalty makes each unit fire rarely. A unit that fires rarely has to fire for something specific, so it becomes a detector for one pattern.
The idea comes from neuroscience. Olshausen and Field (1996) showed that asking for a sparse linear code of natural image patches yields localized, oriented, Gabor-like filters, much like receptive fields in visual cortex. Andrew Ng's CS294A notes (2011) made the neural-network version, with a KL penalty, the standard teaching example. Since 2023 the same tool has become central to interpretability of large language models.
The objective
Encoder with hidden units, possibly larger than the input dimension ; decoder . Over a training set of examples:
- sparsity penalty on the hidden activations (two common choices below)
- how strongly sparsity is traded against reconstruction
- weight decay; stops the network from making activations tiny and decoder weights huge to dodge the penalty
L1 penalty. With ReLU units, . Its gradient is a constant push toward 0 for every active unit, so units that are not needed land exactly at zero, just as L1 zeroes coefficients in the lasso.
KL penalty (Ng's notes). With sigmoid units, . Let be the average activation of unit . Pick a small target , say , and penalize the divergence between a Bernoulli() and a Bernoulli():
- mean activation of hidden unit j over the training set (or the mini-batch)
- sparsity target: unit should be "on" for about a fraction of inputs
- 0 when , and grows without bound as or
The KL penalty constrains each unit's average, not each example. A unit can satisfy it by being strongly on for 5% of inputs (a detector) or weakly on, at 0.05, for all of them (useless). Reconstruction pressure usually prefers the detector, but weight decay matters: without it the encoder can shrink every activation and the decoder scale them back up.
The KL term in backprop
Since and , the penalty simply adds one term to each hidden unit's backpropagated error, before the sigmoid slope:
- output error (linear decoder, squared error)
- positive when (pushes the unit down), negative when (pushes it up)
The term is the same for every example, but it needs , which depends on all of them. Ng's notes run a forward pass over the whole set first to get ; with mini-batches one uses the batch mean or a running average. (With the -averaged reconstruction loss written above, the per-example term is divided by too.)
Worked example
One unit, target ρ = 0.05
- 1The unit fires too often: . Penalty .
- 2Its gradient factor is , so gradient descent lowers its activations.
- 3Now it is nearly silent, : penalty , factor . The negative sign raises activity, and the push is much stronger than in the first case. The KL is asymmetric: it punishes dead units harder than busy ones.
- 4At both the penalty and the gradient are 0.
| Penalty | β | ½‖x̂−x‖² | Units firing / image | Dead units | Bars found |
|---|---|---|---|---|---|
| KL, ρ = 0.02 | 0 | 0.0060 | 23.97 | 0 | 0/8 |
| KL, ρ = 0.02 | 0.1 | 0.0332 | 3.59 | 0 | 4/8 |
| KL, ρ = 0.02 | 0.3 | 0.0542 | 2.20 | 0 | 8/8 |
| KL, ρ = 0.02 | 3 | 0.1138 | 1.93 | 0 | 0/8 |
| L1, ReLU | 0 | 0.0319 | 7.11 | 10 | 0/8 |
| L1, ReLU | 0.1 | 0.1038 | 1.52 | 13 | 8/8 |
| L1, ReLU | 0.3 | 0.4002 | 1.32 | 16 | 3/8 |
Without a penalty the overcomplete layer reconstructs best but explains each image with nearly every unit. There is a sweet spot where the code uses roughly as many units as there are bars (1.56 on average) and each atom is a bar. Beyond it, reconstruction gets worse and the features get worse with it. The L1 rows show the cost of the penalty's constant pull on active units: their activations shrink, so reconstruction is worse than KL at similar sparsity.
Training a sparse autoencoder
- 1Choose an overcomplete width (from up to many times ), a penalty (KL with target , L1, or top-) and its weight .
- 2Forward a batch: , . For KL, compute each unit's mean activation .
- 3Backpropagate the reconstruction error, adding the sparsity term to each hidden unit's error ( for L1, the KL factor above for KL).
- 4Update, keeping decoder atoms bounded (weight decay, or renormalise each column of to unit length).
- 5Monitor three numbers: reconstruction error, average active units per input (), and dead units that never fire. Resample dead units if there are many.
k-sparse (top-k) autoencoders
Instead of penalizing activity, Makhzani and Frey (2013) enforce it: keep the largest pre-activations, set the rest to 0, and backpropagate only through the survivors. Sparsity is exactly by construction, there is no to tune, and the surviving activations are not shrunk. OpenAI's 2024 work on scaling sparse autoencoders to GPT-4 used top- for these reasons.
Modern use: features inside language models
A transformer layer with 512 neurons seems to represent far more than 512 concepts. In superposition (Elhage et al., Toy Models of Superposition, 2022), features are stored as almost-orthogonal directions, more of them than there are dimensions. That works because any one input uses only a few features. Individual neurons are then polysemantic: each responds to a mix of unrelated features.
A sparse autoencoder trained on the model's activations is exactly the tool to undo this. It is dictionary learning: each decoder column is one feature direction, and the sparse code says which few features are present. Anthropic's Towards Monosemanticity (Bricken et al., 2023) trained SAEs on the 512-neuron MLP layer of a one-layer transformer, with dictionaries of up to 131,072 features, many of them human-interpretable (DNA sequences, Arabic script, base64). Scaling Monosemanticity (2024) extended this to Claude 3 Sonnet, with up to 34 million features. Cunningham et al. (2023) reported similar findings independently.
1import torch, torch.nn as nn, torch.nn.functional as F23class SAE(nn.Module):4 def __init__(self, d, m): # m >> d: overcomplete dictionary5 super().__init__()6 self.enc = nn.Linear(d, m)7 self.dec = nn.Linear(m, d)89 def forward(self, x):10 h = F.relu(self.enc(x - self.dec.bias)) # sparse code, exact zeros11 return self.dec(h), h1213def loss_fn(sae, x, beta=5.0):14 x_hat, h = sae(x)15 recon = (x_hat - x).pow(2).sum(-1).mean()16 sparsity = (h * sae.dec.weight.norm(dim=0)).sum(-1).mean() # Σ_j h_j ‖W_dec[:, j]‖17 return recon + beta * sparsity1819def kl_penalty(h, rho=0.05): # Ng-style alternative, sigmoid h20 rho_hat = h.mean(0).clamp(1e-6, 1 - 1e-6) # mean activation per unit over the batch21 return (rho * torch.log(rho / rho_hat) + (1 - rho) * torch.log((1 - rho) / (1 - rho_hat))).sum()Weighting each by its decoder column's norm stops the optimizer from cheating, that is, from shrinking while growing the column. Normalizing the columns to unit length at every step achieves the same.
Use when
- +Overcomplete codes without a bottleneck: capacity comes from width, structure from sparsity.
- +Features are individually meaningful (edges, bars, concepts), which makes them useful for interpretability and as inputs to simple classifiers.
- +The decoder is linear, so each feature is a direction you can inspect, ablate or steer with.
Avoid when
- −Another trade-off knob (, or ) that interacts with learning rate and width; too much sparsity hurts both reconstruction and features.
- −Dead units, feature splitting and shrinkage need dedicated fixes at scale.
- −Sparse features are a hypothesis about the data. An SAE always finds some dictionary, and its interpretability must be checked, not assumed.
Check yourself
With a KL sparsity penalty and target , a hidden unit currently has mean activation . What does the extra backprop term do?
Check yourself
Why can a sparse autoencoder use more hidden units than inputs without learning the identity map?
Remember
- ✓Sparsity, not width, limits capacity: the hidden layer may be overcomplete, but only a few units fire per input.
- ✓KL penalty: on mean activations; backprop adds before .
- ✓L1 on ReLU codes gives exact zeros; top-k fixes the count directly and avoids shrinkage.
- ✓Learned atoms are parts of the data: Gabor-like edges on images, bars in the figure, concepts in LLM activations.
- ✓Interpretability SAEs are dictionary learning on activations, motivated by superposition (Towards Monosemanticity, 2023).