Definition
Autoencoder is a neural network trained to output its own input. An encoder maps to a code , a decoder maps the code back to , and both are trained together to make the reconstruction error small. No labels are needed: the input is its own target.
Copying is trivial, so the interesting part is the constraint that stops the network from just copying. The classic one is a bottleneck: with numbers in the code, the network can only reconstruct well if it finds the few directions or coordinates along which the data actually varies. Real data such as images, audio and sensor readings usually lie near a low-dimensional surface (a manifold) inside the huge input space, so a small code can describe them well.
The idea goes back to the 1980s ("auto-association" with backprop: Rumelhart, Hinton & Williams 1986; Bourlard & Kamp 1988; Baldi & Hornik 1989). Deep autoencoders became practical in 2006, when Hinton & Salakhutdinov pretrained a 784–1000–500–250–30 network layer by layer with RBMs and showed its 30-number codes reconstructed digits and faces far better than 30 principal components.
Architecture and loss
- encoder: any network (MLP, CNN, …)
- the code, latent vector, or bottleneck activation
- decoder , often mirroring the encoder
- reconstruction loss, chosen to match the data type (below)
- real-valued inputs; the negative log-likelihood of a Gaussian decoder with fixed variance (up to constants)
- inputs in (binary or normalised pixels) with a sigmoid output layer; a Bernoulli decoder
Training is ordinary backprop and SGD/Adam on this loss. After training you usually keep only one half: the encoder for features and compression, or the decoder to map codes back to inputs.
Undercomplete and overcomplete
Undercomplete (): the bottleneck itself forces compression. This is the case the rest of this article is about. Overcomplete (): a big enough network can learn the identity and the code is useless, so it needs another constraint:
| Variant | Extra constraint | What it forces |
|---|---|---|
| Sparse AE | penalty , or a KL penalty pushing mean activations toward a small | only a few code units active per input |
| Denoising AE (Vincent et al. 2008) | train on corrupted but reconstruct the clean | learn the data's structure well enough to undo noise |
| Contractive AE (Rifai et al. 2011) | penalty | a code that is insensitive to small input changes |
| Variational AE (Kingma & Welling 2013) | code is a distribution, regularised toward a prior by a KL term | a smooth latent space you can sample new data from |
Linear autoencoder = PCA
Take the simplest case: linear encoder , linear decoder , centred data, MSE loss. The reconstruction has rank at most , so the best it can do is the best rank- approximation of the data, which is projection onto the top- principal subspace. Baldi & Hornik (1989) proved more: this error surface has no bad local minima. Every critical point is either the global minimum or a saddle, so gradient descent finds the PCA solution.
- eigenvalues of the data covariance in decreasing order: the error is the variance in the dropped directions
- the top-k eigenvectors (principal directions)
- any invertible matrix: the code is PCA's coordinates up to an arbitrary linear change of basis, so it spans the same subspace but is not orthogonal or ordered by variance
Worked example: the figure's numbers
| Data | λ₁ | λ₂ = PCA error | Linear AE | Nonlinear AE (tanh) |
|---|---|---|---|---|
| arc | 0.3238 | 0.0517 | 0.0517 | 0.0014 |
| wave | 0.3516 | 0.0227 | 0.0227 | 0.0020 |
| noisy line | 0.2652 | 0.0014 | 0.0014 | 0.0014 |
Reading the table
- 1Linear AE = PCA, every time. The linear network (the same 2–16–1–16–2 shape with identity activations) reaches to four decimals, and its decoder line points in the first principal direction (cosine similarity 1.000000 with the eigenvector). This is the Baldi–Hornik result happening live.
- 2Curved data. PCA keeps 86% of the arc's variance, but the 14% it drops is structure, not noise. The nonlinear AE uses its single code to walk along the arc and gets 37× lower error.
- 3Straight data. On the noisy line there is nothing curved to find, so all three tie at 0.0014, close to the noise variance across the line (). The nonlinear model does not overfit here, but it cost far more to train.
- 4One probe point at , on the arc: PCA projects it to , error ; the nonlinear AE encodes it as and decodes , error .
1import math, torch, torch.nn as nn23torch.manual_seed(0)4a = math.pi * (0.05 + 0.9 * torch.rand(120)) # noisy arc in 2-D5X = torch.stack([0.85 * a.cos(), 0.85 * a.sin() - 0.45], 1) + 0.04 * torch.randn(120, 2)67def autoencoder(act=nn.Tanh):8 enc = nn.Sequential(nn.Linear(2, 16), act(), nn.Linear(16, 1)) # 2 -> 1 bottleneck9 dec = nn.Sequential(nn.Linear(1, 16), act(), nn.Linear(16, 2))10 return enc, dec1112enc, dec = autoencoder() # autoencoder(nn.Identity) gives the linear AE13opt = torch.optim.Adam([*enc.parameters(), *dec.parameters()], lr=0.01)14for step in range(3000):15 loss = ((dec(enc(X)) - X) ** 2).sum(1).mean() # mean ||x - x_hat||^216 opt.zero_grad(); loss.backward(); opt.step()1718# PCA reference: the smallest eigenvalue of the covariance is the best linear 1-D error19lam = torch.linalg.eigvalsh(torch.cov(X.T, correction=0))20print(loss.item(), lam[0].item())21z = enc(X) # 1-D coordinates along the learned curveWhere it's used
Dimensionality reduction
A nonlinear alternative to PCA for visualisation and downstream models; the encoder also maps new points, unlike t-SNE.
Pretraining
Historically, stacked (denoising) autoencoders initialised deep nets layer by layer; today masked autoencoders (MAE) pretrain vision transformers.
Anomaly detection
Train on normal data and flag inputs with high reconstruction error; see the ML-branch article on autoencoder anomaly detection.
Compression and generative models
Learned image codecs, and the latent spaces of latent-diffusion models, are autoencoders (often with quantised or variational codes).
Use when
- +Unsupervised: learns from unlabelled data.
- +Captures curved, nonlinear structure that PCA cannot.
- +Encoder and decoder are reusable networks that work on new inputs.
Avoid when
- −No closed form; needs tuning and can land in poor local minima, unlike PCA.
- −Codes are not ordered, orthogonal or unique (any invertible reparametrisation fits equally well).
- −A plain AE is not a generative model: its latent space has no prior to sample from.
Check yourself
A linear autoencoder with a k-unit code is trained to convergence with MSE on centred data. What does it learn?
Check yourself
In the figure, why does the nonlinear AE beat PCA on the arc but not on the noisy line?
Remember
- ✓Encoder , decoder , trained to minimise reconstruction error; MSE for real values, BCE for inputs.
- ✓Undercomplete: the bottleneck forces compression. Overcomplete: needs sparsity, denoising, contraction or a variational prior.
- ✓Linear AE + MSE spans the PCA subspace with error (Baldi & Hornik 1989); there are no bad local minima.
- ✓Nonlinear hidden layers on both sides let the code follow curved manifolds: on the arc, 37× lower error than PCA with one number.
- ✓Uses: dimensionality reduction, pretraining, anomaly detection, compression and the latent spaces of generative models.