# Autoencoder

> A network trained to reproduce its input through a narrow code; linear ones recover PCA, deep nonlinear ones learn curved low-dimensional manifolds.

Source: https://www.mlfoundry.in/learn/autoencoder  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 10 min read · updated 2026-10-11

**Autoencoder:** is a neural network trained to output its own input. An **encoder** $f_\theta$ maps $x\in\mathbb{R}^d$ to a code $z=f_\theta(x)\in\mathbb{R}^k$, a **decoder** $g_\phi$ maps the code back to $\hat x=g_\phi(z)$, and both are trained together to make the reconstruction error $\ell(x,\hat x)$ small. No labels are needed: the input is its own target.

Copying is trivial, so the interesting part is the constraint that stops the network from just copying. The classic one is a **bottleneck**: with $k<d$ numbers in the code, the network can only reconstruct well if it finds the few directions or coordinates along which the data actually varies. Real data such as images, audio and sensor readings usually lie near a low-dimensional surface (a *manifold*) inside the huge input space, so a small code can describe them well.

The idea goes back to the 1980s ("auto-association" with backprop: Rumelhart, Hinton & Williams 1986; Bourlard & Kamp 1988; Baldi & Hornik 1989). Deep autoencoders became practical in 2006, when Hinton & Salakhutdinov pretrained a 784–1000–500–250–30 network layer by layer with RBMs and showed its 30-number codes reconstructed digits and faces far better than 30 principal components.

## Architecture and loss

$$
z=f_\theta(x),\qquad \hat x=g_\phi(z),\qquad \min_{\theta,\phi}\ \frac1n\sum_{i=1}^n\ell\big(x_i,\ g_\phi(f_\theta(x_i))\big)
$$

- $f_\theta$: encoder: any network $\mathbb{R}^d\to\mathbb{R}^k$ (MLP, CNN, …)
- $z\in\mathbb{R}^k$: the code, latent vector, or bottleneck activation
- $g_\phi$: decoder $\mathbb{R}^k\to\mathbb{R}^d$, often mirroring the encoder
- $\ell$: reconstruction loss, chosen to match the data type (below)

$$
\ell_{\text{MSE}}=\lVert x-\hat x\rVert^2,\qquad \ell_{\text{BCE}}=-\sum_{j=1}^d\Big[x_j\log\hat x_j+(1-x_j)\log(1-\hat x_j)\Big]
$$

- $\ell_{\text{MSE}}$: real-valued inputs; the negative log-likelihood of a Gaussian decoder with fixed variance (up to constants)
- $\ell_{\text{BCE}}$: inputs in $[0,1]$ (binary or normalised pixels) with a sigmoid output layer; a Bernoulli decoder

Training is ordinary backprop and SGD/Adam on this loss. After training you usually keep only one half: the encoder for features and compression, or the decoder to map codes back to inputs.

## Undercomplete and overcomplete

**Undercomplete** ($k<d$): the bottleneck itself forces compression. This is the case the rest of this article is about. **Overcomplete** ($k\ge d$): a big enough network can learn the identity and the code is useless, so it needs another constraint:

| Variant | Extra constraint | What it forces |
| --- | --- | --- |
| Sparse AE | penalty $\lambda\lVert z\rVert_1$, or a KL penalty pushing mean activations toward a small $\rho$ | only a few code units active per input |
| Denoising AE (Vincent et al. 2008) | train on corrupted $\tilde x$ but reconstruct the clean $x$ | learn the data's structure well enough to undo noise |
| Contractive AE (Rifai et al. 2011) | penalty $\lambda\lVert\partial z/\partial x\rVert_F^2$ | a code that is insensitive to small input changes |
| Variational AE (Kingma & Welling 2013) | code is a distribution, regularised toward a prior by a KL term | a smooth latent space you can sample new data from |

## Linear autoencoder = PCA

Take the simplest case: linear encoder $z=Ax$, linear decoder $\hat x=Bz$, centred data, MSE loss. The reconstruction $BAx$ has rank at most $k$, so the best it can do is the best rank-$k$ approximation of the data, which is projection onto the top-$k$ principal subspace. **Baldi & Hornik (1989)** proved more: this error surface has no bad local minima. Every critical point is either the global minimum or a saddle, so gradient descent finds the PCA solution.

$$
\min_{A,B}\ \frac1n\sum_i\lVert x_i-BAx_i\rVert^2=\sum_{j>k}\lambda_j,\qquad\text{attained iff }\ BA=U_kU_k^\top,\ \ B=U_kC,\ A=C^{-1}U_k^\top
$$

- $\lambda_j$: eigenvalues of the data covariance $\tfrac1n\sum_ix_ix_i^\top$ in decreasing order: the error is the variance in the dropped directions
- $U_k$: the top-k eigenvectors (principal directions)
- $C$: any invertible $k\times k$ matrix: the code is PCA's coordinates up to an arbitrary linear change of basis, so it spans the same subspace but is not orthogonal or ordered by variance

> **Nonlinear in one layer is not enough:** Bourlard & Kamp (1988) showed that a single nonlinear hidden layer with a linear output layer still cannot beat PCA's reconstruction error. To capture a **curved** manifold, the encoder and the decoder each need their own nonlinear hidden layers. That is the 2–16–1–16–2 shape in the figure below.

> Figure: 120 noisy points on an arc, autoencoded through a single number z (full-batch Adam, η = 0.01, seeded). Hollow circles are the data; filled dots are their reconstructions, coloured by z; grey lines join each point to its reconstruction. Press play: the nonlinear AE's decoder curve bends onto the arc and the MSE falls from 0.375 to 0.0016 by step 1000, about 3% of PCA's 0.0517. Switch to linear: within 100 steps the curve becomes exactly the dashed PCA line and the loss stops at 0.0517 (ratio 1.000). Click anywhere to send a probe point through both.

## Worked example: the figure's numbers

Mean squared reconstruction error ‖x − x̂‖² per point with a 1-number code, 120 points each, noise std 0.04 per coordinate. PCA error = λ₂ of the data covariance; AEs trained 3000 full-batch Adam steps (η = 0.01, same seed)

| Data | λ₁ | λ₂ = PCA error | Linear AE | Nonlinear AE (tanh) |
| --- | --- | --- | --- | --- |
| arc | 0.3238 | 0.0517 | 0.0517 | 0.0014 |
| wave | 0.3516 | 0.0227 | 0.0227 | 0.0020 |
| noisy line | 0.2652 | 0.0014 | 0.0014 | 0.0014 |

**Reading the table**

1. **Linear AE = PCA, every time.** The linear network (the same 2–16–1–16–2 shape with identity activations) reaches $\lambda_2$ to four decimals, and its decoder line points in the first principal direction (cosine similarity 1.000000 with the eigenvector). This is the Baldi–Hornik result happening live.
2. **Curved data.** PCA keeps 86% of the arc's variance, but the 14% it drops is structure, not noise. The nonlinear AE uses its single code to walk *along* the arc and gets 37× lower error.
3. **Straight data.** On the noisy line there is nothing curved to find, so all three tie at 0.0014, close to the noise variance across the line ($0.04^2=0.0016$). The nonlinear model does not overfit here, but it cost far more to train.
4. **One probe point** at $(0.05,0.45)$, on the arc: PCA projects it to $(0.030,0.143)$, error $0.0947$; the nonlinear AE encodes it as $z=0.155$ and decodes $(0.047,0.396)$, error $0.0029$.

```python
import math, torch, torch.nn as nn

torch.manual_seed(0)
a = math.pi * (0.05 + 0.9 * torch.rand(120))                   # noisy arc in 2-D
X = torch.stack([0.85 * a.cos(), 0.85 * a.sin() - 0.45], 1) + 0.04 * torch.randn(120, 2)

def autoencoder(act=nn.Tanh):
    enc = nn.Sequential(nn.Linear(2, 16), act(), nn.Linear(16, 1))   # 2 -> 1 bottleneck
    dec = nn.Sequential(nn.Linear(1, 16), act(), nn.Linear(16, 2))
    return enc, dec

enc, dec = autoencoder()                  # autoencoder(nn.Identity) gives the linear AE
opt = torch.optim.Adam([*enc.parameters(), *dec.parameters()], lr=0.01)
for step in range(3000):
    loss = ((dec(enc(X)) - X) ** 2).sum(1).mean()                # mean ||x - x_hat||^2
    opt.zero_grad(); loss.backward(); opt.step()

# PCA reference: the smallest eigenvalue of the covariance is the best linear 1-D error
lam = torch.linalg.eigvalsh(torch.cov(X.T, correction=0))
print(loss.item(), lam[0].item())
z = enc(X)                                # 1-D coordinates along the learned curve
```

The figure's autoencoder in PyTorch (same shape and optimiser; your numbers will differ with PyTorch's own initialisation)

> **Low reconstruction error is not a good code:** An autoencoder is judged by what its code is used for, not by its loss alone. Overcomplete or very large networks can reconstruct almost perfectly while learning nothing useful, and nothing in the loss shapes the code space beyond what reconstruction needs. The latent space of a plain AE also has gaps and no density, so decoding random codes rarely gives realistic outputs; that is what VAEs fix.

- **Dimensionality reduction:** A nonlinear alternative to PCA for visualisation and downstream models; the encoder also maps new points, unlike t-SNE.
- **Pretraining:** Historically, stacked (denoising) autoencoders initialised deep nets layer by layer; today masked autoencoders (MAE) pretrain vision transformers.
- **Anomaly detection:** Train on normal data and flag inputs with high reconstruction error; see the ML-branch article on autoencoder anomaly detection.
- **Compression and generative models:** Learned image codecs, and the latent spaces of latent-diffusion models, are autoencoders (often with quantised or variational codes).

**Strengths**

- Unsupervised: learns from unlabelled data.
- Captures curved, nonlinear structure that PCA cannot.
- Encoder and decoder are reusable networks that work on new inputs.

**Limitations**

- No closed form; needs tuning and can land in poor local minima, unlike PCA.
- Codes are not ordered, orthogonal or unique (any invertible reparametrisation fits equally well).
- A plain AE is not a generative model: its latent space has no prior to sample from.

**Question:** A linear autoencoder with a k-unit code is trained to convergence with MSE on centred data. What does it learn?

- A. The top-k principal components, exactly, as orthonormal code units
- B. A code that spans the same subspace as the top-k principal components, possibly in a different basis
- C. A nonlinear manifold
- D. The identity map

**Answer:** B. A code that spans the same subspace as the top-k principal components, possibly in a different basis — The optimum only fixes $BA=U_kU_k^\top$, so $A=C^{-1}U_k^\top$ for any invertible $C$. The reconstruction, and the error $\sum_{j>k}\lambda_j$, equal PCA's, but the individual code units are mixtures of principal coordinates unless you add constraints (e.g. tied, orthonormal weights).

**Question:** In the figure, why does the nonlinear AE beat PCA on the arc but not on the noisy line?

- A. Adam works better on curves
- B. On the arc, the dropped variance is curvature that a 1-D nonlinear code can follow; on the line, the dropped variance is pure noise that no 1-D code can predict
- C. The line has fewer points
- D. PCA overfits the arc

**Answer:** B. On the arc, the dropped variance is curvature that a 1-D nonlinear code can follow; on the line, the dropped variance is pure noise that no 1-D code can predict — A 1-number code can describe position along a curve, so the nonlinear AE removes the arc's curvature error (0.0517 → 0.0014). On the line, what PCA drops is noise across the line, about $0.04^2$, and every 1-D model is stuck at that floor (0.0014).

**Key points**

- Encoder $z=f_\theta(x)$, decoder $\hat x=g_\phi(z)$, trained to minimise reconstruction error; MSE for real values, BCE for $[0,1]$ inputs.
- Undercomplete: the bottleneck forces compression. Overcomplete: needs sparsity, denoising, contraction or a variational prior.
- Linear AE + MSE spans the PCA subspace with error $\sum_{j>k}\lambda_j$ (Baldi & Hornik 1989); there are no bad local minima.
- Nonlinear hidden layers on both sides let the code follow curved manifolds: on the arc, 37× lower error than PCA with one number.
- Uses: dimensionality reduction, pretraining, anomaly detection, compression and the latent spaces of generative models.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
