Denoising autoencoder

Corrupt the input, reconstruct the clean version: the network must learn which way the data manifold lies, and its output points along the score ∇log p(x).

10 min read942 wordscore0 viewsupdated 2026-10-11

Definition

Denoising autoencoder (DAE) is an autoencoder trained to map a corrupted input x~∼q(x~∣x)\tilde x\sim q(\tilde x\mid x) back to the clean xx, minimizing E ∥g(f(x~))−x∥2\mathbb{E}\,\|g(f(\tilde x))-x\|^2. Because the target differs from the input, copying is no longer a solution, and the network has to learn where the data actually lies.

An ordinary autoencoder with a wide enough hidden layer can learn the identity and reconstruct perfectly while learning nothing (see the Autoencoder article). The usual fixes restrict the code: a bottleneck, or a sparsity penalty. Vincent, Larochelle, Bengio and Manzagol (2008) changed the task instead. Destroy part of the input and ask for the original back. The identity now scores badly, because it reproduces the noise.

To undo noise, the network must know what clean data looks like: which pixel values go together, which directions are "on the data" and which are "off it". That knowledge is exactly the structure we want the representation to capture, and it works even with an overcomplete code.

Training objective and corruptions

LDAE=Ex∼pdata  Ex~∼q(x~∣x)  ∥ r(x~)−x ∥2,r=g∘f\mathcal{L}_{\text{DAE}}=\mathbb{E}_{x\sim p_{\text{data}}}\;\mathbb{E}_{\tilde x\sim q(\tilde x\mid x)}\;\big\|\,r(\tilde x)-x\,\big\|^2,\qquad r=g\circ f
q(x~∣x)q(\tilde x\mid x)
corruption process, sampled fresh every time an example is seen
r(x~)r(\tilde x)
reconstruction: encoder f then decoder g, applied to the corrupted input
xx
the clean target (cross-entropy replaces squared error for binary data)
Corruptionq(x̃ | x)Typical use
Additive Gaussianx~=x+σε, ε∼N(0,I)\tilde x=x+\sigma\varepsilon,\ \varepsilon\sim\mathcal N(0,I)real-valued inputs; the case with the cleanest theory (below)
Maskingeach coordinate set to 0 with probability ν\nuVincent et al. 2008 on MNIST; the ancestor of masked-token and masked-patch pre-training
Salt-and-peppera fraction ν\nu of coordinates set to the min or max value at randombinary or bounded images
Common corruption processes

The minimizer over all functions is easy to state. For squared error the best prediction of xx given x~\tilde x is the conditional mean, r∗(x~)=E[x∣x~]r^*(\tilde x)=\mathbb{E}[x\mid\tilde x]. The identity is not this function unless the noise is zero, so the network cannot settle for it: for Gaussian noise it pays E∥x~−x∥2=dσ2\mathbb{E}\|\tilde x-x\|^2=d\sigma^2, while r∗r^* pays less.

The vector field view

Think of r(x~)−x~r(\tilde x)-\tilde x as an arrow at every point of input space: the step the DAE takes when handed x~\tilde x. Corrupted points sit off the data manifold, and the clean point they came from is on it. So the arrows learn to point back toward the manifold, roughly perpendicular to it. Along the manifold the network has no way to know which way the noise pushed, so it barely moves.

For Gaussian corruption this picture is exact. With pσ(x~)=∫N(x~;x,σ2I) p(x) dxp_\sigma(\tilde x)=\int\mathcal N(\tilde x;x,\sigma^2I)\,p(x)\,dx, the density of noisy points, Tweedie's formula gives

r∗(x~)−x~=E[x∣x~]−x~=σ2 ∇x~log⁡pσ(x~)r^*(\tilde x)-\tilde x=\mathbb{E}[x\mid\tilde x]-\tilde x=\sigma^2\,\nabla_{\tilde x}\log p_\sigma(\tilde x)
∇log⁡pσ\nabla\log p_\sigma
the score of the noise-smoothed data density: the direction in which density increases fastest
σ2\sigma^2
noise variance; Alain & Bengio (2014) showed r(x)−x≈σ2∇log⁡p(x)r(x)-x\approx\sigma^2\nabla\log p(x) for small σ\sigma

So a trained DAE is a score estimator: divide its displacement by σ2\sigma^2 and you get ∇log⁡pσ\nabla\log p_\sigma. Vincent (2011) turned this around. Training a DAE with Gaussian noise is equivalent to denoising score matching, fitting a model sθ(x~)≈∇log⁡pσ(x~)s_\theta(\tilde x)\approx\nabla\log p_\sigma(\tilde x).

arrows ½(r(x̃) − x̃) · shade p_σ · click = prober*(x̃) = E[x | x̃], exactnoise σ = 0.20error vs clean x (400 draws) corrupted ‖x̃−x‖² 0.0789 denoised ‖r(x̃)−x‖² 0.0385distance to circle (30 shown) x̃ 0.174 r(x̃) 0.020probe x̃ = (1.45, 0.55)r(x̃) = (0.926, 0.342)(r − x̃)/σ² = (−13.10, −5.21)∇log p_σ(x̃) = (−13.10, −5.21)equal: Tweedie's identityafter 6 passes: ‖r⁶(x̃)‖ = 0.980● training points on the circle● x̃ = x + σε ● r(x̃)→ probe, denoised repeatedly
|
fig.40 training points on the unit circle. Default view: the exact optimal denoiser E[x | x̃] for this training set. The probe readout shows (r − x̃)/σ² equal to a finite-difference ∇log p_σ, which is Tweedie's identity. Switch to the trained DAE (a 2-32-2 tanh MLP, 2000 Adam steps on fresh noise) and it learns nearly the same field near the data: error 0.0440 vs the optimum's 0.0385 at σ = 0.2. Far from the circle it never sees training inputs, and its arrows drift away from the optimal ones. Lower σ toward 0.05 and r* snaps onto individual training points; raise it to 0.5 and r* pulls everything inside the circle.

Worked example: two clean points

Data {(1, 0), (−1, 0)}, σ = 0.5, corrupted input x̃ = (0.3, 0.4)

  1. 1Squared distances to the clean points: 0.72+0.42=0.650.7^2+0.4^2=0.65 and 1.32+0.42=1.851.3^2+0.4^2=1.85.
  2. 2Posterior weights ∝e−d2/2σ2\propto e^{-d^2/2\sigma^2}: e−1.3=0.2725e^{-1.3}=0.2725 and e−3.7=0.0247e^{-3.7}=0.0247, which normalize to 0.91680.9168 and 0.08320.0832.
  3. 3Optimal reconstruction r∗(x~)=0.9168 (1,0)+0.0832 (−1,0)=(0.8337, 0)r^*(\tilde x)=0.9168\,(1,0)+0.0832\,(-1,0)=(0.8337,\,0). The vertical noise is removed completely, and the point is pulled most of the way to (1,0)(1,0).
  4. 4Displacement over σ2\sigma^2: (0.8337−0.3,  0−0.4)/0.25=(2.135, −1.600)(0.8337-0.3,\;0-0.4)/0.25=(2.135,\,-1.600). A finite-difference gradient of log⁡pσ\log p_\sigma at x~\tilde x gives (2.135, −1.600)(2.135,\,-1.600) too.
σIdentity error (≈ 2σ²)Optimal r* errorTrained DAE errorDistance: x̃ → r*(x̃)
0.050.00490.00190.00290.039 → 0.001
0.10.01970.00960.01060.079 → 0.005
0.20.07890.03850.04400.158 → 0.020
0.30.17750.08790.10040.238 → 0.048
0.50.49300.25030.26090.389 → 0.151
The figure's circle, 400 fixed corruptions per row. Error = mean ‖output − x‖²; distance = mean |‖·‖ − 1| to the circle.

Even the optimal denoiser only halves the error, because it removes the component of the noise across the circle, worth about σ2\sigma^2, and cannot remove the component along it, which is the other σ2\sigma^2. Its outputs, however, land almost exactly on the manifold. At σ=0.5\sigma=0.5 the circle is badly blurred: the posterior averages over a wide arc, and E[x∣x~]\mathbb{E}[x\mid\tilde x] falls inside the circle (mean radius 0.850.85).

Training a DAE

  1. 1Pick a corruption qq and its level (σ\sigma for Gaussian noise, ν\nu for masking). For images, masking 10–50% of pixels or σ\sigma around 10–30% of the data range are typical starting points.
  2. 2For each mini-batch, sample a fresh corruption x~∼q(⋅∣x)\tilde x\sim q(\cdot\mid x). Never fix the noise once: the model would memorize it.
  3. 3Forward x~\tilde x through encoder and decoder; compute the loss against the clean xx.
  4. 4Backpropagate and update as usual. At test time feed clean inputs: use f(x)f(x) as features, or r(x~)r(\tilde x) to denoise.
  5. 5Stacked DAEs (Vincent et al. 2010): train one DAE, keep its encoder, train the next DAE on its codes, and so on. Then fine-tune the stack with supervision. This was one of the main pre-training recipes for deep nets before ReLU and better initialization made it unnecessary.
python
1import torch, torch.nn as nn
2
3class DAE(nn.Module):
4 def __init__(self, d=784, h=1024): # overcomplete is fine: h > d
5 super().__init__()
6 self.f = nn.Sequential(nn.Linear(d, h), nn.ReLU())
7 self.g = nn.Sequential(nn.Linear(h, d), nn.Sigmoid())
8 def forward(self, x):
9 return self.g(self.f(x))
10
11def corrupt(x, kind="gauss", sigma=0.3, nu=0.3):
12 if kind == "gauss":
13 return x + sigma * torch.randn_like(x)
14 keep = (torch.rand_like(x) > nu).float() # masking noise: zero a fraction nu
15 return x * keep
16
17model = DAE()
18opt = torch.optim.Adam(model.parameters(), lr=1e-3)
19for x, _ in loader: # x in [0, 1], shape (B, 784)
20 x_tilde = corrupt(x) # fresh noise every step
21 loss = nn.functional.binary_cross_entropy(model(x_tilde), x) # target is the CLEAN x
22 opt.zero_grad(); loss.backward(); opt.step()
23
24# score estimate for Gaussian noise: (model(x) - x) / sigma**2 ≈ ∇ log p_σ(x)
A denoising autoencoder with Gaussian or masking corruption

From denoising to diffusion

If one DAE gives the score at one noise level, a network conditioned on σ\sigma gives it at every level. That is the core of score-based generative models (Song & Ermon, 2019) and denoising diffusion models (Ho et al., 2020). Train a single denoiser across a schedule of noise levels. Then generate by starting from pure noise and repeatedly stepping along the estimated score (or the equivalent noise prediction ε^=−σ∇log⁡pσ\hat\varepsilon=-\sigma\nabla\log p_\sigma) while lowering σ\sigma. The probe in the figure, denoised again and again at fixed σ, is a crude version: it slides onto the circle and stays there.

Where it's used

Representation learning

Stacked DAEs pre-trained deep networks layer by layer (2008–2012), with features that transfer better than plain autoencoder features.

Self-supervised pre-training

Masked language modelling (BERT) and masked autoencoders for images (He et al., 2021) are denoising with masking corruption: reconstruct what was hidden.

Generative modelling

Diffusion and score-based models are DAEs trained across noise levels and sampled iteratively.

Signal cleanup

Image, audio and sensor denoising, where the corruption model matches the real noise.

Check yourself

An overcomplete DAE (hidden layer wider than the input) is trained with Gaussian noise. Why does it not learn the identity map?

Check yourself

A DAE trained with Gaussian noise σ=0.1\sigma=0.1 maps an input x~\tilde x to r(x~)=x~+(0.02, −0.01)r(\tilde x)=\tilde x+(0.02,\,-0.01). What is its estimate of ∇log⁡pσ(x~)\nabla\log p_\sigma(\tilde x)?

Remember

  • ✓Train r(x~)→xr(\tilde x)\to x with fresh corruption x~∼q(x~∣x)\tilde x\sim q(\tilde x\mid x) (Gaussian, masking, salt-and-pepper).
  • ✓The optimum is E[x∣x~]\mathbb{E}[x\mid\tilde x], not the identity, so even an overcomplete DAE must learn structure.
  • ✓r(x~)−x~r(\tilde x)-\tilde x points back toward the data manifold; for Gaussian noise it equals σ2∇log⁡pσ(x~)\sigma^2\nabla\log p_\sigma(\tilde x) (Tweedie, Alain & Bengio 2014).
  • ✓DAE training = denoising score matching (Vincent 2011); conditioning on σ and iterating gives diffusion models.
  • ✓Stacked DAEs were an early deep pre-training recipe; masked modelling is their modern descendant.
← back to Deep Learning roadmap