Definition
Denoising autoencoder (DAE) is an autoencoder trained to map a corrupted input back to the clean , minimizing . Because the target differs from the input, copying is no longer a solution, and the network has to learn where the data actually lies.
An ordinary autoencoder with a wide enough hidden layer can learn the identity and reconstruct perfectly while learning nothing (see the Autoencoder article). The usual fixes restrict the code: a bottleneck, or a sparsity penalty. Vincent, Larochelle, Bengio and Manzagol (2008) changed the task instead. Destroy part of the input and ask for the original back. The identity now scores badly, because it reproduces the noise.
To undo noise, the network must know what clean data looks like: which pixel values go together, which directions are "on the data" and which are "off it". That knowledge is exactly the structure we want the representation to capture, and it works even with an overcomplete code.
Training objective and corruptions
- corruption process, sampled fresh every time an example is seen
- reconstruction: encoder f then decoder g, applied to the corrupted input
- the clean target (cross-entropy replaces squared error for binary data)
| Corruption | q(x̃ | x) | Typical use |
|---|---|---|
| Additive Gaussian | real-valued inputs; the case with the cleanest theory (below) | |
| Masking | each coordinate set to 0 with probability | Vincent et al. 2008 on MNIST; the ancestor of masked-token and masked-patch pre-training |
| Salt-and-pepper | a fraction of coordinates set to the min or max value at random | binary or bounded images |
The minimizer over all functions is easy to state. For squared error the best prediction of given is the conditional mean, . The identity is not this function unless the noise is zero, so the network cannot settle for it: for Gaussian noise it pays , while pays less.
The vector field view
Think of as an arrow at every point of input space: the step the DAE takes when handed . Corrupted points sit off the data manifold, and the clean point they came from is on it. So the arrows learn to point back toward the manifold, roughly perpendicular to it. Along the manifold the network has no way to know which way the noise pushed, so it barely moves.
For Gaussian corruption this picture is exact. With , the density of noisy points, Tweedie's formula gives
- the score of the noise-smoothed data density: the direction in which density increases fastest
- noise variance; Alain & Bengio (2014) showed for small
So a trained DAE is a score estimator: divide its displacement by and you get . Vincent (2011) turned this around. Training a DAE with Gaussian noise is equivalent to denoising score matching, fitting a model .
Worked example: two clean points
Data {(1, 0), (−1, 0)}, σ = 0.5, corrupted input x̃ = (0.3, 0.4)
- 1Squared distances to the clean points: and .
- 2Posterior weights : and , which normalize to and .
- 3Optimal reconstruction . The vertical noise is removed completely, and the point is pulled most of the way to .
- 4Displacement over : . A finite-difference gradient of at gives too.
| σ | Identity error (≈ 2σ²) | Optimal r* error | Trained DAE error | Distance: x̃ → r*(x̃) |
|---|---|---|---|---|
| 0.05 | 0.0049 | 0.0019 | 0.0029 | 0.039 → 0.001 |
| 0.1 | 0.0197 | 0.0096 | 0.0106 | 0.079 → 0.005 |
| 0.2 | 0.0789 | 0.0385 | 0.0440 | 0.158 → 0.020 |
| 0.3 | 0.1775 | 0.0879 | 0.1004 | 0.238 → 0.048 |
| 0.5 | 0.4930 | 0.2503 | 0.2609 | 0.389 → 0.151 |
Even the optimal denoiser only halves the error, because it removes the component of the noise across the circle, worth about , and cannot remove the component along it, which is the other . Its outputs, however, land almost exactly on the manifold. At the circle is badly blurred: the posterior averages over a wide arc, and falls inside the circle (mean radius ).
Training a DAE
- 1Pick a corruption and its level ( for Gaussian noise, for masking). For images, masking 10–50% of pixels or around 10–30% of the data range are typical starting points.
- 2For each mini-batch, sample a fresh corruption . Never fix the noise once: the model would memorize it.
- 3Forward through encoder and decoder; compute the loss against the clean .
- 4Backpropagate and update as usual. At test time feed clean inputs: use as features, or to denoise.
- 5Stacked DAEs (Vincent et al. 2010): train one DAE, keep its encoder, train the next DAE on its codes, and so on. Then fine-tune the stack with supervision. This was one of the main pre-training recipes for deep nets before ReLU and better initialization made it unnecessary.
1import torch, torch.nn as nn23class DAE(nn.Module):4 def __init__(self, d=784, h=1024): # overcomplete is fine: h > d5 super().__init__()6 self.f = nn.Sequential(nn.Linear(d, h), nn.ReLU())7 self.g = nn.Sequential(nn.Linear(h, d), nn.Sigmoid())8 def forward(self, x):9 return self.g(self.f(x))1011def corrupt(x, kind="gauss", sigma=0.3, nu=0.3):12 if kind == "gauss":13 return x + sigma * torch.randn_like(x)14 keep = (torch.rand_like(x) > nu).float() # masking noise: zero a fraction nu15 return x * keep1617model = DAE()18opt = torch.optim.Adam(model.parameters(), lr=1e-3)19for x, _ in loader: # x in [0, 1], shape (B, 784)20 x_tilde = corrupt(x) # fresh noise every step21 loss = nn.functional.binary_cross_entropy(model(x_tilde), x) # target is the CLEAN x22 opt.zero_grad(); loss.backward(); opt.step()2324# score estimate for Gaussian noise: (model(x) - x) / sigma**2 ≈ ∇ log p_σ(x)From denoising to diffusion
If one DAE gives the score at one noise level, a network conditioned on gives it at every level. That is the core of score-based generative models (Song & Ermon, 2019) and denoising diffusion models (Ho et al., 2020). Train a single denoiser across a schedule of noise levels. Then generate by starting from pure noise and repeatedly stepping along the estimated score (or the equivalent noise prediction ) while lowering . The probe in the figure, denoised again and again at fixed σ, is a crude version: it slides onto the circle and stays there.
Where it's used
Representation learning
Stacked DAEs pre-trained deep networks layer by layer (2008–2012), with features that transfer better than plain autoencoder features.
Self-supervised pre-training
Masked language modelling (BERT) and masked autoencoders for images (He et al., 2021) are denoising with masking corruption: reconstruct what was hidden.
Generative modelling
Diffusion and score-based models are DAEs trained across noise levels and sampled iteratively.
Signal cleanup
Image, audio and sensor denoising, where the corruption model matches the real noise.
Check yourself
An overcomplete DAE (hidden layer wider than the input) is trained with Gaussian noise. Why does it not learn the identity map?
Check yourself
A DAE trained with Gaussian noise maps an input to . What is its estimate of ?
Remember
- ✓Train with fresh corruption (Gaussian, masking, salt-and-pepper).
- ✓The optimum is , not the identity, so even an overcomplete DAE must learn structure.
- ✓ points back toward the data manifold; for Gaussian noise it equals (Tweedie, Alain & Bengio 2014).
- ✓DAE training = denoising score matching (Vincent 2011); conditioning on σ and iterating gives diffusion models.
- ✓Stacked DAEs were an early deep pre-training recipe; masked modelling is their modern descendant.