# Denoising autoencoder

> Corrupt the input, reconstruct the clean version: the network must learn which way the data manifold lies, and its output points along the score ∇log p(x).

Source: https://www.mlfoundry.in/learn/denoising-autoencoder  
Section: Deep Learning › Neural Network Algorithms  
Level: core · 10 min read · updated 2026-10-11

**Denoising autoencoder (DAE):** is an autoencoder trained to map a **corrupted** input $\tilde x\sim q(\tilde x\mid x)$ back to the **clean** $x$, minimizing $\mathbb{E}\,\|g(f(\tilde x))-x\|^2$. Because the target differs from the input, copying is no longer a solution, and the network has to learn where the data actually lies.

An ordinary autoencoder with a wide enough hidden layer can learn the identity and reconstruct perfectly while learning nothing (see the Autoencoder article). The usual fixes restrict the code: a bottleneck, or a sparsity penalty. Vincent, Larochelle, Bengio and Manzagol (2008) changed the **task** instead. Destroy part of the input and ask for the original back. The identity now scores badly, because it reproduces the noise.

To undo noise, the network must know what clean data looks like: which pixel values go together, which directions are "on the data" and which are "off it". That knowledge is exactly the structure we want the representation to capture, and it works even with an overcomplete code.

## Training objective and corruptions

$$
\mathcal{L}_{\text{DAE}}=\mathbb{E}_{x\sim p_{\text{data}}}\;\mathbb{E}_{\tilde x\sim q(\tilde x\mid x)}\;\big\|\,r(\tilde x)-x\,\big\|^2,\qquad r=g\circ f
$$

- $q(\tilde x\mid x)$: corruption process, sampled fresh every time an example is seen
- $r(\tilde x)$: reconstruction: encoder f then decoder g, applied to the corrupted input
- $x$: the clean target (cross-entropy replaces squared error for binary data)

Common corruption processes

| Corruption | q(x̃ \| x) | Typical use |
| --- | --- | --- |
| Additive Gaussian | $\tilde x=x+\sigma\varepsilon,\ \varepsilon\sim\mathcal N(0,I)$ | real-valued inputs; the case with the cleanest theory (below) |
| Masking | each coordinate set to 0 with probability $\nu$ | Vincent et al. 2008 on MNIST; the ancestor of masked-token and masked-patch pre-training |
| Salt-and-pepper | a fraction $\nu$ of coordinates set to the min or max value at random | binary or bounded images |

The minimizer over all functions is easy to state. For squared error the best prediction of $x$ given $\tilde x$ is the conditional mean, $r^*(\tilde x)=\mathbb{E}[x\mid\tilde x]$. The identity is not this function unless the noise is zero, so the network cannot settle for it: for Gaussian noise it pays $\mathbb{E}\|\tilde x-x\|^2=d\sigma^2$, while $r^*$ pays less.

## The vector field view

Think of $r(\tilde x)-\tilde x$ as an arrow at every point of input space: the step the DAE takes when handed $\tilde x$. Corrupted points sit **off** the data manifold, and the clean point they came from is on it. So the arrows learn to point back toward the manifold, roughly perpendicular to it. Along the manifold the network has no way to know which way the noise pushed, so it barely moves.

For Gaussian corruption this picture is exact. With $p_\sigma(\tilde x)=\int\mathcal N(\tilde x;x,\sigma^2I)\,p(x)\,dx$, the density of noisy points, Tweedie's formula gives

$$
r^*(\tilde x)-\tilde x=\mathbb{E}[x\mid\tilde x]-\tilde x=\sigma^2\,\nabla_{\tilde x}\log p_\sigma(\tilde x)
$$

- $\nabla\log p_\sigma$: the **score** of the noise-smoothed data density: the direction in which density increases fastest
- $\sigma^2$: noise variance; Alain & Bengio (2014) showed $r(x)-x\approx\sigma^2\nabla\log p(x)$ for small $\sigma$

So a trained DAE is a **score estimator**: divide its displacement by $\sigma^2$ and you get $\nabla\log p_\sigma$. Vincent (2011) turned this around. Training a DAE with Gaussian noise is equivalent to **denoising score matching**, fitting a model $s_\theta(\tilde x)\approx\nabla\log p_\sigma(\tilde x)$.

> Figure: 40 training points on the unit circle. Default view: the exact optimal denoiser E[x | x̃] for this training set. The probe readout shows (r − x̃)/σ² equal to a finite-difference ∇log p_σ, which is Tweedie's identity. Switch to the trained DAE (a 2-32-2 tanh MLP, 2000 Adam steps on fresh noise) and it learns nearly the same field near the data: error 0.0440 vs the optimum's 0.0385 at σ = 0.2. Far from the circle it never sees training inputs, and its arrows drift away from the optimal ones. Lower σ toward 0.05 and r* snaps onto individual training points; raise it to 0.5 and r* pulls everything inside the circle.

## Worked example: two clean points

**Data {(1, 0), (−1, 0)}, σ = 0.5, corrupted input x̃ = (0.3, 0.4)**

1. Squared distances to the clean points: $0.7^2+0.4^2=0.65$ and $1.3^2+0.4^2=1.85$.
2. Posterior weights $\propto e^{-d^2/2\sigma^2}$: $e^{-1.3}=0.2725$ and $e^{-3.7}=0.0247$, which normalize to $0.9168$ and $0.0832$.
3. Optimal reconstruction $r^*(\tilde x)=0.9168\,(1,0)+0.0832\,(-1,0)=(0.8337,\,0)$. The vertical noise is removed completely, and the point is pulled most of the way to $(1,0)$.
4. Displacement over $\sigma^2$: $(0.8337-0.3,\;0-0.4)/0.25=(2.135,\,-1.600)$. A finite-difference gradient of $\log p_\sigma$ at $\tilde x$ gives $(2.135,\,-1.600)$ too.

The figure's circle, 400 fixed corruptions per row. Error = mean ‖output − x‖²; distance = mean |‖·‖ − 1| to the circle.

| σ | Identity error (≈ 2σ²) | Optimal r* error | Trained DAE error | Distance: x̃ → r*(x̃) |
| --- | --- | --- | --- | --- |
| 0.05 | 0.0049 | 0.0019 | 0.0029 | 0.039 → 0.001 |
| 0.1 | 0.0197 | 0.0096 | 0.0106 | 0.079 → 0.005 |
| 0.2 | 0.0789 | 0.0385 | 0.0440 | 0.158 → 0.020 |
| 0.3 | 0.1775 | 0.0879 | 0.1004 | 0.238 → 0.048 |
| 0.5 | 0.4930 | 0.2503 | 0.2609 | 0.389 → 0.151 |

Even the optimal denoiser only halves the error, because it removes the component of the noise **across** the circle, worth about $\sigma^2$, and cannot remove the component **along** it, which is the other $\sigma^2$. Its outputs, however, land almost exactly on the manifold. At $\sigma=0.5$ the circle is badly blurred: the posterior averages over a wide arc, and $\mathbb{E}[x\mid\tilde x]$ falls inside the circle (mean radius $0.85$).

**Training a DAE**

1. Pick a corruption $q$ and its level ($\sigma$ for Gaussian noise, $\nu$ for masking). For images, masking 10–50% of pixels or $\sigma$ around 10–30% of the data range are typical starting points.
2. For each mini-batch, sample a **fresh** corruption $\tilde x\sim q(\cdot\mid x)$. Never fix the noise once: the model would memorize it.
3. Forward $\tilde x$ through encoder and decoder; compute the loss against the clean $x$.
4. Backpropagate and update as usual. At test time feed clean inputs: use $f(x)$ as features, or $r(\tilde x)$ to denoise.
5. **Stacked DAEs** (Vincent et al. 2010): train one DAE, keep its encoder, train the next DAE on its codes, and so on. Then fine-tune the stack with supervision. This was one of the main pre-training recipes for deep nets before ReLU and better initialization made it unnecessary.

```python
import torch, torch.nn as nn

class DAE(nn.Module):
    def __init__(self, d=784, h=1024):              # overcomplete is fine: h > d
        super().__init__()
        self.f = nn.Sequential(nn.Linear(d, h), nn.ReLU())
        self.g = nn.Sequential(nn.Linear(h, d), nn.Sigmoid())
    def forward(self, x):
        return self.g(self.f(x))

def corrupt(x, kind="gauss", sigma=0.3, nu=0.3):
    if kind == "gauss":
        return x + sigma * torch.randn_like(x)
    keep = (torch.rand_like(x) > nu).float()      # masking noise: zero a fraction nu
    return x * keep

model = DAE()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for x, _ in loader:                                 # x in [0, 1], shape (B, 784)
    x_tilde = corrupt(x)                            # fresh noise every step
    loss = nn.functional.binary_cross_entropy(model(x_tilde), x)   # target is the CLEAN x
    opt.zero_grad(); loss.backward(); opt.step()

# score estimate for Gaussian noise: (model(x) - x) / sigma**2  ≈  ∇ log p_σ(x)
```

A denoising autoencoder with Gaussian or masking corruption

## From denoising to diffusion

If one DAE gives the score at one noise level, a network conditioned on $\sigma$ gives it at every level. That is the core of score-based generative models (Song & Ermon, 2019) and denoising diffusion models (Ho et al., 2020). Train a single denoiser across a schedule of noise levels. Then generate by starting from pure noise and repeatedly stepping along the estimated score (or the equivalent noise prediction $\hat\varepsilon=-\sigma\nabla\log p_\sigma$) while lowering $\sigma$. The probe in the figure, denoised again and again at fixed σ, is a crude version: it slides onto the circle and stays there.

> **The optimal denoiser memorizes:** With a finite training set, $\mathbb{E}[x\mid\tilde x]$ is a softmax-weighted average of the **training points**. When $\sigma$ is small compared with the gaps between them, it snaps each input onto its nearest training example (try σ = 0.05 in the figure). A perfect denoiser would therefore reproduce the training set rather than the underlying manifold. Real networks generalize because they cannot represent that function exactly, and this is one lens on memorization in diffusion models.

> **The field is only learned where noise reaches:** A DAE only sees inputs within a few $\sigma$ of the data. Far from it the field is extrapolation, and in the figure the trained DAE's corner arrows disagree with the optimal ones. Small $\sigma$ gives an accurate score near the manifold and a useless one elsewhere; large $\sigma$ covers more space but blurs the structure. Diffusion models sidestep the trade-off by training at many noise levels.

- **Representation learning:** Stacked DAEs pre-trained deep networks layer by layer (2008–2012), with features that transfer better than plain autoencoder features.
- **Self-supervised pre-training:** Masked language modelling (BERT) and masked autoencoders for images (He et al., 2021) are denoising with masking corruption: reconstruct what was hidden.
- **Generative modelling:** Diffusion and score-based models are DAEs trained across noise levels and sampled iteratively.
- **Signal cleanup:** Image, audio and sensor denoising, where the corruption model matches the real noise.

**Question:** An overcomplete DAE (hidden layer wider than the input) is trained with Gaussian noise. Why does it not learn the identity map?

- A. The identity maps x̃ to x̃, which keeps all the noise; E[x | x̃] has strictly lower expected error
- B. Overcomplete networks cannot represent the identity
- C. The noise makes the gradients zero for the identity
- D. Weight decay is required to prevent it

**Answer:** A. The identity maps x̃ to x̃, which keeps all the noise; E[x | x̃] has strictly lower expected error — The loss compares the output with the clean $x$. The identity pays $\mathbb{E}\|\tilde x-x\|^2=d\sigma^2$ (0.0789 at σ = 0.2 on the circle), while the posterior mean pays 0.0385. Gradient descent moves toward the conditional mean whatever the width.

**Question:** A DAE trained with Gaussian noise $\sigma=0.1$ maps an input $\tilde x$ to $r(\tilde x)=\tilde x+(0.02,\,-0.01)$. What is its estimate of $\nabla\log p_\sigma(\tilde x)$?

- A. (0.02, −0.01)
- B. (0.2, −0.1)
- C. (2, −1)
- D. (0.0002, −0.0001)

**Answer:** C. (2, −1) — $\nabla\log p_\sigma\approx(r(\tilde x)-\tilde x)/\sigma^2=(0.02,-0.01)/0.01=(2,-1)$. The score points toward higher density, here mostly in $+x_1$.

**Key points**

- Train $r(\tilde x)\to x$ with fresh corruption $\tilde x\sim q(\tilde x\mid x)$ (Gaussian, masking, salt-and-pepper).
- The optimum is $\mathbb{E}[x\mid\tilde x]$, not the identity, so even an overcomplete DAE must learn structure.
- $r(\tilde x)-\tilde x$ points back toward the data manifold; for Gaussian noise it equals $\sigma^2\nabla\log p_\sigma(\tilde x)$ (Tweedie, Alain & Bengio 2014).
- DAE training = denoising score matching (Vincent 2011); conditioning on σ and iterating gives diffusion models.
- Stacked DAEs were an early deep pre-training recipe; masked modelling is their modern descendant.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
