# Reconstruction loss

> How far a model's output is from the input it should reproduce. MSE, L1 and BCE are Gaussian, Laplace and Bernoulli log-likelihoods. Their per-pixel optimum is a mean or a median, which is why autoencoders blur.

Source: https://www.mlfoundry.in/learn/reconstruction-loss  
Section: Deep Learning › Loss Functions  
Level: core · 11 min read · updated 2026-10-11

**Reconstruction loss:** is the term $\ell(x,\hat x)$ that scores how well a model's output $\hat x=g(f(x))$ reproduces its own input $x$. It is the whole objective of a plain autoencoder and the first term of a VAE's negative ELBO, $-\mathbb E_{q(z|x)}[\log p_\theta(x\mid z)]$. In practice it is a sum over the $D$ dimensions of $x$ (pixels, audio samples, features) of a per-dimension loss: squared error, absolute error or binary cross-entropy.

An autoencoder has no labels. Its target is the input, so the loss decides what "the same" means. Squared error treats every pixel as a real number with Gaussian noise. Cross-entropy treats it as the probability of a coin flip. Absolute error treats it as a Laplace variable. These are modelling choices with visible consequences: the loss decides what the decoder outputs when it is unsure, and that is where blur comes from.

Autoencoders trained to reproduce their input go back to the auto-association networks of the 1980s (Rumelhart, Hinton & Williams, 1986). Hinton & Salakhutdinov (2006) made deep ones work with layer-wise pre-training. Kingma & Welling (2013) turned the reconstruction error into a log-likelihood inside the VAE, with a Bernoulli decoder for MNIST and a Gaussian decoder for real-valued Frey faces. Mathieu, Couprie & LeCun (2016) showed that next-frame video predictors trained with mean squared error produce blurry frames, and proposed gradient-difference and adversarial terms to fix it.

## The three standard losses as likelihoods

$$
\ell_{\text{MSE}}=\sum_{i=1}^{D}(x_i-\hat x_i)^2\qquad \ell_{\text{L1}}=\sum_{i=1}^{D}|x_i-\hat x_i|\qquad \ell_{\text{BCE}}=-\sum_{i=1}^{D}\big[x_i\ln\hat x_i+(1-x_i)\ln(1-\hat x_i)\big]
$$

- $x_i$: target value of dimension $i$, e.g. a pixel in $[0,1]$
- $\hat x_i$: decoder output for dimension $i$; for BCE it is $\sigma(a_i)$ of a logit $a_i$
- $D$: number of output dimensions, 784 for a 28×28 image

Each is a negative log-likelihood of a decoder distribution that factorizes over dimensions, up to a constant and a scale:

$$
\begin{aligned}-\log\mathcal N(x_i;\hat x_i,\sigma^2)&=\frac{(x_i-\hat x_i)^2}{2\sigma^2}+\tfrac12\ln(2\pi\sigma^2)\\-\log\mathrm{Laplace}(x_i;\hat x_i,b)&=\frac{|x_i-\hat x_i|}{b}+\ln(2b)\\-\log\mathrm{Bern}(x_i;\hat x_i)&=-x_i\ln\hat x_i-(1-x_i)\ln(1-\hat x_i)\end{aligned}
$$

- $\sigma^2$: decoder variance. Fixed, it only rescales MSE by $1/2\sigma^2$, which matters once a KL term is added
- $b$: Laplace scale; plays the same role for L1
- $\mathrm{Bern}$: a coin with $P(x_i=1)=\hat x_i$; exact only for binary data

**At a glance**

- **MSE:** Gaussian decoder. Gradient $2(\hat x_i-x_i)$. Optimal prediction: the conditional **mean**
- **L1:** Laplace decoder. Gradient $\operatorname{sign}(\hat x_i-x_i)$. Optimal prediction: the conditional **median**
- **BCE:** Bernoulli decoder. Gradient w.r.t. the logit $\hat x_i-x_i$. Optimal prediction: the conditional **mean**
- **Minimum value:** 0 for MSE and L1; for BCE with grey targets it is the entropy $H(x_i)>0$, not 0
- **Reduction:** a likelihood **sums** over dimensions; `reduction="mean"` divides by $D$ (and the batch)
- **PyTorch:** `F.mse_loss`, `F.l1_loss`, `F.binary_cross_entropy_with_logits`, `F.gaussian_nll_loss`

## Why reconstructions are blurry

A decoder sees a code $z$, not the image. If several images are consistent with the same code, the decoder must commit to one output, and the loss decides which. Minimize the expected per-pixel loss over the possible targets:

$$
\hat x_i^{*}=\arg\min_{c}\ \mathbb E\big[\ell(x_i,c)\mid z\big]=\begin{cases}\mathbb E[x_i\mid z] & \text{MSE, BCE}\\ \operatorname{median}(x_i\mid z) & \text{L1}\end{cases}
$$

- $\mathbb E[x_i\mid z]$: for MSE set $\partial/\partial c\,\mathbb E[(x_i-c)^2]=2(c-\mathbb E x_i)=0$; for BCE the logit gradient $c-\mathbb E x_i$ vanishes at the same point
- $\operatorname{median}$: the L1 subgradient is $P(x_i<c)-P(x_i>c)$, which crosses 0 at the median

An average of several sharp images is a soft image. An edge whose position the code does not pin down becomes a ramp, and fine texture whose phase is unknown averages to flat grey. The decoder is not failing to learn: the blurry output is the exact minimizer of the loss it was given. The median is sharper in this example, because each pixel just takes the majority value, but it is not a cure. For many plausible outcomes it still averages within each pixel, and different pixels can take their medians from different images.

> Figure: A 32-pixel step edge whose position the decoder cannot know: 7 positions (pixels 13 to 19) are equally likely. Under MSE or BCE the best single reconstruction is the per-pixel mean, a 6-pixel grey ramp. Its expected squared error, summed over pixels, is 1.143, while a sharp edge at the middle scores 1.714 (1.50×). Under BCE a sharp guess clipped to [0.01, 0.99] costs 2.42× the ramp. Under L1 the optimum is the per-pixel median, a sharp edge at pixel 16, and both score 1.714. The lower panel probes one pixel: at pixel 17, q = P(y = 1) = 0.714. At p = 0.5 the MSE gradient is −0.429 and the BCE logit gradient is −0.214. Widen the uncertainty and the ramp widens with it (10 grey pixels at ±5).

## Gradients

Per dimension, with the decoder's last layer producing $\hat x_i$ directly (MSE, L1) or through a sigmoid $\hat x_i=\sigma(a_i)$ (BCE):

$$
\frac{\partial\ell_{\text{MSE}}}{\partial\hat x_i}=2(\hat x_i-x_i)\qquad \frac{\partial\ell_{\text{L1}}}{\partial\hat x_i}=\operatorname{sign}(\hat x_i-x_i)\qquad \frac{\partial\ell_{\text{BCE}}}{\partial a_i}=\sigma(a_i)-x_i
$$

- $a_i$: pre-sigmoid logit of dimension $i$
- $\sigma(a_i)-x_i$: the sigmoid's derivative cancels the 1/p of the log: a bounded, non-saturating error signal

If you instead put a sigmoid on the decoder and train it with MSE, the logit gradient is $2(\hat x_i-x_i)\,\hat x_i(1-\hat x_i)$. It vanishes when a pixel is confidently wrong, the same failure the **Loss functions** article describes for classifiers. That is one practical reason BCE is popular for images in $[0,1]$ even when they are not binary.

## Worked example: one 4-pixel image

Target $x=(0,\,0.2,\,0.9,\,1)$, decoder output $\hat x=(0.1,\,0.3,\,0.6,\,0.8)$. All numbers below were computed in Node.

Per-pixel losses and gradients (sum reduction)

| pixel | x | x̂ | (x − x̂)² | \|x − x̂\| | BCE | BCE floor H(x) | ∂BCE/∂a = x̂ − x |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 0 | 0.1 | 0.0100 | 0.1000 | 0.1054 | 0 | 0.100 |
| 2 | 0.2 | 0.3 | 0.0100 | 0.1000 | 0.5261 | 0.5004 | 0.100 |
| 3 | 0.9 | 0.6 | 0.0900 | 0.3000 | 0.5514 | 0.3251 | −0.300 |
| 4 | 1 | 0.8 | 0.0400 | 0.2000 | 0.2231 | 0 | −0.200 |
| **sum** |  |  | **0.1500** | **0.7000** | **1.4060** | **0.8255** |  |

1. MSE: sum $0.15$, mean $0.0375$. L1: sum $0.70$, mean $0.175$. BCE: sum $1.406$ nats.
2. BCE with grey targets cannot reach 0. Its minimum is the entropy of the targets, $0.8255$ nats here, so only $1.406-0.8255=0.5805$ nats (the KL from target to prediction) is actually reducible. Pixel 3 has the worst error ($|r|=0.3$) and also carries the largest reducible part.
3. Gaussian NLL with $\sigma=0.1$: $0.15/(2\cdot0.01)+4\cdot\tfrac12\ln(2\pi\cdot0.01)=7.5-5.5346=1.9654$ nats. The squared error is multiplied by $50$. That factor is what sets the balance against a VAE's KL term.
4. Laplace NLL with $b=0.1$: $0.7/0.1+4\ln0.2=7-6.4378=0.5622$ nats.

## Sum, mean and the KL term

A likelihood is a product over dimensions, so its log is a **sum**. Library defaults are `reduction="mean"`, which divides by the number of elements. For a plain autoencoder this only rescales the learning rate. In a VAE it changes the model, because the reconstruction term is weighed against $\mathrm{KL}(q(z|x)\,\|\,p(z))$, which is summed over the latent dimensions.

Take MNIST with $D=784$. A per-pixel BCE of 0.1 nats is 78.4 nats per image when summed, but only 0.1 when averaged. With a KL of about 20 nats, the mean-reduced objective lets the KL dominate by a factor of 784. That is a β-VAE with $\beta=784$, and it usually ends in **posterior collapse**: the code carries nothing and the decoder outputs the average digit. The same arithmetic explains the Gaussian case: MSE with a fixed $\sigma$ is the true NLL only after multiplying by $1/2\sigma^2$, so "MSE + KL" silently picks $\sigma^2=\tfrac12$, a very noisy decoder, on $[0,1]$ data.

> **Mean reduction inside a VAE:** `F.mse_loss(x_hat, x)` + KL is not the ELBO. Sum the reconstruction over pixels and average over the batch, or equivalently scale the KL by $1/D$ knowingly as a β choice. If samples look like the average image and the KL is near 0, check this first.

```python
import torch, torch.nn.functional as F

x  = torch.tensor([0.0, 0.2, 0.9, 1.0])        # target pixels
xh = torch.tensor([0.1, 0.3, 0.6, 0.8])        # decoder output after the sigmoid
a  = torch.logit(xh)                           # what the decoder should really return

F.mse_loss(xh, x)                              # 0.0375  (mean over 4 pixels)
F.mse_loss(xh, x, reduction="sum")             # 0.1500
F.l1_loss(xh, x, reduction="sum")              # 0.7000
F.binary_cross_entropy_with_logits(a, x, reduction="sum")   # 1.4060, stable: takes logits
F.gaussian_nll_loss(xh, x, var=torch.full_like(x, 0.01),
                    full=True, reduction="sum")             # 1.9654 = 7.5 - 5.5346

def vae_loss(logits, x, mu, logvar, beta=1.0):
    """Negative ELBO per example: sum over pixels and latents, mean over the batch."""
    B = x.shape[0]
    rec = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / B
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / B
    return rec + beta * kl
```

The worked example in PyTorch, and a VAE loss with consistent reductions

## Choosing the loss

For **binary or nearly binary** data (binarized MNIST, masks), BCE is the correct likelihood. For **real-valued images** the Gaussian is the textbook choice, but its variance matters. Rybkin, Daniilidis & Levine (2021) showed that learning a single shared $\sigma$ (the "σ-VAE"), whose optimum is just the mean squared error, balances the two VAE terms automatically and removes the need to hand-tune β. BCE on grey-level pixels is common and works, but it is not a normalized density on $[0,1]$. Loaiza-Ganem & Cunningham (2019) derived the normalizing constant and called the result the **continuous Bernoulli**. For outlier-heavy targets, L1 or Huber is more robust.

None of these fixes blur, because all of them are per-pixel and assume the pixels are independent given $z$. Sharper results need losses that compare structure rather than pixels: **perceptual losses** on network features, **adversarial losses** from a discriminator, or a model that represents the whole distribution of outcomes rather than a single guess, such as a diffusion model or an autoregressive decoder.

> **Blur is uncertainty made visible:** A blurry reconstruction is a picture of $\mathbb E[x\mid z]$. Its grey regions are exactly where the code is uninformative. That makes the reconstruction error useful as a signal: autoencoder-based anomaly detection flags inputs whose reconstruction error is large, because the code cannot pin them down.

> **Denoising and masked variants:** The loss is often computed against a target that differs from the input. Denoising autoencoders reconstruct the clean $x$ from a corrupted $\tilde x$ (Vincent et al., 2008). Masked autoencoders (He et al., 2022) compute MSE only on the masked image patches. The likelihood view is unchanged: the loss is the NLL of the target under the decoder's distribution.

**Strengths**

- Simple, cheap and exactly differentiable; needs no labels
- Each choice is a likelihood, so the value has a meaning (nats) and combines cleanly with a KL term
- BCE on logits gives a bounded, non-saturating gradient for outputs in [0, 1]
- L1 is robust to outliers and gives sharper (median) outputs than MSE

**Limitations**

- Per-pixel and independent: the optimum is a mean (or median), so uncertain detail becomes blur
- Pixel distance is a poor proxy for perceptual similarity: a 1-pixel shift can cost more than a visible blur
- The scale (sum vs mean, the $\sigma$ or $b$ of the likelihood) silently sets the VAE's β
- BCE on non-binary data has a nonzero floor and is not a normalized density

**Question:** An autoencoder trained with MSE outputs a grey ramp where the training images have sharp edges at slightly different positions. What is the best explanation?

- A. The decoder lacks capacity
- B. MSE is minimized by the conditional mean, and the mean of shifted sharp edges is a ramp
- C. The learning rate is too high
- D. MSE has vanishing gradients near the optimum

**Answer:** B. MSE is minimized by the conditional mean, and the mean of shifted sharp edges is a ramp — When the code does not determine the edge position, the expected squared error is minimized by $\mathbb E[x_i\mid z]$, the average over the possible positions. That average is a ramp. In the figure, the ramp scores 1.143 against 1.714 for any single sharp edge, so a more powerful decoder would make the same choice.

**Question:** A VAE on 28×28 images uses `F.mse_loss(x_hat, x)` (mean reduction) plus the summed KL. Samples all look like the same average image. What is the most likely cause?

- A. The KL term is too small
- B. The reconstruction term was divided by 784, so the KL dominates and the posterior collapses
- C. MSE cannot be used in a VAE
- D. The reparameterization trick is missing

**Answer:** B. The reconstruction term was divided by 784, so the KL dominates and the posterior collapses — The ELBO's reconstruction term sums over the $D=784$ pixels. Averaging divides it by 784, which is the same as multiplying the KL weight by 784. The cheapest way to cut the loss is then to set $q(z|x)=p(z)$, so the code carries no information.

**Key points**

- Reconstruction loss = NLL of the input under the decoder: Gaussian → MSE, Laplace → L1, Bernoulli → BCE.
- Per-pixel losses are minimized by the conditional mean (MSE, BCE) or median (L1), so uncertain detail is averaged into blur.
- Use BCE on logits (`binary_cross_entropy_with_logits`): gradient $\hat x-x$, numerically stable.
- A likelihood sums over dimensions. Mean reduction or a hidden $\sigma$ changes a VAE's effective β.
- BCE on grey targets has a floor equal to the target entropy; only the KL part is reducible.
- Sharper outputs need structure-aware losses (perceptual, adversarial) or a distributional decoder (diffusion, autoregressive).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
