Reconstruction loss

How far a model's output is from the input it should reproduce. MSE, L1 and BCE are Gaussian, Laplace and Bernoulli log-likelihoods. Their per-pixel optimum is a mean or a median, which is why autoencoders blur.

11 min read1131 wordscore0 viewsupdated 2026-10-11

Definition

Reconstruction loss is the term ℓ(x,x^)\ell(x,\hat x) that scores how well a model's output x^=g(f(x))\hat x=g(f(x)) reproduces its own input xx. It is the whole objective of a plain autoencoder and the first term of a VAE's negative ELBO, −Eq(z∣x)[log⁡pθ(x∣z)]-\mathbb E_{q(z|x)}[\log p_\theta(x\mid z)]. In practice it is a sum over the DD dimensions of xx (pixels, audio samples, features) of a per-dimension loss: squared error, absolute error or binary cross-entropy.

An autoencoder has no labels. Its target is the input, so the loss decides what "the same" means. Squared error treats every pixel as a real number with Gaussian noise. Cross-entropy treats it as the probability of a coin flip. Absolute error treats it as a Laplace variable. These are modelling choices with visible consequences: the loss decides what the decoder outputs when it is unsure, and that is where blur comes from.

Autoencoders trained to reproduce their input go back to the auto-association networks of the 1980s (Rumelhart, Hinton & Williams, 1986). Hinton & Salakhutdinov (2006) made deep ones work with layer-wise pre-training. Kingma & Welling (2013) turned the reconstruction error into a log-likelihood inside the VAE, with a Bernoulli decoder for MNIST and a Gaussian decoder for real-valued Frey faces. Mathieu, Couprie & LeCun (2016) showed that next-frame video predictors trained with mean squared error produce blurry frames, and proposed gradient-difference and adversarial terms to fix it.

The three standard losses as likelihoods

ℓMSE=∑i=1D(xi−x^i)2ℓL1=∑i=1D∣xi−x^i∣ℓBCE=−∑i=1D[xiln⁡x^i+(1−xi)ln⁡(1−x^i)]\ell_{\text{MSE}}=\sum_{i=1}^{D}(x_i-\hat x_i)^2\qquad \ell_{\text{L1}}=\sum_{i=1}^{D}|x_i-\hat x_i|\qquad \ell_{\text{BCE}}=-\sum_{i=1}^{D}\big[x_i\ln\hat x_i+(1-x_i)\ln(1-\hat x_i)\big]
xix_i
target value of dimension ii, e.g. a pixel in [0,1][0,1]
x^i\hat x_i
decoder output for dimension ii; for BCE it is σ(ai)\sigma(a_i) of a logit aia_i
DD
number of output dimensions, 784 for a 28×28 image

Each is a negative log-likelihood of a decoder distribution that factorizes over dimensions, up to a constant and a scale:

−log⁡N(xi;x^i,σ2)=(xi−x^i)22σ2+12ln⁡(2πσ2)−log⁡Laplace(xi;x^i,b)=∣xi−x^i∣b+ln⁡(2b)−log⁡Bern(xi;x^i)=−xiln⁡x^i−(1−xi)ln⁡(1−x^i)\begin{aligned}-\log\mathcal N(x_i;\hat x_i,\sigma^2)&=\frac{(x_i-\hat x_i)^2}{2\sigma^2}+\tfrac12\ln(2\pi\sigma^2)\\-\log\mathrm{Laplace}(x_i;\hat x_i,b)&=\frac{|x_i-\hat x_i|}{b}+\ln(2b)\\-\log\mathrm{Bern}(x_i;\hat x_i)&=-x_i\ln\hat x_i-(1-x_i)\ln(1-\hat x_i)\end{aligned}
σ2\sigma^2
decoder variance. Fixed, it only rescales MSE by 1/2σ21/2\sigma^2, which matters once a KL term is added
bb
Laplace scale; plays the same role for L1
Bern\mathrm{Bern}
a coin with P(xi=1)=x^iP(x_i=1)=\hat x_i; exact only for binary data

At a glance

MSE
Gaussian decoder. Gradient 2(x^i−xi)2(\hat x_i-x_i). Optimal prediction: the conditional mean
L1
Laplace decoder. Gradient sign⁡(x^i−xi)\operatorname{sign}(\hat x_i-x_i). Optimal prediction: the conditional median
BCE
Bernoulli decoder. Gradient w.r.t. the logit x^i−xi\hat x_i-x_i. Optimal prediction: the conditional mean
Minimum value
0 for MSE and L1; for BCE with grey targets it is the entropy H(xi)>0H(x_i)>0, not 0
Reduction
a likelihood sums over dimensions; reduction="mean" divides by DD (and the batch)
PyTorch
F.mse_loss, F.l1_loss, F.binary_cross_entropy_with_logits, F.gaussian_nll_loss

Why reconstructions are blurry

A decoder sees a code zz, not the image. If several images are consistent with the same code, the decoder must commit to one output, and the loss decides which. Minimize the expected per-pixel loss over the possible targets:

x^i∗=arg⁡min⁡c E[ℓ(xi,c)∣z]={E[xi∣z]MSE, BCEmedian⁡(xi∣z)L1\hat x_i^{*}=\arg\min_{c}\ \mathbb E\big[\ell(x_i,c)\mid z\big]=\begin{cases}\mathbb E[x_i\mid z] & \text{MSE, BCE}\\ \operatorname{median}(x_i\mid z) & \text{L1}\end{cases}
E[xi∣z]\mathbb E[x_i\mid z]
for MSE set ∂/∂c E[(xi−c)2]=2(c−Exi)=0\partial/\partial c\,\mathbb E[(x_i-c)^2]=2(c-\mathbb E x_i)=0; for BCE the logit gradient c−Exic-\mathbb E x_i vanishes at the same point
median⁡\operatorname{median}
the L1 subgradient is P(xi<c)−P(xi>c)P(x_i<c)-P(x_i>c), which crosses 0 at the median

An average of several sharp images is a soft image. An edge whose position the code does not pin down becomes a ramp, and fine texture whose phase is unknown averages to flat grey. The decoder is not failing to learn: the blurry output is the exact minimizer of the loss it was given. The median is sharper in this example, because each pixel just takes the majority value, but it is not a cure. For many plausible outcomes it still averages within each pixel, and different pixels can take their medians from different images.

7 equally likely targets (edge at px 13…19) · best guess under MSE00.51besttrue━ best guess (6 grey px) ┄ sharp guess ━ targets · click a pixelpixel 17: q = P(y=1) = 0.714, E[ℓ] vs p00.511.00mean qprediction p (○ = minimizer)at p = 0.50:E[ℓ] = 0.250∂E/∂p = −0.429 · ∂E/∂logit = −0.107minimizer p* = 0.714 (mean)expected loss per image (sum of 32 px)loss · best · sharp guess · sharp/bestmse · 1.143 · 1.714 · 1.50×l1 · 1.714 · 1.714 · 1.00×bce · 3.383 · 8.199 · 2.42×the mean wins: blur is the optimal answer
fig.A 32-pixel step edge whose position the decoder cannot know: 7 positions (pixels 13 to 19) are equally likely. Under MSE or BCE the best single reconstruction is the per-pixel mean, a 6-pixel grey ramp. Its expected squared error, summed over pixels, is 1.143, while a sharp edge at the middle scores 1.714 (1.50×). Under BCE a sharp guess clipped to [0.01, 0.99] costs 2.42× the ramp. Under L1 the optimum is the per-pixel median, a sharp edge at pixel 16, and both score 1.714. The lower panel probes one pixel: at pixel 17, q = P(y = 1) = 0.714. At p = 0.5 the MSE gradient is −0.429 and the BCE logit gradient is −0.214. Widen the uncertainty and the ramp widens with it (10 grey pixels at ±5).

Gradients

Per dimension, with the decoder's last layer producing x^i\hat x_i directly (MSE, L1) or through a sigmoid x^i=σ(ai)\hat x_i=\sigma(a_i) (BCE):

∂ℓMSE∂x^i=2(x^i−xi)∂ℓL1∂x^i=sign⁡(x^i−xi)∂ℓBCE∂ai=σ(ai)−xi\frac{\partial\ell_{\text{MSE}}}{\partial\hat x_i}=2(\hat x_i-x_i)\qquad \frac{\partial\ell_{\text{L1}}}{\partial\hat x_i}=\operatorname{sign}(\hat x_i-x_i)\qquad \frac{\partial\ell_{\text{BCE}}}{\partial a_i}=\sigma(a_i)-x_i
aia_i
pre-sigmoid logit of dimension ii
σ(ai)−xi\sigma(a_i)-x_i
the sigmoid's derivative cancels the 1/p of the log: a bounded, non-saturating error signal

If you instead put a sigmoid on the decoder and train it with MSE, the logit gradient is 2(x^i−xi) x^i(1−x^i)2(\hat x_i-x_i)\,\hat x_i(1-\hat x_i). It vanishes when a pixel is confidently wrong, the same failure the Loss functions article describes for classifiers. That is one practical reason BCE is popular for images in [0,1][0,1] even when they are not binary.

Worked example: one 4-pixel image

Target x=(0, 0.2, 0.9, 1)x=(0,\,0.2,\,0.9,\,1), decoder output x^=(0.1, 0.3, 0.6, 0.8)\hat x=(0.1,\,0.3,\,0.6,\,0.8). All numbers below were computed in Node.

pixelxx̂(x − x̂)²|x − x̂|BCEBCE floor H(x)∂BCE/∂a = x̂ − x
100.10.01000.10000.105400.100
20.20.30.01000.10000.52610.50040.100
30.90.60.09000.30000.55140.3251−0.300
410.80.04000.20000.22310−0.200
sum0.15000.70001.40600.8255
Per-pixel losses and gradients (sum reduction)
  1. 1MSE: sum 0.150.15, mean 0.03750.0375. L1: sum 0.700.70, mean 0.1750.175. BCE: sum 1.4061.406 nats.
  2. 2BCE with grey targets cannot reach 0. Its minimum is the entropy of the targets, 0.82550.8255 nats here, so only 1.406−0.8255=0.58051.406-0.8255=0.5805 nats (the KL from target to prediction) is actually reducible. Pixel 3 has the worst error (∣r∣=0.3|r|=0.3) and also carries the largest reducible part.
  3. 3Gaussian NLL with σ=0.1\sigma=0.1: 0.15/(2⋅0.01)+4⋅12ln⁡(2π⋅0.01)=7.5−5.5346=1.96540.15/(2\cdot0.01)+4\cdot\tfrac12\ln(2\pi\cdot0.01)=7.5-5.5346=1.9654 nats. The squared error is multiplied by 5050. That factor is what sets the balance against a VAE's KL term.
  4. 4Laplace NLL with b=0.1b=0.1: 0.7/0.1+4ln⁡0.2=7−6.4378=0.56220.7/0.1+4\ln0.2=7-6.4378=0.5622 nats.

Sum, mean and the KL term

A likelihood is a product over dimensions, so its log is a sum. Library defaults are reduction="mean", which divides by the number of elements. For a plain autoencoder this only rescales the learning rate. In a VAE it changes the model, because the reconstruction term is weighed against KL(q(z∣x) ∥ p(z))\mathrm{KL}(q(z|x)\,\|\,p(z)), which is summed over the latent dimensions.

Take MNIST with D=784D=784. A per-pixel BCE of 0.1 nats is 78.4 nats per image when summed, but only 0.1 when averaged. With a KL of about 20 nats, the mean-reduced objective lets the KL dominate by a factor of 784. That is a β-VAE with β=784\beta=784, and it usually ends in posterior collapse: the code carries nothing and the decoder outputs the average digit. The same arithmetic explains the Gaussian case: MSE with a fixed σ\sigma is the true NLL only after multiplying by 1/2σ21/2\sigma^2, so "MSE + KL" silently picks σ2=12\sigma^2=\tfrac12, a very noisy decoder, on [0,1][0,1] data.

python
1import torch, torch.nn.functional as F
2
3x = torch.tensor([0.0, 0.2, 0.9, 1.0]) # target pixels
4xh = torch.tensor([0.1, 0.3, 0.6, 0.8]) # decoder output after the sigmoid
5a = torch.logit(xh) # what the decoder should really return
6
7F.mse_loss(xh, x) # 0.0375 (mean over 4 pixels)
8F.mse_loss(xh, x, reduction="sum") # 0.1500
9F.l1_loss(xh, x, reduction="sum") # 0.7000
10F.binary_cross_entropy_with_logits(a, x, reduction="sum") # 1.4060, stable: takes logits
11F.gaussian_nll_loss(xh, x, var=torch.full_like(x, 0.01),
12 full=True, reduction="sum") # 1.9654 = 7.5 - 5.5346
13
14def vae_loss(logits, x, mu, logvar, beta=1.0):
15 """Negative ELBO per example: sum over pixels and latents, mean over the batch."""
16 B = x.shape[0]
17 rec = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / B
18 kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / B
19 return rec + beta * kl
The worked example in PyTorch, and a VAE loss with consistent reductions

Choosing the loss

For binary or nearly binary data (binarized MNIST, masks), BCE is the correct likelihood. For real-valued images the Gaussian is the textbook choice, but its variance matters. Rybkin, Daniilidis & Levine (2021) showed that learning a single shared σ\sigma (the "σ-VAE"), whose optimum is just the mean squared error, balances the two VAE terms automatically and removes the need to hand-tune β. BCE on grey-level pixels is common and works, but it is not a normalized density on [0,1][0,1]. Loaiza-Ganem & Cunningham (2019) derived the normalizing constant and called the result the continuous Bernoulli. For outlier-heavy targets, L1 or Huber is more robust.

None of these fixes blur, because all of them are per-pixel and assume the pixels are independent given zz. Sharper results need losses that compare structure rather than pixels: perceptual losses on network features, adversarial losses from a discriminator, or a model that represents the whole distribution of outcomes rather than a single guess, such as a diffusion model or an autoregressive decoder.

Use when

  • +Simple, cheap and exactly differentiable; needs no labels
  • +Each choice is a likelihood, so the value has a meaning (nats) and combines cleanly with a KL term
  • +BCE on logits gives a bounded, non-saturating gradient for outputs in [0, 1]
  • +L1 is robust to outliers and gives sharper (median) outputs than MSE

Avoid when

  • −Per-pixel and independent: the optimum is a mean (or median), so uncertain detail becomes blur
  • −Pixel distance is a poor proxy for perceptual similarity: a 1-pixel shift can cost more than a visible blur
  • −The scale (sum vs mean, the σ\sigma or bb of the likelihood) silently sets the VAE's β
  • −BCE on non-binary data has a nonzero floor and is not a normalized density

Check yourself

An autoencoder trained with MSE outputs a grey ramp where the training images have sharp edges at slightly different positions. What is the best explanation?

Check yourself

A VAE on 28×28 images uses F.mse_loss(x_hat, x) (mean reduction) plus the summed KL. Samples all look like the same average image. What is the most likely cause?

Remember

  • ✓Reconstruction loss = NLL of the input under the decoder: Gaussian → MSE, Laplace → L1, Bernoulli → BCE.
  • ✓Per-pixel losses are minimized by the conditional mean (MSE, BCE) or median (L1), so uncertain detail is averaged into blur.
  • ✓Use BCE on logits (binary_cross_entropy_with_logits): gradient x^−x\hat x-x, numerically stable.
  • ✓A likelihood sums over dimensions. Mean reduction or a hidden σ\sigma changes a VAE's effective β.
  • ✓BCE on grey targets has a floor equal to the target entropy; only the KL part is reducible.
  • ✓Sharper outputs need structure-aware losses (perceptual, adversarial) or a distributional decoder (diffusion, autoregressive).
← back to Deep Learning roadmap