Definition
Reconstruction loss is the term that scores how well a model's output reproduces its own input . It is the whole objective of a plain autoencoder and the first term of a VAE's negative ELBO, . In practice it is a sum over the dimensions of (pixels, audio samples, features) of a per-dimension loss: squared error, absolute error or binary cross-entropy.
An autoencoder has no labels. Its target is the input, so the loss decides what "the same" means. Squared error treats every pixel as a real number with Gaussian noise. Cross-entropy treats it as the probability of a coin flip. Absolute error treats it as a Laplace variable. These are modelling choices with visible consequences: the loss decides what the decoder outputs when it is unsure, and that is where blur comes from.
Autoencoders trained to reproduce their input go back to the auto-association networks of the 1980s (Rumelhart, Hinton & Williams, 1986). Hinton & Salakhutdinov (2006) made deep ones work with layer-wise pre-training. Kingma & Welling (2013) turned the reconstruction error into a log-likelihood inside the VAE, with a Bernoulli decoder for MNIST and a Gaussian decoder for real-valued Frey faces. Mathieu, Couprie & LeCun (2016) showed that next-frame video predictors trained with mean squared error produce blurry frames, and proposed gradient-difference and adversarial terms to fix it.
The three standard losses as likelihoods
- target value of dimension , e.g. a pixel in
- decoder output for dimension ; for BCE it is of a logit
- number of output dimensions, 784 for a 28×28 image
Each is a negative log-likelihood of a decoder distribution that factorizes over dimensions, up to a constant and a scale:
- decoder variance. Fixed, it only rescales MSE by , which matters once a KL term is added
- Laplace scale; plays the same role for L1
- a coin with ; exact only for binary data
At a glance
- MSE
- Gaussian decoder. Gradient . Optimal prediction: the conditional mean
- L1
- Laplace decoder. Gradient . Optimal prediction: the conditional median
- BCE
- Bernoulli decoder. Gradient w.r.t. the logit . Optimal prediction: the conditional mean
- Minimum value
- 0 for MSE and L1; for BCE with grey targets it is the entropy , not 0
- Reduction
- a likelihood sums over dimensions;
reduction="mean"divides by (and the batch) - PyTorch
F.mse_loss,F.l1_loss,F.binary_cross_entropy_with_logits,F.gaussian_nll_loss
Why reconstructions are blurry
A decoder sees a code , not the image. If several images are consistent with the same code, the decoder must commit to one output, and the loss decides which. Minimize the expected per-pixel loss over the possible targets:
- for MSE set ; for BCE the logit gradient vanishes at the same point
- the L1 subgradient is , which crosses 0 at the median
An average of several sharp images is a soft image. An edge whose position the code does not pin down becomes a ramp, and fine texture whose phase is unknown averages to flat grey. The decoder is not failing to learn: the blurry output is the exact minimizer of the loss it was given. The median is sharper in this example, because each pixel just takes the majority value, but it is not a cure. For many plausible outcomes it still averages within each pixel, and different pixels can take their medians from different images.
Gradients
Per dimension, with the decoder's last layer producing directly (MSE, L1) or through a sigmoid (BCE):
- pre-sigmoid logit of dimension
- the sigmoid's derivative cancels the 1/p of the log: a bounded, non-saturating error signal
If you instead put a sigmoid on the decoder and train it with MSE, the logit gradient is . It vanishes when a pixel is confidently wrong, the same failure the Loss functions article describes for classifiers. That is one practical reason BCE is popular for images in even when they are not binary.
Worked example: one 4-pixel image
Target , decoder output . All numbers below were computed in Node.
| pixel | x | x̂ | (x − x̂)² | |x − x̂| | BCE | BCE floor H(x) | ∂BCE/∂a = x̂ − x |
|---|---|---|---|---|---|---|---|
| 1 | 0 | 0.1 | 0.0100 | 0.1000 | 0.1054 | 0 | 0.100 |
| 2 | 0.2 | 0.3 | 0.0100 | 0.1000 | 0.5261 | 0.5004 | 0.100 |
| 3 | 0.9 | 0.6 | 0.0900 | 0.3000 | 0.5514 | 0.3251 | −0.300 |
| 4 | 1 | 0.8 | 0.0400 | 0.2000 | 0.2231 | 0 | −0.200 |
| sum | 0.1500 | 0.7000 | 1.4060 | 0.8255 |
- 1MSE: sum , mean . L1: sum , mean . BCE: sum nats.
- 2BCE with grey targets cannot reach 0. Its minimum is the entropy of the targets, nats here, so only nats (the KL from target to prediction) is actually reducible. Pixel 3 has the worst error () and also carries the largest reducible part.
- 3Gaussian NLL with : nats. The squared error is multiplied by . That factor is what sets the balance against a VAE's KL term.
- 4Laplace NLL with : nats.
Sum, mean and the KL term
A likelihood is a product over dimensions, so its log is a sum. Library defaults are reduction="mean", which divides by the number of elements. For a plain autoencoder this only rescales the learning rate. In a VAE it changes the model, because the reconstruction term is weighed against , which is summed over the latent dimensions.
Take MNIST with . A per-pixel BCE of 0.1 nats is 78.4 nats per image when summed, but only 0.1 when averaged. With a KL of about 20 nats, the mean-reduced objective lets the KL dominate by a factor of 784. That is a β-VAE with , and it usually ends in posterior collapse: the code carries nothing and the decoder outputs the average digit. The same arithmetic explains the Gaussian case: MSE with a fixed is the true NLL only after multiplying by , so "MSE + KL" silently picks , a very noisy decoder, on data.
1import torch, torch.nn.functional as F23x = torch.tensor([0.0, 0.2, 0.9, 1.0]) # target pixels4xh = torch.tensor([0.1, 0.3, 0.6, 0.8]) # decoder output after the sigmoid5a = torch.logit(xh) # what the decoder should really return67F.mse_loss(xh, x) # 0.0375 (mean over 4 pixels)8F.mse_loss(xh, x, reduction="sum") # 0.15009F.l1_loss(xh, x, reduction="sum") # 0.700010F.binary_cross_entropy_with_logits(a, x, reduction="sum") # 1.4060, stable: takes logits11F.gaussian_nll_loss(xh, x, var=torch.full_like(x, 0.01),12 full=True, reduction="sum") # 1.9654 = 7.5 - 5.53461314def vae_loss(logits, x, mu, logvar, beta=1.0):15 """Negative ELBO per example: sum over pixels and latents, mean over the batch."""16 B = x.shape[0]17 rec = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / B18 kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / B19 return rec + beta * klChoosing the loss
For binary or nearly binary data (binarized MNIST, masks), BCE is the correct likelihood. For real-valued images the Gaussian is the textbook choice, but its variance matters. Rybkin, Daniilidis & Levine (2021) showed that learning a single shared (the "σ-VAE"), whose optimum is just the mean squared error, balances the two VAE terms automatically and removes the need to hand-tune β. BCE on grey-level pixels is common and works, but it is not a normalized density on . Loaiza-Ganem & Cunningham (2019) derived the normalizing constant and called the result the continuous Bernoulli. For outlier-heavy targets, L1 or Huber is more robust.
None of these fixes blur, because all of them are per-pixel and assume the pixels are independent given . Sharper results need losses that compare structure rather than pixels: perceptual losses on network features, adversarial losses from a discriminator, or a model that represents the whole distribution of outcomes rather than a single guess, such as a diffusion model or an autoregressive decoder.
Use when
- +Simple, cheap and exactly differentiable; needs no labels
- +Each choice is a likelihood, so the value has a meaning (nats) and combines cleanly with a KL term
- +BCE on logits gives a bounded, non-saturating gradient for outputs in [0, 1]
- +L1 is robust to outliers and gives sharper (median) outputs than MSE
Avoid when
- −Per-pixel and independent: the optimum is a mean (or median), so uncertain detail becomes blur
- −Pixel distance is a poor proxy for perceptual similarity: a 1-pixel shift can cost more than a visible blur
- −The scale (sum vs mean, the or of the likelihood) silently sets the VAE's β
- −BCE on non-binary data has a nonzero floor and is not a normalized density
Check yourself
An autoencoder trained with MSE outputs a grey ramp where the training images have sharp edges at slightly different positions. What is the best explanation?
Check yourself
A VAE on 28×28 images uses F.mse_loss(x_hat, x) (mean reduction) plus the summed KL. Samples all look like the same average image. What is the most likely cause?
Remember
- ✓Reconstruction loss = NLL of the input under the decoder: Gaussian → MSE, Laplace → L1, Bernoulli → BCE.
- ✓Per-pixel losses are minimized by the conditional mean (MSE, BCE) or median (L1), so uncertain detail is averaged into blur.
- ✓Use BCE on logits (
binary_cross_entropy_with_logits): gradient , numerically stable. - ✓A likelihood sums over dimensions. Mean reduction or a hidden changes a VAE's effective β.
- ✓BCE on grey targets has a floor equal to the target entropy; only the KL part is reducible.
- ✓Sharper outputs need structure-aware losses (perceptual, adversarial) or a distributional decoder (diffusion, autoregressive).