Image super-resolution

Super-resolution predicts a high-res image from a low-res one, an ill-posed inverse problem: MSE-trained CNNs (SRCNN, EDSR with pixel shuffle) maximise PSNR but blur, while GAN and perceptual losses (SRGAN, ESRGAN) invent sharp texture at lower PSNR.

8 min read871 wordscore0 viewsupdated 2026-10-15

Definition

Single-image super-resolution (SISR) is the task of estimating a high-resolution (HR) image xx from one low-resolution (LR) observation yy, usually at an integer scale r=2,3,4r=2,3,4. The first deep model was SRCNN (Dong, Loy, He and Tang, "Learning a Deep Convolutional Network for Image Super-Resolution", ECCV 2014). Later milestones are ESPCN's sub-pixel convolution (Shi et al., CVPR 2016), EDSR (Lim et al., CVPRW 2017), and the perceptually trained SRGAN (Ledig et al., CVPR 2017) and ESRGAN (Wang et al., ECCVW 2018).

The LR image is modelled as a degraded HR image, y=(x∗k)↓r+ny=(x*k)\downarrow_r+n: blur with kernel kk, subsample by rr, add noise. Benchmarks usually take a simpler form, yy = bicubic downsampling of xx. The output has r2r^2 times more pixels than the input, so for ×4\times4 the model must produce 16 values per input pixel.

The problem is ill-posed. Downsampling discards every frequency above the LR Nyquist limit, so many different HR images map to exactly the same LR image. Classical methods interpolated (bicubic), used example patches, or learned sparse-coding dictionaries. A network instead learns a prior over natural images and picks a plausible HR image. Which one it picks depends on the loss.

Architectures: where to upsample

SRCNN first upsamples the LR image with bicubic interpolation, then applies three convolutions to the luminance channel: 9×99\times9 with 64 filters (patch extraction), 1×11\times1 with 32 (non-linear mapping) and 5×55\times5 with 1 (reconstruction). That is 92⋅64+64⋅32+52⋅32=8,0329^2\cdot64+64\cdot32+5^2\cdot32=8{,}032 weights, trained with MSE. It runs every convolution at HR size, which wastes r2r^2 times the compute.

ESPCN keeps all layers at LR size and upsamples once, at the end, with a sub-pixel convolution. A convolution outputs C⋅r2C\cdot r^2 channels, and a fixed periodic shuffle (PixelShuffle in PyTorch) rearranges them into an image rr times larger:

PS(T)c,  rh+i,  rw+j=Tc r2+i r+j,  h,  w,T∈RCr2×H×W → RC×rH×rW\mathrm{PS}(T)_{c,\;rh+i,\;rw+j}=T_{c\,r^2+i\,r+j,\;h,\;w},\qquad T\in\mathbb R^{Cr^2\times H\times W}\ \to\ \mathbb R^{C\times rH\times rW}
(i,j)(i,j)
sub-pixel offset inside each r×rr\times r output block, 0≤i,j<r0\le i,j<r
c r2+i r+jc\,r^2+i\,r+j
the input channel that supplies that offset; each channel fills one position of every block

It is equivalent to a strided Transposed convolution but cheaper, and it avoids the checkerboard artefacts that uneven kernel overlap can cause. EDSR (Enhanced Deep Residual Networks, winner of NTIRE 2017) uses the same post-upsampling layout with a deep residual body: 32 residual blocks of 256 channels, 43M parameters. It removed batch normalisation, which the authors found hurt SR. Residuals are scaled by 0.1 for stable training, and the loss is L1.

a · SRCNN (2014): upsample first, then 3 convs at HR sizeLRH×Wbicubic ↑rrH×rW9×9 conv, 64patch features1×1 conv, 32non-linear map5×5 conv, 1reconstruct YSRrH×rWevery conv runs on r² times more pixels than the input has · trained with MSEb · ESPCN (2016) / EDSR (2017): stay at LR size, upsample lastLRH×W×3residual blocksEDSR: 32 × 256 ch,no BN, scale 0.1conv→ 3·r² chr² = 4 maps, H×Wshuffleone map, rH×rWSR2H×2W×3PixelShuffle(r): (C·r², H, W) → (C, rH, rW) · EDSR: L1 loss, 43M parametersc · SRGAN (2017) / ESRGAN (2018): same kind of generator, perceptual trainingLRH×Wgenerator GSRGAN: 16 res blocksESRGAN: RRDBSR4H×4WVGG-19 features φ‖φ(SR) − φ(HR)‖²discriminator Dreal HR or SR?l_content+ 10⁻³ l_advthe feature loss and D reward realistic texture over pixel accuracy: sharper images, lower PSNR(Set5 ×4: SRResNet-MSE 32.05 dB vs SRGAN 29.40 dB, yet SRGAN wins on mean opinion score)
fig.Three designs. (a) SRCNN upsamples with bicubic first and runs 9×9, 1×1 and 5×5 convolutions at HR size. (b) ESPCN and EDSR compute at LR size; a last convolution emits 3·r² channels, and pixel shuffle interleaves them: for r = 2, four H×W maps become one 2H×2W map, each map filling one corner of every 2×2 block. (c) SRGAN and ESRGAN keep such a generator but train it with VGG-feature and adversarial losses.

Losses and metrics

Evaluation uses PSNR and SSIM (Wang, Bovik, Sheikh and Simoncelli, IEEE TIP 2004), by convention on the Y channel of YCbCr with rr border pixels cropped, on Set5, Set14, BSD100 and Urban100. Most models train on DIV2K (800 training images at 2K resolution).

PSNR=10log⁡10MAX2MSE,SSIM(a,b)=(2μaμb+C1)(2σab+C2)(μa2+μb2+C1)(σa2+σb2+C2)\mathrm{PSNR}=10\log_{10}\frac{\mathrm{MAX}^2}{\mathrm{MSE}},\qquad \mathrm{SSIM}(a,b)=\frac{(2\mu_a\mu_b+C_1)(2\sigma_{ab}+C_2)}{(\mu_a^2+\mu_b^2+C_1)(\sigma_a^2+\sigma_b^2+C_2)}
MAX\mathrm{MAX}
largest pixel value: 255 for 8-bit images, 1 for images in [0, 1]
μ, σ2, σab\mu,\ \sigma^2,\ \sigma_{ab}
local means, variances and covariance in a sliding window; SSIM is averaged over windows
C1,C2C_1, C_2
stabilisers (0.01 MAX)2(0.01\,\mathrm{MAX})^2 and (0.03 MAX)2(0.03\,\mathrm{MAX})^2

Training with MSE maximises PSNR directly, and the optimum is the posterior mean: the average of all HR images consistent with the input. Averaging many sharp textures with different phases gives a smooth one, which is why PSNR-optimal outputs look blurry.

SRGAN replaced pixel MSE with a Perceptual loss and an adversarial term. The content loss compares VGG-19 feature maps ϕ\phi (the authors' best variant used the conv5_4 features); a discriminator DD (see Generative adversarial network (GAN)) pushes outputs onto the manifold of natural images:

ℓSR=1WϕHϕ∥ϕ(x)−ϕ(G(y))∥22⏟VGG content loss+10−3(−log⁡D(G(y)))⏟adversarial loss\ell^{SR}=\underbrace{\frac{1}{W_\phi H_\phi}\big\lVert\phi(x)-\phi(G(y))\big\rVert_2^2}_{\text{VGG content loss}}+10^{-3}\underbrace{\big(-\log D(G(y))\big)}_{\text{adversarial loss}}
G(y)G(y)
generator output; in SRGAN a ResNet with 16 residual blocks and two ×2 pixel-shuffle stages
ϕ\phi
frozen VGG-19 activations; the loss is summed over feature positions and normalised by their count
DD
discriminator trained to tell real HR images from generator outputs

ESRGAN improved the recipe: Residual-in-Residual Dense Blocks (RRDB) without batch norm, a relativistic discriminator, and VGG features taken before the activation. It won the PIRM 2018 perceptual SR challenge. Real-ESRGAN (2021) then trained the same network on synthetic high-order degradations (blur, noise, resizing, JPEG) so it works on real photos, not only on bicubic inputs.

Reference numbers (PSNR on the Y channel)

Set5 ×3
bicubic 30.39 dB, SRCNN 32.39 dB (ECCV 2014)
EDSR
32 blocks × 256 channels, 43M parameters, L1 loss; Set5 ×4 32.46 dB
SRGAN vs SRResNet
Set5 ×4: SRResNet (same generator, MSE loss) 32.05 dB, SRGAN 29.40 dB; in the paper's mean-opinion-score test, raters still clearly preferred SRGAN
Perception–distortion
Blau and Michaeli (CVPR 2018) proved that no algorithm can be both minimal in distortion (e.g. MSE) and perfect in perceptual quality: lowering one beyond a frontier raises the other
×2 super-resolution · LR = 2×2 box average · scores against the HR truthHR truth 32×32LR input 16×16bicubicanother HR, stripes ↘LR differs by ≤ 0.123→→→PSNR (dB)22.04bicubic: 22.04SSIM0.800bicubic: 0.800PSNR, stripe patch17.21bicubic: 17.21detail energy / HR0.16bicubic: 0.16PSNR vs α · peak 22.88 at α 1.1242102sharpening adds detail energy, but past the peak it adds error; inventedtexture looks detailed yet scores below blur (perception–distortion)
scaleupsample
fig.A 32×32 synthetic image with a patch of period-4 diagonal stripes, box-downsampled and upscaled. At the default (×2, bicubic) PSNR is 22.04 dB and SSIM 0.800, but only 16% of the true detail energy survives; bilinear gives 20.94 dB. Sharpening with α = 1.0 is the PSNR peak, 22.88 dB (detail 0.40); α = 2.0 adds more detail (0.74) but falls to 22.22 dB. At ×4 the stripes vanish from the LR image, and the rotated-stripe image has exactly the same LR. Inventing its texture restores detail energy (0.71 vs 0.01) yet drops PSNR from 18.35 to 16.54 dB, and the stripe patch from 15.16 to 12.17 dB.

Worked example: why MSE prefers blur

A 4-pixel HR row is either stripes A =(0.2,0.8,0.2,0.8)=(0.2, 0.8, 0.2, 0.8) or the shifted stripes B =(0.8,0.2,0.8,0.2)=(0.8, 0.2, 0.8, 0.2). Averaging pairs (r=2r=2) maps both to the LR row (0.5,0.5)(0.5, 0.5), so the input cannot tell them apart. Assume each is equally likely and compare three outputs:

OutputMSE if AMSE if BExpected MSEPSNR of expected MSE
flat (0.5, 0.5, 0.5, 0.5)0.090.090.0910.46 dB
sharp guess A00.360.187.45 dB
sharp guess B0.3600.187.45 dB
MSE against each possible truth; the flat answer, the posterior mean, wins on expected MSE

The flat row has no texture at all, yet it beats either sharp guess by 3 dB. A guess has the right look, but half the time it is wrong at every pixel, and a wrong-phase stripe costs four times the squared error of a flat one. A GAN chooses a sharp, plausible answer and accepts the PSNR penalty, which is the perception–distortion trade-off in four pixels.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class ESPCN(nn.Module):
4 def __init__(self, r=3, c=1):
5 super().__init__()
6 self.body = nn.Sequential(
7 nn.Conv2d(c, 64, 5, padding=2), nn.Tanh(),
8 nn.Conv2d(64, 32, 3, padding=1), nn.Tanh(),
9 nn.Conv2d(32, c * r * r, 3, padding=1), # all layers run at LR size
10 )
11 self.shuffle = nn.PixelShuffle(r) # (N, c*r*r, H, W) -> (N, c, rH, rW)
12
13 def forward(self, lr):
14 return self.shuffle(self.body(lr))
15
16def psnr(sr, hr, border=3): # Y channel in [0, 1], crop r border pixels
17 sr, hr = sr[..., border:-border, border:-border], hr[..., border:-border, border:-border]
18 return 10 * torch.log10(1.0 / F.mse_loss(sr.clamp(0, 1), hr))
19
20model = ESPCN(r=3)
21lr = torch.rand(1, 1, 64, 64)
22print(model(lr).shape) # torch.Size([1, 1, 192, 192])
23loss = F.l1_loss(model(lr), torch.rand(1, 1, 192, 192)) # EDSR-style L1 objective
An ESPCN-style ×3 network with PixelShuffle, and PSNR on [0, 1] images (PyTorch; not run here)

Check yourself

Why does an SR network trained with MSE produce blurry textures?

Check yourself

A ×4 sub-pixel layer must output a 3-channel 256×256 image from a 64×64 feature map. How many channels does the convolution before PixelShuffle produce?

Remember

  • ✓SR inverts y=(x∗k)↓r+ny=(x*k)\downarrow_r+n; many HR images share one LR image, so the model's prior and loss decide the answer.
  • ✓SRCNN (2014): bicubic upsampling, then 9×9 → 1×1 → 5×5 convolutions, 8,032 weights, MSE.
  • ✓Sub-pixel convolution (ESPCN, 2016): compute at LR size, emit Cr2C r^2 channels, PixelShuffle them into an r×r\times larger image.
  • ✓EDSR (2017): 32 residual blocks without batch norm, 43M parameters, L1 loss; Set5 ×4 32.46 dB.
  • ✓MSE/L1 → posterior mean → high PSNR but blur. SRGAN/ESRGAN: VGG perceptual loss + adversarial loss → sharp, invented texture at lower PSNR.
  • ✓Perception–distortion trade-off (Blau and Michaeli, 2018): low distortion and high realism cannot both be optimal.
← back to Deep Learning roadmap