Definition
Single-image super-resolution (SISR) is the task of estimating a high-resolution (HR) image from one low-resolution (LR) observation , usually at an integer scale . The first deep model was SRCNN (Dong, Loy, He and Tang, "Learning a Deep Convolutional Network for Image Super-Resolution", ECCV 2014). Later milestones are ESPCN's sub-pixel convolution (Shi et al., CVPR 2016), EDSR (Lim et al., CVPRW 2017), and the perceptually trained SRGAN (Ledig et al., CVPR 2017) and ESRGAN (Wang et al., ECCVW 2018).
The LR image is modelled as a degraded HR image, : blur with kernel , subsample by , add noise. Benchmarks usually take a simpler form, = bicubic downsampling of . The output has times more pixels than the input, so for the model must produce 16 values per input pixel.
The problem is ill-posed. Downsampling discards every frequency above the LR Nyquist limit, so many different HR images map to exactly the same LR image. Classical methods interpolated (bicubic), used example patches, or learned sparse-coding dictionaries. A network instead learns a prior over natural images and picks a plausible HR image. Which one it picks depends on the loss.
Architectures: where to upsample
SRCNN first upsamples the LR image with bicubic interpolation, then applies three convolutions to the luminance channel: with 64 filters (patch extraction), with 32 (non-linear mapping) and with 1 (reconstruction). That is weights, trained with MSE. It runs every convolution at HR size, which wastes times the compute.
ESPCN keeps all layers at LR size and upsamples once, at the end, with a sub-pixel convolution. A convolution outputs channels, and a fixed periodic shuffle (PixelShuffle in PyTorch) rearranges them into an image times larger:
- sub-pixel offset inside each output block,
- the input channel that supplies that offset; each channel fills one position of every block
It is equivalent to a strided Transposed convolution but cheaper, and it avoids the checkerboard artefacts that uneven kernel overlap can cause. EDSR (Enhanced Deep Residual Networks, winner of NTIRE 2017) uses the same post-upsampling layout with a deep residual body: 32 residual blocks of 256 channels, 43M parameters. It removed batch normalisation, which the authors found hurt SR. Residuals are scaled by 0.1 for stable training, and the loss is L1.
Losses and metrics
Evaluation uses PSNR and SSIM (Wang, Bovik, Sheikh and Simoncelli, IEEE TIP 2004), by convention on the Y channel of YCbCr with border pixels cropped, on Set5, Set14, BSD100 and Urban100. Most models train on DIV2K (800 training images at 2K resolution).
- largest pixel value: 255 for 8-bit images, 1 for images in [0, 1]
- local means, variances and covariance in a sliding window; SSIM is averaged over windows
- stabilisers and
Training with MSE maximises PSNR directly, and the optimum is the posterior mean: the average of all HR images consistent with the input. Averaging many sharp textures with different phases gives a smooth one, which is why PSNR-optimal outputs look blurry.
SRGAN replaced pixel MSE with a Perceptual loss and an adversarial term. The content loss compares VGG-19 feature maps (the authors' best variant used the conv5_4 features); a discriminator (see Generative adversarial network (GAN)) pushes outputs onto the manifold of natural images:
- generator output; in SRGAN a ResNet with 16 residual blocks and two ×2 pixel-shuffle stages
- frozen VGG-19 activations; the loss is summed over feature positions and normalised by their count
- discriminator trained to tell real HR images from generator outputs
ESRGAN improved the recipe: Residual-in-Residual Dense Blocks (RRDB) without batch norm, a relativistic discriminator, and VGG features taken before the activation. It won the PIRM 2018 perceptual SR challenge. Real-ESRGAN (2021) then trained the same network on synthetic high-order degradations (blur, noise, resizing, JPEG) so it works on real photos, not only on bicubic inputs.
Reference numbers (PSNR on the Y channel)
- Set5 ×3
- bicubic 30.39 dB, SRCNN 32.39 dB (ECCV 2014)
- EDSR
- 32 blocks × 256 channels, 43M parameters, L1 loss; Set5 ×4 32.46 dB
- SRGAN vs SRResNet
- Set5 ×4: SRResNet (same generator, MSE loss) 32.05 dB, SRGAN 29.40 dB; in the paper's mean-opinion-score test, raters still clearly preferred SRGAN
- Perception–distortion
- Blau and Michaeli (CVPR 2018) proved that no algorithm can be both minimal in distortion (e.g. MSE) and perfect in perceptual quality: lowering one beyond a frontier raises the other
Worked example: why MSE prefers blur
A 4-pixel HR row is either stripes A or the shifted stripes B . Averaging pairs () maps both to the LR row , so the input cannot tell them apart. Assume each is equally likely and compare three outputs:
| Output | MSE if A | MSE if B | Expected MSE | PSNR of expected MSE |
|---|---|---|---|---|
| flat (0.5, 0.5, 0.5, 0.5) | 0.09 | 0.09 | 0.09 | 10.46 dB |
| sharp guess A | 0 | 0.36 | 0.18 | 7.45 dB |
| sharp guess B | 0.36 | 0 | 0.18 | 7.45 dB |
The flat row has no texture at all, yet it beats either sharp guess by 3 dB. A guess has the right look, but half the time it is wrong at every pixel, and a wrong-phase stripe costs four times the squared error of a flat one. A GAN chooses a sharp, plausible answer and accepts the PSNR penalty, which is the perception–distortion trade-off in four pixels.
1import torch, torch.nn as nn, torch.nn.functional as F23class ESPCN(nn.Module):4 def __init__(self, r=3, c=1):5 super().__init__()6 self.body = nn.Sequential(7 nn.Conv2d(c, 64, 5, padding=2), nn.Tanh(),8 nn.Conv2d(64, 32, 3, padding=1), nn.Tanh(),9 nn.Conv2d(32, c * r * r, 3, padding=1), # all layers run at LR size10 )11 self.shuffle = nn.PixelShuffle(r) # (N, c*r*r, H, W) -> (N, c, rH, rW)1213 def forward(self, lr):14 return self.shuffle(self.body(lr))1516def psnr(sr, hr, border=3): # Y channel in [0, 1], crop r border pixels17 sr, hr = sr[..., border:-border, border:-border], hr[..., border:-border, border:-border]18 return 10 * torch.log10(1.0 / F.mse_loss(sr.clamp(0, 1), hr))1920model = ESPCN(r=3)21lr = torch.rand(1, 1, 64, 64)22print(model(lr).shape) # torch.Size([1, 1, 192, 192])23loss = F.l1_loss(model(lr), torch.rand(1, 1, 192, 192)) # EDSR-style L1 objectiveCheck yourself
Why does an SR network trained with MSE produce blurry textures?
Check yourself
A ×4 sub-pixel layer must output a 3-channel 256×256 image from a 64×64 feature map. How many channels does the convolution before PixelShuffle produce?
Remember
- ✓SR inverts ; many HR images share one LR image, so the model's prior and loss decide the answer.
- ✓SRCNN (2014): bicubic upsampling, then 9×9 → 1×1 → 5×5 convolutions, 8,032 weights, MSE.
- ✓Sub-pixel convolution (ESPCN, 2016): compute at LR size, emit channels, PixelShuffle them into an larger image.
- ✓EDSR (2017): 32 residual blocks without batch norm, 43M parameters, L1 loss; Set5 ×4 32.46 dB.
- ✓MSE/L1 → posterior mean → high PSNR but blur. SRGAN/ESRGAN: VGG perceptual loss + adversarial loss → sharp, invented texture at lower PSNR.
- ✓Perception–distortion trade-off (Blau and Michaeli, 2018): low distortion and high realism cannot both be optimal.