# Image super-resolution

> Super-resolution predicts a high-res image from a low-res one, an ill-posed inverse problem: MSE-trained CNNs (SRCNN, EDSR with pixel shuffle) maximise PSNR but blur, while GAN and perceptual losses (SRGAN, ESRGAN) invent sharp texture at lower PSNR.

Source: https://www.mlfoundry.in/learn/image-super-resolution  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Single-image super-resolution:** (SISR) is the task of estimating a high-resolution (HR) image $x$ from one low-resolution (LR) observation $y$, usually at an integer scale $r=2,3,4$. The first deep model was SRCNN (Dong, Loy, He and Tang, "Learning a Deep Convolutional Network for Image Super-Resolution", ECCV 2014). Later milestones are ESPCN's sub-pixel convolution (Shi et al., CVPR 2016), EDSR (Lim et al., CVPRW 2017), and the perceptually trained SRGAN (Ledig et al., CVPR 2017) and ESRGAN (Wang et al., ECCVW 2018).

The LR image is modelled as a degraded HR image, $y=(x*k)\downarrow_r+n$: blur with kernel $k$, subsample by $r$, add noise. Benchmarks usually take a simpler form, $y$ = bicubic downsampling of $x$. The output has $r^2$ times more pixels than the input, so for $\times4$ the model must produce 16 values per input pixel.

The problem is **ill-posed**. Downsampling discards every frequency above the LR Nyquist limit, so many different HR images map to exactly the same LR image. Classical methods interpolated (bicubic), used example patches, or learned sparse-coding dictionaries. A network instead learns a prior over natural images and picks a plausible HR image. Which one it picks depends on the loss.

## Architectures: where to upsample

**SRCNN** first upsamples the LR image with bicubic interpolation, then applies three convolutions to the luminance channel: $9\times9$ with 64 filters (patch extraction), $1\times1$ with 32 (non-linear mapping) and $5\times5$ with 1 (reconstruction). That is $9^2\cdot64+64\cdot32+5^2\cdot32=8{,}032$ weights, trained with MSE. It runs every convolution at HR size, which wastes $r^2$ times the compute.

**ESPCN** keeps all layers at LR size and upsamples once, at the end, with a **sub-pixel convolution**. A convolution outputs $C\cdot r^2$ channels, and a fixed periodic shuffle (`PixelShuffle` in PyTorch) rearranges them into an image $r$ times larger:

$$
\mathrm{PS}(T)_{c,\;rh+i,\;rw+j}=T_{c\,r^2+i\,r+j,\;h,\;w},\qquad T\in\mathbb R^{Cr^2\times H\times W}\ \to\ \mathbb R^{C\times rH\times rW}
$$

- $(i,j)$: sub-pixel offset inside each $r\times r$ output block, $0\le i,j<r$
- $c\,r^2+i\,r+j$: the input channel that supplies that offset; each channel fills one position of every block

It is equivalent to a strided **Transposed convolution** but cheaper, and it avoids the checkerboard artefacts that uneven kernel overlap can cause. **EDSR** (Enhanced Deep Residual Networks, winner of NTIRE 2017) uses the same post-upsampling layout with a deep residual body: 32 residual blocks of 256 channels, 43M parameters. It removed batch normalisation, which the authors found hurt SR. Residuals are scaled by 0.1 for stable training, and the loss is L1.

> Figure: Three designs. (a) SRCNN upsamples with bicubic first and runs 9×9, 1×1 and 5×5 convolutions at HR size. (b) ESPCN and EDSR compute at LR size; a last convolution emits 3·r² channels, and pixel shuffle interleaves them: for r = 2, four H×W maps become one 2H×2W map, each map filling one corner of every 2×2 block. (c) SRGAN and ESRGAN keep such a generator but train it with VGG-feature and adversarial losses.

## Losses and metrics

Evaluation uses **PSNR** and **SSIM** (Wang, Bovik, Sheikh and Simoncelli, IEEE TIP 2004), by convention on the Y channel of YCbCr with $r$ border pixels cropped, on Set5, Set14, BSD100 and Urban100. Most models train on DIV2K (800 training images at 2K resolution).

$$
\mathrm{PSNR}=10\log_{10}\frac{\mathrm{MAX}^2}{\mathrm{MSE}},\qquad \mathrm{SSIM}(a,b)=\frac{(2\mu_a\mu_b+C_1)(2\sigma_{ab}+C_2)}{(\mu_a^2+\mu_b^2+C_1)(\sigma_a^2+\sigma_b^2+C_2)}
$$

- $\mathrm{MAX}$: largest pixel value: 255 for 8-bit images, 1 for images in [0, 1]
- $\mu,\ \sigma^2,\ \sigma_{ab}$: local means, variances and covariance in a sliding window; SSIM is averaged over windows
- $C_1, C_2$: stabilisers $(0.01\,\mathrm{MAX})^2$ and $(0.03\,\mathrm{MAX})^2$

Training with MSE maximises PSNR directly, and the optimum is the **posterior mean**: the average of all HR images consistent with the input. Averaging many sharp textures with different phases gives a smooth one, which is why PSNR-optimal outputs look blurry.

**SRGAN** replaced pixel MSE with a **Perceptual loss** and an adversarial term. The content loss compares VGG-19 feature maps $\phi$ (the authors' best variant used the conv5_4 features); a discriminator $D$ (see **Generative adversarial network (GAN)**) pushes outputs onto the manifold of natural images:

$$
\ell^{SR}=\underbrace{\frac{1}{W_\phi H_\phi}\big\lVert\phi(x)-\phi(G(y))\big\rVert_2^2}_{\text{VGG content loss}}+10^{-3}\underbrace{\big(-\log D(G(y))\big)}_{\text{adversarial loss}}
$$

- $G(y)$: generator output; in SRGAN a ResNet with 16 residual blocks and two ×2 pixel-shuffle stages
- $\phi$: frozen VGG-19 activations; the loss is summed over feature positions and normalised by their count
- $D$: discriminator trained to tell real HR images from generator outputs

**ESRGAN** improved the recipe: Residual-in-Residual Dense Blocks (RRDB) without batch norm, a relativistic discriminator, and VGG features taken before the activation. It won the PIRM 2018 perceptual SR challenge. Real-ESRGAN (2021) then trained the same network on synthetic high-order degradations (blur, noise, resizing, JPEG) so it works on real photos, not only on bicubic inputs.

**Reference numbers (PSNR on the Y channel)**

- **Set5 ×3:** bicubic 30.39 dB, SRCNN 32.39 dB (ECCV 2014)
- **EDSR:** 32 blocks × 256 channels, 43M parameters, L1 loss; Set5 ×4 32.46 dB
- **SRGAN vs SRResNet:** Set5 ×4: SRResNet (same generator, MSE loss) 32.05 dB, SRGAN 29.40 dB; in the paper's mean-opinion-score test, raters still clearly preferred SRGAN
- **Perception–distortion:** Blau and Michaeli (CVPR 2018) proved that no algorithm can be both minimal in distortion (e.g. MSE) and perfect in perceptual quality: lowering one beyond a frontier raises the other

> Figure: A 32×32 synthetic image with a patch of period-4 diagonal stripes, box-downsampled and upscaled. At the default (×2, bicubic) PSNR is 22.04 dB and SSIM 0.800, but only 16% of the true detail energy survives; bilinear gives 20.94 dB. Sharpening with α = 1.0 is the PSNR peak, 22.88 dB (detail 0.40); α = 2.0 adds more detail (0.74) but falls to 22.22 dB. At ×4 the stripes vanish from the LR image, and the rotated-stripe image has exactly the same LR. Inventing its texture restores detail energy (0.71 vs 0.01) yet drops PSNR from 18.35 to 16.54 dB, and the stripe patch from 15.16 to 12.17 dB.

## Worked example: why MSE prefers blur

A 4-pixel HR row is either stripes A $=(0.2, 0.8, 0.2, 0.8)$ or the shifted stripes B $=(0.8, 0.2, 0.8, 0.2)$. Averaging pairs ($r=2$) maps both to the LR row $(0.5, 0.5)$, so the input cannot tell them apart. Assume each is equally likely and compare three outputs:

MSE against each possible truth; the flat answer, the posterior mean, wins on expected MSE

| Output | MSE if A | MSE if B | Expected MSE | PSNR of expected MSE |
| --- | --- | --- | --- | --- |
| flat (0.5, 0.5, 0.5, 0.5) | 0.09 | 0.09 | 0.09 | 10.46 dB |
| sharp guess A | 0 | 0.36 | 0.18 | 7.45 dB |
| sharp guess B | 0.36 | 0 | 0.18 | 7.45 dB |

The flat row has no texture at all, yet it beats either sharp guess by 3 dB. A guess has the right look, but half the time it is wrong at every pixel, and a wrong-phase stripe costs four times the squared error of a flat one. A GAN chooses a sharp, plausible answer and accepts the PSNR penalty, which is the perception–distortion trade-off in four pixels.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class ESPCN(nn.Module):
    def __init__(self, r=3, c=1):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(c, 64, 5, padding=2), nn.Tanh(),
            nn.Conv2d(64, 32, 3, padding=1), nn.Tanh(),
            nn.Conv2d(32, c * r * r, 3, padding=1),    # all layers run at LR size
        )
        self.shuffle = nn.PixelShuffle(r)              # (N, c*r*r, H, W) -> (N, c, rH, rW)

    def forward(self, lr):
        return self.shuffle(self.body(lr))

def psnr(sr, hr, border=3):                            # Y channel in [0, 1], crop r border pixels
    sr, hr = sr[..., border:-border, border:-border], hr[..., border:-border, border:-border]
    return 10 * torch.log10(1.0 / F.mse_loss(sr.clamp(0, 1), hr))

model = ESPCN(r=3)
lr = torch.rand(1, 1, 64, 64)
print(model(lr).shape)                                 # torch.Size([1, 1, 192, 192])
loss = F.l1_loss(model(lr), torch.rand(1, 1, 192, 192)) # EDSR-style L1 objective
```

An ESPCN-style ×3 network with PixelShuffle, and PSNR on [0, 1] images (PyTorch; not run here)

> **Trained on bicubic, tested on real photos:** A model trained only on bicubic downsampling learns that one degradation. Real low-resolution images also contain sensor noise, blur from optics and motion, and JPEG artefacts, so such a model sharpens the noise and blocking along with the content. Real-world SR trains on synthetic mixtures of degradations (Real-ESRGAN) or on paired real captures. For the general degradation model, see **Image restoration**.

> **PSNR is not the goal, but it is not useless:** PSNR and SSIM measure fidelity: whether the output equals the truth. Perceptual metrics such as LPIPS (Zhang et al., CVPR 2018) and human opinion scores measure realism. When an SR output feeds measurement or diagnosis, as in medical imaging, invented detail is a liability and PSNR-oriented models are the safer choice.

**Question:** Why does an SR network trained with MSE produce blurry textures?

- A. MSE gradients vanish at high frequencies
- B. The MSE-optimal output is the mean of all HR images consistent with the input, and averaging misaligned textures smooths them
- C. Convolutions cannot represent high frequencies
- D. Pixel shuffle blurs its input

**Answer:** B. The MSE-optimal output is the mean of all HR images consistent with the input, and averaging misaligned textures smooths them — Minimising expected squared error gives the conditional mean $\mathbb E[x\mid y]$. When many sharp textures fit the LR input, their mean is flat, as in the worked example, where the flat row beats either sharp guess by 3 dB.

**Question:** A ×4 sub-pixel layer must output a 3-channel 256×256 image from a 64×64 feature map. How many channels does the convolution before PixelShuffle produce?

- A. 3
- B. 12
- C. 48
- D. 192

**Answer:** C. 48 — PixelShuffle maps $C r^2$ channels at $H\times W$ to $C$ channels at $rH\times rW$. With $C=3$ and $r=4$ that is $3\cdot16=48$ channels.

**Key points**

- SR inverts $y=(x*k)\downarrow_r+n$; many HR images share one LR image, so the model's prior and loss decide the answer.
- SRCNN (2014): bicubic upsampling, then 9×9 → 1×1 → 5×5 convolutions, 8,032 weights, MSE.
- Sub-pixel convolution (ESPCN, 2016): compute at LR size, emit $C r^2$ channels, PixelShuffle them into an $r\times$ larger image.
- EDSR (2017): 32 residual blocks without batch norm, 43M parameters, L1 loss; Set5 ×4 32.46 dB.
- MSE/L1 → posterior mean → high PSNR but blur. SRGAN/ESRGAN: VGG perceptual loss + adversarial loss → sharp, invented texture at lower PSNR.
- Perception–distortion trade-off (Blau and Michaeli, 2018): low distortion and high realism cannot both be optimal.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
