Definition
Image restoration is the recovery of a clean image from a degraded observation , usually modelled as : a known or unknown linear operator followed by noise . Denoising, deblurring and JPEG artefact removal are the main tasks. Super-resolution and inpainting fit the same model and have their own articles. The landmark deep restorer is DnCNN (Zhang, Zuo, Chen, Meng and Zhang, "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising", IEEE TIP 2017).
Input and output are images of the same size. The model says what happened to the image. For denoising and is sensor noise; for deblurring convolves with a blur kernel (defocus, camera shake, motion); for super-resolution blurs and subsamples; for inpainting is a pixel mask. JPEG compression is not linear: it quantises DCT coefficients, which causes blocking and ringing.
Restoration is non-blind when and the noise level are known and blind when they must be estimated too, as with real camera shake or unknown noise. Most real photos are blind problems, and their noise is not white Gaussian: it is signal-dependent (Poisson–Gaussian at the sensor) and spatially correlated after demosaicing and in-camera processing.
Why it is hard: ill-posedness
A blur kernel is a low-pass filter. In the Fourier domain , and for a Gaussian blur falls towards zero at high frequencies. The naive inverse divides the noise by those tiny values and explodes. The classical fix adds a regulariser, a prior that says what images look like:
- data fidelity: the estimate, degraded again, should match the observation
- prior or regulariser; with the squared norm the solution is the closed-form Tikhonov filter on the right
- trade-off; the Wiener filter uses , the noise-to-signal power ratio
Better hand-made priors followed: total variation (piecewise-smooth images), sparsity in wavelets, and non-local self-similarity. BM3D (Dabov et al., IEEE TIP 2007) groups similar patches into 3-D stacks and filters them jointly. It was the denoising reference that deep models had to beat.
DnCNN: learn the residual
DnCNN learns the prior from data. It is a plain fully convolutional network: layer 1 is Conv+ReLU, layers 2–16 are Conv+BN+ReLU, and layer 17 is a Conv with one output channel, all with 64 channels and no pooling. Its key choice is residual learning. The network predicts the noise rather than the clean image, so the estimate is :
- noisy and clean training patches; the noise is synthesised, e.g. Gaussian with a given σ
- the target: the noise itself, which is close to zero-mean and has far less structure than the image
It is residual learning at the scale of the whole network: in ResNet each block learns a correction added to its input, here the full network learns what to subtract from . The authors found that residual learning and batch normalisation together trained faster and denoised better. DnCNN-S is trained for one noise level. Blind DnCNN-B (20 layers) is trained on σ ∈ [0, 55] and needs no noise level at test time. One DnCNN-3 model handled blind denoising, ×2–×4 super-resolution and JPEG deblocking.
Methods and numbers
- BSD68, Gaussian noise
- PSNR at σ = 15 / 25 / 50: BM3D 31.07 / 28.57 / 25.62 dB, DnCNN-S 31.73 / 29.23 / 26.23 dB
- FFDNet (2018)
- takes a noise-level map as an extra input, so one network covers a range of σ and spatially varying noise
- JPEG and deblurring
- ARCNN (Dong et al., ICCV 2015) for JPEG artefacts; DeepDeblur (Nah et al., CVPR 2017) and its GoPro dataset of real motion blur; DeblurGAN (Kupyn et al., CVPR 2018)
- Transformers
- SwinIR (Liang et al., ICCVW 2021): Swin Transformer blocks with shifted windows. Restormer (Zamir et al., CVPR 2022): attention across channels instead of pixels, so cost grows linearly with image size
- Real noise
- SIDD (Abdelhamed et al., CVPR 2018): smartphone images paired with low-noise ground truth; Noise2Noise (Lehtinen et al., ICML 2018) trains on pairs of independent noisy shots, no clean target needed
Worked example: gain per frequency
Take the figure's default blur ( samples, ) and look at a few frequencies . The inverse filter applies gain ; Tikhonov with applies . The product is how much of the true signal survives.
| k (ω) | H(ω) | inverse gain 1/H | Tikhonov gain G | signal kept G·H |
|---|---|---|---|---|
| 8 (0.125π) | 0.7346 | 1.36 | 1.336 | 0.982 |
| 16 (0.25π) | 0.2912 | 3.43 | 3.072 | 0.895 |
| 24 (0.375π) | 0.0623 | 16.1 | 4.488 | 0.280 |
| 32 (0.5π) | 0.0072 | 139 | 0.716 | 0.005 |
| 40 (0.625π) | 0.0004 | 2268 | 0.044 | 0.000 |
Where , Tikhonov matches the inverse and restores the signal. Where , it switches the frequency off, giving up detail rather than amplifying noise by 139× or 2268×. A learned restorer does better than any such linear filter, because its prior is non-linear and depends on content: it can keep a sharp edge and still smooth a flat region.
Beyond feed-forward CNNs
Learned priors can also be plugged into the optimisation above. Plug-and-play methods alternate a data-fidelity step on with a call to a pretrained denoiser, which acts as the prior. Any can then be handled without retraining (for example DPIR, Zhang et al., 2021).
Diffusion models take this further. A pretrained unconditional diffusion prior is guided during sampling to agree with : DDRM (Kawar et al., NeurIPS 2022) for linear and DPS (Chung et al., ICLR 2023) for general, even non-linear, operators. They produce sharp, realistic results under heavy degradation, but slowly and with the same risk as GAN super-resolution (see Image super-resolution): plausible detail that is not the true detail.
1import torch, torch.nn as nn, torch.nn.functional as F23class DnCNN(nn.Module):4 def __init__(self, depth=17, ch=64, c=1):5 super().__init__()6 layers = [nn.Conv2d(c, ch, 3, padding=1), nn.ReLU(inplace=True)]7 for _ in range(depth - 2):8 layers += [nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch), nn.ReLU(inplace=True)]9 layers += [nn.Conv2d(ch, c, 3, padding=1)]10 self.net = nn.Sequential(*layers)1112 def forward(self, y):13 return y - self.net(y) # the network predicts the noise v; x_hat = y - v_hat1415model = DnCNN()16opt = torch.optim.Adam(model.parameters(), lr=1e-3)17x = torch.rand(128, 1, 40, 40) # clean 40x40 patches in [0, 1]18y = x + (25 / 255) * torch.randn_like(x) # sigma = 25 on the 0-255 scale19loss = 0.5 * F.mse_loss(model.net(y), y - x, reduction="sum") / x.shape[0] # the DnCNN residual loss20opt.zero_grad(); loss.backward(); opt.step()21psnr = 10 * torch.log10(1 / F.mse_loss(model(y).clamp(0, 1), x))Use when
- +Learned priors beat hand-crafted ones: DnCNN gains about 0.6 dB over BM3D on BSD68 at every tested σ
- +One fully convolutional network runs on any image size and is fast on a GPU
- +Plug-and-play and diffusion methods reuse one prior for many operators H
Avoid when
- −A feed-forward restorer is tied to the degradations it was trained on
- −MSE-trained outputs over-smooth fine texture; GAN or diffusion outputs can invent detail
- −Blind real-world restoration still needs realistic training degradations or paired data
Check yourself
Why does DnCNN output the noise v = y − x instead of the clean image x?
Check yourself
In the figure, the inverse filter recovers the signal almost exactly when σ_n = 0 but collapses with σ_n = 0.02. Why?
Remember
- ✓Degradation model : (denoise), blur (deblur), ↓ (super-resolution), mask (inpaint); JPEG is non-linear quantisation.
- ✓Non-blind: H and σ known; blind: they must be estimated too. Real noise is signal-dependent and correlated.
- ✓Inverting is ill-posed: amplifies noise. Regularise: (Tikhonov/Wiener) or use a prior.
- ✓DnCNN (2017): 17 conv layers with BN, predicts the residual noise, x̂ = y − R(y); beats BM3D by about 0.6 dB on BSD68.
- ✓Transformers (SwinIR, Restormer) and diffusion priors (DDRM, DPS) are the newer generation; diffusion can hallucinate detail.