Image restoration

Image restoration inverts a degradation y = Hx + n (noise, blur, JPEG artefacts). The inverse is ill-posed, so it needs a prior: once hand-crafted, now learned by CNNs such as DnCNN (2017), which predicts the noise residual, and by transformers and diffusion models.

8 min read796 wordscore0 viewsupdated 2026-10-15

Definition

Image restoration is the recovery of a clean image xx from a degraded observation yy, usually modelled as y=Hx+ny=Hx+n: a known or unknown linear operator HH followed by noise nn. Denoising, deblurring and JPEG artefact removal are the main tasks. Super-resolution and inpainting fit the same model and have their own articles. The landmark deep restorer is DnCNN (Zhang, Zuo, Chen, Meng and Zhang, "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising", IEEE TIP 2017).

Input and output are images of the same size. The model says what happened to the image. For denoising H=IH=I and nn is sensor noise; for deblurring HH convolves with a blur kernel kk (defocus, camera shake, motion); for super-resolution HH blurs and subsamples; for inpainting HH is a pixel mask. JPEG compression is not linear: it quantises 8×88\times8 DCT coefficients, which causes blocking and ringing.

Restoration is non-blind when HH and the noise level are known and blind when they must be estimated too, as with real camera shake or unknown noise. Most real photos are blind problems, and their noise is not white Gaussian: it is signal-dependent (Poisson–Gaussian at the sensor) and spatially correlated after demosaicing and in-camera processing.

Why it is hard: ill-posedness

A blur kernel is a low-pass filter. In the Fourier domain Y(ω)=H(ω)X(ω)+N(ω)Y(\omega)=H(\omega)X(\omega)+N(\omega), and for a Gaussian blur ∣H(ω)∣|H(\omega)| falls towards zero at high frequencies. The naive inverse X=Y/HX=Y/H divides the noise by those tiny values and explodes. The classical fix adds a regulariser, a prior that says what images look like:

x^=arg⁡min⁡x 12∥y−Hx∥22+λ R(x),R(x)=∥x∥22 ⇒ X^(ω)=H(ω)‾ Y(ω)∣H(ω)∣2+λ\hat x=\arg\min_x\ \tfrac12\lVert y-Hx\rVert_2^2+\lambda\,\mathcal R(x),\qquad \mathcal R(x)=\lVert x\rVert_2^2\ \Rightarrow\ \hat X(\omega)=\frac{\overline{H(\omega)}\,Y(\omega)}{|H(\omega)|^2+\lambda}
∥y−Hx∥2\lVert y-Hx\rVert^2
data fidelity: the estimate, degraded again, should match the observation
R(x)\mathcal R(x)
prior or regulariser; with the squared norm the solution is the closed-form Tikhonov filter on the right
λ\lambda
trade-off; the Wiener filter uses λ(ω)=Sn(ω)/Sx(ω)\lambda(\omega)=S_n(\omega)/S_x(\omega), the noise-to-signal power ratio

Better hand-made priors followed: total variation (piecewise-smooth images), sparsity in wavelets, and non-local self-similarity. BM3D (Dabov et al., IEEE TIP 2007) groups similar patches into 3-D stacks and filters them jointly. It was the denoising reference that deep models had to beat.

degradation model y = H x + n, and a restorer that inverts itclean xH = IdenoisingH = blur kdeblurringJPEG QdeblockingH = ↓rsuper-resolutionH = maskinpainting+ nN(0, σ²)observed yf_θ(y)learnedestimate x̂non-blind: H, σ known · blind: estimate them too · H is ill-conditioned, so a prior is neededDnCNN (2017): predict the noise, not the imagenoisy yConv+ReLU3×3, 64layer 1Conv+BN+ReLU3×3, 64 channelslayers 2 … 16Conv3×3 → 1layer 17residual v̂ = R(y)−skip: y itselfx̂ = y − v̂loss = 1/(2N) · ∑ᵢ ‖R(yᵢ; Θ) − (yᵢ − xᵢ)‖² · target is the noise v = y − x, not x17 layers of 3×3 → 35×35 receptive field · no pooling, output size = input sizeDnCNN-S: one model per noise level σ · DnCNN-B: blind, trained on σ ∈ [0, 55] (20 layers)residual learning + batch norm train fast; the noise is simpler to model than the image
fig.Top: one degradation model covers many tasks. H is the identity for denoising, a blur for deblurring, a subsampler for super-resolution and a mask for inpainting; JPEG adds non-linear quantisation. Noise n is added, and a learned restorer f_θ maps y back to x̂. Bottom: DnCNN stacks 17 3×3 convolution layers with 64 channels (BN + ReLU in layers 2–16, a 35×35 receptive field). It outputs the noise residual v̂, and the clean estimate is y − v̂.

DnCNN: learn the residual

DnCNN learns the prior from data. It is a plain fully convolutional network: layer 1 is Conv+ReLU, layers 2–16 are Conv+BN+ReLU, and layer 17 is a Conv with one output channel, all 3×33\times3 with 64 channels and no pooling. Its key choice is residual learning. The network R(y;Θ)\mathcal R(y;\Theta) predicts the noise v=y−xv=y-x rather than the clean image, so the estimate is x^=y−R(y)\hat x=y-\mathcal R(y):

ℓ(Θ)=12N∑i=1N∥R(yi;Θ)−(yi−xi)∥F2\ell(\Theta)=\frac{1}{2N}\sum_{i=1}^{N}\big\lVert\mathcal R(y_i;\Theta)-(y_i-x_i)\big\rVert_F^2
(yi,xi)(y_i, x_i)
noisy and clean training patches; the noise is synthesised, e.g. Gaussian with a given σ
yi−xiy_i-x_i
the target: the noise itself, which is close to zero-mean and has far less structure than the image

It is residual learning at the scale of the whole network: in ResNet each block learns a correction added to its input, here the full network learns what to subtract from yy. The authors found that residual learning and batch normalisation together trained faster and denoised better. DnCNN-S is trained for one noise level. Blind DnCNN-B (20 layers) is trained on σ ∈ [0, 55] and needs no noise level at test time. One DnCNN-3 model handled blind denoising, ×2–×4 super-resolution and JPEG deblocking.

Methods and numbers

BSD68, Gaussian noise
PSNR at σ = 15 / 25 / 50: BM3D 31.07 / 28.57 / 25.62 dB, DnCNN-S 31.73 / 29.23 / 26.23 dB
FFDNet (2018)
takes a noise-level map as an extra input, so one network covers a range of σ and spatially varying noise
JPEG and deblurring
ARCNN (Dong et al., ICCV 2015) for JPEG artefacts; DeepDeblur (Nah et al., CVPR 2017) and its GoPro dataset of real motion blur; DeblurGAN (Kupyn et al., CVPR 2018)
Transformers
SwinIR (Liang et al., ICCVW 2021): Swin Transformer blocks with shifted windows. Restormer (Zamir et al., CVPR 2022): attention across channels instead of pixels, so cost grows linearly with image size
Real noise
SIDD (Abdelhamed et al., CVPR 2018): smartphone images paired with low-noise ground truth; Noise2Noise (Lehtinen et al., ICML 2018) trains on pairs of independent noisy shots, no clean target needed
y = h ∗ x + n · blur σ_b 2.0 · noise σ_n 0.020 · Tikhonov, λ = 10^-2.0000.51━ clean x ━ observed y ━ restored x̂gain per frequency (log scale): |H| blur, |G| restorer1e-311e3ω = 0π (Nyquist)PSNR observed y20.00PSNR restored x̂20.12best λ = 10^-1.5020.52PSNR in dB, bar scale 0–30; best λ isan oracle (it peeks at the clean x)|H|² + λ: frequencies with |H|² ≫ λ are inverted, those with |H|² ≪ λ are switched off
restorer
fig.A 128-sample scanline blurred (σ_b = 2) and corrupted with noise (σ_n = 0.02), then restored in the Fourier domain. The observation scores 20.00 dB. The inverse filter divides by |H|, whose smallest value gives a gain of 3.9e5, and collapses to −63.37 dB. Tikhonov with λ = 10⁻² gives 20.12 dB; the best λ on the grid, 10⁻¹·⁵, reaches 20.52 dB; λ = 10⁻⁴ under-regularises and falls to 9.16 dB. Set the noise to 0 and the inverse filter recovers x almost exactly (over 99 dB): blur alone is invertible, the noise is what makes the problem ill-posed.

Worked example: gain per frequency

Take the figure's default blur (σb=2\sigma_b=2 samples, N=128N=128) and look at a few frequencies ω=2πk/128\omega=2\pi k/128. The inverse filter applies gain 1/H1/H; Tikhonov with λ=0.01\lambda=0.01 applies G=H/(H2+λ)G=H/(H^2+\lambda). The product GHGH is how much of the true signal survives.

k (ω)H(ω)inverse gain 1/HTikhonov gain Gsignal kept G·H
8 (0.125π)0.73461.361.3360.982
16 (0.25π)0.29123.433.0720.895
24 (0.375π)0.062316.14.4880.280
32 (0.5π)0.00721390.7160.005
40 (0.625π)0.000422680.0440.000
Default blur of the figure; noise at each frequency is multiplied by the gain

Where H2≫λH^2\gg\lambda, Tikhonov matches the inverse and restores the signal. Where H2≪λH^2\ll\lambda, it switches the frequency off, giving up detail rather than amplifying noise by 139× or 2268×. A learned restorer does better than any such linear filter, because its prior is non-linear and depends on content: it can keep a sharp edge and still smooth a flat region.

Beyond feed-forward CNNs

Learned priors can also be plugged into the optimisation above. Plug-and-play methods alternate a data-fidelity step on ∥y−Hx∥2\lVert y-Hx\rVert^2 with a call to a pretrained denoiser, which acts as the prior. Any HH can then be handled without retraining (for example DPIR, Zhang et al., 2021).

Diffusion models take this further. A pretrained unconditional diffusion prior is guided during sampling to agree with yy: DDRM (Kawar et al., NeurIPS 2022) for linear HH and DPS (Chung et al., ICLR 2023) for general, even non-linear, operators. They produce sharp, realistic results under heavy degradation, but slowly and with the same risk as GAN super-resolution (see Image super-resolution): plausible detail that is not the true detail.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class DnCNN(nn.Module):
4 def __init__(self, depth=17, ch=64, c=1):
5 super().__init__()
6 layers = [nn.Conv2d(c, ch, 3, padding=1), nn.ReLU(inplace=True)]
7 for _ in range(depth - 2):
8 layers += [nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch), nn.ReLU(inplace=True)]
9 layers += [nn.Conv2d(ch, c, 3, padding=1)]
10 self.net = nn.Sequential(*layers)
11
12 def forward(self, y):
13 return y - self.net(y) # the network predicts the noise v; x_hat = y - v_hat
14
15model = DnCNN()
16opt = torch.optim.Adam(model.parameters(), lr=1e-3)
17x = torch.rand(128, 1, 40, 40) # clean 40x40 patches in [0, 1]
18y = x + (25 / 255) * torch.randn_like(x) # sigma = 25 on the 0-255 scale
19loss = 0.5 * F.mse_loss(model.net(y), y - x, reduction="sum") / x.shape[0] # the DnCNN residual loss
20opt.zero_grad(); loss.backward(); opt.step()
21psnr = 10 * torch.log10(1 / F.mse_loss(model(y).clamp(0, 1), x))
DnCNN with residual learning, and one training step on synthetic Gaussian noise (PyTorch; not run here)

Use when

  • +Learned priors beat hand-crafted ones: DnCNN gains about 0.6 dB over BM3D on BSD68 at every tested σ
  • +One fully convolutional network runs on any image size and is fast on a GPU
  • +Plug-and-play and diffusion methods reuse one prior for many operators H

Avoid when

  • −A feed-forward restorer is tied to the degradations it was trained on
  • −MSE-trained outputs over-smooth fine texture; GAN or diffusion outputs can invent detail
  • −Blind real-world restoration still needs realistic training degradations or paired data

Check yourself

Why does DnCNN output the noise v = y − x instead of the clean image x?

Check yourself

In the figure, the inverse filter recovers the signal almost exactly when σ_n = 0 but collapses with σ_n = 0.02. Why?

Remember

  • ✓Degradation model y=Hx+ny=Hx+n: H=IH=I (denoise), blur (deblur), ↓ (super-resolution), mask (inpaint); JPEG is non-linear quantisation.
  • ✓Non-blind: H and σ known; blind: they must be estimated too. Real noise is signal-dependent and correlated.
  • ✓Inverting HH is ill-posed: 1/∣H∣1/|H| amplifies noise. Regularise: X^=HˉY/(∣H∣2+λ)\hat X=\bar H Y/(|H|^2+\lambda) (Tikhonov/Wiener) or use a prior.
  • ✓DnCNN (2017): 17 conv layers with BN, predicts the residual noise, x̂ = y − R(y); beats BM3D by about 0.6 dB on BSD68.
  • ✓Transformers (SwinIR, Restormer) and diffusion priors (DDRM, DPS) are the newer generation; diffusion can hallucinate detail.
← back to Deep Learning roadmap