# Image restoration

> Image restoration inverts a degradation y = Hx + n (noise, blur, JPEG artefacts). The inverse is ill-posed, so it needs a prior: once hand-crafted, now learned by CNNs such as DnCNN (2017), which predicts the noise residual, and by transformers and diffusion models.

Source: https://www.mlfoundry.in/learn/image-restoration  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Image restoration:** is the recovery of a clean image $x$ from a degraded observation $y$, usually modelled as $y=Hx+n$: a known or unknown linear operator $H$ followed by noise $n$. Denoising, deblurring and JPEG artefact removal are the main tasks. Super-resolution and inpainting fit the same model and have their own articles. The landmark deep restorer is DnCNN (Zhang, Zuo, Chen, Meng and Zhang, "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising", IEEE TIP 2017).

Input and output are images of the same size. The model says what happened to the image. For denoising $H=I$ and $n$ is sensor noise; for deblurring $H$ convolves with a blur kernel $k$ (defocus, camera shake, motion); for super-resolution $H$ blurs and subsamples; for inpainting $H$ is a pixel mask. JPEG compression is not linear: it quantises $8\times8$ DCT coefficients, which causes blocking and ringing.

Restoration is **non-blind** when $H$ and the noise level are known and **blind** when they must be estimated too, as with real camera shake or unknown noise. Most real photos are blind problems, and their noise is not white Gaussian: it is signal-dependent (Poisson–Gaussian at the sensor) and spatially correlated after demosaicing and in-camera processing.

## Why it is hard: ill-posedness

A blur kernel is a low-pass filter. In the Fourier domain $Y(\omega)=H(\omega)X(\omega)+N(\omega)$, and for a Gaussian blur $|H(\omega)|$ falls towards zero at high frequencies. The naive inverse $X=Y/H$ divides the noise by those tiny values and explodes. The classical fix adds a regulariser, a prior that says what images look like:

$$
\hat x=\arg\min_x\ \tfrac12\lVert y-Hx\rVert_2^2+\lambda\,\mathcal R(x),\qquad \mathcal R(x)=\lVert x\rVert_2^2\ \Rightarrow\ \hat X(\omega)=\frac{\overline{H(\omega)}\,Y(\omega)}{|H(\omega)|^2+\lambda}
$$

- $\lVert y-Hx\rVert^2$: data fidelity: the estimate, degraded again, should match the observation
- $\mathcal R(x)$: prior or regulariser; with the squared norm the solution is the closed-form Tikhonov filter on the right
- $\lambda$: trade-off; the Wiener filter uses $\lambda(\omega)=S_n(\omega)/S_x(\omega)$, the noise-to-signal power ratio

Better hand-made priors followed: total variation (piecewise-smooth images), sparsity in wavelets, and non-local self-similarity. BM3D (Dabov et al., IEEE TIP 2007) groups similar patches into 3-D stacks and filters them jointly. It was the denoising reference that deep models had to beat.

> Figure: Top: one degradation model covers many tasks. H is the identity for denoising, a blur for deblurring, a subsampler for super-resolution and a mask for inpainting; JPEG adds non-linear quantisation. Noise n is added, and a learned restorer f_θ maps y back to x̂. Bottom: DnCNN stacks 17 3×3 convolution layers with 64 channels (BN + ReLU in layers 2–16, a 35×35 receptive field). It outputs the noise residual v̂, and the clean estimate is y − v̂.

## DnCNN: learn the residual

DnCNN learns the prior from data. It is a plain fully convolutional network: layer 1 is Conv+ReLU, layers 2–16 are Conv+BN+ReLU, and layer 17 is a Conv with one output channel, all $3\times3$ with 64 channels and no pooling. Its key choice is **residual learning**. The network $\mathcal R(y;\Theta)$ predicts the noise $v=y-x$ rather than the clean image, so the estimate is $\hat x=y-\mathcal R(y)$:

$$
\ell(\Theta)=\frac{1}{2N}\sum_{i=1}^{N}\big\lVert\mathcal R(y_i;\Theta)-(y_i-x_i)\big\rVert_F^2
$$

- $(y_i, x_i)$: noisy and clean training patches; the noise is synthesised, e.g. Gaussian with a given σ
- $y_i-x_i$: the target: the noise itself, which is close to zero-mean and has far less structure than the image

It is residual learning at the scale of the whole network: in **ResNet** each block learns a correction added to its input, here the full network learns what to subtract from $y$. The authors found that residual learning and batch normalisation together trained faster and denoised better. DnCNN-S is trained for one noise level. Blind DnCNN-B (20 layers) is trained on σ ∈ [0, 55] and needs no noise level at test time. One DnCNN-3 model handled blind denoising, ×2–×4 super-resolution and JPEG deblocking.

**Methods and numbers**

- **BSD68, Gaussian noise:** PSNR at σ = 15 / 25 / 50: BM3D 31.07 / 28.57 / 25.62 dB, DnCNN-S 31.73 / 29.23 / 26.23 dB
- **FFDNet (2018):** takes a noise-level map as an extra input, so one network covers a range of σ and spatially varying noise
- **JPEG and deblurring:** ARCNN (Dong et al., ICCV 2015) for JPEG artefacts; DeepDeblur (Nah et al., CVPR 2017) and its GoPro dataset of real motion blur; DeblurGAN (Kupyn et al., CVPR 2018)
- **Transformers:** SwinIR (Liang et al., ICCVW 2021): **Swin Transformer** blocks with shifted windows. Restormer (Zamir et al., CVPR 2022): attention across channels instead of pixels, so cost grows linearly with image size
- **Real noise:** SIDD (Abdelhamed et al., CVPR 2018): smartphone images paired with low-noise ground truth; Noise2Noise (Lehtinen et al., ICML 2018) trains on pairs of independent noisy shots, no clean target needed

> Figure: A 128-sample scanline blurred (σ_b = 2) and corrupted with noise (σ_n = 0.02), then restored in the Fourier domain. The observation scores 20.00 dB. The inverse filter divides by |H|, whose smallest value gives a gain of 3.9e5, and collapses to −63.37 dB. Tikhonov with λ = 10⁻² gives 20.12 dB; the best λ on the grid, 10⁻¹·⁵, reaches 20.52 dB; λ = 10⁻⁴ under-regularises and falls to 9.16 dB. Set the noise to 0 and the inverse filter recovers x almost exactly (over 99 dB): blur alone is invertible, the noise is what makes the problem ill-posed.

## Worked example: gain per frequency

Take the figure's default blur ($\sigma_b=2$ samples, $N=128$) and look at a few frequencies $\omega=2\pi k/128$. The inverse filter applies gain $1/H$; Tikhonov with $\lambda=0.01$ applies $G=H/(H^2+\lambda)$. The product $GH$ is how much of the true signal survives.

Default blur of the figure; noise at each frequency is multiplied by the gain

| k (ω) | H(ω) | inverse gain 1/H | Tikhonov gain G | signal kept G·H |
| --- | --- | --- | --- | --- |
| 8 (0.125π) | 0.7346 | 1.36 | 1.336 | 0.982 |
| 16 (0.25π) | 0.2912 | 3.43 | 3.072 | 0.895 |
| 24 (0.375π) | 0.0623 | 16.1 | 4.488 | 0.280 |
| 32 (0.5π) | 0.0072 | 139 | 0.716 | 0.005 |
| 40 (0.625π) | 0.0004 | 2268 | 0.044 | 0.000 |

Where $H^2\gg\lambda$, Tikhonov matches the inverse and restores the signal. Where $H^2\ll\lambda$, it switches the frequency off, giving up detail rather than amplifying noise by 139× or 2268×. A learned restorer does better than any such linear filter, because its prior is non-linear and depends on content: it can keep a sharp edge and still smooth a flat region.

## Beyond feed-forward CNNs

Learned priors can also be plugged into the optimisation above. **Plug-and-play** methods alternate a data-fidelity step on $\lVert y-Hx\rVert^2$ with a call to a pretrained denoiser, which acts as the prior. Any $H$ can then be handled without retraining (for example DPIR, Zhang et al., 2021).

**Diffusion models** take this further. A pretrained unconditional diffusion prior is guided during sampling to agree with $y$: DDRM (Kawar et al., NeurIPS 2022) for linear $H$ and DPS (Chung et al., ICLR 2023) for general, even non-linear, operators. They produce sharp, realistic results under heavy degradation, but slowly and with the same risk as GAN super-resolution (see **Image super-resolution**): plausible detail that is not the true detail.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class DnCNN(nn.Module):
    def __init__(self, depth=17, ch=64, c=1):
        super().__init__()
        layers = [nn.Conv2d(c, ch, 3, padding=1), nn.ReLU(inplace=True)]
        for _ in range(depth - 2):
            layers += [nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch), nn.ReLU(inplace=True)]
        layers += [nn.Conv2d(ch, c, 3, padding=1)]
        self.net = nn.Sequential(*layers)

    def forward(self, y):
        return y - self.net(y)                  # the network predicts the noise v; x_hat = y - v_hat

model = DnCNN()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
x = torch.rand(128, 1, 40, 40)                   # clean 40x40 patches in [0, 1]
y = x + (25 / 255) * torch.randn_like(x)         # sigma = 25 on the 0-255 scale
loss = 0.5 * F.mse_loss(model.net(y), y - x, reduction="sum") / x.shape[0]   # the DnCNN residual loss
opt.zero_grad(); loss.backward(); opt.step()
psnr = 10 * torch.log10(1 / F.mse_loss(model(y).clamp(0, 1), x))
```

DnCNN with residual learning, and one training step on synthetic Gaussian noise (PyTorch; not run here)

> **Synthetic noise does not transfer:** A denoiser trained on additive white Gaussian noise often fails on real photos, whose noise is signal-dependent, spatially correlated and altered by the camera pipeline: it leaves blotches or smears texture. Train on real pairs (SIDD), on noise synthesised with a camera model, or use a non-blind network such as FFDNet with an estimated noise map.

**Strengths**

- Learned priors beat hand-crafted ones: DnCNN gains about 0.6 dB over BM3D on BSD68 at every tested σ
- One fully convolutional network runs on any image size and is fast on a GPU
- Plug-and-play and diffusion methods reuse one prior for many operators H

**Limitations**

- A feed-forward restorer is tied to the degradations it was trained on
- MSE-trained outputs over-smooth fine texture; GAN or diffusion outputs can invent detail
- Blind real-world restoration still needs realistic training degradations or paired data

**Question:** Why does DnCNN output the noise v = y − x instead of the clean image x?

- A. The noise has fewer pixels than the image
- B. With a residual target the network models a nearly structureless, zero-mean signal, which together with batch norm trains faster and denoises better
- C. It makes the network blind to the noise level
- D. Predicting x would need pooling layers

**Answer:** B. With a residual target the network models a nearly structureless, zero-mean signal, which together with batch norm trains faster and denoises better — For moderate noise, $y$ is already close to $x$, so learning $x$ directly means mostly learning an identity mapping. The residual is small and much simpler. The paper reports that residual learning and batch normalisation help each other.

**Question:** In the figure, the inverse filter recovers the signal almost exactly when σ_n = 0 but collapses with σ_n = 0.02. Why?

- A. The blur kernel changes when noise is added
- B. Noise at high frequencies is divided by |H|, which is tiny there, so it is amplified by up to 3.9e5
- C. PSNR is undefined for noisy signals
- D. The inverse filter ignores low frequencies

**Answer:** B. Noise at high frequencies is divided by |H|, which is tiny there, so it is amplified by up to 3.9e5 — $\hat X=Y/H=X+N/H$. Without noise the second term vanishes. With noise, the frequencies where the blur almost removed the signal now carry noise scaled by $1/|H|$. Regularisation ($|H|^2+\lambda$ in the denominator) or a learned prior prevents this.

**Key points**

- Degradation model $y=Hx+n$: $H=I$ (denoise), blur (deblur), ↓ (super-resolution), mask (inpaint); JPEG is non-linear quantisation.
- Non-blind: H and σ known; blind: they must be estimated too. Real noise is signal-dependent and correlated.
- Inverting $H$ is ill-posed: $1/|H|$ amplifies noise. Regularise: $\hat X=\bar H Y/(|H|^2+\lambda)$ (Tikhonov/Wiener) or use a prior.
- DnCNN (2017): 17 conv layers with BN, predicts the residual noise, x̂ = y − R(y); beats BM3D by about 0.6 dB on BSD68.
- Transformers (SwinIR, Restormer) and diffusion priors (DDRM, DPS) are the newer generation; diffusion can hallucinate detail.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
