# Image inpainting

> Inpainting fills a masked region plausibly. Partial and gated convolutions stop networks reading hole pixels as data, LaMa's Fourier convolutions give image-wide context for large masks, and diffusion models sample many plausible fills.

Source: https://www.mlfoundry.in/learn/image-inpainting  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Image inpainting:** takes an image $x$ and a binary mask $m$ ($1$ = known, $0$ = missing) and produces $\hat x$ that equals $x$ where $m=1$ and is plausible and consistent with the surroundings where $m=0$. The first deep model was the **Context Encoder** (Pathak, Krähenbühl, Donahue, Darrell and Efros, CVPR 2016). Later milestones are **partial convolutions** (Liu et al., ECCV 2018), **gated convolutions** (Yu et al., ICCV 2019, "DeepFill v2"), **LaMa** with Fourier convolutions (Suvorov et al., WACV 2022) and diffusion inpainting such as RePaint (Lugmayr et al., CVPR 2022).

Uses include object removal, restoring damaged photos, filling borders after rotation, and editing a region from a text prompt. The task is ill-posed: a hole over a doorway could hold a door, a window or a wall. A good fill continues edges and textures across the boundary and matches colour so no seam shows. It does not have to match the original pixels.

Classical methods either propagate colours inward with PDEs (Bertalmio et al., SIGGRAPH 2000; Telea's fast marching, 2004, in OpenCV) or copy patches from elsewhere in the image (Criminisi, Pérez and Toyama, 2004; PatchMatch, Barnes et al., 2009). Patch copying makes sharp textures but cannot invent content that appears nowhere else, such as the missing half of a face.

## Learned inpainting: encoder–decoder plus a GAN

The Context Encoder dropped the central $64\times64$ block of a $128\times128$ image and trained an encoder–decoder to predict it. A channel-wise fully connected bottleneck let every output pixel depend on the whole image. L2 alone gives blurry fills, because the mean of many plausible answers is blurry, so an adversarial term was added:

$$
\mathcal L=\lambda_{\text{rec}}\big\|(1-m)\odot\big(x-G(m\odot x)\big)\big\|_2^2+\lambda_{\text{adv}}\,\mathcal L_{\text{adv}},\qquad \lambda_{\text{rec}}=0.999,\ \lambda_{\text{adv}}=0.001
$$

- $(1-m)\odot$: the reconstruction loss is measured only inside the hole
- $G$: the encoder–decoder generator; its input has the hole blanked out
- $\mathcal L_{\text{adv}}$: GAN loss that pushes fills towards real-looking texture (see **Generative adversarial network (GAN)**)

Iizuka, Simo-Serra and Ishikawa (SIGGRAPH 2017) added dilated convolutions for context and two discriminators: a global one on the whole image and a local one on a patch around the hole. Yu et al. (CVPR 2018, "DeepFill v1") added **contextual attention**, which lets hole features copy from matching background patches. All of these used ordinary convolutions on an image whose hole was filled with a constant.

> Figure: Top: the masked image and the mask are concatenated into 4 channels. An encoder, a context stage and a decoder predict ŷ, and only the hole is pasted back: x̂ = m⊙x + (1−m)⊙ŷ. Bottom: four ways a layer can handle the hole. A vanilla conv reads the placeholder as data. A partial conv averages only valid inputs and grows the mask. A gated conv learns a soft mask. LaMa's Fourier conv mixes the whole image in one layer.

## Why plain convolutions leak, and two fixes

A convolution cannot tell a real black pixel from a hole filled with $0$. Windows on the hole boundary average real pixels with the placeholder, so the output depends on an arbitrary value. Stacked layers spread the error outward, and the results show colour shifts, blur and seams. Liu et al. fixed this with **partial convolutions** for irregular holes:

$$
x'=\begin{cases}W^\top(X\odot M)\,\dfrac{\operatorname{sum}(\mathbf 1)}{\operatorname{sum}(M)}+b, & \operatorname{sum}(M)>0\\[4pt] 0, & \text{otherwise}\end{cases}\qquad m'=\mathbb 1\big[\operatorname{sum}(M)>0\big]
$$

- $X,\ M$: the feature values and mask values inside the current sliding window
- $\operatorname{sum}(\mathbf 1)/\operatorname{sum}(M)$: rescales by the fraction of valid inputs, so the output does not shrink when part of the window is a hole (9/6 when 6 of 9 pixels are known)
- $m'$: mask update: a location becomes valid once any input in its window was valid, so the hole shrinks layer by layer

The mask update is a hard rule. Once one pixel in a window is valid, the location counts as fully valid, and every channel shares the same mask. **Gated convolution** replaces the rule with a learned, soft, per-channel gate:

$$
\text{Gating}=\sum W_g\cdot I,\qquad \text{Feature}=\sum W_f\cdot I,\qquad O=\phi(\text{Feature})\odot\sigma(\text{Gating})
$$

- $I$: layer input; in the first layer it includes the mask and an optional user sketch channel
- $\sigma(\text{Gating})\in(0,1)$: a learned soft validity value for every channel and position
- $\phi$: activation such as ELU or LeakyReLU

**Key systems and their settings**

- **Context Encoder (2016):** 128×128 input, central 64×64 hole; L2 + adversarial with weights 0.999 / 0.001
- **Partial conv (2018):** U-Net-style network on 512×512 images with irregular masks. Loss: L_valid + 6·L_hole + 0.05·L_perceptual + 120·L_style + 0.1·L_tv (see **Perceptual loss**)
- **Gated conv (2019):** free-form masks drawn as random brush strokes; SN-PatchGAN discriminator; accepts a user sketch as an extra input channel
- **LaMa (2022):** fast Fourier convolutions (Chi, Jiang and Mu, NeurIPS 2020), very large masks during training, high-receptive-field perceptual loss; trained on 256×256 crops and works at much higher resolution
- **Stable Diffusion inpainting:** the U-Net takes 9 input channels: 4 noisy latent + 4 latent of the masked image + 1 down-sampled mask (see **Stable Diffusion**)

> Figure: A 20×20 ramp with a dark bar and a 6×6 hole, passed through L stacked 3×3 mean filters. Errors are measured against the same filters applied to the image without a hole. At the defaults (L = 2, placeholder 0), the naive conv has hole MAE 0.271 and changes 60 known pixels by more than 0.01. The partial conv has MAE 0.041 on the 32 hole pixels it has filled; 4 of 36 are still empty, and only 8 known pixels change. At L = 3 the hole is gone. Placeholder 0.5 lowers the naive error to 0.084, and placeholder 1 raises it to 0.416. The partial column never changes with the placeholder.

## Large masks, global context and diffusion

Real edits often remove a whole person, so masks can cover half the image. A $3\times3$ layer's receptive field grows by only 2 pixels per layer, so the hole centre gets no context until late in the network. LaMa's **fast Fourier convolution** applies a $1\times1$ convolution to the 2D FFT of the feature map. Since every frequency depends on every pixel, each output sees the whole image in one layer. That also suits periodic structure like bricks, fences and windows. LaMa trains on wide, aggressive masks and uses a high-receptive-field perceptual loss built on a dilated segmentation network.

Diffusion models handle the many-plausible-answers problem by sampling (see **Diffusion models**). RePaint uses an unconditional DDPM and, at every reverse step, replaces the known region with a noised copy of the original:

$$
x_{t-1}=m\odot x_{t-1}^{\text{known}}+(1-m)\odot x_{t-1}^{\text{unknown}},\qquad x_{t-1}^{\text{known}}\sim\mathcal N\big(\sqrt{\bar\alpha_{t-1}}\,x_0,\ (1-\bar\alpha_{t-1})I\big)
$$

- $x_{t-1}^{\text{unknown}}$: the model's ordinary reverse-step sample
- $x_0$: the input image; only its known pixels are used
- $m$: RePaint also jumps back to a noisier step and re-denoises (resampling), which harmonises the boundary

Text-guided editors instead fine-tune a latent diffusion model with the mask and masked image as extra inputs, as in the 9-channel Stable Diffusion inpainting model. They are much slower than LaMa's single forward pass but can create new content from a prompt.

## Worked example: one 3×3 window on the hole boundary

The kernel is a $3\times3$ mean ($W=1/9$ everywhere, $b=0$). Six pixels in the window are known with value $0.8$ and three lie in the hole.

Output of one window (computed in Python)

| Layer | Computation | Output | New mask |
| --- | --- | --- | --- |
| Vanilla, placeholder 0 | 6 × 0.8 / 9 | 0.533 | (none) |
| Vanilla, placeholder 1 | (6 × 0.8 + 3 × 1) / 9 | 0.867 | (none) |
| Partial conv | (6 × 0.8 / 9) × 9/6 | 0.800 | 1 |
| Partial conv, window fully in hole | sum(M) = 0 | 0 | 0 |

The vanilla output moves by $0.33$ when only the placeholder changes. The partial conv returns the mean of the valid pixels, $0.8$, whatever the hole holds.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class PartialConv2d(nn.Conv2d):
    """Convolve only valid pixels, rescale by sum(1)/sum(M), return the updated mask."""
    def forward(self, x, m):                       # x: (B,C,H,W); m: (B,1,H,W), 1 = known
        ones = torch.ones(1, 1, *self.kernel_size, device=x.device)
        with torch.no_grad():
            cnt = F.conv2d(m, ones, stride=self.stride, padding=self.padding)   # sum(M)
        valid = (cnt > 0).float()                  # m'
        scale = self.kernel_size[0] * self.kernel_size[1] / cnt.clamp(min=1)
        out = F.conv2d(x * m, self.weight, None, self.stride, self.padding) * scale * valid
        if self.bias is not None:
            out = out + self.bias.view(1, -1, 1, 1) * valid
        return out, valid

layer = PartialConv2d(3, 64, kernel_size=7, stride=2, padding=3)
img, mask = torch.rand(1, 3, 512, 512), torch.ones(1, 1, 512, 512)
mask[..., 200:300, 150:350] = 0                    # a rectangular hole
feat, mask1 = layer(img * mask, mask)              # (1,64,256,256) and the shrunken hole
```

A partial convolution layer in PyTorch (not run here: no torch in the scratch environment)

> **Pixel metrics punish valid answers:** PSNR, SSIM and L1 against the original reward the blurry average of all plausible fills. A sharp door where the original had a window scores worse than a grey smear. Papers on large masks therefore report FID and LPIPS, and user studies. Also state the mask distribution: results on small central squares say little about free-form masks covering 40% of the image.

> **Train on the masks you will meet:** Mask shape is part of the training distribution. A model trained only on central squares learns where the hole will be and does poorly on free-form holes. Partial and gated convolution introduced irregular and brush-stroke masks, and LaMa made very large, wide masks one of its three main ingredients, alongside Fourier convolutions and the high-receptive-field loss.

**Strengths**

- Feed-forward models (partial/gated conv, LaMa) fill a hole in one pass and keep known pixels exactly through compositing
- Mask-aware layers make the output independent of the placeholder value
- Diffusion inpainting samples diverse fills and can follow a text prompt

**Limitations**

- With local receptive fields, the centre of a large hole gets little context
- GAN-trained models give one deterministic fill and can show repeated textures or seams
- Diffusion inpainting needs tens of denoising steps, and boundary consistency needs extra care (resampling, mask-conditioned fine-tuning)

**Question:** A partial convolution with a 3×3 mean kernel sees a window with 3 valid pixels of value 0.6 and 6 hole pixels. What does it output, and what is the new mask value?

- A. 0.2 and 0
- B. 0.6 and 1
- C. 0.2 and 1
- D. 0 and 0

**Answer:** B. 0.6 and 1 — Vanilla output would be $3\times0.6/9=0.2$. The renormalisation multiplies by $9/3$, giving $0.6$, the mean of the valid inputs. Since $\operatorname{sum}(M)=3>0$, the location becomes valid: $m'=1$.

**Question:** Why does LaMa use fast Fourier convolutions?

- A. They are cheaper than 3×3 convolutions at every resolution
- B. Every output depends on the whole image after one layer, which large holes need
- C. They remove the need for a mask input
- D. They make the generator invertible

**Answer:** B. Every output depends on the whole image after one layer, which large holes need — A 1×1 convolution on the FFT mixes all spatial positions, giving an image-wide receptive field in the first layers. With stacked 3×3 layers the centre of a large hole gets no context until deep in the network.

**Key points**

- Inpainting predicts the missing region and composites: $\hat x=m\odot x+(1-m)\odot G(m\odot x,m)$.
- Context Encoders (2016) used L2 + adversarial loss (0.999 / 0.001) on a central 64×64 hole. GAN losses avoid blurry averages.
- Plain convolutions read the placeholder as data. Partial conv rescales by $\operatorname{sum}(\mathbf 1)/\operatorname{sum}(M)$ and updates the mask; gated conv learns a soft mask.
- LaMa uses Fourier convolutions and very large training masks for an image-wide receptive field.
- Diffusion inpainting (RePaint, 9-channel Stable Diffusion) samples many plausible fills. Evaluate with FID/LPIPS and a stated mask distribution, not only PSNR.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
