Definition
Image inpainting takes an image and a binary mask ( = known, = missing) and produces that equals where and is plausible and consistent with the surroundings where . The first deep model was the Context Encoder (Pathak, Krähenbühl, Donahue, Darrell and Efros, CVPR 2016). Later milestones are partial convolutions (Liu et al., ECCV 2018), gated convolutions (Yu et al., ICCV 2019, "DeepFill v2"), LaMa with Fourier convolutions (Suvorov et al., WACV 2022) and diffusion inpainting such as RePaint (Lugmayr et al., CVPR 2022).
Uses include object removal, restoring damaged photos, filling borders after rotation, and editing a region from a text prompt. The task is ill-posed: a hole over a doorway could hold a door, a window or a wall. A good fill continues edges and textures across the boundary and matches colour so no seam shows. It does not have to match the original pixels.
Classical methods either propagate colours inward with PDEs (Bertalmio et al., SIGGRAPH 2000; Telea's fast marching, 2004, in OpenCV) or copy patches from elsewhere in the image (Criminisi, Pérez and Toyama, 2004; PatchMatch, Barnes et al., 2009). Patch copying makes sharp textures but cannot invent content that appears nowhere else, such as the missing half of a face.
Learned inpainting: encoder–decoder plus a GAN
The Context Encoder dropped the central block of a image and trained an encoder–decoder to predict it. A channel-wise fully connected bottleneck let every output pixel depend on the whole image. L2 alone gives blurry fills, because the mean of many plausible answers is blurry, so an adversarial term was added:
- the reconstruction loss is measured only inside the hole
- the encoder–decoder generator; its input has the hole blanked out
- GAN loss that pushes fills towards real-looking texture (see Generative adversarial network (GAN))
Iizuka, Simo-Serra and Ishikawa (SIGGRAPH 2017) added dilated convolutions for context and two discriminators: a global one on the whole image and a local one on a patch around the hole. Yu et al. (CVPR 2018, "DeepFill v1") added contextual attention, which lets hole features copy from matching background patches. All of these used ordinary convolutions on an image whose hole was filled with a constant.
Why plain convolutions leak, and two fixes
A convolution cannot tell a real black pixel from a hole filled with . Windows on the hole boundary average real pixels with the placeholder, so the output depends on an arbitrary value. Stacked layers spread the error outward, and the results show colour shifts, blur and seams. Liu et al. fixed this with partial convolutions for irregular holes:
- the feature values and mask values inside the current sliding window
- rescales by the fraction of valid inputs, so the output does not shrink when part of the window is a hole (9/6 when 6 of 9 pixels are known)
- mask update: a location becomes valid once any input in its window was valid, so the hole shrinks layer by layer
The mask update is a hard rule. Once one pixel in a window is valid, the location counts as fully valid, and every channel shares the same mask. Gated convolution replaces the rule with a learned, soft, per-channel gate:
- layer input; in the first layer it includes the mask and an optional user sketch channel
- a learned soft validity value for every channel and position
- activation such as ELU or LeakyReLU
Key systems and their settings
- Context Encoder (2016)
- 128×128 input, central 64×64 hole; L2 + adversarial with weights 0.999 / 0.001
- Partial conv (2018)
- U-Net-style network on 512×512 images with irregular masks. Loss: L_valid + 6·L_hole + 0.05·L_perceptual + 120·L_style + 0.1·L_tv (see Perceptual loss)
- Gated conv (2019)
- free-form masks drawn as random brush strokes; SN-PatchGAN discriminator; accepts a user sketch as an extra input channel
- LaMa (2022)
- fast Fourier convolutions (Chi, Jiang and Mu, NeurIPS 2020), very large masks during training, high-receptive-field perceptual loss; trained on 256×256 crops and works at much higher resolution
- Stable Diffusion inpainting
- the U-Net takes 9 input channels: 4 noisy latent + 4 latent of the masked image + 1 down-sampled mask (see Stable Diffusion)
Large masks, global context and diffusion
Real edits often remove a whole person, so masks can cover half the image. A layer's receptive field grows by only 2 pixels per layer, so the hole centre gets no context until late in the network. LaMa's fast Fourier convolution applies a convolution to the 2D FFT of the feature map. Since every frequency depends on every pixel, each output sees the whole image in one layer. That also suits periodic structure like bricks, fences and windows. LaMa trains on wide, aggressive masks and uses a high-receptive-field perceptual loss built on a dilated segmentation network.
Diffusion models handle the many-plausible-answers problem by sampling (see Diffusion models). RePaint uses an unconditional DDPM and, at every reverse step, replaces the known region with a noised copy of the original:
- the model's ordinary reverse-step sample
- the input image; only its known pixels are used
- RePaint also jumps back to a noisier step and re-denoises (resampling), which harmonises the boundary
Text-guided editors instead fine-tune a latent diffusion model with the mask and masked image as extra inputs, as in the 9-channel Stable Diffusion inpainting model. They are much slower than LaMa's single forward pass but can create new content from a prompt.
Worked example: one 3×3 window on the hole boundary
The kernel is a mean ( everywhere, ). Six pixels in the window are known with value and three lie in the hole.
| Layer | Computation | Output | New mask |
|---|---|---|---|
| Vanilla, placeholder 0 | 6 × 0.8 / 9 | 0.533 | (none) |
| Vanilla, placeholder 1 | (6 × 0.8 + 3 × 1) / 9 | 0.867 | (none) |
| Partial conv | (6 × 0.8 / 9) × 9/6 | 0.800 | 1 |
| Partial conv, window fully in hole | sum(M) = 0 | 0 | 0 |
The vanilla output moves by when only the placeholder changes. The partial conv returns the mean of the valid pixels, , whatever the hole holds.
1import torch, torch.nn as nn, torch.nn.functional as F23class PartialConv2d(nn.Conv2d):4 """Convolve only valid pixels, rescale by sum(1)/sum(M), return the updated mask."""5 def forward(self, x, m): # x: (B,C,H,W); m: (B,1,H,W), 1 = known6 ones = torch.ones(1, 1, *self.kernel_size, device=x.device)7 with torch.no_grad():8 cnt = F.conv2d(m, ones, stride=self.stride, padding=self.padding) # sum(M)9 valid = (cnt > 0).float() # m'10 scale = self.kernel_size[0] * self.kernel_size[1] / cnt.clamp(min=1)11 out = F.conv2d(x * m, self.weight, None, self.stride, self.padding) * scale * valid12 if self.bias is not None:13 out = out + self.bias.view(1, -1, 1, 1) * valid14 return out, valid1516layer = PartialConv2d(3, 64, kernel_size=7, stride=2, padding=3)17img, mask = torch.rand(1, 3, 512, 512), torch.ones(1, 1, 512, 512)18mask[..., 200:300, 150:350] = 0 # a rectangular hole19feat, mask1 = layer(img * mask, mask) # (1,64,256,256) and the shrunken holeUse when
- +Feed-forward models (partial/gated conv, LaMa) fill a hole in one pass and keep known pixels exactly through compositing
- +Mask-aware layers make the output independent of the placeholder value
- +Diffusion inpainting samples diverse fills and can follow a text prompt
Avoid when
- −With local receptive fields, the centre of a large hole gets little context
- −GAN-trained models give one deterministic fill and can show repeated textures or seams
- −Diffusion inpainting needs tens of denoising steps, and boundary consistency needs extra care (resampling, mask-conditioned fine-tuning)
Check yourself
A partial convolution with a 3×3 mean kernel sees a window with 3 valid pixels of value 0.6 and 6 hole pixels. What does it output, and what is the new mask value?
Check yourself
Why does LaMa use fast Fourier convolutions?
Remember
- ✓Inpainting predicts the missing region and composites: .
- ✓Context Encoders (2016) used L2 + adversarial loss (0.999 / 0.001) on a central 64×64 hole. GAN losses avoid blurry averages.
- ✓Plain convolutions read the placeholder as data. Partial conv rescales by and updates the mask; gated conv learns a soft mask.
- ✓LaMa uses Fourier convolutions and very large training masks for an image-wide receptive field.
- ✓Diffusion inpainting (RePaint, 9-channel Stable Diffusion) samples many plausible fills. Evaluate with FID/LPIPS and a stated mask distribution, not only PSNR.