Image inpainting

Inpainting fills a masked region plausibly. Partial and gated convolutions stop networks reading hole pixels as data, LaMa's Fourier convolutions give image-wide context for large masks, and diffusion models sample many plausible fills.

8 min read859 wordscore0 viewsupdated 2026-10-15

Definition

Image inpainting takes an image xx and a binary mask mm (11 = known, 00 = missing) and produces x^\hat x that equals xx where m=1m=1 and is plausible and consistent with the surroundings where m=0m=0. The first deep model was the Context Encoder (Pathak, Krähenbühl, Donahue, Darrell and Efros, CVPR 2016). Later milestones are partial convolutions (Liu et al., ECCV 2018), gated convolutions (Yu et al., ICCV 2019, "DeepFill v2"), LaMa with Fourier convolutions (Suvorov et al., WACV 2022) and diffusion inpainting such as RePaint (Lugmayr et al., CVPR 2022).

Uses include object removal, restoring damaged photos, filling borders after rotation, and editing a region from a text prompt. The task is ill-posed: a hole over a doorway could hold a door, a window or a wall. A good fill continues edges and textures across the boundary and matches colour so no seam shows. It does not have to match the original pixels.

Classical methods either propagate colours inward with PDEs (Bertalmio et al., SIGGRAPH 2000; Telea's fast marching, 2004, in OpenCV) or copy patches from elsewhere in the image (Criminisi, Pérez and Toyama, 2004; PatchMatch, Barnes et al., 2009). Patch copying makes sharp textures but cannot invent content that appears nowhere else, such as the missing half of a face.

Learned inpainting: encoder–decoder plus a GAN

The Context Encoder dropped the central 64×6464\times64 block of a 128×128128\times128 image and trained an encoder–decoder to predict it. A channel-wise fully connected bottleneck let every output pixel depend on the whole image. L2 alone gives blurry fills, because the mean of many plausible answers is blurry, so an adversarial term was added:

L=λrec∥(1−m)⊙(x−G(m⊙x))∥22+λadv Ladv,λrec=0.999, λadv=0.001\mathcal L=\lambda_{\text{rec}}\big\|(1-m)\odot\big(x-G(m\odot x)\big)\big\|_2^2+\lambda_{\text{adv}}\,\mathcal L_{\text{adv}},\qquad \lambda_{\text{rec}}=0.999,\ \lambda_{\text{adv}}=0.001
(1−m)⊙(1-m)\odot
the reconstruction loss is measured only inside the hole
GG
the encoder–decoder generator; its input has the hole blanked out
Ladv\mathcal L_{\text{adv}}
GAN loss that pushes fills towards real-looking texture (see Generative adversarial network (GAN))

Iizuka, Simo-Serra and Ishikawa (SIGGRAPH 2017) added dilated convolutions for context and two discriminators: a global one on the whole image and a local one on a patch around the hole. Yu et al. (CVPR 2018, "DeepFill v1") added contextual attention, which lets hole features copy from matching background patches. All of these used ordinary convolutions on an image whose hole was filled with a constant.

inpainting: predict the hole from the known pixels, then paste backx⊙m, 512×512×3mask m1 = knownconcat → 4 channelsenccontextdeccontext: dilated convs,contextual attention, FFCŷ = G(x⊙m, m)x̂ = m⊙x+ (1−m)⊙ŷkeep knowntraining: random masks on full images · losses: L1 (hole, valid) + perceptual/style + adversarialhow one layer treats the holevanilla convy = Wᵀx + bhole placeholder (0 ormean) is read as data→ colour shift, blur, visible seamsContext Encoders 2016,Iizuka et al. 2017partial conv (2018)y = Wᵀ(x⊙m)·9/Σm + b if Σm > 0, else 0m′ = 1[Σm > 0]hole shrinks 1 px perside per 3×3 layerhard, rule-based maskLiu et al., ECCV 2018gated conv (2019)g = σ(W_g ∗ x)f = φ(W_f ∗ x)y = f ⊙ glearned soft mask perchannel and pixel;mask, sketch as inputDeepFill v2, ICCV 2019Fourier conv (LaMa)FFT → 1×1 conv on thespectrum → inverse FFTeach output sees thewhole image at layer 1suits large masks andperiodic texturesSuvorov et al. 2022
fig.Top: the masked image and the mask are concatenated into 4 channels. An encoder, a context stage and a decoder predict ŷ, and only the hole is pasted back: x̂ = m⊙x + (1−m)⊙ŷ. Bottom: four ways a layer can handle the hole. A vanilla conv reads the placeholder as data. A partial conv averages only valid inputs and grows the mask. A gated conv learns a soft mask. LaMa's Fourier conv mixes the whole image in one layer.

Why plain convolutions leak, and two fixes

A convolution cannot tell a real black pixel from a hole filled with 00. Windows on the hole boundary average real pixels with the placeholder, so the output depends on an arbitrary value. Stacked layers spread the error outward, and the results show colour shifts, blur and seams. Liu et al. fixed this with partial convolutions for irregular holes:

x′={W⊤(X⊙M) sum⁡(1)sum⁡(M)+b,sum⁡(M)>00,otherwisem′=1[sum⁡(M)>0]x'=\begin{cases}W^\top(X\odot M)\,\dfrac{\operatorname{sum}(\mathbf 1)}{\operatorname{sum}(M)}+b, & \operatorname{sum}(M)>0\\[4pt] 0, & \text{otherwise}\end{cases}\qquad m'=\mathbb 1\big[\operatorname{sum}(M)>0\big]
X, MX,\ M
the feature values and mask values inside the current sliding window
sum⁡(1)/sum⁡(M)\operatorname{sum}(\mathbf 1)/\operatorname{sum}(M)
rescales by the fraction of valid inputs, so the output does not shrink when part of the window is a hole (9/6 when 6 of 9 pixels are known)
m′m'
mask update: a location becomes valid once any input in its window was valid, so the hole shrinks layer by layer

The mask update is a hard rule. Once one pixel in a window is valid, the location counts as fully valid, and every channel shares the same mask. Gated convolution replaces the rule with a learned, soft, per-channel gate:

Gating=∑Wg⋅I,Feature=∑Wf⋅I,O=ϕ(Feature)⊙σ(Gating)\text{Gating}=\sum W_g\cdot I,\qquad \text{Feature}=\sum W_f\cdot I,\qquad O=\phi(\text{Feature})\odot\sigma(\text{Gating})
II
layer input; in the first layer it includes the mask and an optional user sketch channel
σ(Gating)∈(0,1)\sigma(\text{Gating})\in(0,1)
a learned soft validity value for every channel and position
ϕ\phi
activation such as ELU or LeakyReLU

Key systems and their settings

Context Encoder (2016)
128×128 input, central 64×64 hole; L2 + adversarial with weights 0.999 / 0.001
Partial conv (2018)
U-Net-style network on 512×512 images with irregular masks. Loss: L_valid + 6·L_hole + 0.05·L_perceptual + 120·L_style + 0.1·L_tv (see Perceptual loss)
Gated conv (2019)
free-form masks drawn as random brush strokes; SN-PatchGAN discriminator; accepts a user sketch as an extra input channel
LaMa (2022)
fast Fourier convolutions (Chi, Jiang and Mu, NeurIPS 2020), very large masks during training, high-receptive-field perceptual loss; trained on 256×256 crops and works at much higher resolution
Stable Diffusion inpainting
the U-Net takes 9 input channels: 4 noisy latent + 4 latent of the masked image + 1 down-sampled mask (see Stable Diffusion)
2 stacked 3×3 mean layers · hole 6×6 · placeholder 0input x⊙m (+ placeholder)naive convpartial convdashed = hole · • = known pixel changed by the hole · pink = mask still 0naivepartialhole MAE (filled pixels) vs no-hole0.2710.041known pixels moved > 0.01 (•)608hole pixels not yet fillednever tracked4 of 36
hole placeholder
the partial-conv column never changes with the placeholder: hole values are multiplied by m = 0
fig.A 20×20 ramp with a dark bar and a 6×6 hole, passed through L stacked 3×3 mean filters. Errors are measured against the same filters applied to the image without a hole. At the defaults (L = 2, placeholder 0), the naive conv has hole MAE 0.271 and changes 60 known pixels by more than 0.01. The partial conv has MAE 0.041 on the 32 hole pixels it has filled; 4 of 36 are still empty, and only 8 known pixels change. At L = 3 the hole is gone. Placeholder 0.5 lowers the naive error to 0.084, and placeholder 1 raises it to 0.416. The partial column never changes with the placeholder.

Large masks, global context and diffusion

Real edits often remove a whole person, so masks can cover half the image. A 3×33\times3 layer's receptive field grows by only 2 pixels per layer, so the hole centre gets no context until late in the network. LaMa's fast Fourier convolution applies a 1×11\times1 convolution to the 2D FFT of the feature map. Since every frequency depends on every pixel, each output sees the whole image in one layer. That also suits periodic structure like bricks, fences and windows. LaMa trains on wide, aggressive masks and uses a high-receptive-field perceptual loss built on a dilated segmentation network.

Diffusion models handle the many-plausible-answers problem by sampling (see Diffusion models). RePaint uses an unconditional DDPM and, at every reverse step, replaces the known region with a noised copy of the original:

xt−1=m⊙xt−1known+(1−m)⊙xt−1unknown,xt−1known∼N(αˉt−1 x0, (1−αˉt−1)I)x_{t-1}=m\odot x_{t-1}^{\text{known}}+(1-m)\odot x_{t-1}^{\text{unknown}},\qquad x_{t-1}^{\text{known}}\sim\mathcal N\big(\sqrt{\bar\alpha_{t-1}}\,x_0,\ (1-\bar\alpha_{t-1})I\big)
xt−1unknownx_{t-1}^{\text{unknown}}
the model's ordinary reverse-step sample
x0x_0
the input image; only its known pixels are used
mm
RePaint also jumps back to a noisier step and re-denoises (resampling), which harmonises the boundary

Text-guided editors instead fine-tune a latent diffusion model with the mask and masked image as extra inputs, as in the 9-channel Stable Diffusion inpainting model. They are much slower than LaMa's single forward pass but can create new content from a prompt.

Worked example: one 3×3 window on the hole boundary

The kernel is a 3×33\times3 mean (W=1/9W=1/9 everywhere, b=0b=0). Six pixels in the window are known with value 0.80.8 and three lie in the hole.

LayerComputationOutputNew mask
Vanilla, placeholder 06 × 0.8 / 90.533(none)
Vanilla, placeholder 1(6 × 0.8 + 3 × 1) / 90.867(none)
Partial conv(6 × 0.8 / 9) × 9/60.8001
Partial conv, window fully in holesum(M) = 000
Output of one window (computed in Python)

The vanilla output moves by 0.330.33 when only the placeholder changes. The partial conv returns the mean of the valid pixels, 0.80.8, whatever the hole holds.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class PartialConv2d(nn.Conv2d):
4 """Convolve only valid pixels, rescale by sum(1)/sum(M), return the updated mask."""
5 def forward(self, x, m): # x: (B,C,H,W); m: (B,1,H,W), 1 = known
6 ones = torch.ones(1, 1, *self.kernel_size, device=x.device)
7 with torch.no_grad():
8 cnt = F.conv2d(m, ones, stride=self.stride, padding=self.padding) # sum(M)
9 valid = (cnt > 0).float() # m'
10 scale = self.kernel_size[0] * self.kernel_size[1] / cnt.clamp(min=1)
11 out = F.conv2d(x * m, self.weight, None, self.stride, self.padding) * scale * valid
12 if self.bias is not None:
13 out = out + self.bias.view(1, -1, 1, 1) * valid
14 return out, valid
15
16layer = PartialConv2d(3, 64, kernel_size=7, stride=2, padding=3)
17img, mask = torch.rand(1, 3, 512, 512), torch.ones(1, 1, 512, 512)
18mask[..., 200:300, 150:350] = 0 # a rectangular hole
19feat, mask1 = layer(img * mask, mask) # (1,64,256,256) and the shrunken hole
A partial convolution layer in PyTorch (not run here: no torch in the scratch environment)

Use when

  • +Feed-forward models (partial/gated conv, LaMa) fill a hole in one pass and keep known pixels exactly through compositing
  • +Mask-aware layers make the output independent of the placeholder value
  • +Diffusion inpainting samples diverse fills and can follow a text prompt

Avoid when

  • −With local receptive fields, the centre of a large hole gets little context
  • −GAN-trained models give one deterministic fill and can show repeated textures or seams
  • −Diffusion inpainting needs tens of denoising steps, and boundary consistency needs extra care (resampling, mask-conditioned fine-tuning)

Check yourself

A partial convolution with a 3×3 mean kernel sees a window with 3 valid pixels of value 0.6 and 6 hole pixels. What does it output, and what is the new mask value?

Check yourself

Why does LaMa use fast Fourier convolutions?

Remember

  • ✓Inpainting predicts the missing region and composites: x^=m⊙x+(1−m)⊙G(m⊙x,m)\hat x=m\odot x+(1-m)\odot G(m\odot x,m).
  • ✓Context Encoders (2016) used L2 + adversarial loss (0.999 / 0.001) on a central 64×64 hole. GAN losses avoid blurry averages.
  • ✓Plain convolutions read the placeholder as data. Partial conv rescales by sum⁡(1)/sum⁡(M)\operatorname{sum}(\mathbf 1)/\operatorname{sum}(M) and updates the mask; gated conv learns a soft mask.
  • ✓LaMa uses Fourier convolutions and very large training masks for an image-wide receptive field.
  • ✓Diffusion inpainting (RePaint, 9-channel Stable Diffusion) samples many plausible fills. Evaluate with FID/LPIPS and a stated mask distribution, not only PSNR.
← back to Deep Learning roadmap