Image editing and inpainting

Diffusion editing reuses a text-to-image model: SDEdit noises an input part way and denoises it, inpainting regenerates only a masked region, InstructPix2Pix follows written instructions, and Prompt-to-Prompt edits the prompt while keeping the cross-attention layout.

8 min read887 wordscore0 viewsupdated 2026-10-21

Definition

Diffusion-based image editing changes an existing image with a pretrained diffusion model instead of generating from pure noise. Four methods cover most practice. SDEdit (Meng et al., arXiv 2021, ICLR 2022) noises the input to an intermediate step and denoises it. Masked inpainting regenerates only a region marked by a mask. InstructPix2Pix (Brooks, Holynski and Efros, CVPR 2023) edits from an instruction such as "make it snowy". Prompt-to-Prompt (Hertz et al., arXiv 2022, ICLR 2023) edits the prompt and reuses the original image's cross-attention maps.

Editing needs two things that plain Text-to-image generation does not: the output must stay faithful to the parts of the input you did not ask to change, and it must still look real. Every method below trades those two against each other, with a different knob. Architectures for classical and learned inpainting (partial and gated convolutions, LaMa, RePaint) are covered in Image inpainting; this article covers how the diffusion-era tools are used.

Editing with a pretrained diffusion model: what goes in, what comes outSDEdit / img2img (Meng et al., 2021)guide xnoise to t₀z(t₀)denoise t₀→0editedz(t₀) = √ᾱ(t₀)·E(x) + √(1−ᾱ(t₀))·ε, t₀ set by strength0.3: 15 of 50 steps, √ᾱ = 0.789 · 0.8: 40 steps, 0.208Masked inpainting (SD inpainting model)x, mask mU-Net input, 9 ch:4 noisy latent z_t4 E(x ⊙ (1 − m))1 mask m at 64 × 64filledm = 1 marks pixels to repaint; after decoding:x̂ = m ⊙ D(z₀) + (1 − m) ⊙ x (paste back)InstructPix2Pix (Brooks et al., 2023)input I“make it snowy”U-Net(z_t ⊕ E(I), text)8 input channelseditedε̃ = ε(∅,∅) + s_I[ε(I,∅) − ε(∅,∅)] + s_T[ε(I,T) − ε(I,∅)]s_I = 1.5 (stay close to I), s_T = 7.5 (follow T)3 U-Net evaluations per step; no inversion neededPrompt-to-Prompt (Hertz et al., 2022)“a cat…”same seed, edited promptinject source cross-attnmaps for early steps“a dog…”word swap cat → dog: layout comes from the mapsalso: add a phrase, re-weight a word’s attentionreal photos: invert first (DDIM / null-text)
fig.Inputs and outputs of the four methods. SDEdit noises a rough guide to t₀ and denoises it; at 50 steps, strength 0.3 runs 15 steps and keeps √ᾱ = 0.789 of the guide's amplitude, strength 0.8 runs 40 and keeps 0.208. The SD inpainting U-Net reads 9 channels and the original is pasted back outside the mask. InstructPix2Pix concatenates the input image's latent and uses two guidance scales. Prompt-to-Prompt keeps the seed, swaps a word and injects the source cross-attention maps.

SDEdit: noise, then denoise

A rough input, such as a stroke painting or a photo you want restyled, is off the manifold of real images in its details but right in its layout. SDEdit adds just enough noise to hide the details and lets the model's reverse process fill them in:

zt0=αˉt0 E(xguide)+1−αˉt0 ϵ,t0=strength⋅T,then run the sampler from t0 to 0z_{t_0}=\sqrt{\bar\alpha_{t_0}}\,\mathcal E(x^{\text{guide}})+\sqrt{1-\bar\alpha_{t_0}}\,\epsilon,\qquad t_0=\text{strength}\cdot T,\qquad \text{then run the sampler from }t_0\text{ to }0
E\mathcal E
the VAE encoder of a Latent diffusion models pipeline; SDEdit itself worked in pixel space
strength\text{strength}
fraction of the schedule redone; diffusers runs ⌊strength⋅N⌋\lfloor\text{strength}\cdot N\rfloor of NN steps (default strength 0.8 in the img2img pipeline)
αˉt0\sqrt{\bar\alpha_{t_0}}
how much of the input's amplitude survives in the starting latent

Low strength keeps the input almost unchanged, including its flaws. High strength produces a realistic image that may ignore the input. The paper found t0∈[0.3,0.6]t_0\in[0.3,0.6] a good range for stroke-based generation, and reported that human raters preferred it to GAN-based baselines on realism. It needs no training, no mask and no paired data, which is why img2img is in every toolkit.

Masked inpainting

With a mask mm (m=1m=1 where content should be regenerated, the convention of diffusers), there are two ways to use a diffusion model. The training-free way runs an ordinary model and, after every step, overwrites the unmasked region with a correctly noised copy of the original:

zt−1←m⊙zt−1gen+(1−m)⊙(αˉt−1 z0orig+1−αˉt−1 ϵ)z_{t-1}\leftarrow m\odot z_{t-1}^{\text{gen}}+(1-m)\odot\big(\sqrt{\bar\alpha_{t-1}}\,z_0^{\text{orig}}+\sqrt{1-\bar\alpha_{t-1}}\,\epsilon\big)
zt−1genz_{t-1}^{\text{gen}}
the sampler's output for the whole latent
z0origz_0^{\text{orig}}
the latent of the original image

The model only ever sees the known region through noise, so early steps can commit the hole to content that contradicts its surroundings. RePaint (Lugmayr et al., CVPR 2022) fixes this by resampling: after a step, it diffuses back to the noisier step and denoises again, letting the hole and the context agree, at a multiple of the cost. The trained way is a dedicated inpainting model. The Stable Diffusion inpainting checkpoints feed the U-Net 9 channels: the noisy latent (4), the latent of the masked image (4) and the mask downsampled to 64×6464\times64 (1). The 5 extra input channels start from zero weights, and the model is fine-tuned with synthetic masks, so it sees the clean context at every step.

SDEdit: noise the guide to strength 0.30, then denoise (15 of 50 steps)guide (rough sketch)dashed: real images● ≤ 45° off ● moretrade-off over strength 0 → 10°45°90°00.40.8angle error (unfaithful →)off-ring distance (unreal ↑)start timestep t₀281√ᾱ(t₀): guide kept0.789mean angle error49.9°mean off-ring distance0.106denoiser evaluations15
task
fig.Two toy 2-D latent spaces with exact noise predictors, SD v1's schedule, 50 DDIM steps, 64 seeds. SDEdit (real images on a ring of 8 modes, guide inside the ring): strength 0 leaves the guide 0.800 off the ring. The default 0.3 runs 15 steps from t₀ = 281 and gives a mean angle error of 49.9° with 0.106 off the ring; 0.1 gives 25.8° and 0.258; diffusers' default 0.8 gives 73.4° and 0.091 (uniform would be 90°). Inpainting (known pixel 0.6; real images have both pixels agree; 80% of images are dark): replacing the known region gives 13 of 64 contradicting fills (20.3%); RePaint with U = 2 gives 9.4% and U = 3 gives 0% at 150 evaluations; the inpainting model gives 0% at 50. At p(dark) = 0.9, replacement fails 32.8% of the time.

Editing by instruction or by prompt

InstructPix2Pix is trained for the job. A fine-tuned GPT-3 wrote edit instructions and edited captions for image captions, Stable Diffusion with Prompt-to-Prompt rendered each before/after pair, and CLIP-based filters kept the consistent ones, giving roughly 450,000 training examples. The model is Stable Diffusion with extra input channels for the input image's latent. Image and text conditions are each dropped 5% of the time (and both together 5%), which enables two guidance scales:

ϵ~=ϵ(zt,∅,∅)+sI[ϵ(zt,cI,∅)−ϵ(zt,∅,∅)]+sT[ϵ(zt,cI,cT)−ϵ(zt,cI,∅)]\tilde\epsilon=\epsilon(z_t,\varnothing,\varnothing)+s_I\big[\epsilon(z_t,c_I,\varnothing)-\epsilon(z_t,\varnothing,\varnothing)\big]+s_T\big[\epsilon(z_t,c_I,c_T)-\epsilon(z_t,c_I,\varnothing)\big]
cI, cTc_I,\ c_T
the input image and the instruction
sIs_I
image guidance: higher stays closer to the input (image_guidance_scale, 1.5 by default in diffusers)
sTs_T
text guidance: higher follows the instruction more (guidance_scale, default 7.5)

Prompt-to-Prompt needs no training. Cross-attention maps decide where each word's content goes (see Cross-attention). Generate with the source prompt, then generate with the edited prompt from the same seed while injecting the source attention maps for the early steps. Swapping "cat" for "dog" then changes the animal and keeps the pose and the sofa. It supports word swaps, added phrases and re-weighting one word's attention. Editing a real photo first requires inverting it to noise, for example by DDIM inversion plus null-text inversion (Mokady et al., CVPR 2023).

MethodInputTrainingMain knobGood for
SDEdit / img2imgimage + promptnonestrengthrestyling, sketch → image
Blended / RePaint inpaintingimage + mask + promptnoneresampling countany base model, small holes
Inpainting checkpointimage + mask + promptfine-tuned, 9 channelsmask, promptobject removal or replacement
InstructPix2Piximage + instructionabout 450k generated pairss_I, s_Tglobal edits: weather, style, season
Prompt-to-Promptprompt pair (or inverted photo)noneinjection steps τword-level edits keeping layout
Which tool for which edit

Worked example

  1. 1img2img at 50 steps. Strength 0.3 runs ⌊0.3⋅50⌋=15\lfloor0.3\cdot50\rfloor=15 steps from t0=281t_0=281, where αˉ=0.789\sqrt{\bar\alpha}=0.789. Strength 0.5 runs 25 steps from t0=481t_0=481 (αˉ=0.550\sqrt{\bar\alpha}=0.550). Strength 0.8 runs 40 steps from t0=781t_0=781 (αˉ=0.208\sqrt{\bar\alpha}=0.208): most of the input is gone.
  2. 2InstructPix2Pix guidance on one latent value with ϵ(∅,∅)=0.10\epsilon(\varnothing,\varnothing)=0.10, ϵ(I,∅)=0.30\epsilon(I,\varnothing)=0.30, ϵ(I,T)=0.50\epsilon(I,T)=0.50: ϵ~=0.10+1.5⋅0.20+7.5⋅0.20=0.10+0.30+1.50=1.90\tilde\epsilon=0.10+1.5\cdot0.20+7.5\cdot0.20=0.10+0.30+1.50=1.90. That is three U-Net evaluations per step.
  3. 3Mask resolution. A 512×512512\times512 mask becomes 64×6464\times64 in latent space, so each mask cell covers 8×8=648\times8=64 pixels. Dilate thin masks by a few pixels, or edges of the object survive.
python
1import torch
2from diffusers import AutoPipelineForInpainting, StableDiffusionInstructPix2PixPipeline
3from diffusers.utils import load_image
4
5image = load_image("room.png").resize((512, 512))
6mask = load_image("room_mask.png").resize((512, 512)) # white = regenerate
7
8inpaint = AutoPipelineForInpainting.from_pretrained(
9 "stabilityai/stable-diffusion-2-inpainting", torch_dtype=torch.float16).to("cuda")
10filled = inpaint(prompt="an empty wooden floor", image=image, mask_image=mask,
11 num_inference_steps=30, guidance_scale=7.5,
12 generator=torch.Generator("cuda").manual_seed(0)).images[0]
13
14ip2p = StableDiffusionInstructPix2PixPipeline.from_pretrained(
15 "timbrooks/instruct-pix2pix", torch_dtype=torch.float16).to("cuda")
16snowy = ip2p("make it look like winter", image=image, num_inference_steps=20,
17 image_guidance_scale=1.5, # s_I: stay close to the input
18 guidance_scale=7.5).images[0] # s_T: follow the instruction
Inpainting and instruction editing with Hugging Face diffusers (needs a GPU and the downloaded weights; not run here)

Check yourself

An img2img edit at strength 0.9 ignores the layout of the input photo. What is the most direct fix?

Check yourself

Why does a dedicated 9-channel inpainting model produce more coherent fills than overwriting the known region at every step?

Remember

  • ✓SDEdit/img2img: zt0=αˉt0E(x)+1−αˉt0ϵz_{t_0}=\sqrt{\bar\alpha_{t_0}}\mathcal E(x)+\sqrt{1-\bar\alpha_{t_0}}\epsilon, then denoise; strength trades faithfulness for realism (t0∈[0.3,0.6]t_0\in[0.3,0.6] in the paper).
  • ✓Inpainting: either re-impose the noised known region each step (training-free, add RePaint resampling) or use a 9-channel model that sees the masked image and mask.
  • ✓Always paste the original back outside the mask; the VAE round trip is lossy, and latent masks have 8 × 8-pixel cells.
  • ✓InstructPix2Pix: trained on about 450k generated pairs; two guidance scales sI≈1.5s_I\approx1.5, sT≈7.5s_T\approx7.5, three evaluations per step.
  • ✓Prompt-to-Prompt: same seed, edited prompt, injected source cross-attention maps; real photos need inversion first.
← back to Generative AI roadmap