# Image editing and inpainting

> Diffusion editing reuses a text-to-image model: SDEdit noises an input part way and denoises it, inpainting regenerates only a masked region, InstructPix2Pix follows written instructions, and Prompt-to-Prompt edits the prompt while keeping the cross-attention layout.

Source: https://www.mlfoundry.in/learn/image-editing-and-inpainting  
Section: Generative AI › Multimodal Generative AI  
Level: core · 8 min read · updated 2026-10-21

**Diffusion-based image editing:** changes an existing image with a pretrained diffusion model instead of generating from pure noise. Four methods cover most practice. **SDEdit** (Meng et al., arXiv 2021, ICLR 2022) noises the input to an intermediate step and denoises it. **Masked inpainting** regenerates only a region marked by a mask. **InstructPix2Pix** (Brooks, Holynski and Efros, CVPR 2023) edits from an instruction such as "make it snowy". **Prompt-to-Prompt** (Hertz et al., arXiv 2022, ICLR 2023) edits the prompt and reuses the original image's cross-attention maps.

Editing needs two things that plain **Text-to-image generation** does not: the output must stay faithful to the parts of the input you did not ask to change, and it must still look real. Every method below trades those two against each other, with a different knob. Architectures for classical and learned inpainting (partial and gated convolutions, LaMa, RePaint) are covered in **Image inpainting**; this article covers how the diffusion-era tools are used.

> Figure: Inputs and outputs of the four methods. SDEdit noises a rough guide to t₀ and denoises it; at 50 steps, strength 0.3 runs 15 steps and keeps √ᾱ = 0.789 of the guide's amplitude, strength 0.8 runs 40 and keeps 0.208. The SD inpainting U-Net reads 9 channels and the original is pasted back outside the mask. InstructPix2Pix concatenates the input image's latent and uses two guidance scales. Prompt-to-Prompt keeps the seed, swaps a word and injects the source cross-attention maps.

## SDEdit: noise, then denoise

A rough input, such as a stroke painting or a photo you want restyled, is off the manifold of real images in its details but right in its layout. SDEdit adds just enough noise to hide the details and lets the model's reverse process fill them in:

$$
z_{t_0}=\sqrt{\bar\alpha_{t_0}}\,\mathcal E(x^{\text{guide}})+\sqrt{1-\bar\alpha_{t_0}}\,\epsilon,\qquad t_0=\text{strength}\cdot T,\qquad \text{then run the sampler from }t_0\text{ to }0
$$

- $\mathcal E$: the VAE encoder of a **Latent diffusion models** pipeline; SDEdit itself worked in pixel space
- $\text{strength}$: fraction of the schedule redone; diffusers runs $\lfloor\text{strength}\cdot N\rfloor$ of $N$ steps (default strength 0.8 in the img2img pipeline)
- $\sqrt{\bar\alpha_{t_0}}$: how much of the input's amplitude survives in the starting latent

Low strength keeps the input almost unchanged, including its flaws. High strength produces a realistic image that may ignore the input. The paper found $t_0\in[0.3,0.6]$ a good range for stroke-based generation, and reported that human raters preferred it to GAN-based baselines on realism. It needs no training, no mask and no paired data, which is why img2img is in every toolkit.

## Masked inpainting

With a mask $m$ ($m=1$ where content should be regenerated, the convention of diffusers), there are two ways to use a diffusion model. The training-free way runs an ordinary model and, after every step, overwrites the unmasked region with a correctly noised copy of the original:

$$
z_{t-1}\leftarrow m\odot z_{t-1}^{\text{gen}}+(1-m)\odot\big(\sqrt{\bar\alpha_{t-1}}\,z_0^{\text{orig}}+\sqrt{1-\bar\alpha_{t-1}}\,\epsilon\big)
$$

- $z_{t-1}^{\text{gen}}$: the sampler's output for the whole latent
- $z_0^{\text{orig}}$: the latent of the original image

The model only ever sees the known region through noise, so early steps can commit the hole to content that contradicts its surroundings. RePaint (Lugmayr et al., CVPR 2022) fixes this by **resampling**: after a step, it diffuses back to the noisier step and denoises again, letting the hole and the context agree, at a multiple of the cost. The trained way is a dedicated inpainting model. The Stable Diffusion inpainting checkpoints feed the U-Net 9 channels: the noisy latent (4), the latent of the masked image (4) and the mask downsampled to $64\times64$ (1). The 5 extra input channels start from zero weights, and the model is fine-tuned with synthetic masks, so it sees the clean context at every step.

> Figure: Two toy 2-D latent spaces with exact noise predictors, SD v1's schedule, 50 DDIM steps, 64 seeds. SDEdit (real images on a ring of 8 modes, guide inside the ring): strength 0 leaves the guide 0.800 off the ring. The default 0.3 runs 15 steps from t₀ = 281 and gives a mean angle error of 49.9° with 0.106 off the ring; 0.1 gives 25.8° and 0.258; diffusers' default 0.8 gives 73.4° and 0.091 (uniform would be 90°). Inpainting (known pixel 0.6; real images have both pixels agree; 80% of images are dark): replacing the known region gives 13 of 64 contradicting fills (20.3%); RePaint with U = 2 gives 9.4% and U = 3 gives 0% at 150 evaluations; the inpainting model gives 0% at 50. At p(dark) = 0.9, replacement fails 32.8% of the time.

## Editing by instruction or by prompt

**InstructPix2Pix** is trained for the job. A fine-tuned GPT-3 wrote edit instructions and edited captions for image captions, Stable Diffusion with Prompt-to-Prompt rendered each before/after pair, and CLIP-based filters kept the consistent ones, giving roughly 450,000 training examples. The model is Stable Diffusion with extra input channels for the input image's latent. Image and text conditions are each dropped 5% of the time (and both together 5%), which enables two guidance scales:

$$
\tilde\epsilon=\epsilon(z_t,\varnothing,\varnothing)+s_I\big[\epsilon(z_t,c_I,\varnothing)-\epsilon(z_t,\varnothing,\varnothing)\big]+s_T\big[\epsilon(z_t,c_I,c_T)-\epsilon(z_t,c_I,\varnothing)\big]
$$

- $c_I,\ c_T$: the input image and the instruction
- $s_I$: image guidance: higher stays closer to the input (`image_guidance_scale`, 1.5 by default in diffusers)
- $s_T$: text guidance: higher follows the instruction more (`guidance_scale`, default 7.5)

**Prompt-to-Prompt** needs no training. Cross-attention maps decide where each word's content goes (see **Cross-attention**). Generate with the source prompt, then generate with the edited prompt from the same seed while injecting the source attention maps for the early steps. Swapping "cat" for "dog" then changes the animal and keeps the pose and the sofa. It supports word swaps, added phrases and re-weighting one word's attention. Editing a real photo first requires inverting it to noise, for example by DDIM inversion plus **null-text inversion** (Mokady et al., CVPR 2023).

Which tool for which edit

| Method | Input | Training | Main knob | Good for |
| --- | --- | --- | --- | --- |
| SDEdit / img2img | image + prompt | none | strength | restyling, sketch → image |
| Blended / RePaint inpainting | image + mask + prompt | none | resampling count | any base model, small holes |
| Inpainting checkpoint | image + mask + prompt | fine-tuned, 9 channels | mask, prompt | object removal or replacement |
| InstructPix2Pix | image + instruction | about 450k generated pairs | s_I, s_T | global edits: weather, style, season |
| Prompt-to-Prompt | prompt pair (or inverted photo) | none | injection steps τ | word-level edits keeping layout |

## Worked example

1. **img2img at 50 steps.** Strength 0.3 runs $\lfloor0.3\cdot50\rfloor=15$ steps from $t_0=281$, where $\sqrt{\bar\alpha}=0.789$. Strength 0.5 runs 25 steps from $t_0=481$ ($\sqrt{\bar\alpha}=0.550$). Strength 0.8 runs 40 steps from $t_0=781$ ($\sqrt{\bar\alpha}=0.208$): most of the input is gone.
2. **InstructPix2Pix guidance** on one latent value with $\epsilon(\varnothing,\varnothing)=0.10$, $\epsilon(I,\varnothing)=0.30$, $\epsilon(I,T)=0.50$: $\tilde\epsilon=0.10+1.5\cdot0.20+7.5\cdot0.20=0.10+0.30+1.50=1.90$. That is three U-Net evaluations per step.
3. **Mask resolution.** A $512\times512$ mask becomes $64\times64$ in latent space, so each mask cell covers $8\times8=64$ pixels. Dilate thin masks by a few pixels, or edges of the object survive.

```python
import torch
from diffusers import AutoPipelineForInpainting, StableDiffusionInstructPix2PixPipeline
from diffusers.utils import load_image

image = load_image("room.png").resize((512, 512))
mask = load_image("room_mask.png").resize((512, 512))    # white = regenerate

inpaint = AutoPipelineForInpainting.from_pretrained(
    "stabilityai/stable-diffusion-2-inpainting", torch_dtype=torch.float16).to("cuda")
filled = inpaint(prompt="an empty wooden floor", image=image, mask_image=mask,
                 num_inference_steps=30, guidance_scale=7.5,
                 generator=torch.Generator("cuda").manual_seed(0)).images[0]

ip2p = StableDiffusionInstructPix2PixPipeline.from_pretrained(
    "timbrooks/instruct-pix2pix", torch_dtype=torch.float16).to("cuda")
snowy = ip2p("make it look like winter", image=image, num_inference_steps=20,
             image_guidance_scale=1.5,      # s_I: stay close to the input
             guidance_scale=7.5).images[0]  # s_T: follow the instruction
```

Inpainting and instruction editing with Hugging Face diffusers (needs a GPU and the downloaded weights; not run here)

> **The VAE changes pixels you did not mask:** The whole image passes through the VAE encoder and decoder, which is lossy: faces, small text and fine textures outside the mask come back slightly altered. Composite the decoded result into the original with the (feathered) mask, $\hat x=m\odot\mathcal D(z_0)+(1-m)\odot x$, so unmasked pixels stay exact. Feathering the seam by a few pixels hides the boundary.

> **Strength and image guidance are the same dial:** Each method has one knob that moves along the faithfulness–realism curve: strength for SDEdit, the resampling count or a dedicated model for inpainting, $s_I$ for InstructPix2Pix, and the number of injected steps for Prompt-to-Prompt. Sweep it on a few test images per use case instead of fixing one value for everything. For spatial structure that must not move at all (poses, edges, depth), condition on it with **ControlNet**.

**Question:** An img2img edit at strength 0.9 ignores the layout of the input photo. What is the most direct fix?

- A. Raise the guidance scale
- B. Lower the strength, so fewer steps are redone and more of the input survives in z(t₀)
- C. Use more denoising steps
- D. Switch to an ancestral sampler

**Answer:** B. Lower the strength, so fewer steps are redone and more of the input survives in z(t₀) — Strength sets $t_0$. At 50 steps, 0.9 starts at $t_0=881$ with $\sqrt{\bar\alpha}=0.131$, so almost nothing of the input remains. At 0.5 the start is $t_0=481$ with $\sqrt{\bar\alpha}=0.550$. More steps or guidance do not bring the input back.

**Question:** Why does a dedicated 9-channel inpainting model produce more coherent fills than overwriting the known region at every step?

- A. It uses a larger VAE
- B. It sees the clean masked image and the mask at every step, so the hole is generated in agreement with its context from the start
- C. It runs more steps
- D. It skips classifier-free guidance

**Answer:** B. It sees the clean masked image and the mask at every step, so the hole is generated in agreement with its context from the start — With replacement, the model sees the context only through noise, and early steps can commit the hole to content that contradicts it. In the figure that gives 20.3% contradicting fills; resampling (U = 3) or the context-aware model brings it to 0%.

**Key points**

- SDEdit/img2img: $z_{t_0}=\sqrt{\bar\alpha_{t_0}}\mathcal E(x)+\sqrt{1-\bar\alpha_{t_0}}\epsilon$, then denoise; strength trades faithfulness for realism ($t_0\in[0.3,0.6]$ in the paper).
- Inpainting: either re-impose the noised known region each step (training-free, add RePaint resampling) or use a 9-channel model that sees the masked image and mask.
- Always paste the original back outside the mask; the VAE round trip is lossy, and latent masks have 8 × 8-pixel cells.
- InstructPix2Pix: trained on about 450k generated pairs; two guidance scales $s_I\approx1.5$, $s_T\approx7.5$, three evaluations per step.
- Prompt-to-Prompt: same seed, edited prompt, injected source cross-attention maps; real photos need inversion first.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
