Definition
Diffusion-based image editing changes an existing image with a pretrained diffusion model instead of generating from pure noise. Four methods cover most practice. SDEdit (Meng et al., arXiv 2021, ICLR 2022) noises the input to an intermediate step and denoises it. Masked inpainting regenerates only a region marked by a mask. InstructPix2Pix (Brooks, Holynski and Efros, CVPR 2023) edits from an instruction such as "make it snowy". Prompt-to-Prompt (Hertz et al., arXiv 2022, ICLR 2023) edits the prompt and reuses the original image's cross-attention maps.
Editing needs two things that plain Text-to-image generation does not: the output must stay faithful to the parts of the input you did not ask to change, and it must still look real. Every method below trades those two against each other, with a different knob. Architectures for classical and learned inpainting (partial and gated convolutions, LaMa, RePaint) are covered in Image inpainting; this article covers how the diffusion-era tools are used.
SDEdit: noise, then denoise
A rough input, such as a stroke painting or a photo you want restyled, is off the manifold of real images in its details but right in its layout. SDEdit adds just enough noise to hide the details and lets the model's reverse process fill them in:
- the VAE encoder of a Latent diffusion models pipeline; SDEdit itself worked in pixel space
- fraction of the schedule redone; diffusers runs of steps (default strength 0.8 in the img2img pipeline)
- how much of the input's amplitude survives in the starting latent
Low strength keeps the input almost unchanged, including its flaws. High strength produces a realistic image that may ignore the input. The paper found a good range for stroke-based generation, and reported that human raters preferred it to GAN-based baselines on realism. It needs no training, no mask and no paired data, which is why img2img is in every toolkit.
Masked inpainting
With a mask ( where content should be regenerated, the convention of diffusers), there are two ways to use a diffusion model. The training-free way runs an ordinary model and, after every step, overwrites the unmasked region with a correctly noised copy of the original:
- the sampler's output for the whole latent
- the latent of the original image
The model only ever sees the known region through noise, so early steps can commit the hole to content that contradicts its surroundings. RePaint (Lugmayr et al., CVPR 2022) fixes this by resampling: after a step, it diffuses back to the noisier step and denoises again, letting the hole and the context agree, at a multiple of the cost. The trained way is a dedicated inpainting model. The Stable Diffusion inpainting checkpoints feed the U-Net 9 channels: the noisy latent (4), the latent of the masked image (4) and the mask downsampled to (1). The 5 extra input channels start from zero weights, and the model is fine-tuned with synthetic masks, so it sees the clean context at every step.
Editing by instruction or by prompt
InstructPix2Pix is trained for the job. A fine-tuned GPT-3 wrote edit instructions and edited captions for image captions, Stable Diffusion with Prompt-to-Prompt rendered each before/after pair, and CLIP-based filters kept the consistent ones, giving roughly 450,000 training examples. The model is Stable Diffusion with extra input channels for the input image's latent. Image and text conditions are each dropped 5% of the time (and both together 5%), which enables two guidance scales:
- the input image and the instruction
- image guidance: higher stays closer to the input (
image_guidance_scale, 1.5 by default in diffusers) - text guidance: higher follows the instruction more (
guidance_scale, default 7.5)
Prompt-to-Prompt needs no training. Cross-attention maps decide where each word's content goes (see Cross-attention). Generate with the source prompt, then generate with the edited prompt from the same seed while injecting the source attention maps for the early steps. Swapping "cat" for "dog" then changes the animal and keeps the pose and the sofa. It supports word swaps, added phrases and re-weighting one word's attention. Editing a real photo first requires inverting it to noise, for example by DDIM inversion plus null-text inversion (Mokady et al., CVPR 2023).
| Method | Input | Training | Main knob | Good for |
|---|---|---|---|---|
| SDEdit / img2img | image + prompt | none | strength | restyling, sketch → image |
| Blended / RePaint inpainting | image + mask + prompt | none | resampling count | any base model, small holes |
| Inpainting checkpoint | image + mask + prompt | fine-tuned, 9 channels | mask, prompt | object removal or replacement |
| InstructPix2Pix | image + instruction | about 450k generated pairs | s_I, s_T | global edits: weather, style, season |
| Prompt-to-Prompt | prompt pair (or inverted photo) | none | injection steps τ | word-level edits keeping layout |
Worked example
- 1img2img at 50 steps. Strength 0.3 runs steps from , where . Strength 0.5 runs 25 steps from (). Strength 0.8 runs 40 steps from (): most of the input is gone.
- 2InstructPix2Pix guidance on one latent value with , , : . That is three U-Net evaluations per step.
- 3Mask resolution. A mask becomes in latent space, so each mask cell covers pixels. Dilate thin masks by a few pixels, or edges of the object survive.
1import torch2from diffusers import AutoPipelineForInpainting, StableDiffusionInstructPix2PixPipeline3from diffusers.utils import load_image45image = load_image("room.png").resize((512, 512))6mask = load_image("room_mask.png").resize((512, 512)) # white = regenerate78inpaint = AutoPipelineForInpainting.from_pretrained(9 "stabilityai/stable-diffusion-2-inpainting", torch_dtype=torch.float16).to("cuda")10filled = inpaint(prompt="an empty wooden floor", image=image, mask_image=mask,11 num_inference_steps=30, guidance_scale=7.5,12 generator=torch.Generator("cuda").manual_seed(0)).images[0]1314ip2p = StableDiffusionInstructPix2PixPipeline.from_pretrained(15 "timbrooks/instruct-pix2pix", torch_dtype=torch.float16).to("cuda")16snowy = ip2p("make it look like winter", image=image, num_inference_steps=20,17 image_guidance_scale=1.5, # s_I: stay close to the input18 guidance_scale=7.5).images[0] # s_T: follow the instructionCheck yourself
An img2img edit at strength 0.9 ignores the layout of the input photo. What is the most direct fix?
Check yourself
Why does a dedicated 9-channel inpainting model produce more coherent fills than overwriting the known region at every step?
Remember
- ✓SDEdit/img2img: , then denoise; strength trades faithfulness for realism ( in the paper).
- ✓Inpainting: either re-impose the noised known region each step (training-free, add RePaint resampling) or use a 9-channel model that sees the masked image and mask.
- ✓Always paste the original back outside the mask; the VAE round trip is lossy, and latent masks have 8 × 8-pixel cells.
- ✓InstructPix2Pix: trained on about 450k generated pairs; two guidance scales , , three evaluations per step.
- ✓Prompt-to-Prompt: same seed, edited prompt, injected source cross-attention maps; real photos need inversion first.