Definition
ControlNet a side network that adds spatial conditioning to a pretrained, frozen diffusion model. It is a trainable copy of the denoiser's encoder that reads a condition image (Canny edges, a depth map, a human pose skeleton, a segmentation map, a scribble) and adds its features back into the frozen network through zero convolutions: 1×1 convolutions whose weights and biases start at zero. Introduced by Lvmin Zhang, Anyi Rao and Maneesh Agrawala, "Adding Conditional Control to Text-to-Image Diffusion Models" (ICCV 2023).
A prompt cannot pin down layout. "A dancer, arms raised" gives a different pose with every seed, and words like "left" or "behind" are followed loosely. Designers, game studios and product teams often already have the structure they want: a sketch, a 3-D render, a pose from a reference photo, a floor plan. ControlNet turns that structure into a hard constraint while the prompt still decides content and style.
In a pipeline it sits next to the base model: a preprocessor (Canny, a monocular depth estimator, OpenPose) turns a reference image into a condition map, the ControlNet reads that map at every denoising step, and the base model, its LoRAs and its sampler stay unchanged. The base itself is covered in Stable Diffusion and Latent diffusion models; the generation call in Text-to-image generation.
The trainable copy and zero convolutions
Take a frozen block that maps features to . ControlNet locks , makes a trainable clone , and joins the clone to the original with two zero convolutions :
- the frozen pretrained block (locked weights)
- trainable copy, initialised to
- condition features, from a small encoder over the condition image
- 1×1 convolutions with weight and bias initialised to 0
At step 0 both zero convolutions output 0, so : the combined model is exactly the pretrained one, and no random noise is injected into deep features. Training still moves. For with , the gradient is non-zero because , the copy's output, is non-zero. After the first update , gradients flow back into and , and the branch learns.
On Stable Diffusion the clone covers the U-Net's 12 encoder blocks and the middle block. Its 13 outputs, each through its own zero convolution, are added to the 12 skip connections and to the middle block's output. The 512×512 condition image is first reduced to the 64×64 latent grid by a tiny encoder of four convolutions (4×4 kernels, stride 2, 16, 32, 64 and 128 channels, ReLU).
Training
- the noised latent of the training image at step t
- text prompt and task-specific condition (edge map, depth, pose)
- frozen U-Net plus ControlNet; only the copy, the small condition encoder and the zero convs get gradients
It is the ordinary denoising loss on (condition, image, caption) triples, which are cheap to make: run the preprocessor over captioned photos. During training 50% of the prompts are replaced by empty strings, so the network learns to read meaning from the condition map alone. The paper reports a "sudden convergence": the model does not follow the map gradually but switches abruptly from ignoring it to following it, usually in fewer than 10K optimisation steps.
From the paper and the reference implementation
- Released conditions
- Canny edges, depth (MiDaS), HED soft edges, M-LSD straight lines, OpenPose skeletons, ADE20K segmentation, normal maps, user scribbles
- Training cost
- about 23% more GPU memory and 34% more time per iteration than fine-tuning SD alone (single A100, as reported)
- Size
- for SD 1.5 the copy is roughly 360M parameters, against about 860M in the U-Net
- Data
- trains robustly on small (under 50K) and large (over 1M) datasets
- Reuse
- a ControlNet trained on SD 1.5 also works on community fine-tunes of SD 1.5, because they share the architecture
Zero init versus random init, live
The figure trains a 12-dimensional toy ControlNet, , with Adam on 48 samples. is frozen, starts as a copy of it, and start either at zero or at Gaussian noise of scale σ.
Worked example: one scalar ControlNet
with , , input , condition , target , loss , plain gradient descent with step 0.5.
| Step | w_c, z₁, z₂ | Output y | Loss | ∂L/∂z₂ | ∂L/∂w_c | ∂L/∂z₁ |
|---|---|---|---|---|---|---|
| 0 | 0.5, 0, 0 | 0.5000 | 0.6400 | −0.8000 | 0 | 0 |
| 1 | 0.5, 0, 0.4 | 0.7000 | 0.3600 | −0.6000 | −0.4800 | −0.4800 |
| 2 | 0.74, 0.24, 0.7 | 1.2666 | 0.0011 | −0.0731 | −0.0691 | −0.0691 |
Step 0 matches the frozen model (output 0.5) and only has a gradient, . One update later , and the gradients of and switch on. By step 2 the loss has fallen from 0.64 to 0.0011.
Using it
1import cv2, numpy as np, torch2from PIL import Image3from diffusers import ControlNetModel, StableDiffusionControlNetPipeline45ref = np.array(Image.open("room.jpg").convert("RGB").resize((512, 512)))6edges = cv2.Canny(ref, 100, 200) # low / high hysteresis thresholds7cond = Image.fromarray(np.stack([edges] * 3, axis=-1)) # 3-channel edge map89controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)10pipe = StableDiffusionControlNetPipeline.from_pretrained(11 "stable-diffusion-v1-5/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float1612).to("cuda")1314image = pipe(15 "a scandinavian living room, morning light", image=cond,16 num_inference_steps=30, guidance_scale=7.5,17 controlnet_conditioning_scale=0.8, # s: multiplies every residual18 control_guidance_end=0.6, # stop control after 60% of the steps19 generator=torch.Generator("cuda").manual_seed(0),20).images[0]- 1Pick the condition that encodes what must not change: Canny or line art for exact outlines, depth for layout and perspective, OpenPose for body pose, segmentation for region-by-region content.
- 2Match resolution and base: a ControlNet works only with the base family it was trained for (SD 1.5, SDXL and so on), and the map should have the generation size.
- 3Tune the conditioning scale
s, usually 0.5–1.0. Too high and the image copies preprocessing artefacts; too low and the structure drifts. - 4Limit the window with
control_guidance_start/control_guidance_end: early steps decide layout, so ending control at 50–70% keeps structure and frees fine detail. - 5Stack several ControlNets (pose plus depth) by passing lists of models, maps and scales; their residuals are summed.
Use when
- +Exact spatial control from maps that are cheap to produce
- +Base model untouched: one ControlNet serves its fine-tunes, LoRAs and samplers
- +Composable: several conditions, each with its own scale and step window
- +Trains on a single GPU from modest datasets
Avoid when
- −Extra compute at every denoising step (a second encoder pass)
- −Tied to one base architecture; SD 1.5 ControlNets do not run on SDXL
- −Inherits preprocessor errors (missed limbs in OpenPose, depth artefacts)
- −Lighter alternatives (T2I-Adapter, ControlNet-XS) trade some fidelity for speed
Check yourself
At the first training step, which ControlNet parameters receive a non-zero gradient?
Check yourself
The generated image copies the edge map so closely that wallpaper texture shows up as stripes. What should you change first?
Remember
- ✓ControlNet = frozen base + trainable copy of its encoder; .
- ✓Zero convolutions (1×1, weights and bias 0) make step 0 identical to the base model; Z₂ gets the first gradient, then the rest learn.
- ✓On SD: copy of 12 encoder blocks + middle block, added to 12 skips + middle; the condition is reduced 512² → 64² by four small convs.
- ✓Training is the normal denoising loss on (map, image, caption) triples with 50% empty prompts.
- ✓At inference: conditioning scale s, start/end window and stacking several ControlNets are the main controls.