ControlNet

ControlNet steers a frozen text-to-image diffusion model with a spatial map (edges, depth, pose): a trainable copy of the encoder reads the map and feeds zero-initialised 1×1 convolutions, so training starts from the unchanged model.

7 min read893 wordsadvanced0 viewsupdated 2026-10-21

Definition

ControlNet a side network that adds spatial conditioning to a pretrained, frozen diffusion model. It is a trainable copy of the denoiser's encoder that reads a condition image (Canny edges, a depth map, a human pose skeleton, a segmentation map, a scribble) and adds its features back into the frozen network through zero convolutions: 1×1 convolutions whose weights and biases start at zero. Introduced by Lvmin Zhang, Anyi Rao and Maneesh Agrawala, "Adding Conditional Control to Text-to-Image Diffusion Models" (ICCV 2023).

A prompt cannot pin down layout. "A dancer, arms raised" gives a different pose with every seed, and words like "left" or "behind" are followed loosely. Designers, game studios and product teams often already have the structure they want: a sketch, a 3-D render, a pose from a reference photo, a floor plan. ControlNet turns that structure into a hard constraint while the prompt still decides content and style.

In a pipeline it sits next to the base model: a preprocessor (Canny, a monocular depth estimator, OpenPose) turns a reference image into a condition map, the ControlNet reads that map at every denoising step, and the base model, its LoRAs and its sampler stay unchanged. The base itself is covered in Stable Diffusion and Latent diffusion models; the generation call in Text-to-image generation.

The trainable copy and zero convolutions

Take a frozen block F(⋅;Θ)\mathcal F(\cdot;\Theta) that maps features xx to yy. ControlNet locks Θ\Theta, makes a trainable clone Θc\Theta_c, and joins the clone to the original with two zero convolutions Z\mathcal Z:

yc=F(x;Θ)+Z(F(x+Z(c;Θz1); Θc); Θz2)y_c=\mathcal F(x;\Theta)+\mathcal Z\big(\mathcal F\big(x+\mathcal Z(c;\Theta_{z1});\,\Theta_c\big);\,\Theta_{z2}\big)
F(x;Θ)\mathcal F(x;\Theta)
the frozen pretrained block (locked weights)
Θc\Theta_c
trainable copy, initialised to Θ\Theta
cc
condition features, from a small encoder over the condition image
Z(⋅;Θz1), Z(⋅;Θz2)\mathcal Z(\cdot;\Theta_{z1}),\ \mathcal Z(\cdot;\Theta_{z2})
1×1 convolutions with weight and bias initialised to 0

At step 0 both zero convolutions output 0, so yc=yy_c=y: the combined model is exactly the pretrained one, and no random noise is injected into deep features. Training still moves. For y=Why=Wh with W=0W=0, the gradient ∂L/∂W=(∂L/∂y) h⊤\partial\mathcal L/\partial W=(\partial\mathcal L/\partial y)\,h^\top is non-zero because hh, the copy's output, is non-zero. After the first update Θz2≠0\Theta_{z2}\neq0, gradients flow back into Θc\Theta_c and Θz1\Theta_{z1}, and the branch learns.

On Stable Diffusion the clone covers the U-Net's 12 encoder blocks and the middle block. Its 13 outputs, each through its own zero convolution, are added to the 12 skip connections and to the middle block's output. The 512×512 condition image is first reduced to the 64×64 latent grid by a tiny encoder of four convolutions (4×4 kernels, stride 2, 16, 32, 64 and 128 channels, ReLU).

ControlNet on SD 1.5: frozen U-Net (top two rows), trainable copy (bottom)z_t 64²×4E1 64²E2 32²E3 16²E4 8²mid 8²ε_θ 64²×4D1 64²D2 32²D3 16²D4 8²+skipZ₂ zero 1×1Z₂ zero 1×1Z₂ zero 1×1Z₂ zero 1×1Z₂ zero 1×1+ s·Z₂(·)z_t + Z₁(c)C1 64²C2 32²C3 16²C4 8²C-mid 8²initialised as a copy of E1–E4 and mid, then trainededges c 512²4 convs →64²Z₁ zero 1×1channels 16·32·64·128Z₁, Z₂: 1×1 convs, weight = bias = 0 at initstep 0: every Z₂ outputs 0, so output = frozen SDs = conditioning scale on all residualsprompt "a bird on a branch" feeds thecross-attention of both networksfrozen (locked)trainablezero convolutioncondition image
fig.ControlNet on SD 1.5. Top: the frozen U-Net, encoder E1–E4 and the middle block going right, decoder D4–D1 coming back, with skip connections. Bottom: the trainable copy reads the noisy latent plus Z₁ of the condition features (Canny edges, 512² reduced to 64² by four convolutions). Each copy block's output goes through its own zero convolution Z₂ and is added, times the conditioning scale s, to a decoder input or the middle output. At initialisation every Z outputs 0.

Training

L=Ez0,t,ct,cf,ϵ∼N(0,I)[∥ϵ−ϵθ(zt, t, ct, cf)∥22]\mathcal L=\mathbb E_{z_0,t,c_t,c_f,\epsilon\sim\mathcal N(0,I)}\Big[\big\lVert\epsilon-\epsilon_\theta(z_t,\,t,\,c_t,\,c_f)\big\rVert_2^2\Big]
ztz_t
the noised latent of the training image at step t
ct, cfc_t,\ c_f
text prompt and task-specific condition (edge map, depth, pose)
ϵθ\epsilon_\theta
frozen U-Net plus ControlNet; only the copy, the small condition encoder and the zero convs get gradients

It is the ordinary denoising loss on (condition, image, caption) triples, which are cheap to make: run the preprocessor over captioned photos. During training 50% of the prompts are replaced by empty strings, so the network learns to read meaning from the condition map alone. The paper reports a "sudden convergence": the model does not follow the map gradually but switches abruptly from ignoring it to following it, usually in fewer than 10K optimisation steps.

From the paper and the reference implementation

Released conditions
Canny edges, depth (MiDaS), HED soft edges, M-LSD straight lines, OpenPose skeletons, ADE20K segmentation, normal maps, user scribbles
Training cost
about 23% more GPU memory and 34% more time per iteration than fine-tuning SD alone (single A100, as reported)
Size
for SD 1.5 the copy is roughly 360M parameters, against about 860M in the U-Net
Data
trains robustly on small (under 50K) and large (over 1M) datasets
Reuse
a ControlNet trained on SD 1.5 also works on community fine-tunes of SD 1.5, because they share the architecture

Zero init versus random init, live

The figure trains a 12-dimensional toy ControlNet, y=Wbx+s Z2 relu(Wc(x+Z1c))y=W_bx+s\,Z_2\,\mathrm{relu}(W_c(x+Z_1c)), with Adam on 48 samples. WbW_b is frozen, WcW_c starts as a copy of it, and Z1,Z2Z_1,Z_2 start either at zero or at Gaussian noise of scale σ.

y = W_b·x + s·Z₂·relu(W_c·(x + Z₁·c)) · 12-d toy, 48 samples, Adam lr 0.01loss (log) vs stepdrift ‖W_c − W_b‖ / ‖W_b‖1e-31e-21e-111e11e200.5100150150300300zero init: step-0 loss 3.992 · grad ‖W_c‖ 0.00 ‖Z₁‖ 0.00 ‖Z₂‖ 4.19random σ = 0.10: step-0 loss 5.189 · grad ‖W_c‖ 1.03 ‖Z₁‖ 0.83 ‖Z₂‖ 5.26step 300, scale s = 1.0: loss 0.004 (zero) vs 0.052 (random) · frozen base alone 3.992sample 1, 12 outputs (zero-init run): base prediction targets = 0 switches the control branch off: every run falls back to the frozen model exactly
learning ratebars show
fig.Default (lr 0.01, σ = 0.1, step 300, s = 1). With zero init the step-0 loss is 3.992, exactly the frozen base model's, and the step-0 gradients are 0.00 for W_c and Z₁ and 4.19 for Z₂ alone. Random init starts worse, at 5.189, and ends at 0.052 against 0.004. At σ = 0.3 the random run starts at 17.105 and ends at 0.180. The random runs also drift the copy further from the frozen weights (0.783 against 0.474). Drag s on the trained zero-init run: s = 0.5 gives loss 1.002, and s = 0 gives back 3.992, the base model.

Worked example: one scalar ControlNet

y=wbx+z2 relu(wc(x+z1c))y=w_bx+z_2\,\mathrm{relu}(w_c(x+z_1c)) with wb=wc=0.5w_b=w_c=0.5, z1=z2=0z_1=z_2=0, input x=1x=1, condition c=2c=2, target 1.31.3, loss (y−1.3)2(y-1.3)^2, plain gradient descent with step 0.5.

Stepw_c, z₁, z₂Output yLoss∂L/∂z₂∂L/∂w_c∂L/∂z₁
00.5, 0, 00.50000.6400−0.800000
10.5, 0, 0.40.70000.3600−0.6000−0.4800−0.4800
20.74, 0.24, 0.71.26660.0011−0.0731−0.0691−0.0691
Computed in Python

Step 0 matches the frozen model (output 0.5) and only z2z_2 has a gradient, 2(0.5−1.3)⋅relu(0.5)=−0.82(0.5-1.3)\cdot\mathrm{relu}(0.5)=-0.8. One update later z2=0.4z_2=0.4, and the gradients of wcw_c and z1z_1 switch on. By step 2 the loss has fallen from 0.64 to 0.0011.

Using it

python
1import cv2, numpy as np, torch
2from PIL import Image
3from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
4
5ref = np.array(Image.open("room.jpg").convert("RGB").resize((512, 512)))
6edges = cv2.Canny(ref, 100, 200) # low / high hysteresis thresholds
7cond = Image.fromarray(np.stack([edges] * 3, axis=-1)) # 3-channel edge map
8
9controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
10pipe = StableDiffusionControlNetPipeline.from_pretrained(
11 "stable-diffusion-v1-5/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
12).to("cuda")
13
14image = pipe(
15 "a scandinavian living room, morning light", image=cond,
16 num_inference_steps=30, guidance_scale=7.5,
17 controlnet_conditioning_scale=0.8, # s: multiplies every residual
18 control_guidance_end=0.6, # stop control after 60% of the steps
19 generator=torch.Generator("cuda").manual_seed(0),
20).images[0]
Canny ControlNet with Hugging Face diffusers and OpenCV (needs a GPU and model downloads; not run here)
  1. 1Pick the condition that encodes what must not change: Canny or line art for exact outlines, depth for layout and perspective, OpenPose for body pose, segmentation for region-by-region content.
  2. 2Match resolution and base: a ControlNet works only with the base family it was trained for (SD 1.5, SDXL and so on), and the map should have the generation size.
  3. 3Tune the conditioning scale s, usually 0.5–1.0. Too high and the image copies preprocessing artefacts; too low and the structure drifts.
  4. 4Limit the window with control_guidance_start / control_guidance_end: early steps decide layout, so ending control at 50–70% keeps structure and frees fine detail.
  5. 5Stack several ControlNets (pose plus depth) by passing lists of models, maps and scales; their residuals are summed.

Use when

  • +Exact spatial control from maps that are cheap to produce
  • +Base model untouched: one ControlNet serves its fine-tunes, LoRAs and samplers
  • +Composable: several conditions, each with its own scale and step window
  • +Trains on a single GPU from modest datasets

Avoid when

  • −Extra compute at every denoising step (a second encoder pass)
  • −Tied to one base architecture; SD 1.5 ControlNets do not run on SDXL
  • −Inherits preprocessor errors (missed limbs in OpenPose, depth artefacts)
  • −Lighter alternatives (T2I-Adapter, ControlNet-XS) trade some fidelity for speed

Check yourself

At the first training step, which ControlNet parameters receive a non-zero gradient?

Check yourself

The generated image copies the edge map so closely that wallpaper texture shows up as stripes. What should you change first?

Remember

  • ✓ControlNet = frozen base + trainable copy of its encoder; yc=F(x;Θ)+Z(F(x+Z(c);Θc))y_c=\mathcal F(x;\Theta)+\mathcal Z(\mathcal F(x+\mathcal Z(c);\Theta_c)).
  • ✓Zero convolutions (1×1, weights and bias 0) make step 0 identical to the base model; Z₂ gets the first gradient, then the rest learn.
  • ✓On SD: copy of 12 encoder blocks + middle block, added to 12 skips + middle; the condition is reduced 512² → 64² by four small convs.
  • ✓Training is the normal denoising loss on (map, image, caption) triples with 50% empty prompts.
  • ✓At inference: conditioning scale s, start/end window and stacking several ControlNets are the main controls.
← back to Generative AI roadmap