# ControlNet

> ControlNet steers a frozen text-to-image diffusion model with a spatial map (edges, depth, pose): a trainable copy of the encoder reads the map and feeds zero-initialised 1×1 convolutions, so training starts from the unchanged model.

Source: https://www.mlfoundry.in/learn/controlnet  
Section: Generative AI › Multimodal Generative AI  
Level: advanced · 7 min read · updated 2026-10-21

**ControlNet:** a side network that adds **spatial conditioning** to a pretrained, frozen diffusion model. It is a trainable copy of the denoiser's encoder that reads a condition image (Canny edges, a depth map, a human pose skeleton, a segmentation map, a scribble) and adds its features back into the frozen network through **zero convolutions**: 1×1 convolutions whose weights and biases start at zero. Introduced by Lvmin Zhang, Anyi Rao and Maneesh Agrawala, "Adding Conditional Control to Text-to-Image Diffusion Models" (ICCV 2023).

A prompt cannot pin down layout. "A dancer, arms raised" gives a different pose with every seed, and words like "left" or "behind" are followed loosely. Designers, game studios and product teams often already have the structure they want: a sketch, a 3-D render, a pose from a reference photo, a floor plan. ControlNet turns that structure into a hard constraint while the prompt still decides content and style.

In a pipeline it sits next to the base model: a preprocessor (Canny, a monocular depth estimator, OpenPose) turns a reference image into a condition map, the ControlNet reads that map at every denoising step, and the base model, its LoRAs and its sampler stay unchanged. The base itself is covered in **Stable Diffusion** and **Latent diffusion models**; the generation call in **Text-to-image generation**.

## The trainable copy and zero convolutions

Take a frozen block $\mathcal F(\cdot;\Theta)$ that maps features $x$ to $y$. ControlNet locks $\Theta$, makes a trainable clone $\Theta_c$, and joins the clone to the original with two zero convolutions $\mathcal Z$:

$$
y_c=\mathcal F(x;\Theta)+\mathcal Z\big(\mathcal F\big(x+\mathcal Z(c;\Theta_{z1});\,\Theta_c\big);\,\Theta_{z2}\big)
$$

- $\mathcal F(x;\Theta)$: the frozen pretrained block (locked weights)
- $\Theta_c$: trainable copy, initialised to $\Theta$
- $c$: condition features, from a small encoder over the condition image
- $\mathcal Z(\cdot;\Theta_{z1}),\ \mathcal Z(\cdot;\Theta_{z2})$: 1×1 convolutions with weight and bias initialised to 0

At step 0 both zero convolutions output 0, so $y_c=y$: the combined model is exactly the pretrained one, and no random noise is injected into deep features. Training still moves. For $y=Wh$ with $W=0$, the gradient $\partial\mathcal L/\partial W=(\partial\mathcal L/\partial y)\,h^\top$ is non-zero because $h$, the copy's output, is non-zero. After the first update $\Theta_{z2}\neq0$, gradients flow back into $\Theta_c$ and $\Theta_{z1}$, and the branch learns.

On Stable Diffusion the clone covers the U-Net's 12 encoder blocks and the middle block. Its 13 outputs, each through its own zero convolution, are added to the 12 skip connections and to the middle block's output. The 512×512 condition image is first reduced to the 64×64 latent grid by a tiny encoder of four convolutions (4×4 kernels, stride 2, 16, 32, 64 and 128 channels, ReLU).

> Figure: ControlNet on SD 1.5. Top: the frozen U-Net, encoder E1–E4 and the middle block going right, decoder D4–D1 coming back, with skip connections. Bottom: the trainable copy reads the noisy latent plus Z₁ of the condition features (Canny edges, 512² reduced to 64² by four convolutions). Each copy block's output goes through its own zero convolution Z₂ and is added, times the conditioning scale s, to a decoder input or the middle output. At initialisation every Z outputs 0.

## Training

$$
\mathcal L=\mathbb E_{z_0,t,c_t,c_f,\epsilon\sim\mathcal N(0,I)}\Big[\big\lVert\epsilon-\epsilon_\theta(z_t,\,t,\,c_t,\,c_f)\big\rVert_2^2\Big]
$$

- $z_t$: the noised latent of the training image at step t
- $c_t,\ c_f$: text prompt and task-specific condition (edge map, depth, pose)
- $\epsilon_\theta$: frozen U-Net plus ControlNet; only the copy, the small condition encoder and the zero convs get gradients

It is the ordinary denoising loss on (condition, image, caption) triples, which are cheap to make: run the preprocessor over captioned photos. During training 50% of the prompts are replaced by empty strings, so the network learns to read meaning from the condition map alone. The paper reports a "sudden convergence": the model does not follow the map gradually but switches abruptly from ignoring it to following it, usually in fewer than 10K optimisation steps.

**From the paper and the reference implementation**

- **Released conditions:** Canny edges, depth (MiDaS), HED soft edges, M-LSD straight lines, OpenPose skeletons, ADE20K segmentation, normal maps, user scribbles
- **Training cost:** about 23% more GPU memory and 34% more time per iteration than fine-tuning SD alone (single A100, as reported)
- **Size:** for SD 1.5 the copy is roughly 360M parameters, against about 860M in the U-Net
- **Data:** trains robustly on small (under 50K) and large (over 1M) datasets
- **Reuse:** a ControlNet trained on SD 1.5 also works on community fine-tunes of SD 1.5, because they share the architecture

## Zero init versus random init, live

The figure trains a 12-dimensional toy ControlNet, $y=W_bx+s\,Z_2\,\mathrm{relu}(W_c(x+Z_1c))$, with Adam on 48 samples. $W_b$ is frozen, $W_c$ starts as a copy of it, and $Z_1,Z_2$ start either at zero or at Gaussian noise of scale σ.

> Figure: Default (lr 0.01, σ = 0.1, step 300, s = 1). With zero init the step-0 loss is 3.992, exactly the frozen base model's, and the step-0 gradients are 0.00 for W_c and Z₁ and 4.19 for Z₂ alone. Random init starts worse, at 5.189, and ends at 0.052 against 0.004. At σ = 0.3 the random run starts at 17.105 and ends at 0.180. The random runs also drift the copy further from the frozen weights (0.783 against 0.474). Drag s on the trained zero-init run: s = 0.5 gives loss 1.002, and s = 0 gives back 3.992, the base model.

## Worked example: one scalar ControlNet

$y=w_bx+z_2\,\mathrm{relu}(w_c(x+z_1c))$ with $w_b=w_c=0.5$, $z_1=z_2=0$, input $x=1$, condition $c=2$, target $1.3$, loss $(y-1.3)^2$, plain gradient descent with step 0.5.

Computed in Python

| Step | w_c, z₁, z₂ | Output y | Loss | ∂L/∂z₂ | ∂L/∂w_c | ∂L/∂z₁ |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | 0.5, 0, 0 | 0.5000 | 0.6400 | −0.8000 | 0 | 0 |
| 1 | 0.5, 0, 0.4 | 0.7000 | 0.3600 | −0.6000 | −0.4800 | −0.4800 |
| 2 | 0.74, 0.24, 0.7 | 1.2666 | 0.0011 | −0.0731 | −0.0691 | −0.0691 |

Step 0 matches the frozen model (output 0.5) and only $z_2$ has a gradient, $2(0.5-1.3)\cdot\mathrm{relu}(0.5)=-0.8$. One update later $z_2=0.4$, and the gradients of $w_c$ and $z_1$ switch on. By step 2 the loss has fallen from 0.64 to 0.0011.

## Using it

```python
import cv2, numpy as np, torch
from PIL import Image
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline

ref = np.array(Image.open("room.jpg").convert("RGB").resize((512, 512)))
edges = cv2.Canny(ref, 100, 200)                       # low / high hysteresis thresholds
cond = Image.fromarray(np.stack([edges] * 3, axis=-1))  # 3-channel edge map

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
).to("cuda")

image = pipe(
    "a scandinavian living room, morning light", image=cond,
    num_inference_steps=30, guidance_scale=7.5,
    controlnet_conditioning_scale=0.8,   # s: multiplies every residual
    control_guidance_end=0.6,            # stop control after 60% of the steps
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
```

Canny ControlNet with Hugging Face diffusers and OpenCV (needs a GPU and model downloads; not run here)

1. **Pick the condition** that encodes what must not change: Canny or line art for exact outlines, depth for layout and perspective, OpenPose for body pose, segmentation for region-by-region content.
2. **Match resolution and base**: a ControlNet works only with the base family it was trained for (SD 1.5, SDXL and so on), and the map should have the generation size.
3. **Tune the conditioning scale** `s`, usually 0.5–1.0. Too high and the image copies preprocessing artefacts; too low and the structure drifts.
4. **Limit the window** with `control_guidance_start` / `control_guidance_end`: early steps decide layout, so ending control at 50–70% keeps structure and frees fine detail.
5. **Stack** several ControlNets (pose plus depth) by passing lists of models, maps and scales; their residuals are summed.

> **The map is a constraint, not a suggestion:** If the prompt contradicts the map (a "cat" over a human pose skeleton), the result is usually a malformed hybrid, not a cat. Also check the preprocessed map before generating: a Canny threshold that picks up wallpaper texture will be reproduced as texture. The map is the input; what the preprocessor missed or invented, the ControlNet follows.

> **Why not fine-tune the base, or train from scratch?:** Fine-tuning the whole U-Net on a few hundred thousand condition pairs risks forgetting what billions of images taught it. Adding new layers with random weights injects noise into deep features and damages the model at step 0. The locked copy plus zero convolutions avoids both: the original weights never change, and the new path starts silent. The same "start as identity" idea appears as Flamingo's zero-initialised tanh gates and LoRA's zero-initialised $B$ matrix.

**Strengths**

- Exact spatial control from maps that are cheap to produce
- Base model untouched: one ControlNet serves its fine-tunes, LoRAs and samplers
- Composable: several conditions, each with its own scale and step window
- Trains on a single GPU from modest datasets

**Limitations**

- Extra compute at every denoising step (a second encoder pass)
- Tied to one base architecture; SD 1.5 ControlNets do not run on SDXL
- Inherits preprocessor errors (missed limbs in OpenPose, depth artefacts)
- Lighter alternatives (T2I-Adapter, ControlNet-XS) trade some fidelity for speed

**Question:** At the first training step, which ControlNet parameters receive a non-zero gradient?

- A. None: with zero weights everywhere the gradient is zero
- B. Only the output zero convolutions Z₂
- C. Only the trainable copy Θ_c
- D. All of them equally

**Answer:** B. Only the output zero convolutions Z₂ — The gradient of a linear layer's weight is (upstream gradient) × (its input). Z₂'s input is the copy's output, which is non-zero, so Z₂ moves. Θ_c and Z₁ sit behind Z₂, whose zero weights block the gradient to them. After one update Z₂ is non-zero and they start to learn.

**Question:** The generated image copies the edge map so closely that wallpaper texture shows up as stripes. What should you change first?

- A. Raise the classifier-free guidance scale
- B. Lower controlnet_conditioning_scale and/or end control earlier, and clean the map (higher Canny thresholds)
- C. Switch to a different sampler
- D. Retrain the ControlNet

**Answer:** B. Lower controlnet_conditioning_scale and/or end control earlier, and clean the map (higher Canny thresholds) — The stripes come from the condition: the map contains them and the control is strong across all steps. Weaker or earlier-ending control and a cleaner map fix it; CFG changes how strongly the prompt is followed, not the map.

**Key points**

- ControlNet = frozen base + trainable copy of its encoder; $y_c=\mathcal F(x;\Theta)+\mathcal Z(\mathcal F(x+\mathcal Z(c);\Theta_c))$.
- Zero convolutions (1×1, weights and bias 0) make step 0 identical to the base model; Z₂ gets the first gradient, then the rest learn.
- On SD: copy of 12 encoder blocks + middle block, added to 12 skips + middle; the condition is reduced 512² → 64² by four small convs.
- Training is the normal denoising loss on (map, image, caption) triples with 50% empty prompts.
- At inference: conditioning scale s, start/end window and stacking several ControlNets are the main controls.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
