U-Net

U-Net (Ronneberger, Fischer, Brox, 2015) pairs a contracting path with a symmetric expanding path joined by concatenated skips; with valid convolutions a 572² tile gives a 388² mask, and the design became the backbone of diffusion models.

8 min read660 wordscore0 viewsupdated 2026-10-15

Definition

U-Net is an encoder–decoder network for Image segmentation in which every decoder level receives, by concatenation, the feature map of the encoder level at the same resolution. Ronneberger, Fischer and Brox introduced it in "U-Net: Convolutional Networks for Biomedical Image Segmentation" (MICCAI 2015). It was trained from scratch on 30 annotated electron-microscopy images.

The task: label each pixel of a microscopy image as cell or membrane/background. Annotated data is scarce, and cells touch, so the thin borders between them decide whether two cells are segmented as one. FCN had shown that a network can be trained end to end for dense labels. Its decoder was a few upsampled score maps, however, and its skips were summed class scores. U-Net makes the decoder as deep as the encoder and passes whole feature maps across.

Architecture

The contracting path repeats two unpadded 3×33\times3 convolutions with ReLU and a 2×22\times2 max-pool with stride 2. The channels double after each pool: 64, 128, 256, 512, 1024. The expanding path repeats a 2×22\times2 up-convolution that halves the channels. It then concatenates the centre-cropped encoder map at that level and applies two 3×33\times3 convolutions. A final 1×11\times1 convolution maps 64 channels to the 2 classes. The network has 23 convolutional layers and no fully connected layer.

Because the convolutions are "valid", each one removes a 1-pixel border, so the output is smaller than the input. The encoder map at 568² must be cropped to 392² before it is concatenated. Every pooled map must also have an even size. Working backwards, the legal input sides are 188+16k188+16k:

nout=nin−184,nin∈{188, 204, …, 572, … }=188+16kn_{\text{out}} = n_{\text{in}}-184,\qquad n_{\text{in}}\in\{188,\,204,\,\dots,\,572,\,\dots\}=188+16k
184184
pixels lost to the 18 unpadded 3×33\times3 convs, in input-resolution units (2⋅922\cdot92)
16k16k
the four 2× poolings need the size before each pool to be even
U-Net (2015): every 3×3 conv is unpadded, so each one trims 2 pixelsinput tile 572 × 572 × 11×1 conv → 388 × 388 × 2572 → 570 → 56864 ch392 → 390 → 388128 → 64 chcrop 568 → 392, concat284 → 282 → 280128 ch200 → 198 → 196256 → 128 chcrop 280 → 200, concat140 → 138 → 136256 ch104 → 102 → 100512 → 256 chcrop 136 → 104, concat68 → 66 → 64512 ch56 → 54 → 521024 → 512 chcrop 64 → 56, concat32 → 30 → 281024 ch (bottleneck)up-conv 2×2: 28 → 56max-pool 2×2: 64 → 32↓ max-pool 2×2 (channels ×2 after) · ↑ up-conv 2×2 (channels ÷2) · ⇢ copy, centre-crop, concatenate23 conv layers · output = centre 388² of the 572² input; adjacent input tiles overlap by 184 pxnumbers: side length after each layer · channels (right side: after concat → after convs)
fig.The original U-Net on a 572 × 572 tile. Left: the contracting path; each box shows the side after its two valid 3×3 convs and the channels. Right: the expanding path. Each level up-convolves the level below (2×2, channels halved), concatenates the centre-cropped encoder map (dashed), and convolves again. The 1×1 output layer gives 388 × 388 × 2, the centre of the input tile.

Overlap-tile and the weighted loss

Each 388² output needs the 572² input around it. To segment a large image, U-Net slides the tile with stride 388 so that the outputs abut. At the image border the missing context is filled by mirroring the image. The architecture works on tiles of any legal size, so it is limited only by GPU memory, not by the image size.

Training uses a pixel-wise softmax and a cross-entropy in which every pixel has a precomputed weight:

E=−∑x∈Ωw(x) log⁡pℓ(x)(x),w(x)=wc(x)+w0exp⁡ ⁣(−(d1(x)+d2(x))22σ2)E=-\sum_{x\in\Omega}w(x)\,\log p_{\ell(x)}(x),\qquad w(x)=w_c(x)+w_0\exp\!\Big(-\frac{\big(d_1(x)+d_2(x)\big)^2}{2\sigma^2}\Big)
pℓ(x)(x)p_{\ell(x)}(x)
softmax probability of the true label ℓ(x)\ell(x) at pixel xx
wcw_c
weight map that balances the class frequencies
d1, d2d_1,\,d_2
distances to the border of the nearest and the second-nearest cell
w0, σw_0,\ \sigma
w0=10w_0=10 and σ≈5\sigma\approx5 pixels in the paper

Where two cells nearly touch, d1+d2d_1+d_2 is small and the background pixels between them get weight close to 1+w01+w_0. Missing a 2-pixel gap would otherwise cost almost nothing, because it is a tiny fraction of the pixels. With the weights, it costs as much as misclassifying about ten ordinary pixels per gap pixel.

w(x) = w_c + w0 · exp(−(d1 + d2)² / 2σ²), w0 = 10, σ = 5 px, gap = 2 pxcell 1cell 2w along the dashed row (1 … 11)1611inside cells and far background: w = 1 · gap pixels: d1 + d2 = gap + 1gap pixel weight1 + 10·e^(−9/50)9.35pixels with w > 2120 of 2304 = 5.2%their share of Σw21.0% of the loss weightmax w = 9.35w_c = 1 (no classbalancing here)the loss is−Σ w(x) log p(x),so the thin gap mustbe predicted asbackground
fig.The weight map for two cells, computed with w0 = 10 and w_c = 1. Default (gap 2 px, σ = 5): every gap pixel on the centre row has d1 + d2 = 3 and weight 1 + 10·e^(−9/50) = 9.35. The 120 of 2,304 pixels (5.2%) with w > 2 carry 21.0% of the total loss weight. Widen the gap to 8 px and the gap weight falls to 2.98, with a 9.6% share. Keep gap 2 and raise σ to 8: the ridge widens and 40.8% of the weight sits on 12.8% of the pixels.

Training and results (from the paper)

Data
ISBI 2012 EM segmentation: 30 training images of 512 × 512 (Drosophila ventral nerve cord)
Augmentation
random elastic deformations: displacements on a coarse 3 × 3 grid, std 10 px, bicubic interpolation; dropout at the end of the contracting path
Optimiser
SGD, momentum 0.99, one large tile per batch; init Gaussian with std √(2/N), N = incoming nodes
Cost
10 hours of training on an NVIDIA Titan GPU (6 GB); a 512 × 512 image is segmented in under a second
EM challenge
warping error 0.000353 and rand error 0.0382, first on the leaderboard at the time
Cell tracking 2015
IoU 92.03% on PhC-U373 (second best 83%) and 77.56% on DIC-HeLa (second best 46%)

Worked example: tiling a 512 × 512 image

Tile in → outTilesInput pixelsOverheadOutput / input per tile
508 → 3242 × 2 = 44 × 508² = 1,032,2563.94×0.407
572 → 3882 × 2 = 44 × 572² = 1,308,7364.99×0.460
700 → 5161700² = 490,0001.87×0.543
Input pixels the network must process to cover a 512 × 512 image (262,144 px), for three legal tile sizes

The 184-pixel context margin is fixed, so larger tiles waste less. The paper prefers large input tiles over a large batch for this reason, and sets the momentum to 0.99 so that many earlier samples still shape each update.

python
1import numpy as np
2
3def unet_out(n, depth=4):
4 """Output side of the original U-Net (two valid 3×3 convs per level) or None if a pool sees an odd size."""
5 for _ in range(depth):
6 n -= 4
7 if n <= 0 or n % 2:
8 return None
9 n //= 2
10 n -= 4 # bottleneck convs
11 for _ in range(depth):
12 n = 2 * n - 4 # up-conv 2×2, then two valid convs
13 return n if n > 0 else None
14
15print([(n, unet_out(n)) for n in range(560, 590) if unet_out(n)]) # [(572, 388), (588, 404)]
16
17def overlap_tile(image, model, n_in=572):
18 """Predict a large image tile by tile; context missing at the border is mirrored."""
19 n_out = unet_out(n_in)
20 m = (n_in - n_out) // 2 # 92-pixel margin on each side
21 H, W = image.shape
22 th, tw = -(-H // n_out), -(-W // n_out) # tiles per axis (ceil)
23 padded = np.pad(image, ((m, th * n_out - H + m), (m, tw * n_out - W + m)), mode="reflect")
24 out = np.zeros((th * n_out, tw * n_out))
25 for i in range(th):
26 for j in range(tw):
27 tile = padded[i * n_out:i * n_out + n_in, j * n_out:j * n_out + n_in]
28 out[i * n_out:(i + 1) * n_out, j * n_out:(j + 1) * n_out] = model(tile)
29 return out[:H, :W], th * tw
30
31fake_unet = lambda t: t[92:-92, 92:-92] # stand-in with the right geometry: centre crop
32img = np.random.default_rng(0).random((512, 512))
33pred, n_tiles = overlap_tile(img, fake_unet)
34print(n_tiles, np.allclose(pred, img)) # 4 True
Legal tile sizes and overlap-tile inference with mirrored borders (numpy, runs as is; a centre-crop stands in for a trained U-Net)

From microscopy to diffusion

Most modern U-Nets use padded ("same") convolutions, so output size equals input size and no cropping is needed. Descendants include 3D U-Net for volumes (Çiçek et al., 2016), U-Net++ with nested dense skips, and nnU-Net (Isensee et al., 2021), which configures a U-Net automatically for each medical dataset.

The same shape became the standard denoiser of Diffusion models. A denoiser must output an image-sized noise estimate in which both global layout and pixel detail are right; the U-Net's multi-scale path and full-resolution skips give exactly that. DDPM (Ho, Jain and Abbeel, 2020) used a U-Net with group normalisation, self-attention at the 16×1616\times16 level and a sinusoidal time-step embedding added in every residual block. Stable Diffusion runs a U-Net in a latent space and injects the text through Cross-attention.

Use when

  • +Concatenated skips keep exact boundary detail; the decoder learns how to use it.
  • +Trains from very few images with strong elastic augmentation.
  • +Fully convolutional: any legal tile size, and seamless overlap-tile inference on huge images.

Avoid when

  • −Valid convolutions waste a 92-px margin per side of each tile.
  • −Plain U-Net cannot separate touching objects by itself; the border weights or instance-aware heads are needed.
  • −Memory grows with the full-resolution skip maps, which limits 3D and high-resolution use.

Check yourself

Why does the original U-Net crop the encoder map before concatenating it?

Check yourself

What is the output size of the original U-Net for a 700 × 700 input?

Remember

  • ✓Contracting path (2 × valid 3×3 conv + 2×2 max-pool, channels 64 → 1024) and a symmetric expanding path (2×2 up-conv, concat, 2 × 3×3 conv).
  • ✓Valid convs: 572² in → 388² out; legal inputs 188+16k188+16k, output = input − 184; encoder maps are centre-cropped before concatenation.
  • ✓Overlap-tile inference with mirrored borders segments images of any size.
  • ✓Weighted cross-entropy w=wc+w0e−(d1+d2)2/2σ2w=w_c+w_0e^{-(d_1+d_2)^2/2\sigma^2}, w0=10w_0=10, σ≈5\sigma\approx5 px, forces the gaps between touching cells to be learned.
  • ✓Won the ISBI 2012 EM and 2015 cell-tracking challenges from 30 training images and heavy elastic augmentation.
  • ✓With padded convs, attention and time embeddings it became the denoiser in DDPM and Stable Diffusion.
← back to Deep Learning roadmap