Definition
U-Net is an encoder–decoder network for Image segmentation in which every decoder level receives, by concatenation, the feature map of the encoder level at the same resolution. Ronneberger, Fischer and Brox introduced it in "U-Net: Convolutional Networks for Biomedical Image Segmentation" (MICCAI 2015). It was trained from scratch on 30 annotated electron-microscopy images.
The task: label each pixel of a microscopy image as cell or membrane/background. Annotated data is scarce, and cells touch, so the thin borders between them decide whether two cells are segmented as one. FCN had shown that a network can be trained end to end for dense labels. Its decoder was a few upsampled score maps, however, and its skips were summed class scores. U-Net makes the decoder as deep as the encoder and passes whole feature maps across.
Architecture
The contracting path repeats two unpadded convolutions with ReLU and a max-pool with stride 2. The channels double after each pool: 64, 128, 256, 512, 1024. The expanding path repeats a up-convolution that halves the channels. It then concatenates the centre-cropped encoder map at that level and applies two convolutions. A final convolution maps 64 channels to the 2 classes. The network has 23 convolutional layers and no fully connected layer.
Because the convolutions are "valid", each one removes a 1-pixel border, so the output is smaller than the input. The encoder map at 568² must be cropped to 392² before it is concatenated. Every pooled map must also have an even size. Working backwards, the legal input sides are :
- pixels lost to the 18 unpadded convs, in input-resolution units ()
- the four 2× poolings need the size before each pool to be even
Overlap-tile and the weighted loss
Each 388² output needs the 572² input around it. To segment a large image, U-Net slides the tile with stride 388 so that the outputs abut. At the image border the missing context is filled by mirroring the image. The architecture works on tiles of any legal size, so it is limited only by GPU memory, not by the image size.
Training uses a pixel-wise softmax and a cross-entropy in which every pixel has a precomputed weight:
- softmax probability of the true label at pixel
- weight map that balances the class frequencies
- distances to the border of the nearest and the second-nearest cell
- and pixels in the paper
Where two cells nearly touch, is small and the background pixels between them get weight close to . Missing a 2-pixel gap would otherwise cost almost nothing, because it is a tiny fraction of the pixels. With the weights, it costs as much as misclassifying about ten ordinary pixels per gap pixel.
Training and results (from the paper)
- Data
- ISBI 2012 EM segmentation: 30 training images of 512 × 512 (Drosophila ventral nerve cord)
- Augmentation
- random elastic deformations: displacements on a coarse 3 × 3 grid, std 10 px, bicubic interpolation; dropout at the end of the contracting path
- Optimiser
- SGD, momentum 0.99, one large tile per batch; init Gaussian with std √(2/N), N = incoming nodes
- Cost
- 10 hours of training on an NVIDIA Titan GPU (6 GB); a 512 × 512 image is segmented in under a second
- EM challenge
- warping error 0.000353 and rand error 0.0382, first on the leaderboard at the time
- Cell tracking 2015
- IoU 92.03% on PhC-U373 (second best 83%) and 77.56% on DIC-HeLa (second best 46%)
Worked example: tiling a 512 × 512 image
| Tile in → out | Tiles | Input pixels | Overhead | Output / input per tile |
|---|---|---|---|---|
| 508 → 324 | 2 × 2 = 4 | 4 × 508² = 1,032,256 | 3.94× | 0.407 |
| 572 → 388 | 2 × 2 = 4 | 4 × 572² = 1,308,736 | 4.99× | 0.460 |
| 700 → 516 | 1 | 700² = 490,000 | 1.87× | 0.543 |
The 184-pixel context margin is fixed, so larger tiles waste less. The paper prefers large input tiles over a large batch for this reason, and sets the momentum to 0.99 so that many earlier samples still shape each update.
1import numpy as np23def unet_out(n, depth=4):4 """Output side of the original U-Net (two valid 3×3 convs per level) or None if a pool sees an odd size."""5 for _ in range(depth):6 n -= 47 if n <= 0 or n % 2:8 return None9 n //= 210 n -= 4 # bottleneck convs11 for _ in range(depth):12 n = 2 * n - 4 # up-conv 2×2, then two valid convs13 return n if n > 0 else None1415print([(n, unet_out(n)) for n in range(560, 590) if unet_out(n)]) # [(572, 388), (588, 404)]1617def overlap_tile(image, model, n_in=572):18 """Predict a large image tile by tile; context missing at the border is mirrored."""19 n_out = unet_out(n_in)20 m = (n_in - n_out) // 2 # 92-pixel margin on each side21 H, W = image.shape22 th, tw = -(-H // n_out), -(-W // n_out) # tiles per axis (ceil)23 padded = np.pad(image, ((m, th * n_out - H + m), (m, tw * n_out - W + m)), mode="reflect")24 out = np.zeros((th * n_out, tw * n_out))25 for i in range(th):26 for j in range(tw):27 tile = padded[i * n_out:i * n_out + n_in, j * n_out:j * n_out + n_in]28 out[i * n_out:(i + 1) * n_out, j * n_out:(j + 1) * n_out] = model(tile)29 return out[:H, :W], th * tw3031fake_unet = lambda t: t[92:-92, 92:-92] # stand-in with the right geometry: centre crop32img = np.random.default_rng(0).random((512, 512))33pred, n_tiles = overlap_tile(img, fake_unet)34print(n_tiles, np.allclose(pred, img)) # 4 TrueFrom microscopy to diffusion
Most modern U-Nets use padded ("same") convolutions, so output size equals input size and no cropping is needed. Descendants include 3D U-Net for volumes (Çiçek et al., 2016), U-Net++ with nested dense skips, and nnU-Net (Isensee et al., 2021), which configures a U-Net automatically for each medical dataset.
The same shape became the standard denoiser of Diffusion models. A denoiser must output an image-sized noise estimate in which both global layout and pixel detail are right; the U-Net's multi-scale path and full-resolution skips give exactly that. DDPM (Ho, Jain and Abbeel, 2020) used a U-Net with group normalisation, self-attention at the level and a sinusoidal time-step embedding added in every residual block. Stable Diffusion runs a U-Net in a latent space and injects the text through Cross-attention.
Use when
- +Concatenated skips keep exact boundary detail; the decoder learns how to use it.
- +Trains from very few images with strong elastic augmentation.
- +Fully convolutional: any legal tile size, and seamless overlap-tile inference on huge images.
Avoid when
- −Valid convolutions waste a 92-px margin per side of each tile.
- −Plain U-Net cannot separate touching objects by itself; the border weights or instance-aware heads are needed.
- −Memory grows with the full-resolution skip maps, which limits 3D and high-resolution use.
Check yourself
Why does the original U-Net crop the encoder map before concatenating it?
Check yourself
What is the output size of the original U-Net for a 700 × 700 input?
Remember
- ✓Contracting path (2 × valid 3×3 conv + 2×2 max-pool, channels 64 → 1024) and a symmetric expanding path (2×2 up-conv, concat, 2 × 3×3 conv).
- ✓Valid convs: 572² in → 388² out; legal inputs , output = input − 184; encoder maps are centre-cropped before concatenation.
- ✓Overlap-tile inference with mirrored borders segments images of any size.
- ✓Weighted cross-entropy , , px, forces the gaps between touching cells to be learned.
- ✓Won the ISBI 2012 EM and 2015 cell-tracking challenges from 30 training images and heavy elastic augmentation.
- ✓With padded convs, attention and time embeddings it became the denoiser in DDPM and Stable Diffusion.