# U-Net

> U-Net (Ronneberger, Fischer, Brox, 2015) pairs a contracting path with a symmetric expanding path joined by concatenated skips; with valid convolutions a 572² tile gives a 388² mask, and the design became the backbone of diffusion models.

Source: https://www.mlfoundry.in/learn/u-net  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**U-Net:** is an encoder–decoder network for **Image segmentation** in which every decoder level receives, by concatenation, the feature map of the encoder level at the same resolution. Ronneberger, Fischer and Brox introduced it in "U-Net: Convolutional Networks for Biomedical Image Segmentation" (MICCAI 2015). It was trained from scratch on 30 annotated electron-microscopy images.

The task: label each pixel of a microscopy image as cell or membrane/background. Annotated data is scarce, and cells touch, so the thin borders between them decide whether two cells are segmented as one. **FCN** had shown that a network can be trained end to end for dense labels. Its decoder was a few upsampled score maps, however, and its skips were summed class scores. U-Net makes the decoder as deep as the encoder and passes whole feature maps across.

## Architecture

The **contracting path** repeats two unpadded $3\times3$ convolutions with ReLU and a $2\times2$ max-pool with stride 2. The channels double after each pool: 64, 128, 256, 512, 1024. The **expanding path** repeats a $2\times2$ up-convolution that halves the channels. It then concatenates the centre-cropped encoder map at that level and applies two $3\times3$ convolutions. A final $1\times1$ convolution maps 64 channels to the 2 classes. The network has 23 convolutional layers and no fully connected layer.

Because the convolutions are "valid", each one removes a 1-pixel border, so the output is smaller than the input. The encoder map at 568² must be cropped to 392² before it is concatenated. Every pooled map must also have an even size. Working backwards, the legal input sides are $188+16k$:

$$
n_{\text{out}} = n_{\text{in}}-184,\qquad n_{\text{in}}\in\{188,\,204,\,\dots,\,572,\,\dots\}=188+16k
$$

- $184$: pixels lost to the 18 unpadded $3\times3$ convs, in input-resolution units ($2\cdot92$)
- $16k$: the four 2× poolings need the size before each pool to be even

> Figure: The original U-Net on a 572 × 572 tile. Left: the contracting path; each box shows the side after its two valid 3×3 convs and the channels. Right: the expanding path. Each level up-convolves the level below (2×2, channels halved), concatenates the centre-cropped encoder map (dashed), and convolves again. The 1×1 output layer gives 388 × 388 × 2, the centre of the input tile.

## Overlap-tile and the weighted loss

Each 388² output needs the 572² input around it. To segment a large image, U-Net slides the tile with stride 388 so that the outputs abut. At the image border the missing context is filled by **mirroring** the image. The architecture works on tiles of any legal size, so it is limited only by GPU memory, not by the image size.

Training uses a pixel-wise softmax and a cross-entropy in which every pixel has a precomputed weight:

$$
E=-\sum_{x\in\Omega}w(x)\,\log p_{\ell(x)}(x),\qquad w(x)=w_c(x)+w_0\exp\!\Big(-\frac{\big(d_1(x)+d_2(x)\big)^2}{2\sigma^2}\Big)
$$

- $p_{\ell(x)}(x)$: softmax probability of the true label $\ell(x)$ at pixel $x$
- $w_c$: weight map that balances the class frequencies
- $d_1,\,d_2$: distances to the border of the nearest and the second-nearest cell
- $w_0,\ \sigma$: $w_0=10$ and $\sigma\approx5$ pixels in the paper

Where two cells nearly touch, $d_1+d_2$ is small and the background pixels between them get weight close to $1+w_0$. Missing a 2-pixel gap would otherwise cost almost nothing, because it is a tiny fraction of the pixels. With the weights, it costs as much as misclassifying about ten ordinary pixels per gap pixel.

> Figure: The weight map for two cells, computed with w0 = 10 and w_c = 1. Default (gap 2 px, σ = 5): every gap pixel on the centre row has d1 + d2 = 3 and weight 1 + 10·e^(−9/50) = 9.35. The 120 of 2,304 pixels (5.2%) with w > 2 carry 21.0% of the total loss weight. Widen the gap to 8 px and the gap weight falls to 2.98, with a 9.6% share. Keep gap 2 and raise σ to 8: the ridge widens and 40.8% of the weight sits on 12.8% of the pixels.

**Training and results (from the paper)**

- **Data:** ISBI 2012 EM segmentation: 30 training images of 512 × 512 (Drosophila ventral nerve cord)
- **Augmentation:** random elastic deformations: displacements on a coarse 3 × 3 grid, std 10 px, bicubic interpolation; dropout at the end of the contracting path
- **Optimiser:** SGD, momentum 0.99, one large tile per batch; init Gaussian with std √(2/N), N = incoming nodes
- **Cost:** 10 hours of training on an NVIDIA Titan GPU (6 GB); a 512 × 512 image is segmented in under a second
- **EM challenge:** warping error 0.000353 and rand error 0.0382, first on the leaderboard at the time
- **Cell tracking 2015:** IoU 92.03% on PhC-U373 (second best 83%) and 77.56% on DIC-HeLa (second best 46%)

## Worked example: tiling a 512 × 512 image

Input pixels the network must process to cover a 512 × 512 image (262,144 px), for three legal tile sizes

| Tile in → out | Tiles | Input pixels | Overhead | Output / input per tile |
| --- | --- | --- | --- | --- |
| 508 → 324 | 2 × 2 = 4 | 4 × 508² = 1,032,256 | 3.94× | 0.407 |
| 572 → 388 | 2 × 2 = 4 | 4 × 572² = 1,308,736 | 4.99× | 0.460 |
| 700 → 516 | 1 | 700² = 490,000 | 1.87× | 0.543 |

The 184-pixel context margin is fixed, so larger tiles waste less. The paper prefers large input tiles over a large batch for this reason, and sets the momentum to 0.99 so that many earlier samples still shape each update.

```python
import numpy as np

def unet_out(n, depth=4):
    """Output side of the original U-Net (two valid 3×3 convs per level) or None if a pool sees an odd size."""
    for _ in range(depth):
        n -= 4
        if n <= 0 or n % 2:
            return None
        n //= 2
    n -= 4                                    # bottleneck convs
    for _ in range(depth):
        n = 2 * n - 4                         # up-conv 2×2, then two valid convs
    return n if n > 0 else None

print([(n, unet_out(n)) for n in range(560, 590) if unet_out(n)])   # [(572, 388), (588, 404)]

def overlap_tile(image, model, n_in=572):
    """Predict a large image tile by tile; context missing at the border is mirrored."""
    n_out = unet_out(n_in)
    m = (n_in - n_out) // 2                   # 92-pixel margin on each side
    H, W = image.shape
    th, tw = -(-H // n_out), -(-W // n_out)   # tiles per axis (ceil)
    padded = np.pad(image, ((m, th * n_out - H + m), (m, tw * n_out - W + m)), mode="reflect")
    out = np.zeros((th * n_out, tw * n_out))
    for i in range(th):
        for j in range(tw):
            tile = padded[i * n_out:i * n_out + n_in, j * n_out:j * n_out + n_in]
            out[i * n_out:(i + 1) * n_out, j * n_out:(j + 1) * n_out] = model(tile)
    return out[:H, :W], th * tw

fake_unet = lambda t: t[92:-92, 92:-92]       # stand-in with the right geometry: centre crop
img = np.random.default_rng(0).random((512, 512))
pred, n_tiles = overlap_tile(img, fake_unet)
print(n_tiles, np.allclose(pred, img))        # 4 True
```

Legal tile sizes and overlap-tile inference with mirrored borders (numpy, runs as is; a centre-crop stands in for a trained U-Net)

## From microscopy to diffusion

Most modern U-Nets use padded ("same") convolutions, so output size equals input size and no cropping is needed. Descendants include 3D U-Net for volumes (Çiçek et al., 2016), **U-Net++** with nested dense skips, and nnU-Net (Isensee et al., 2021), which configures a U-Net automatically for each medical dataset.

The same shape became the standard denoiser of **Diffusion models**. A denoiser must output an image-sized noise estimate in which both global layout and pixel detail are right; the U-Net's multi-scale path and full-resolution skips give exactly that. DDPM (Ho, Jain and Abbeel, 2020) used a U-Net with group normalisation, self-attention at the $16\times16$ level and a sinusoidal time-step embedding added in every residual block. **Stable Diffusion** runs a U-Net in a latent space and injects the text through **Cross-attention**.

**Strengths**

- Concatenated skips keep exact boundary detail; the decoder learns how to use it.
- Trains from very few images with strong elastic augmentation.
- Fully convolutional: any legal tile size, and seamless overlap-tile inference on huge images.

**Limitations**

- Valid convolutions waste a 92-px margin per side of each tile.
- Plain U-Net cannot separate touching objects by itself; the border weights or instance-aware heads are needed.
- Memory grows with the full-resolution skip maps, which limits 3D and high-resolution use.

**Question:** Why does the original U-Net crop the encoder map before concatenating it?

- A. To save memory
- B. Unpadded convolutions shrink every map, so the encoder map (e.g. 568²) is larger than the upsampled decoder map (392²)
- C. Border pixels of the encoder map are mirrored and must be removed
- D. Cropping acts as data augmentation

**Answer:** B. Unpadded convolutions shrink every map, so the encoder map (e.g. 568²) is larger than the upsampled decoder map (392²) — Each valid $3\times3$ conv trims one pixel per side. By the time the decoder returns to the top level it has lost more than the encoder had, so the encoder map is centre-cropped to match: $568\to392$ at the top, $64\to56$ at the deepest skip.

**Question:** What is the output size of the original U-Net for a 700 × 700 input?

- A. 700 × 700
- B. 516 × 516
- C. 388 × 388
- D. The input is not legal

**Answer:** B. 516 × 516 — $700=188+16\cdot32$ is legal (sizes before the pools: 696, 344, 168, 80). The output is $700-184=516$.

**Key points**

- Contracting path (2 × valid 3×3 conv + 2×2 max-pool, channels 64 → 1024) and a symmetric expanding path (2×2 up-conv, concat, 2 × 3×3 conv).
- Valid convs: 572² in → 388² out; legal inputs $188+16k$, output = input − 184; encoder maps are centre-cropped before concatenation.
- Overlap-tile inference with mirrored borders segments images of any size.
- Weighted cross-entropy $w=w_c+w_0e^{-(d_1+d_2)^2/2\sigma^2}$, $w_0=10$, $\sigma\approx5$ px, forces the gaps between touching cells to be learned.
- Won the ISBI 2012 EM and 2015 cell-tracking challenges from 30 training images and heavy elastic augmentation.
- With padded convs, attention and time embeddings it became the denoiser in DDPM and **Stable Diffusion**.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
