U-Net++

U-Net++ (Zhou et al., 2018) replaces U-Net's skips with a grid of nested, densely connected convolution nodes, supervises its four full-resolution outputs, and can be pruned at inference to the sub-network of any depth.

7 min read677 wordsadvanced0 viewsupdated 2026-10-15

Definition

U-Net++ is an encoder–decoder segmentation network in which the plain skip connections of U-Net are replaced by nested, dense skip pathways: a grid of convolution nodes Xi,jX^{i,j} that fuse every earlier node on the same level with an upsampled node from the level below. It was introduced by Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh and Jianming Liang in "UNet++: A Nested U-Net Architecture for Medical Image Segmentation" (DLMIA workshop, MICCAI 2018), with an extended journal version in IEEE Transactions on Medical Imaging (2020).

The task is the same as U-Net's: an image in, a per-pixel mask out, often from a few hundred labelled medical images. U-Net concatenates the encoder map at level ii directly with the decoder map at level ii. The authors argue that this joins features that are semantically far apart: shallow encoder features are edges and textures, the decoder features have passed through the whole network.

A second problem is depth. The best U-Net depth differs between datasets and is found by training several networks. U-Net++ embeds U-Nets of depths 1 to 4 in one model that shares a single encoder.

Nested skip pathways

Index nodes by level ii (downsampling step, top row i=0i=0) and column jj (convolution node along the skip pathway). Column j=0j=0 is the encoder. Every other node concatenates all earlier nodes of its row with the upsampled output of the node diagonally below:

xi,j={H(D(xi−1,0)),j=0H([ [xi,k]k=0j−1, U(xi+1,j−1)]),j>0x^{i,j}=\begin{cases}\mathcal H\big(\mathcal D(x^{i-1,0})\big), & j=0\\[2pt] \mathcal H\Big(\big[\,[x^{i,k}]_{k=0}^{j-1},\ \mathcal U\big(x^{i+1,j-1}\big)\big]\Big), & j>0\end{cases}
H\mathcal H
a convolution block (two 3×33\times3 convolutions with activations); its input for j=0j=0 is D\mathcal D, 2×22\times2 max-pooling of the node above
U\mathcal U
upsampling by 2, a 2×22\times2 stride-2 transposed convolution in the reference code
[⋅][\cdot]
channel concatenation; node Xi,jX^{i,j} receives (j+1)(j+1) maps of the level's width

U-Net is the special case that keeps only the outer edge of this grid: the encoder Xi,0X^{i,0} and the nodes with i+j=4i+j=4, each fed by one skip. The inner nodes are new. They form a dense block along each row, in the spirit of DenseNet, so the encoder map is refined by a few convolutions before it meets the decoder.

U-Net++: nested dense skip pathways between encoder and decoder (hover a node)level 0: 32 ch96×96level 1: 64 ch48×48level 2: 128 ch24×24level 3: 256 ch12×12level 4: 512 ch6×6L1L2L31×1, σ → ŷ11×1, σ → ŷ21×1, σ → ŷ31×1, σ → ŷ4X0,0X1,0X2,0X3,0X4,0X0,1X1,1X2,1X3,1X0,2X1,2X2,2X0,3X1,3X0,4node rule (j ≥ 1)X(i,j) = H( [ X(i,0), …,X(i,j−1), U(X(i+1,j−1))])H = two 3×3 convU = 2×2 up-conv[ ] = concatenationhovered nodenone——inferenceaccurate: mean ŷ1…ŷ4fast: keep i+j ≤ d,output ŷd (U-Net++ Ld)encoder X(i,0)U-Net decoder (i+j = 4)nested nodes (new)deep supervision↓ max-pool · ↗ upsample · ⇢ dense skip (arcs = longer skips) · violet = pruning cut Ld
fig.The U-Net++ grid with the paper's filter widths (32 to 512) at a 96×96 input. Blue nodes are the encoder, orange nodes the decoder that a depth-4 U-Net also has, green nodes the nested additions. Dashed arrows and arcs are the dense skips; diagonal arrows are 2×2 up-convolutions. Each X0,j for j = 1…4 has its own 1×1 sigmoid head. The violet cuts mark the pruned networks L1 to L3. Hover a node to see what it concatenates.

Deep supervision and pruning

The four top-row nodes X0,1,…,X0,4X^{0,1},\dots,X^{0,4} are full-resolution outputs of U-Nets of depth 1 to 4. With deep supervision, each gets a 1×11\times1 convolution and a sigmoid, and each output is trained against the same mask. The paper uses a combined binary cross-entropy and Dice loss on every head:

L(Y,Y^)=−1N∑b=1N(12 Yblog⁡Y^b+2 YbY^bYb+Y^b)\mathcal L(Y,\hat Y)=-\frac1N\sum_{b=1}^{N}\Big(\tfrac12\,Y_b\log\hat Y_b+\frac{2\,Y_b\hat Y_b}{Y_b+\hat Y_b}\Big)
Yb, Y^bY_b,\ \hat Y_b
flattened ground-truth mask and predicted probabilities of image bb; the products and sums run over pixels
NN
batch size; the loss is applied to each of the four heads

At test time there are two modes. Accurate mode averages the four outputs. Fast mode keeps one output X0,dX^{0,d} and deletes every node with i+j>di+j>d. This is exact, not an approximation: by the node rule, X0,dX^{0,d} depends only on nodes with i+j≤di+j\le d. The deleted nodes still mattered during training, because their gradients flowed back into the shared nodes. The result is called U-Net++ LdL^d.

Configuration and reported results (2018 paper)

Widths
32, 64, 128, 256, 512 filters at levels 0 to 4; four 2× downsamplings
Parameters
U-Net 7.76M, U-Net++ 9.04M (with or without deep supervision); the counts follow from the widths above
Tasks
four medical segmentation tasks: cell nuclei (microscopy), colon polyps (colonoscopy), liver (CT), lung nodules (CT, LIDC-IDRI)
Baselines
U-Net and a wide U-Net with more filters, to show that the gain is not just from parameter count
Result
U-Net++ with deep supervision: average IoU gain of 3.9 points over U-Net and 3.4 over wide U-Net
Pruning
U-Net++ L3L^3 cut inference time by 32.2% on average for a 0.6-point IoU drop
U-Net++ L4: 15 nodes · 9.04M params · 4.19G MACs at 96×96filters 32, 64, 128, 256, 512 · click a node · 100.0% of full U-Net++ compute0,01,02,03,04,00,11,12,13,10,21,22,20,31,30,4ŷ4parameters (M)0.100.100.470.511.932.217.769.04L1L2L3L4MACs at 96×96 (G)0.490.490.901.251.302.461.704.19L1L2L3L4U-NetU-Net++ (pruned to Ld)X0,4 at 96×96: in = X0,0 32 + X0,1 32 + X0,2 32 + X0,3 32 + up(X1,3) 32 = 160 ch → 32 ch2×2 up-conv + two 3×3 convs: 63,584 params · 528.5M MACs (12.6% of this net)params live in the deep levels (512 ch); compute lives in the top row (96×96), where U-Net++ adds most nodes
architecture
U-Net++ Ld keeps the nodes with i + j ≤ d and reads its mask from X0,d: the pruned model used at inference.
fig.Parameters and multiply-accumulates (MACs) of U-Net and U-Net++ at depths 1 to 4, counted layer by layer for a 96×96 RGB input. At the default, U-Net++ L4 has 15 nodes, 9.04M parameters and 4.19 GMACs, against 7.76M and 1.70 GMACs for U-Net: 17% more weights but 2.47× the compute. The extra cost sits in the top row, which holds 40.5% of all MACs and only 2.3% of the parameters. Click X0,4: it concatenates 160 channels and costs 63,584 parameters. Drag the depth to 3: U-Net++ L3 needs 2.21M parameters and 2.46 GMACs, 58.7% of the full model.

Worked example: the cost of one nested node

Node X0,2X^{0,2} at 96×9696\times96 concatenates X0,0X^{0,0}, X0,1X^{0,1} (32 channels each) and U(X1,1)\mathcal U(X^{1,1}), which maps 64 channels down to 32. A convolution with cinc_{in} inputs, coutc_{out} outputs and a k×kk\times k kernel has cink2cout+coutc_{in}k^2c_{out}+c_{out} parameters and cink2coutc_{in}k^2c_{out} MACs per output pixel.

LayerChannelsParametersMACs
up-conv 2×2 of X1,164 → 3264·4·32 + 32 = 8,22464·32·9,216 = 18.9M
3×3 conv on the concat32 + 32 + 32 = 96 → 3296·9·32 + 32 = 27,68096·9·32·9,216 = 254.8M
3×3 conv32 → 3232·9·32 + 32 = 9,24832·9·32·9,216 = 84.9M
node total45,152358.6M
Node X0,2 of U-Net++ (96×96 input, 9,216 output pixels per layer)

The node has under 0.5% of the model's 9.04M parameters but 8.6% of its 4.19 GMACs. Nested nodes at full resolution are cheap in memory for weights and expensive in compute and activations, the reverse of the deep levels, where levels 3 and 4 hold 74% of the parameters.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def block(cin, cout): # H: two 3x3 conv + ReLU
4 return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.ReLU(inplace=True),
5 nn.Conv2d(cout, cout, 3, padding=1), nn.ReLU(inplace=True))
6
7class UNetPP(nn.Module):
8 def __init__(self, cin=3, f=(32, 64, 128, 256, 512), L=4):
9 super().__init__()
10 self.L, self.pool = L, nn.MaxPool2d(2)
11 self.node, self.up = nn.ModuleDict(), nn.ModuleDict()
12 for i in range(L + 1):
13 self.node[f"{i}_0"] = block(cin if i == 0 else f[i - 1], f[i])
14 for j in range(1, L + 1):
15 for i in range(L + 1 - j):
16 self.up[f"{i}_{j}"] = nn.ConvTranspose2d(f[i + 1], f[i], 2, stride=2)
17 self.node[f"{i}_{j}"] = block((j + 1) * f[i], f[i])
18 self.head = nn.ModuleList(nn.Conv2d(f[0], 1, 1) for _ in range(L))
19
20 def forward(self, x, depth=None): # depth d < L runs only nodes with i + j <= d
21 d, X = depth or self.L, {}
22 for i in range(d + 1):
23 X[i, 0] = self.node[f"{i}_0"](x if i == 0 else self.pool(X[i - 1, 0]))
24 for j in range(1, d + 1):
25 for i in range(d + 1 - j):
26 skips = [X[i, k] for k in range(j)] + [self.up[f"{i}_{j}"](X[i + 1, j - 1])]
27 X[i, j] = self.node[f"{i}_{j}"](torch.cat(skips, dim=1))
28 return [self.head[j - 1](X[0, j]) for j in range(1, d + 1)] # logits of heads 1..d
29
30def bce_dice(logit, y, eps=1.0):
31 p = torch.sigmoid(logit)
32 dice = (2 * (p * y).sum() + eps) / (p.sum() + y.sum() + eps)
33 return 0.5 * F.binary_cross_entropy_with_logits(logit, y) + (1 - dice)
34
35model = UNetPP()
36x, y = torch.randn(2, 3, 96, 96), (torch.rand(2, 1, 96, 96) > 0.7).float()
37loss = sum(bce_dice(o, y) for o in model(x)) / 4 # deep supervision on X0,1..X0,4
38accurate = torch.stack([o.sigmoid() for o in model(x)]).mean(0) # accurate mode
39fast = model(x, depth=2)[-1].sigmoid() # fast mode: U-Net++ L2
A minimal U-Net++ in PyTorch with deep supervision and depth pruning (not run here: no PyTorch in the sandbox). Without batch norm its parameter count equals the 9.04M above plus three extra 1×1 heads.

Use when

  • +Encoder features pass through refining convolutions before they meet the decoder, narrowing the semantic gap the design targets
  • +One training run yields U-Nets of four depths; choose the speed/accuracy point afterwards
  • +Only about 17% more parameters than U-Net at the same widths

Avoid when

  • −About 2.5× U-Net's compute at depth 4, and much more activation memory, because the extra nodes run at the highest resolutions
  • −The gains over a well-tuned U-Net are a few IoU points and vary by dataset
  • −Dense concatenation makes the top-row inputs wide (160 channels at X0,4)

Check yourself

Why can U-Net++ drop every node with i + j > 2 at inference and still produce exactly the same output at X0,2?

Check yourself

Node X1,2 sits on level 1 (64 filters). How many channels enter its first 3×3 convolution?

Remember

  • ✓Nodes Xi,j=H([xi,0,…,xi,j−1,U(xi+1,j−1)])X^{i,j}=\mathcal H([x^{i,0},\dots,x^{i,j-1},\mathcal U(x^{i+1,j-1})]): dense skips along each level, upsampling from below.
  • ✓U-Net is the outer edge of the grid; U-Net++ adds the inner nodes and nests U-Nets of depths 1 to 4 on one encoder.
  • ✓Deep supervision: a 1×1 sigmoid head on each of X0,1…X0,4, trained with BCE plus Dice.
  • ✓Inference: average the heads (accurate) or prune to L^d by keeping i + j ≤ d (fast, exact).
  • ✓Paper widths 32 to 512: 9.04M vs 7.76M parameters; about 2.5× U-Net's MACs, mostly in the full-resolution row.
← back to Deep Learning roadmap