Definition
U-Net++ is an encoder–decoder segmentation network in which the plain skip connections of U-Net are replaced by nested, dense skip pathways: a grid of convolution nodes that fuse every earlier node on the same level with an upsampled node from the level below. It was introduced by Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh and Jianming Liang in "UNet++: A Nested U-Net Architecture for Medical Image Segmentation" (DLMIA workshop, MICCAI 2018), with an extended journal version in IEEE Transactions on Medical Imaging (2020).
The task is the same as U-Net's: an image in, a per-pixel mask out, often from a few hundred labelled medical images. U-Net concatenates the encoder map at level directly with the decoder map at level . The authors argue that this joins features that are semantically far apart: shallow encoder features are edges and textures, the decoder features have passed through the whole network.
A second problem is depth. The best U-Net depth differs between datasets and is found by training several networks. U-Net++ embeds U-Nets of depths 1 to 4 in one model that shares a single encoder.
Nested skip pathways
Index nodes by level (downsampling step, top row ) and column (convolution node along the skip pathway). Column is the encoder. Every other node concatenates all earlier nodes of its row with the upsampled output of the node diagonally below:
- a convolution block (two convolutions with activations); its input for is , max-pooling of the node above
- upsampling by 2, a stride-2 transposed convolution in the reference code
- channel concatenation; node receives maps of the level's width
U-Net is the special case that keeps only the outer edge of this grid: the encoder and the nodes with , each fed by one skip. The inner nodes are new. They form a dense block along each row, in the spirit of DenseNet, so the encoder map is refined by a few convolutions before it meets the decoder.
Deep supervision and pruning
The four top-row nodes are full-resolution outputs of U-Nets of depth 1 to 4. With deep supervision, each gets a convolution and a sigmoid, and each output is trained against the same mask. The paper uses a combined binary cross-entropy and Dice loss on every head:
- flattened ground-truth mask and predicted probabilities of image ; the products and sums run over pixels
- batch size; the loss is applied to each of the four heads
At test time there are two modes. Accurate mode averages the four outputs. Fast mode keeps one output and deletes every node with . This is exact, not an approximation: by the node rule, depends only on nodes with . The deleted nodes still mattered during training, because their gradients flowed back into the shared nodes. The result is called U-Net++ .
Configuration and reported results (2018 paper)
- Widths
- 32, 64, 128, 256, 512 filters at levels 0 to 4; four 2× downsamplings
- Parameters
- U-Net 7.76M, U-Net++ 9.04M (with or without deep supervision); the counts follow from the widths above
- Tasks
- four medical segmentation tasks: cell nuclei (microscopy), colon polyps (colonoscopy), liver (CT), lung nodules (CT, LIDC-IDRI)
- Baselines
- U-Net and a wide U-Net with more filters, to show that the gain is not just from parameter count
- Result
- U-Net++ with deep supervision: average IoU gain of 3.9 points over U-Net and 3.4 over wide U-Net
- Pruning
- U-Net++ cut inference time by 32.2% on average for a 0.6-point IoU drop
Worked example: the cost of one nested node
Node at concatenates , (32 channels each) and , which maps 64 channels down to 32. A convolution with inputs, outputs and a kernel has parameters and MACs per output pixel.
| Layer | Channels | Parameters | MACs |
|---|---|---|---|
| up-conv 2×2 of X1,1 | 64 → 32 | 64·4·32 + 32 = 8,224 | 64·32·9,216 = 18.9M |
| 3×3 conv on the concat | 32 + 32 + 32 = 96 → 32 | 96·9·32 + 32 = 27,680 | 96·9·32·9,216 = 254.8M |
| 3×3 conv | 32 → 32 | 32·9·32 + 32 = 9,248 | 32·9·32·9,216 = 84.9M |
| node total | 45,152 | 358.6M |
The node has under 0.5% of the model's 9.04M parameters but 8.6% of its 4.19 GMACs. Nested nodes at full resolution are cheap in memory for weights and expensive in compute and activations, the reverse of the deep levels, where levels 3 and 4 hold 74% of the parameters.
1import torch, torch.nn as nn, torch.nn.functional as F23def block(cin, cout): # H: two 3x3 conv + ReLU4 return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.ReLU(inplace=True),5 nn.Conv2d(cout, cout, 3, padding=1), nn.ReLU(inplace=True))67class UNetPP(nn.Module):8 def __init__(self, cin=3, f=(32, 64, 128, 256, 512), L=4):9 super().__init__()10 self.L, self.pool = L, nn.MaxPool2d(2)11 self.node, self.up = nn.ModuleDict(), nn.ModuleDict()12 for i in range(L + 1):13 self.node[f"{i}_0"] = block(cin if i == 0 else f[i - 1], f[i])14 for j in range(1, L + 1):15 for i in range(L + 1 - j):16 self.up[f"{i}_{j}"] = nn.ConvTranspose2d(f[i + 1], f[i], 2, stride=2)17 self.node[f"{i}_{j}"] = block((j + 1) * f[i], f[i])18 self.head = nn.ModuleList(nn.Conv2d(f[0], 1, 1) for _ in range(L))1920 def forward(self, x, depth=None): # depth d < L runs only nodes with i + j <= d21 d, X = depth or self.L, {}22 for i in range(d + 1):23 X[i, 0] = self.node[f"{i}_0"](x if i == 0 else self.pool(X[i - 1, 0]))24 for j in range(1, d + 1):25 for i in range(d + 1 - j):26 skips = [X[i, k] for k in range(j)] + [self.up[f"{i}_{j}"](X[i + 1, j - 1])]27 X[i, j] = self.node[f"{i}_{j}"](torch.cat(skips, dim=1))28 return [self.head[j - 1](X[0, j]) for j in range(1, d + 1)] # logits of heads 1..d2930def bce_dice(logit, y, eps=1.0):31 p = torch.sigmoid(logit)32 dice = (2 * (p * y).sum() + eps) / (p.sum() + y.sum() + eps)33 return 0.5 * F.binary_cross_entropy_with_logits(logit, y) + (1 - dice)3435model = UNetPP()36x, y = torch.randn(2, 3, 96, 96), (torch.rand(2, 1, 96, 96) > 0.7).float()37loss = sum(bce_dice(o, y) for o in model(x)) / 4 # deep supervision on X0,1..X0,438accurate = torch.stack([o.sigmoid() for o in model(x)]).mean(0) # accurate mode39fast = model(x, depth=2)[-1].sigmoid() # fast mode: U-Net++ L2Use when
- +Encoder features pass through refining convolutions before they meet the decoder, narrowing the semantic gap the design targets
- +One training run yields U-Nets of four depths; choose the speed/accuracy point afterwards
- +Only about 17% more parameters than U-Net at the same widths
Avoid when
- −About 2.5× U-Net's compute at depth 4, and much more activation memory, because the extra nodes run at the highest resolutions
- −The gains over a well-tuned U-Net are a few IoU points and vary by dataset
- −Dense concatenation makes the top-row inputs wide (160 channels at X0,4)
Check yourself
Why can U-Net++ drop every node with i + j > 2 at inference and still produce exactly the same output at X0,2?
Check yourself
Node X1,2 sits on level 1 (64 filters). How many channels enter its first 3×3 convolution?
Remember
- ✓Nodes : dense skips along each level, upsampling from below.
- ✓U-Net is the outer edge of the grid; U-Net++ adds the inner nodes and nests U-Nets of depths 1 to 4 on one encoder.
- ✓Deep supervision: a 1×1 sigmoid head on each of X0,1…X0,4, trained with BCE plus Dice.
- ✓Inference: average the heads (accurate) or prune to L^d by keeping i + j ≤ d (fast, exact).
- ✓Paper widths 32 to 512: 9.04M vs 7.76M parameters; about 2.5× U-Net's MACs, mostly in the full-resolution row.