Definition
DenseNet (densely connected convolutional network) is a CNN made of dense blocks. Inside a block, layer takes as input the channel-wise concatenation of the block input and the outputs of all layers before it, , and contributes new feature maps, where is the growth rate. Transition layers between blocks shrink the channel count and the resolution.
Huang, Liu, van der Maaten and Weinberger introduced DenseNet in "Densely Connected Convolutional Networks" (CVPR 2017, best paper award). It came out of the same line of work as Stochastic depth (Huang et al., 2016), which had shown that randomly dropping whole residual blocks during training does not hurt and often helps. That suggested many layers of a deep ResNet contribute little and that the important thing is short paths from early layers to late ones. DenseNet makes such paths explicit: in a block of layers there are direct connections instead of .
The difference from Residual neural network (ResNet) is how a shortcut is merged. ResNet adds, , so the identity and the new features share the same channels and later layers cannot tell them apart. DenseNet concatenates, so every earlier feature map stays available unchanged, and each layer only has to add a small amount of new information. That is why its layers can be very narrow: channels per layer in the ImageNet models, and on CIFAR.
At a glance
- Paper
- Huang, Liu, van der Maaten, Weinberger, CVPR 2017
- Connectivity
- concatenation of all earlier maps in a block: connections for layers
- Growth rate
- new channels per layer; after layers the block has channels
- Layer (BC)
- BN → ReLU → conv to → BN → ReLU → conv to
- Transition
- BN → ReLU → conv to channels → avg-pool,
- DenseNet-121
- 7.98M params, 2.83G multiply-adds, 74.43% top-1 (torchvision weights)
- Main cost
- activation memory and memory bandwidth from repeated concatenation
Dense connectivity and growth rate
- input of the dense block, channels
- concatenation along the channel axis (all maps in a block have the same spatial size)
- composite function of layer : BN → ReLU → conv (pre-activation order, as in ResNet v2)
- growth rate: channels produced by each
- input width of layer , growing linearly through the block
The block's output is the concatenation of everything, channels, so the network's width grows linearly inside a block. The paper interprets this as a shared collective knowledge: each layer reads the whole state and writes channels to it. The growth rate controls how much new information each layer adds. Small values work; the ImageNet models use (DenseNet-161 uses ).
Bottleneck layers and compression
Late in a block the input is wide: in DenseNet-121's third block, layer 24 reads channels. A conv straight from 992 to 32 channels would cost weights. The bottleneck version (DenseNet-B) first applies a conv to channels: weights. The layer is then BN-ReLU-Conv(1×1)-BN-ReLU-Conv(3×3).
Compression (DenseNet-C) lets the transition's conv output channels for a block that produced , with . With the width is halved at every transition, so the next block starts narrower. Both together give DenseNet-BC, which is what "DenseNet" usually means. Counted on the 121-layer ImageNet layout:
| Variant | bottleneck | θ | params | multiply-adds |
|---|---|---|---|---|
| DenseNet | no | 1 | 20.81M | 4.72G |
| DenseNet-B | 4k | 1 | 13.91M | 3.92G |
| DenseNet-C | no | 0.5 | 10.87M | 3.05G |
| DenseNet-BC | 4k | 0.5 | 7.98M | 2.83G |
The bottleneck does not always save weights. In the first block, layer inputs have 64 to 224 channels, fewer than for the first layers, and the block's bottlenecked version has 335,040 parameters against 250,560 without it. It pays off in the wide later blocks, which hold most of the parameters.
Architecture of DenseNet-121
| Stage | output | layers | params | MACs |
|---|---|---|---|---|
| stem: 7×7/2 conv, 3×3/2 max-pool | 64 × 56² | 1 | 0.01M | 118M |
| dense block 1 | 256 × 56² | 6 | 0.34M | 1040M |
| transition 1 | 128 × 28² | 1 | 0.03M | 103M |
| dense block 2 | 512 × 28² | 12 | 0.92M | 713M |
| transition 2 | 256 × 14² | 1 | 0.13M | 103M |
| dense block 3 | 1024 × 14² | 24 | 2.84M | 549M |
| transition 3 | 512 × 7² | 1 | 0.53M | 103M |
| dense block 4 | 1024 × 7² | 16 | 2.16M | 104M |
| BN, global avg-pool, FC | 1000 | 1 | 1.03M | 1M |
The name counts conv and FC layers: stem conv dense-layer convs transition convs classifier . DenseNet-169, -201 and -264 deepen blocks 3 and 4: (6, 12, 32, 32), (6, 12, 48, 32) and (6, 12, 64, 48).
Worked example: the first dense block
DenseNet-121, block 1 (k₀ = 64, k = 32, L = 6, bottleneck 4k = 128), at 56×56
- 1Layer 1 reads channels. BN on 64 channels (128 params), conv (), BN on 128 (256), conv (): 45,440 parameters. It outputs 32 channels.
- 2Layer 2 reads , channels: the grows to weights, the stays , total 49,600. Each layer adds more ( input channels × outputs + 64 BN params).
- 3Layers 3 to 6 read 128, 160, 192, 224 channels: 53,760, 57,920, 62,080, 66,240 parameters. The block total is 335,040, and its output is channels.
- 4Connections: layer receives inputs, so . A ResNet stage of 6 blocks has 6 shortcuts.
- 5Transition 1: BN on 256 (512), conv (): 33,280 parameters, then average pooling to .
Why it trains well: gradients through concatenation
The backward pass of a concatenation simply splits the incoming gradient into the slices that belong to each source. Since is part of the input of every later layer in its block (and of the transition), its gradient is a sum of direct contributions:
- concatenated input of layer ( stands for the transition layer)
- the channel slice of that gradient belonging to
- gradient computed by layer 's BN-ReLU-conv backward
Each term is only one layer away from the transition and the loss, so early layers get strong, short-path gradients. The paper calls this implicit deep supervision, comparing it to deeply-supervised nets, which attach a classifier to every hidden layer. Unlike ResNet's additive identity, the paths are not summed into one tensor, so nothing forces later layers to keep reusing a representation; the paper's heat-maps of trained filter weights show that layers do spread weight over many earlier layers, including far ones.
1import torch, torch.nn as nn2from torchvision.models import densenet121, DenseNet121_Weights34class DenseLayer(nn.Module):5 def __init__(self, c_in, k=32, bn_size=4):6 super().__init__()7 self.f = nn.Sequential(8 nn.BatchNorm2d(c_in), nn.ReLU(inplace=True),9 nn.Conv2d(c_in, bn_size * k, 1, bias=False), # bottleneck to 4k10 nn.BatchNorm2d(bn_size * k), nn.ReLU(inplace=True),11 nn.Conv2d(bn_size * k, k, 3, padding=1, bias=False)) # k new maps1213 def forward(self, feats): # feats: list of all earlier maps14 return self.f(torch.cat(feats, dim=1))1516class DenseBlock(nn.Module):17 def __init__(self, c0, L, k=32):18 super().__init__()19 self.layers = nn.ModuleList(DenseLayer(c0 + i * k, k) for i in range(L))2021 def forward(self, x):22 feats = [x]23 for layer in self.layers:24 feats.append(layer(feats))25 return torch.cat(feats, dim=1) # c0 + L*k channels2627blk = DenseBlock(64, 6)28print(sum(p.numel() for p in blk.parameters())) # 33504029print(blk(torch.randn(1, 64, 56, 56)).shape) # torch.Size([1, 256, 56, 56])3031model = densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1)32print(sum(p.numel() for p in model.parameters())) # 797885633# memory_efficient=True recomputes the concatenation + BN + ReLU + 1x1 in backward34lean = densenet121(memory_efficient=True)| Model | paper top-1 err. | torchvision acc@1 | Params | GFLOPs |
|---|---|---|---|---|
| densenet121 | 25.02% | 74.43% | 8.0M | 2.83 |
| densenet169 | 23.80% | 75.60% | 14.1M | 3.36 |
| densenet201 | 22.58% | 76.90% | 20.0M | 4.29 |
| densenet161 (k = 48) | – | 77.14% | 28.7M | 7.73 |
| resnet50 (for scale) | – | 76.13% | 25.6M | 4.09 |
The paper reports that a DenseNet-201 with 20M parameters reaches about the validation error of a ResNet-101 with more than 40M, and on CIFAR it reaches 3.46% (C10+) and 17.18% (C100+) error with a 190-layer DenseNet-BC at . The parameter efficiency is real. Measured speed is a different matter.
Use when
- +Very parameter-efficient: 8.0M parameters for ResNet-50-like accuracy
- +Short gradient paths to every layer (implicit deep supervision); trains easily at depth
- +Feature reuse: low-level maps reach the classifier directly
- +Strong on small data; the paper reports it is less prone to overfitting than ResNets
Avoid when
- −High activation memory and memory traffic from repeated concatenation
- −Slower on GPUs than its FLOPs suggest: many thin convs and copies
- −Channel width varies inside a block, which complicates pruning and kernel tuning
- −Overtaken by ResNeXt/RegNet/EfficientNet-style networks on accuracy per unit of latency
Check yourself
A dense block has k₀ = 64 input channels, growth rate k = 32 and L = 6 layers. How many channels does it output, and how many direct connections does it contain?
Check yourself
Why does a DenseNet layer need far fewer output channels (k = 32) than a ResNet layer of comparable depth?
Remember
- ✓: every layer reads the concatenation of all earlier maps in its block and adds channels.
- ✓A block of layers has connections and outputs channels.
- ✓DenseNet-BC: bottleneck BN-ReLU-1×1 ()-BN-ReLU-3×3 () layers, transitions with compression .
- ✓DenseNet-121: 7.98M parameters, 2.83G multiply-adds, 74.4% top-1 with torchvision weights.
- ✓The price is activation memory; use shared buffers / checkpointing (memory_efficient=True).