DenseNet

Each layer receives the concatenation of every earlier feature map in its block and adds only k new channels. Features are reused instead of relearned, so DenseNet-121 reaches ResNet-level accuracy with 8.0M parameters, at the cost of heavy activation memory.

13 min read1051 wordscore0 viewsupdated 2026-10-11

Definition

DenseNet (densely connected convolutional network) is a CNN made of dense blocks. Inside a block, layer ℓ\ell takes as input the channel-wise concatenation of the block input and the outputs of all layers before it, xℓ=Hℓ([x0,x1,…,xℓ−1])x_\ell=H_\ell([x_0,x_1,\dots,x_{\ell-1}]), and contributes kk new feature maps, where kk is the growth rate. Transition layers between blocks shrink the channel count and the resolution.

Huang, Liu, van der Maaten and Weinberger introduced DenseNet in "Densely Connected Convolutional Networks" (CVPR 2017, best paper award). It came out of the same line of work as Stochastic depth (Huang et al., 2016), which had shown that randomly dropping whole residual blocks during training does not hurt and often helps. That suggested many layers of a deep ResNet contribute little and that the important thing is short paths from early layers to late ones. DenseNet makes such paths explicit: in a block of LL layers there are L(L+1)/2L(L+1)/2 direct connections instead of LL.

The difference from Residual neural network (ResNet) is how a shortcut is merged. ResNet adds, xℓ=Hℓ(xℓ−1)+xℓ−1x_\ell=H_\ell(x_{\ell-1})+x_{\ell-1}, so the identity and the new features share the same channels and later layers cannot tell them apart. DenseNet concatenates, so every earlier feature map stays available unchanged, and each layer only has to add a small amount of new information. That is why its layers can be very narrow: k=32k=32 channels per layer in the ImageNet models, and k=12k=12 on CIFAR.

At a glance

Paper
Huang, Liu, van der Maaten, Weinberger, CVPR 2017
Connectivity
concatenation of all earlier maps in a block: L(L+1)/2L(L+1)/2 connections for LL layers
Growth rate
kk new channels per layer; after ℓ\ell layers the block has k0+ℓkk_0+\ell k channels
Layer (BC)
BN → ReLU → 1×11\times1 conv to 4k4k → BN → ReLU → 3×33\times3 conv to kk
Transition
BN → ReLU → 1×11\times1 conv to ⌊θm⌋\lfloor\theta m\rfloor channels → 2×22\times2 avg-pool, θ=0.5\theta=0.5
DenseNet-121
7.98M params, 2.83G multiply-adds, 74.43% top-1 (torchvision weights)
Main cost
activation memory and memory bandwidth from repeated concatenation

Dense connectivity and growth rate

xℓ=Hℓ([x0,x1,…,xℓ−1]),cℓin=k0+(ℓ−1) k,cblockout=k0+L kx_\ell=H_\ell\big([x_0,x_1,\dots,x_{\ell-1}]\big),\qquad c_\ell^{\text{in}}=k_0+(\ell-1)\,k,\qquad c^{\text{out}}_{\text{block}}=k_0+L\,k
x0x_0
input of the dense block, k0k_0 channels
[⋅,…,⋅][\cdot,\dots,\cdot]
concatenation along the channel axis (all maps in a block have the same spatial size)
HℓH_\ell
composite function of layer ℓ\ell: BN → ReLU → conv (pre-activation order, as in ResNet v2)
kk
growth rate: channels produced by each HℓH_\ell
cℓinc_\ell^{\text{in}}
input width of layer ℓ\ell, growing linearly through the block

The block's output is the concatenation of everything, k0+Lkk_0+Lk channels, so the network's width grows linearly inside a block. The paper interprets this as a shared collective knowledge: each layer reads the whole state and writes kk channels to it. The growth rate controls how much new information each layer adds. Small values work; the ImageNet models use k=32k=32 (DenseNet-161 uses k=48k=48).

xℓ = Hℓ([x₀, x₁, …, xℓ₋₁]) · 21 connections in 6 layersx₀:64+32+32+32+32+32+321234566outbar = input channels of layer ℓ, coloured by sourceselected layer 6: input 224 ch = 64 + 5·32 → 1×1 to 128 → 3×3 to 3266,240 params (1×1: 28,672, 3×3: 36,864, BN: 704)one dense block · DenseNet-BCout channels k₀+Lk256transition θ=0.5256 → 128block params335,040connections21maps kept, naive concat864maps kept, shared buffer256ImageNet (6,12,24,16), k₀=2kstem 64×56²0.01Mdense1 256×56²0.34Mtrans1 128×28²0.03Mdense2 512×28²0.92Mtrans2 256×14²0.13Mdense3 1024×14²2.84Mtrans3 512×7²0.53Mdense4 1024×7²2.16Mhead 1024→10001.03Mtotal params7.98Mmultiply-adds2.83G= torchvision7,978,856
click a bar or node
fig.Top left: the connections of one dense block. Click a layer and its incoming arcs light up, one colour per source. Below, each bar is a layer's input, stacked from the block input and every earlier layer's k channels. The default is DenseNet-121's first block (k₀ = 64, k = 32, L = 6): 21 connections, 256 output channels and 335,040 parameters, with the transition compressing 256 → 128. The right panel recounts the full ImageNet network with k₀ = 2k for the chosen variant. DenseNet-BC at k = 32 gives 7.98M parameters and 2.83G multiply-adds, exactly torchvision's densenet121. The plain variant (no bottleneck, no compression) needs 20.81M and 4.72G for the same depth. The memory rows count the channel-maps kept for the backward pass: 864 if every layer stores its own concatenated input, 256 with a shared buffer.

Bottleneck layers and compression

Late in a block the input is wide: in DenseNet-121's third block, layer 24 reads 256+23⋅32=992256+23\cdot32=992 channels. A 3×33\times3 conv straight from 992 to 32 channels would cost 9⋅992⋅32=285,6969\cdot992\cdot32=285{,}696 weights. The bottleneck version (DenseNet-B) first applies a 1×11\times1 conv to 4k=1284k=128 channels: 992⋅128+9⋅128⋅32=163,840992\cdot128+9\cdot128\cdot32=163{,}840 weights. The layer is then BN-ReLU-Conv(1×1)-BN-ReLU-Conv(3×3).

Compression (DenseNet-C) lets the transition's 1×11\times1 conv output ⌊θm⌋\lfloor\theta m\rfloor channels for a block that produced mm, with 0<θ≤10<\theta\le1. With θ=0.5\theta=0.5 the width is halved at every transition, so the next block starts narrower. Both together give DenseNet-BC, which is what "DenseNet" usually means. Counted on the 121-layer ImageNet layout:

Variantbottleneckθparamsmultiply-adds
DenseNetno120.81M4.72G
DenseNet-B4k113.91M3.92G
DenseNet-Cno0.510.87M3.05G
DenseNet-BC4k0.57.98M2.83G
Variants on the DenseNet-121 layout (blocks 6, 12, 24, 16; k = 32; k₀ = 64; 224×224 input), counted by the figure's code. BN parameters included.

The bottleneck does not always save weights. In the first block, layer inputs have 64 to 224 channels, fewer than 4k=1284k=128 for the first layers, and the block's bottlenecked version has 335,040 parameters against 250,560 without it. It pays off in the wide later blocks, which hold most of the parameters.

Architecture of DenseNet-121

StageoutputlayersparamsMACs
stem: 7×7/2 conv, 3×3/2 max-pool64 × 56²10.01M118M
dense block 1256 × 56²60.34M1040M
transition 1128 × 28²10.03M103M
dense block 2512 × 28²120.92M713M
transition 2256 × 14²10.13M103M
dense block 31024 × 14²242.84M549M
transition 3512 × 7²10.53M103M
dense block 41024 × 7²162.16M104M
BN, global avg-pool, FC100011.03M1M
DenseNet-121 (torchvision layout), per stage. Parameters and multiply-adds from the figure's code; the total, 7,978,856 parameters, matches torchvision's densenet121.

The name counts conv and FC layers: 11 stem conv + 2⋅(6+12+24+16)=116+\,2\cdot(6+12+24+16)=116 dense-layer convs + 3+\,3 transition convs + 1+\,1 classifier =121=121. DenseNet-169, -201 and -264 deepen blocks 3 and 4: (6, 12, 32, 32), (6, 12, 48, 32) and (6, 12, 64, 48).

Worked example: the first dense block

DenseNet-121, block 1 (k₀ = 64, k = 32, L = 6, bottleneck 4k = 128), at 56×56

  1. 1Layer 1 reads 6464 channels. BN on 64 channels (128 params), 1×11\times1 conv 64→12864\to128 (8,1928{,}192), BN on 128 (256), 3×33\times3 conv 128→32128\to32 (36,86436{,}864): 45,440 parameters. It outputs 32 channels.
  2. 2Layer 2 reads [x0,x1][x_0,x_1], 64+32=9664+32=96 channels: the 1×11\times1 grows to 12,28812{,}288 weights, the 3×33\times3 stays 36,86436{,}864, total 49,600. Each layer adds 4,1604{,}160 more (3232 input channels × 128128 outputs + 64 BN params).
  3. 3Layers 3 to 6 read 128, 160, 192, 224 channels: 53,760, 57,920, 62,080, 66,240 parameters. The block total is 335,040, and its output is 64+6⋅32=25664+6\cdot32=256 channels.
  4. 4Connections: layer ℓ\ell receives ℓ\ell inputs, so 1+2+⋯+6=211+2+\dots+6=21. A ResNet stage of 6 blocks has 6 shortcuts.
  5. 5Transition 1: BN on 256 (512), 1×11\times1 conv 256→128256\to128 (32,76832{,}768): 33,280 parameters, then 2×22\times2 average pooling to 28×2828\times28.

Why it trains well: gradients through concatenation

The backward pass of a concatenation simply splits the incoming gradient into the slices that belong to each source. Since xix_i is part of the input of every later layer in its block (and of the transition), its gradient is a sum of direct contributions:

∂L∂xi=∑ℓ=i+1L+1[∂L∂uℓ]xi,uℓ=[x0,…,xℓ−1]\frac{\partial\mathcal L}{\partial x_i}=\sum_{\ell=i+1}^{L+1}\Big[\frac{\partial\mathcal L}{\partial u_\ell}\Big]_{x_i},\qquad u_\ell=[x_0,\dots,x_{\ell-1}]
uℓu_\ell
concatenated input of layer ℓ\ell (ℓ=L+1\ell=L+1 stands for the transition layer)
[ ⋅ ]xi[\,\cdot\,]_{x_i}
the channel slice of that gradient belonging to xix_i
∂L/∂uℓ\partial\mathcal L/\partial u_\ell
gradient computed by layer ℓ\ell's BN-ReLU-conv backward

Each term is only one layer away from the transition and the loss, so early layers get strong, short-path gradients. The paper calls this implicit deep supervision, comparing it to deeply-supervised nets, which attach a classifier to every hidden layer. Unlike ResNet's additive identity, the paths are not summed into one tensor, so nothing forces later layers to keep reusing a representation; the paper's heat-maps of trained filter weights show that layers do spread weight over many earlier layers, including far ones.

python
1import torch, torch.nn as nn
2from torchvision.models import densenet121, DenseNet121_Weights
3
4class DenseLayer(nn.Module):
5 def __init__(self, c_in, k=32, bn_size=4):
6 super().__init__()
7 self.f = nn.Sequential(
8 nn.BatchNorm2d(c_in), nn.ReLU(inplace=True),
9 nn.Conv2d(c_in, bn_size * k, 1, bias=False), # bottleneck to 4k
10 nn.BatchNorm2d(bn_size * k), nn.ReLU(inplace=True),
11 nn.Conv2d(bn_size * k, k, 3, padding=1, bias=False)) # k new maps
12
13 def forward(self, feats): # feats: list of all earlier maps
14 return self.f(torch.cat(feats, dim=1))
15
16class DenseBlock(nn.Module):
17 def __init__(self, c0, L, k=32):
18 super().__init__()
19 self.layers = nn.ModuleList(DenseLayer(c0 + i * k, k) for i in range(L))
20
21 def forward(self, x):
22 feats = [x]
23 for layer in self.layers:
24 feats.append(layer(feats))
25 return torch.cat(feats, dim=1) # c0 + L*k channels
26
27blk = DenseBlock(64, 6)
28print(sum(p.numel() for p in blk.parameters())) # 335040
29print(blk(torch.randn(1, 64, 56, 56)).shape) # torch.Size([1, 256, 56, 56])
30
31model = densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1)
32print(sum(p.numel() for p in model.parameters())) # 7978856
33# memory_efficient=True recomputes the concatenation + BN + ReLU + 1x1 in backward
34lean = densenet121(memory_efficient=True)
A DenseNet-BC dense layer and block, the torchvision model, and its parameter count
Modelpaper top-1 err.torchvision acc@1ParamsGFLOPs
densenet12125.02%74.43%8.0M2.83
densenet16923.80%75.60%14.1M3.36
densenet20122.58%76.90%20.0M4.29
densenet161 (k = 48)–77.14%28.7M7.73
resnet50 (for scale)–76.13%25.6M4.09
ImageNet. Top-1 error: Huang et al. 2017, Table 3 (single 224 crop; DenseNet-161 is not in that table). Params, GFLOPs and acc@1: torchvision's documented weights (IMAGENET1K_V1).

The paper reports that a DenseNet-201 with 20M parameters reaches about the validation error of a ResNet-101 with more than 40M, and on CIFAR it reaches 3.46% (C10+) and 17.18% (C100+) error with a 190-layer DenseNet-BC at k=40k=40. The parameter efficiency is real. Measured speed is a different matter.

Use when

  • +Very parameter-efficient: 8.0M parameters for ResNet-50-like accuracy
  • +Short gradient paths to every layer (implicit deep supervision); trains easily at depth
  • +Feature reuse: low-level maps reach the classifier directly
  • +Strong on small data; the paper reports it is less prone to overfitting than ResNets

Avoid when

  • −High activation memory and memory traffic from repeated concatenation
  • −Slower on GPUs than its FLOPs suggest: many thin convs and copies
  • −Channel width varies inside a block, which complicates pruning and kernel tuning
  • −Overtaken by ResNeXt/RegNet/EfficientNet-style networks on accuracy per unit of latency

Check yourself

A dense block has k₀ = 64 input channels, growth rate k = 32 and L = 6 layers. How many channels does it output, and how many direct connections does it contain?

Check yourself

Why does a DenseNet layer need far fewer output channels (k = 32) than a ResNet layer of comparable depth?

Remember

  • ✓xℓ=Hℓ([x0,…,xℓ−1])x_\ell=H_\ell([x_0,\dots,x_{\ell-1}]): every layer reads the concatenation of all earlier maps in its block and adds kk channels.
  • ✓A block of LL layers has L(L+1)/2L(L+1)/2 connections and outputs k0+Lkk_0+Lk channels.
  • ✓DenseNet-BC: bottleneck BN-ReLU-1×1 (4k4k)-BN-ReLU-3×3 (kk) layers, transitions with compression θ=0.5\theta=0.5.
  • ✓DenseNet-121: 7.98M parameters, 2.83G multiply-adds, 74.4% top-1 with torchvision weights.
  • ✓The price is activation memory; use shared buffers / checkpointing (memory_efficient=True).
← back to Deep Learning roadmap