# DenseNet

> Each layer receives the concatenation of every earlier feature map in its block and adds only k new channels. Features are reused instead of relearned, so DenseNet-121 reaches ResNet-level accuracy with 8.0M parameters, at the cost of heavy activation memory.

Source: https://www.mlfoundry.in/learn/densenet  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 13 min read · updated 2026-10-11

**DenseNet:** (densely connected convolutional network) is a CNN made of **dense blocks**. Inside a block, layer $\ell$ takes as input the channel-wise **concatenation** of the block input and the outputs of all layers before it, $x_\ell=H_\ell([x_0,x_1,\dots,x_{\ell-1}])$, and contributes $k$ new feature maps, where $k$ is the **growth rate**. **Transition layers** between blocks shrink the channel count and the resolution.

Huang, Liu, van der Maaten and Weinberger introduced DenseNet in "Densely Connected Convolutional Networks" (CVPR 2017, best paper award). It came out of the same line of work as **Stochastic depth** (Huang et al., 2016), which had shown that randomly dropping whole residual blocks during training does not hurt and often helps. That suggested many layers of a deep ResNet contribute little and that the important thing is short paths from early layers to late ones. DenseNet makes such paths explicit: in a block of $L$ layers there are $L(L+1)/2$ direct connections instead of $L$.

The difference from **Residual neural network (ResNet)** is how a shortcut is merged. ResNet **adds**, $x_\ell=H_\ell(x_{\ell-1})+x_{\ell-1}$, so the identity and the new features share the same channels and later layers cannot tell them apart. DenseNet **concatenates**, so every earlier feature map stays available unchanged, and each layer only has to add a small amount of new information. That is why its layers can be very narrow: $k=32$ channels per layer in the ImageNet models, and $k=12$ on CIFAR.

**At a glance**

- **Paper:** Huang, Liu, van der Maaten, Weinberger, CVPR 2017
- **Connectivity:** concatenation of all earlier maps in a block: $L(L+1)/2$ connections for $L$ layers
- **Growth rate:** $k$ new channels per layer; after $\ell$ layers the block has $k_0+\ell k$ channels
- **Layer (BC):** BN → ReLU → $1\times1$ conv to $4k$ → BN → ReLU → $3\times3$ conv to $k$
- **Transition:** BN → ReLU → $1\times1$ conv to $\lfloor\theta m\rfloor$ channels → $2\times2$ avg-pool, $\theta=0.5$
- **DenseNet-121:** 7.98M params, 2.83G multiply-adds, 74.43% top-1 (torchvision weights)
- **Main cost:** activation memory and memory bandwidth from repeated concatenation

## Dense connectivity and growth rate

$$
x_\ell=H_\ell\big([x_0,x_1,\dots,x_{\ell-1}]\big),\qquad c_\ell^{\text{in}}=k_0+(\ell-1)\,k,\qquad c^{\text{out}}_{\text{block}}=k_0+L\,k
$$

- $x_0$: input of the dense block, $k_0$ channels
- $[\cdot,\dots,\cdot]$: concatenation along the channel axis (all maps in a block have the same spatial size)
- $H_\ell$: composite function of layer $\ell$: BN → ReLU → conv (pre-activation order, as in ResNet v2)
- $k$: growth rate: channels produced by each $H_\ell$
- $c_\ell^{\text{in}}$: input width of layer $\ell$, growing linearly through the block

The block's output is the concatenation of everything, $k_0+Lk$ channels, so the network's width grows linearly inside a block. The paper interprets this as a shared **collective knowledge**: each layer reads the whole state and writes $k$ channels to it. The growth rate controls how much new information each layer adds. Small values work; the ImageNet models use $k=32$ (DenseNet-161 uses $k=48$).

> Figure: Top left: the connections of one dense block. Click a layer and its incoming arcs light up, one colour per source. Below, each bar is a layer's input, stacked from the block input and every earlier layer's k channels. The default is DenseNet-121's first block (k₀ = 64, k = 32, L = 6): 21 connections, 256 output channels and 335,040 parameters, with the transition compressing 256 → 128. The right panel recounts the full ImageNet network with k₀ = 2k for the chosen variant. DenseNet-BC at k = 32 gives 7.98M parameters and 2.83G multiply-adds, exactly torchvision's densenet121. The plain variant (no bottleneck, no compression) needs 20.81M and 4.72G for the same depth. The memory rows count the channel-maps kept for the backward pass: 864 if every layer stores its own concatenated input, 256 with a shared buffer.

## Bottleneck layers and compression

Late in a block the input is wide: in DenseNet-121's third block, layer 24 reads $256+23\cdot32=992$ channels. A $3\times3$ conv straight from 992 to 32 channels would cost $9\cdot992\cdot32=285{,}696$ weights. The **bottleneck** version (DenseNet-B) first applies a $1\times1$ conv to $4k=128$ channels: $992\cdot128+9\cdot128\cdot32=163{,}840$ weights. The layer is then BN-ReLU-Conv(1×1)-BN-ReLU-Conv(3×3).

**Compression** (DenseNet-C) lets the transition's $1\times1$ conv output $\lfloor\theta m\rfloor$ channels for a block that produced $m$, with $0<\theta\le1$. With $\theta=0.5$ the width is halved at every transition, so the next block starts narrower. Both together give **DenseNet-BC**, which is what "DenseNet" usually means. Counted on the 121-layer ImageNet layout:

Variants on the DenseNet-121 layout (blocks 6, 12, 24, 16; k = 32; k₀ = 64; 224×224 input), counted by the figure's code. BN parameters included.

| Variant | bottleneck | θ | params | multiply-adds |
| --- | --- | --- | --- | --- |
| DenseNet | no | 1 | 20.81M | 4.72G |
| DenseNet-B | 4k | 1 | 13.91M | 3.92G |
| DenseNet-C | no | 0.5 | 10.87M | 3.05G |
| DenseNet-BC | 4k | 0.5 | 7.98M | 2.83G |

The bottleneck does not always save weights. In the first block, layer inputs have 64 to 224 channels, fewer than $4k=128$ for the first layers, and the block's bottlenecked version has 335,040 parameters against 250,560 without it. It pays off in the wide later blocks, which hold most of the parameters.

## Architecture of DenseNet-121

DenseNet-121 (torchvision layout), per stage. Parameters and multiply-adds from the figure's code; the total, 7,978,856 parameters, matches torchvision's densenet121.

| Stage | output | layers | params | MACs |
| --- | --- | --- | --- | --- |
| stem: 7×7/2 conv, 3×3/2 max-pool | 64 × 56² | 1 | 0.01M | 118M |
| dense block 1 | 256 × 56² | 6 | 0.34M | 1040M |
| transition 1 | 128 × 28² | 1 | 0.03M | 103M |
| dense block 2 | 512 × 28² | 12 | 0.92M | 713M |
| transition 2 | 256 × 14² | 1 | 0.13M | 103M |
| dense block 3 | 1024 × 14² | 24 | 2.84M | 549M |
| transition 3 | 512 × 7² | 1 | 0.53M | 103M |
| dense block 4 | 1024 × 7² | 16 | 2.16M | 104M |
| BN, global avg-pool, FC | 1000 | 1 | 1.03M | 1M |

The name counts conv and FC layers: $1$ stem conv $+\,2\cdot(6+12+24+16)=116$ dense-layer convs $+\,3$ transition convs $+\,1$ classifier $=121$. DenseNet-169, -201 and -264 deepen blocks 3 and 4: (6, 12, 32, 32), (6, 12, 48, 32) and (6, 12, 64, 48).

## Worked example: the first dense block

**DenseNet-121, block 1 (k₀ = 64, k = 32, L = 6, bottleneck 4k = 128), at 56×56**

1. Layer 1 reads $64$ channels. BN on 64 channels (128 params), $1\times1$ conv $64\to128$ ($8{,}192$), BN on 128 (256), $3\times3$ conv $128\to32$ ($36{,}864$): **45,440** parameters. It outputs 32 channels.
2. Layer 2 reads $[x_0,x_1]$, $64+32=96$ channels: the $1\times1$ grows to $12{,}288$ weights, the $3\times3$ stays $36{,}864$, total **49,600**. Each layer adds $4{,}160$ more ($32$ input channels × $128$ outputs + 64 BN params).
3. Layers 3 to 6 read 128, 160, 192, 224 channels: 53,760, 57,920, 62,080, 66,240 parameters. The block total is **335,040**, and its output is $64+6\cdot32=256$ channels.
4. Connections: layer $\ell$ receives $\ell$ inputs, so $1+2+\dots+6=21$. A ResNet stage of 6 blocks has 6 shortcuts.
5. Transition 1: BN on 256 (512), $1\times1$ conv $256\to128$ ($32{,}768$): **33,280** parameters, then $2\times2$ average pooling to $28\times28$.

## Why it trains well: gradients through concatenation

The backward pass of a concatenation simply splits the incoming gradient into the slices that belong to each source. Since $x_i$ is part of the input of every later layer in its block (and of the transition), its gradient is a sum of direct contributions:

$$
\frac{\partial\mathcal L}{\partial x_i}=\sum_{\ell=i+1}^{L+1}\Big[\frac{\partial\mathcal L}{\partial u_\ell}\Big]_{x_i},\qquad u_\ell=[x_0,\dots,x_{\ell-1}]
$$

- $u_\ell$: concatenated input of layer $\ell$ ($\ell=L+1$ stands for the transition layer)
- $[\,\cdot\,]_{x_i}$: the channel slice of that gradient belonging to $x_i$
- $\partial\mathcal L/\partial u_\ell$: gradient computed by layer $\ell$'s BN-ReLU-conv backward

Each term is only one layer away from the transition and the loss, so early layers get strong, short-path gradients. The paper calls this **implicit deep supervision**, comparing it to deeply-supervised nets, which attach a classifier to every hidden layer. Unlike ResNet's additive identity, the paths are not summed into one tensor, so nothing forces later layers to keep reusing a representation; the paper's heat-maps of trained filter weights show that layers do spread weight over many earlier layers, including far ones.

```python
import torch, torch.nn as nn
from torchvision.models import densenet121, DenseNet121_Weights

class DenseLayer(nn.Module):
    def __init__(self, c_in, k=32, bn_size=4):
        super().__init__()
        self.f = nn.Sequential(
            nn.BatchNorm2d(c_in), nn.ReLU(inplace=True),
            nn.Conv2d(c_in, bn_size * k, 1, bias=False),          # bottleneck to 4k
            nn.BatchNorm2d(bn_size * k), nn.ReLU(inplace=True),
            nn.Conv2d(bn_size * k, k, 3, padding=1, bias=False))   # k new maps

    def forward(self, feats):                 # feats: list of all earlier maps
        return self.f(torch.cat(feats, dim=1))

class DenseBlock(nn.Module):
    def __init__(self, c0, L, k=32):
        super().__init__()
        self.layers = nn.ModuleList(DenseLayer(c0 + i * k, k) for i in range(L))

    def forward(self, x):
        feats = [x]
        for layer in self.layers:
            feats.append(layer(feats))
        return torch.cat(feats, dim=1)       # c0 + L*k channels

blk = DenseBlock(64, 6)
print(sum(p.numel() for p in blk.parameters()))   # 335040
print(blk(torch.randn(1, 64, 56, 56)).shape)      # torch.Size([1, 256, 56, 56])

model = densenet121(weights=DenseNet121_Weights.IMAGENET1K_V1)
print(sum(p.numel() for p in model.parameters()))  # 7978856
# memory_efficient=True recomputes the concatenation + BN + ReLU + 1x1 in backward
lean = densenet121(memory_efficient=True)
```

A DenseNet-BC dense layer and block, the torchvision model, and its parameter count

ImageNet. Top-1 error: Huang et al. 2017, Table 3 (single 224 crop; DenseNet-161 is not in that table). Params, GFLOPs and acc@1: torchvision's documented weights (IMAGENET1K_V1).

| Model | paper top-1 err. | torchvision acc@1 | Params | GFLOPs |
| --- | --- | --- | --- | --- |
| densenet121 | 25.02% | 74.43% | 8.0M | 2.83 |
| densenet169 | 23.80% | 75.60% | 14.1M | 3.36 |
| densenet201 | 22.58% | 76.90% | 20.0M | 4.29 |
| densenet161 (k = 48) | – | 77.14% | 28.7M | 7.73 |
| resnet50 (for scale) | – | 76.13% | 25.6M | 4.09 |

The paper reports that a DenseNet-201 with 20M parameters reaches about the validation error of a ResNet-101 with more than 40M, and on CIFAR it reaches 3.46% (C10+) and 17.18% (C100+) error with a 190-layer DenseNet-BC at $k=40$. The parameter efficiency is real. Measured speed is a different matter.

> **Memory: the naive implementation is quadratic:** If each layer materialises its own concatenated input, layer $\ell$ stores $k_0+(\ell-1)k$ channels, so a block holds $Lk_0+kL(L-1)/2$ channel-maps for backward, quadratic in $L$. In DenseNet-121's third block ($k_0=256$, $L=24$, at $14\times14$) that is 14,976 maps, about 11.7 MB per image in float32, against 1,024 maps (0.80 MB) if every feature map is stored once. Pleiss et al. (2017), "Memory-Efficient Implementation of DenseNets", write all outputs into a shared buffer and recompute the concatenation, BN and ReLU in the backward pass. torchvision exposes this as `memory_efficient=True` (gradient checkpointing).

> **Concatenate vs add:** Addition keeps the width fixed and forces new and old features into the same channels; concatenation keeps them separate and grows the width. A ResNet layer must both preserve and modify the state, while a DenseNet layer only appends. Later designs mostly went back to addition (ResNeXt, RegNet, EfficientNet, ConvNeXt), but concatenation survives in U-Net skip connections, in feature pyramids and in DenseNet-style CSPNet backbones.

> **Ordering: pre-activation:** DenseNet's layers are BN → ReLU → conv, the pre-activation order of ResNet v2. One consequence: the stored maps are raw conv outputs, and each consuming layer applies its **own** BN to them, so the same feature map is normalised differently for each reader. See **Batch normalization** for why those per-layer statistics matter at inference.

**Strengths**

- Very parameter-efficient: 8.0M parameters for ResNet-50-like accuracy
- Short gradient paths to every layer (implicit deep supervision); trains easily at depth
- Feature reuse: low-level maps reach the classifier directly
- Strong on small data; the paper reports it is less prone to overfitting than ResNets

**Limitations**

- High activation memory and memory traffic from repeated concatenation
- Slower on GPUs than its FLOPs suggest: many thin convs and copies
- Channel width varies inside a block, which complicates pruning and kernel tuning
- Overtaken by ResNeXt/RegNet/EfficientNet-style networks on accuracy per unit of latency

**Question:** A dense block has k₀ = 64 input channels, growth rate k = 32 and L = 6 layers. How many channels does it output, and how many direct connections does it contain?

- A. 256 channels, 21 connections
- B. 96 channels, 6 connections
- C. 256 channels, 36 connections
- D. 224 channels, 15 connections

**Answer:** A. 256 channels, 21 connections — The output concatenates the input and all 6 layer outputs: $64+6\cdot32=256$ channels. Layer $\ell$ receives $\ell$ inputs ($x_0$ through $x_{\ell-1}$), so the block has $1+2+\dots+6=L(L+1)/2=21$ connections.

**Question:** Why does a DenseNet layer need far fewer output channels (k = 32) than a ResNet layer of comparable depth?

- A. Because it uses larger kernels
- B. Because concatenation keeps every earlier feature map available, so each layer only adds a few new maps instead of rewriting the whole state
- C. Because BatchNorm is applied twice
- D. Because transition layers double the channels

**Answer:** B. Because concatenation keeps every earlier feature map available, so each layer only adds a few new maps instead of rewriting the whole state — In a ResNet the state is overwritten by addition, so every block must carry the full width. In a DenseNet the state is appended to, and the classifier and later layers can read early features directly. Each layer only needs to contribute $k$ new channels.

**Key points**

- $x_\ell=H_\ell([x_0,\dots,x_{\ell-1}])$: every layer reads the concatenation of all earlier maps in its block and adds $k$ channels.
- A block of $L$ layers has $L(L+1)/2$ connections and outputs $k_0+Lk$ channels.
- DenseNet-BC: bottleneck BN-ReLU-1×1 ($4k$)-BN-ReLU-3×3 ($k$) layers, transitions with compression $\theta=0.5$.
- DenseNet-121: 7.98M parameters, 2.83G multiply-adds, 74.4% top-1 with torchvision weights.
- The price is activation memory; use shared buffers / checkpointing (memory_efficient=True).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
