Definition
Stochastic depth is a regularizer for residual networks: during training, each residual branch is switched off at random, so that block becomes the identity . Block survives with probability . At test time every block runs, with its branch scaled to match the training-time expectation. Its per-sample form is called DropPath in vision-transformer code.
Very deep ResNets train slowly and can overfit, and in a deep stack many blocks contribute only small corrections. Huang, Sun, Liu, Sedra and Weinberger (2016, "Deep networks with stochastic depth", ECCV) proposed training a short network and testing a deep one. For each mini-batch they drop a random subset of residual blocks and bypass them with the identity shortcut. The shortcut is what makes this possible: in a Residual neural network (ResNet) removing leaves a valid network, while removing a layer of a plain network breaks it.
On CIFAR-10, a 110-layer ResNet with stochastic depth reached 5.25% test error against 6.41% with constant depth, and on CIFAR-100 24.98% against 27.22%, while training about a quarter faster. They also trained a 1202-layer ResNet with it. The method was later adopted by EfficientNet (where the code calls it drop_connect), DeiT, Swin, ConvNeXt and most vision transformers, usually as drop_path_rate.
Formula
- residual branch of block l (e.g. conv-BN-ReLU-conv-BN)
- 1 if the block runs on this step, 0 if it is skipped (identity)
- survival probability of block ; the input has
- survival of the last block; Huang et al. use
The original ResNet applied a ReLU after the addition, ; with pre-activation blocks and Transformers the sum is left as written above. Early blocks extract low-level features that every later block depends on, so the linear decay rule keeps them almost always and drops the late ones most. Huang et al. found it better than a uniform . The gradient of a skipped block is exactly zero on that step, but the shortcut still carries the gradient to all earlier blocks.
- residual blocks in ResNet-110 (3 stages × 18 blocks of 2 conv layers)
- expected blocks per step: 25.5% less residual-branch compute than the full 54
Test time: the scaling
At test time all blocks run. Each branch was present only a fraction of the time during training, so the original paper scales it by , exactly like dropout's weight-scaling rule. Modern implementations (torchvision.ops.stochastic_depth, timm's DropPath) use the inverted convention instead: divide the surviving branch by during training, and run the plain network at test time.
- the expected contribution of the branch during training
- mean 1, so matches the unscaled test network (exactly for linear branches, approximately otherwise)
At a glance
- Drops
- whole residual branches; the shortcut always stays
- Rule
- linear decay , in the paper; timm uses drop rates
torch.linspace(0, drop_path_rate, depth) - Typical rates
- drop 0.1 (DeiT-B), 0.2 / 0.3 / 0.5 (Swin-T / S / B); larger models need more
- Mask granularity
- per mini-batch in the paper (
mode="batch"), per example in DropPath (mode="row") - Train speed
- only batch mode can skip computation: about 25% faster for ResNet-110 at
- Test time
- full depth, deterministic; scale branches by p_l, or nothing with inverted scaling
Worked example: four blocks
Take , , input , and make every branch output the constant so the arithmetic is exact (checked in Python by enumerating all 16 masks).
| quantity | value | how |
|---|---|---|
| expected active blocks | ||
| P(all four run) | ||
| P(all four skipped) | ||
| in training (original) | ||
| test, branches | matches | |
| test, scaling forgotten | : 26% too large | |
| in training (inverted, ) | the plain test network gives 3.0: matches |
Each training step samples one of sub-networks of depth 0 to 4. In general there are , which ties stochastic depth to the view of Veit, Wilber and Belongie (2016) that a ResNet behaves like an ensemble of many relatively shallow paths: stochastic depth trains that ensemble explicitly, with shared weights, and the full network at test time approximates its average.
Why it helps
- 1Shorter networks during training. The expected depth is about three quarters of the full depth for , so the gradient reaches the early blocks through fewer residual branches, and forward-backward is cheaper when whole blocks can be skipped.
- 2Regularization. Every block must be useful when some of its predecessors are missing, a coarse-grained version of dropout's argument against co-adaptation.
- 3An implicit ensemble of depths. Test-time prediction averages over shared-weight sub-networks.
- 4Robustness. A network trained this way degrades gracefully if blocks are removed at test time, because it has seen every block go missing.
1import torch, torch.nn as nn2from torchvision.ops import StochasticDepth, stochastic_depth34# torchvision: p is the DROP probability; survivors are divided by (1 - p) (inverted)5x = torch.ones(4, 3)6print(stochastic_depth(x, p=0.5, mode="row", training=True))7# each ROW (example) is all 0.0 or all 2.08print(stochastic_depth(x, p=0.5, mode="batch", training=True))9# the whole batch is all 0.0 or all 2.010print(stochastic_depth(x, p=0.5, mode="row", training=False)) # identity1112class Block(nn.Module):13 def __init__(self, ch, drop):14 super().__init__()15 self.f = nn.Sequential(16 nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch), nn.ReLU(),17 nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch),18 )19 self.sd = StochasticDepth(p=drop, mode="row") # applied to the BRANCH only20 def forward(self, x):21 return torch.relu(x + self.sd(self.f(x))) # never drop the shortcut2223# Linear decay: drop rate grows with depth (Huang et al.: survival 1 -> 0.5)24L, p_drop_last = 54, 0.525blocks = nn.Sequential(*[Block(16, drop=(l / L) * p_drop_last) for l in range(1, L + 1)])2627# timm-style DropPath (what ViTs use): rates torch.linspace(0, drop_path_rate, depth)28def drop_path(x, drop_prob=0.0, training=False):29 if drop_prob == 0.0 or not training:30 return x31 keep = 1 - drop_prob32 shape = (x.shape[0],) + (1,) * (x.dim() - 1) # one draw per example33 return x * x.new_empty(shape).bernoulli_(keep) / keep3435# In a transformer block: x = x + drop_path(attn(norm1(x)), dpr[i], self.training)36# x = x + drop_path(mlp(norm2(x)), dpr[i], self.training)Use when
- +Strong regularizer for deep residual networks and ViTs; standard in modern recipes
- +Batch mode trains faster: about 25% fewer block evaluations for ResNet-110
- +Trains an implicit ensemble of 2^L depths with shared weights
- +No cost at test time with inverted scaling; one hyperparameter (the final rate)
Avoid when
- −Needs residual connections: there must be an identity to fall back on
- −Row mode (the usual choice) saves no compute
- −Drop vs survival conventions differ between the paper and libraries
- −Too high a rate hurts small models and short training runs
Check yourself
A ViT has 12 blocks and drop_path_rate = 0.1 with timm's linear schedule. What is the drop probability of the first and the last block?
Check yourself
With the original (non-inverted) convention, what goes wrong if you run all blocks at test time without scaling them by p_l?
Remember
- ✓ with : skipped blocks become the identity.
- ✓Linear decay with ; expected depth , 40.25 of 54 blocks for ResNet-110.
- ✓Test time: all blocks, branch (original) or train-time (inverted, torchvision and timm).
- ✓Batch mode (paper) saves compute; row mode (DropPath in ViTs) draws per example and only regularizes.
- ✓Library arguments are drop probabilities; the paper's p_l are survival probabilities.