Stochastic depth

During training, skip whole residual blocks at random, keeping block l with a probability that decays linearly to p_L = 0.5. The network is shorter and cheaper to train, regularized like an ensemble of depths, and complete at test time.

11 min read774 wordsadvanced0 viewsupdated 2026-10-11

Definition

Stochastic depth is a regularizer for residual networks: during training, each residual branch flf_l is switched off at random, so that block becomes the identity xl=xl−1x_l=x_{l-1}. Block ll survives with probability plp_l. At test time every block runs, with its branch scaled to match the training-time expectation. Its per-sample form is called DropPath in vision-transformer code.

Very deep ResNets train slowly and can overfit, and in a deep stack many blocks contribute only small corrections. Huang, Sun, Liu, Sedra and Weinberger (2016, "Deep networks with stochastic depth", ECCV) proposed training a short network and testing a deep one. For each mini-batch they drop a random subset of residual blocks and bypass them with the identity shortcut. The shortcut is what makes this possible: in a Residual neural network (ResNet) removing flf_l leaves a valid network, while removing a layer of a plain network breaks it.

On CIFAR-10, a 110-layer ResNet with stochastic depth reached 5.25% test error against 6.41% with constant depth, and on CIFAR-100 24.98% against 27.22%, while training about a quarter faster. They also trained a 1202-layer ResNet with it. The method was later adopted by EfficientNet (where the code calls it drop_connect), DeiT, Swin, ConvNeXt and most vision transformers, usually as drop_path_rate.

Formula

xl=xl−1+bl fl(xl−1),bl∼Bernoulli(pl),pl=1−lL (1−pL)x_l=x_{l-1}+b_l\,f_l(x_{l-1}),\qquad b_l\sim\mathrm{Bernoulli}(p_l),\qquad p_l=1-\frac{l}{L}\,(1-p_L)
flf_l
residual branch of block l (e.g. conv-BN-ReLU-conv-BN)
blb_l
1 if the block runs on this step, 0 if it is skipped (identity)
plp_l
survival probability of block l=1,…,Ll=1,\dots,L; the input has p0=1p_0=1
pLp_L
survival of the last block; Huang et al. use pL=0.5p_L=0.5

The original ResNet applied a ReLU after the addition, xl=ReLU(blfl(xl−1)+xl−1)x_l=\mathrm{ReLU}\big(b_lf_l(x_{l-1})+x_{l-1}\big); with pre-activation blocks and Transformers the sum is left as written above. Early blocks extract low-level features that every later block depends on, so the linear decay rule keeps them almost always and drops the late ones most. Huang et al. found it better than a uniform pl=pLp_l=p_L. The gradient of a skipped block is exactly zero on that step, but the shortcut still carries the gradient to all earlier blocks.

E[active blocks]=∑l=1Lpl=L−(1−pL)(L+1)2⇒L=54, pL=0.5:  54−0.5⋅552=40.25\mathbb E[\text{active blocks}]=\sum_{l=1}^{L}p_l=L-\frac{(1-p_L)(L+1)}{2}\qquad\Rightarrow\qquad L=54,\ p_L=0.5:\ \ 54-\frac{0.5\cdot55}{2}=40.25
L=54L=54
residual blocks in ResNet-110 (3 stages × 18 blocks of 2 conv layers)
40.2540.25
expected blocks per step: 25.5% less residual-branch compute than the full 54

Test time: the scaling

At test time all blocks run. Each branch was present only a fraction plp_l of the time during training, so the original paper scales it by plp_l, exactly like dropout's weight-scaling rule. Modern implementations (torchvision.ops.stochastic_depth, timm's DropPath) use the inverted convention instead: divide the surviving branch by plp_l during training, and run the plain network at test time.

original:  xltest=xl−1test+pl fl(xl−1test)inverted:  xltrain=xl−1+blpl fl(xl−1),  xltest=xl−1+fl(xl−1)\text{original: }\ x_l^{\text{test}}=x_{l-1}^{\text{test}}+p_l\,f_l\big(x_{l-1}^{\text{test}}\big)\qquad\qquad\text{inverted: }\ x_l^{\text{train}}=x_{l-1}+\frac{b_l}{p_l}\,f_l(x_{l-1}),\ \ x_l^{\text{test}}=x_{l-1}+f_l(x_{l-1})
plflp_l f_l
the expected contribution of the branch during training
bl/plb_l/p_l
mean 1, so E[xtrain]\mathbb E[x^{\text{train}}] matches the unscaled test network (exactly for linear branches, approximately otherwise)

At a glance

Drops
whole residual branches; the shortcut always stays
Rule
linear decay pl=1−lL(1−pL)p_l=1-\frac lL(1-p_L), pL=0.5p_L=0.5 in the paper; timm uses drop rates torch.linspace(0, drop_path_rate, depth)
Typical rates
drop 0.1 (DeiT-B), 0.2 / 0.3 / 0.5 (Swin-T / S / B); larger models need more
Mask granularity
per mini-batch in the paper (mode="batch"), per example in DropPath (mode="row")
Train speed
only batch mode can skip computation: about 25% faster for ResNet-110 at pL=0.5p_L=0.5
Test time
full depth, deterministic; scale branches by p_l, or nothing with inverted scaling
one step, row mode: block l runs w.p. p_l123456789101112x18x211x38x411x56x66x710x810number at right = blocks that ran for that examplesurvival p_l (linear decay 1 → p_L)10■ block runs ⬚ skipped: x_l = x_(l−1)residual stream ‖x_l‖, 12 blocks0612113test-time output vs average training output:branch × p_l at test: off by 4.7%forgot × p_l: off by 33.2%expected depth Σp_l = 8.75 of 12 blocksResNet-110 (54 blocks): 40.25 → 25.5% lessresidual-branch compute per step━ sampled paths ━ ‖mean training x_l‖ ━ test net ┅ wrong scaling
maskrulescaling
fig.Left: one training step on a batch of 8 through 12 residual blocks with the linear rule and p_L = 0.5. In row mode every example skips its own blocks (with the default seed the 8 examples run 8, 11, 8, 11, 6, 6, 10 and 10 blocks); batch mode skips the same blocks for all of them, as in the original paper. The expected depth is 8.75 of 12 (6.00 with the uniform rule); for ResNet-110's 54 blocks it is 40.25, 25.5% less residual-branch compute. Right: the residual stream of a fixed toy network (16-d, tanh branches). Grey: six sampled training paths. Thick blue: the norm of the average training stream. With the original convention, scaling branches by p_l at test (green) puts the output within 4.7% of the average training output; forgetting the scaling (dashed red) is off by 33.2%. With the inverted convention the plain test network is within 9.3%, and wrongly multiplying by p_l on top is off by 22.5%.

Worked example: four blocks

Take L=4L=4, pL=0.5p_L=0.5, input x0=1x_0=1, and make every branch output the constant fl=0.5f_l=0.5 so the arithmetic is exact (checked in Python by enumerating all 16 masks).

quantityvaluehow
p1,…,p4p_1,\dots,p_40.875, 0.75, 0.625, 0.50.875,\ 0.75,\ 0.625,\ 0.51−l4⋅0.51-\tfrac l4\cdot0.5
expected active blocks2.752.75∑pl\sum p_l
P(all four run)0.2050.2050.875⋅0.75⋅0.625⋅0.50.875\cdot0.75\cdot0.625\cdot0.5
P(all four skipped)0.00590.00590.125⋅0.25⋅0.375⋅0.50.125\cdot0.25\cdot0.375\cdot0.5
E[x4]\mathbb E[x_4] in training (original)2.3752.3751+0.5∑pl1+0.5\sum p_l
test, branches ×pl\times p_l2.3752.375matches
test, scaling forgotten3.03.01+4⋅0.51+4\cdot0.5: 26% too large
E[x4]\mathbb E[x_4] in training (inverted, ÷pl\div p_l)3.03.0the plain test network gives 3.0: matches
Linear decay with L = 4, p_L = 0.5

Each training step samples one of 24=162^4=16 sub-networks of depth 0 to 4. In general there are 2L2^L, which ties stochastic depth to the view of Veit, Wilber and Belongie (2016) that a ResNet behaves like an ensemble of many relatively shallow paths: stochastic depth trains that ensemble explicitly, with shared weights, and the full network at test time approximates its average.

Why it helps

  1. 1Shorter networks during training. The expected depth is about three quarters of the full depth for pL=0.5p_L=0.5, so the gradient reaches the early blocks through fewer residual branches, and forward-backward is cheaper when whole blocks can be skipped.
  2. 2Regularization. Every block must be useful when some of its predecessors are missing, a coarse-grained version of dropout's argument against co-adaptation.
  3. 3An implicit ensemble of depths. Test-time prediction averages over 2L2^L shared-weight sub-networks.
  4. 4Robustness. A network trained this way degrades gracefully if blocks are removed at test time, because it has seen every block go missing.
python
1import torch, torch.nn as nn
2from torchvision.ops import StochasticDepth, stochastic_depth
3
4# torchvision: p is the DROP probability; survivors are divided by (1 - p) (inverted)
5x = torch.ones(4, 3)
6print(stochastic_depth(x, p=0.5, mode="row", training=True))
7# each ROW (example) is all 0.0 or all 2.0
8print(stochastic_depth(x, p=0.5, mode="batch", training=True))
9# the whole batch is all 0.0 or all 2.0
10print(stochastic_depth(x, p=0.5, mode="row", training=False)) # identity
11
12class Block(nn.Module):
13 def __init__(self, ch, drop):
14 super().__init__()
15 self.f = nn.Sequential(
16 nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch), nn.ReLU(),
17 nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch),
18 )
19 self.sd = StochasticDepth(p=drop, mode="row") # applied to the BRANCH only
20 def forward(self, x):
21 return torch.relu(x + self.sd(self.f(x))) # never drop the shortcut
22
23# Linear decay: drop rate grows with depth (Huang et al.: survival 1 -> 0.5)
24L, p_drop_last = 54, 0.5
25blocks = nn.Sequential(*[Block(16, drop=(l / L) * p_drop_last) for l in range(1, L + 1)])
26
27# timm-style DropPath (what ViTs use): rates torch.linspace(0, drop_path_rate, depth)
28def drop_path(x, drop_prob=0.0, training=False):
29 if drop_prob == 0.0 or not training:
30 return x
31 keep = 1 - drop_prob
32 shape = (x.shape[0],) + (1,) * (x.dim() - 1) # one draw per example
33 return x * x.new_empty(shape).bernoulli_(keep) / keep
34
35# In a transformer block: x = x + drop_path(attn(norm1(x)), dpr[i], self.training)
36# x = x + drop_path(mlp(norm2(x)), dpr[i], self.training)
Stochastic depth with torchvision, timm-style DropPath, and a linear schedule

Use when

  • +Strong regularizer for deep residual networks and ViTs; standard in modern recipes
  • +Batch mode trains faster: about 25% fewer block evaluations for ResNet-110
  • +Trains an implicit ensemble of 2^L depths with shared weights
  • +No cost at test time with inverted scaling; one hyperparameter (the final rate)

Avoid when

  • −Needs residual connections: there must be an identity to fall back on
  • −Row mode (the usual choice) saves no compute
  • −Drop vs survival conventions differ between the paper and libraries
  • −Too high a rate hurts small models and short training runs

Check yourself

A ViT has 12 blocks and drop_path_rate = 0.1 with timm's linear schedule. What is the drop probability of the first and the last block?

Check yourself

With the original (non-inverted) convention, what goes wrong if you run all blocks at test time without scaling them by p_l?

Remember

  • ✓xl=xl−1+blfl(xl−1)x_l=x_{l-1}+b_lf_l(x_{l-1}) with bl∼Bernoulli(pl)b_l\sim\mathrm{Bernoulli}(p_l): skipped blocks become the identity.
  • ✓Linear decay pl=1−lL(1−pL)p_l=1-\frac lL(1-p_L) with pL=0.5p_L=0.5; expected depth L−(1−pL)(L+1)/2L-(1-p_L)(L+1)/2, 40.25 of 54 blocks for ResNet-110.
  • ✓Test time: all blocks, branch ×pl\times p_l (original) or train-time ÷pl\div p_l (inverted, torchvision and timm).
  • ✓Batch mode (paper) saves compute; row mode (DropPath in ViTs) draws per example and only regularizes.
  • ✓Library arguments are drop probabilities; the paper's p_l are survival probabilities.
← back to Deep Learning roadmap