# Stochastic depth

> During training, skip whole residual blocks at random, keeping block l with a probability that decays linearly to p_L = 0.5. The network is shorter and cheaper to train, regularized like an ensemble of depths, and complete at test time.

Source: https://www.mlfoundry.in/learn/stochastic-depth  
Section: Deep Learning › Regularization & Normalization  
Level: advanced · 11 min read · updated 2026-10-11

**Stochastic depth:** is a regularizer for residual networks: during training, each residual branch $f_l$ is switched off at random, so that block becomes the identity $x_l=x_{l-1}$. Block $l$ survives with probability $p_l$. At test time every block runs, with its branch scaled to match the training-time expectation. Its per-sample form is called **DropPath** in vision-transformer code.

Very deep ResNets train slowly and can overfit, and in a deep stack many blocks contribute only small corrections. Huang, Sun, Liu, Sedra and Weinberger (2016, "Deep networks with stochastic depth", ECCV) proposed training a **short network** and testing a **deep** one. For each mini-batch they drop a random subset of residual blocks and bypass them with the identity shortcut. The shortcut is what makes this possible: in a **Residual neural network (ResNet)** removing $f_l$ leaves a valid network, while removing a layer of a plain network breaks it.

On CIFAR-10, a 110-layer ResNet with stochastic depth reached 5.25% test error against 6.41% with constant depth, and on CIFAR-100 24.98% against 27.22%, while training about a quarter faster. They also trained a 1202-layer ResNet with it. The method was later adopted by EfficientNet (where the code calls it `drop_connect`), DeiT, Swin, ConvNeXt and most vision transformers, usually as `drop_path_rate`.

## Formula

$$
x_l=x_{l-1}+b_l\,f_l(x_{l-1}),\qquad b_l\sim\mathrm{Bernoulli}(p_l),\qquad p_l=1-\frac{l}{L}\,(1-p_L)
$$

- $f_l$: residual branch of block l (e.g. conv-BN-ReLU-conv-BN)
- $b_l$: 1 if the block runs on this step, 0 if it is skipped (identity)
- $p_l$: survival probability of block $l=1,\dots,L$; the input has $p_0=1$
- $p_L$: survival of the last block; Huang et al. use $p_L=0.5$

The original ResNet applied a ReLU after the addition, $x_l=\mathrm{ReLU}\big(b_lf_l(x_{l-1})+x_{l-1}\big)$; with pre-activation blocks and Transformers the sum is left as written above. Early blocks extract low-level features that every later block depends on, so the **linear decay rule** keeps them almost always and drops the late ones most. Huang et al. found it better than a uniform $p_l=p_L$. The gradient of a skipped block is exactly zero on that step, but the shortcut still carries the gradient to all earlier blocks.

$$
\mathbb E[\text{active blocks}]=\sum_{l=1}^{L}p_l=L-\frac{(1-p_L)(L+1)}{2}\qquad\Rightarrow\qquad L=54,\ p_L=0.5:\ \ 54-\frac{0.5\cdot55}{2}=40.25
$$

- $L=54$: residual blocks in ResNet-110 (3 stages × 18 blocks of 2 conv layers)
- $40.25$: expected blocks per step: 25.5% less residual-branch compute than the full 54

## Test time: the scaling

At test time all blocks run. Each branch was present only a fraction $p_l$ of the time during training, so the original paper scales it by $p_l$, exactly like dropout's weight-scaling rule. Modern implementations (`torchvision.ops.stochastic_depth`, timm's `DropPath`) use the **inverted** convention instead: divide the surviving branch by $p_l$ during training, and run the plain network at test time.

$$
\text{original: }\ x_l^{\text{test}}=x_{l-1}^{\text{test}}+p_l\,f_l\big(x_{l-1}^{\text{test}}\big)\qquad\qquad\text{inverted: }\ x_l^{\text{train}}=x_{l-1}+\frac{b_l}{p_l}\,f_l(x_{l-1}),\ \ x_l^{\text{test}}=x_{l-1}+f_l(x_{l-1})
$$

- $p_l f_l$: the expected contribution of the branch during training
- $b_l/p_l$: mean 1, so $\mathbb E[x^{\text{train}}]$ matches the unscaled test network (exactly for linear branches, approximately otherwise)

**At a glance**

- **Drops:** whole residual branches; the shortcut always stays
- **Rule:** linear decay $p_l=1-\frac lL(1-p_L)$, $p_L=0.5$ in the paper; timm uses drop rates `torch.linspace(0, drop_path_rate, depth)`
- **Typical rates:** drop 0.1 (DeiT-B), 0.2 / 0.3 / 0.5 (Swin-T / S / B); larger models need more
- **Mask granularity:** per mini-batch in the paper (`mode="batch"`), per example in DropPath (`mode="row"`)
- **Train speed:** only batch mode can skip computation: about 25% faster for ResNet-110 at $p_L=0.5$
- **Test time:** full depth, deterministic; scale branches by p_l, or nothing with inverted scaling

> Figure: Left: one training step on a batch of 8 through 12 residual blocks with the linear rule and p_L = 0.5. In row mode every example skips its own blocks (with the default seed the 8 examples run 8, 11, 8, 11, 6, 6, 10 and 10 blocks); batch mode skips the same blocks for all of them, as in the original paper. The expected depth is 8.75 of 12 (6.00 with the uniform rule); for ResNet-110's 54 blocks it is 40.25, 25.5% less residual-branch compute. Right: the residual stream of a fixed toy network (16-d, tanh branches). Grey: six sampled training paths. Thick blue: the norm of the average training stream. With the original convention, scaling branches by p_l at test (green) puts the output within 4.7% of the average training output; forgetting the scaling (dashed red) is off by 33.2%. With the inverted convention the plain test network is within 9.3%, and wrongly multiplying by p_l on top is off by 22.5%.

## Worked example: four blocks

Take $L=4$, $p_L=0.5$, input $x_0=1$, and make every branch output the constant $f_l=0.5$ so the arithmetic is exact (checked in Python by enumerating all 16 masks).

Linear decay with L = 4, p_L = 0.5

| quantity | value | how |
| --- | --- | --- |
| $p_1,\dots,p_4$ | $0.875,\ 0.75,\ 0.625,\ 0.5$ | $1-\tfrac l4\cdot0.5$ |
| expected active blocks | $2.75$ | $\sum p_l$ |
| P(all four run) | $0.205$ | $0.875\cdot0.75\cdot0.625\cdot0.5$ |
| P(all four skipped) | $0.0059$ | $0.125\cdot0.25\cdot0.375\cdot0.5$ |
| $\mathbb E[x_4]$ in training (original) | $2.375$ | $1+0.5\sum p_l$ |
| test, branches $\times p_l$ | $2.375$ | matches |
| test, scaling forgotten | $3.0$ | $1+4\cdot0.5$: 26% too large |
| $\mathbb E[x_4]$ in training (inverted, $\div p_l$) | $3.0$ | the plain test network gives 3.0: matches |

Each training step samples one of $2^4=16$ sub-networks of depth 0 to 4. In general there are $2^L$, which ties stochastic depth to the view of Veit, Wilber and Belongie (2016) that a ResNet behaves like an ensemble of many relatively shallow paths: stochastic depth trains that ensemble explicitly, with shared weights, and the full network at test time approximates its average.

**Why it helps**

1. **Shorter networks during training.** The expected depth is about three quarters of the full depth for $p_L=0.5$, so the gradient reaches the early blocks through fewer residual branches, and forward-backward is cheaper when whole blocks can be skipped.
2. **Regularization.** Every block must be useful when some of its predecessors are missing, a coarse-grained version of dropout's argument against co-adaptation.
3. **An implicit ensemble of depths.** Test-time prediction averages over $2^L$ shared-weight sub-networks.
4. **Robustness.** A network trained this way degrades gracefully if blocks are removed at test time, because it has seen every block go missing.

```python
import torch, torch.nn as nn
from torchvision.ops import StochasticDepth, stochastic_depth

# torchvision: p is the DROP probability; survivors are divided by (1 - p) (inverted)
x = torch.ones(4, 3)
print(stochastic_depth(x, p=0.5, mode="row", training=True))
# each ROW (example) is all 0.0 or all 2.0
print(stochastic_depth(x, p=0.5, mode="batch", training=True))
# the whole batch is all 0.0 or all 2.0
print(stochastic_depth(x, p=0.5, mode="row", training=False))   # identity

class Block(nn.Module):
    def __init__(self, ch, drop):
        super().__init__()
        self.f = nn.Sequential(
            nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch), nn.ReLU(),
            nn.Conv2d(ch, ch, 3, padding=1, bias=False), nn.BatchNorm2d(ch),
        )
        self.sd = StochasticDepth(p=drop, mode="row")   # applied to the BRANCH only
    def forward(self, x):
        return torch.relu(x + self.sd(self.f(x)))       # never drop the shortcut

# Linear decay: drop rate grows with depth (Huang et al.: survival 1 -> 0.5)
L, p_drop_last = 54, 0.5
blocks = nn.Sequential(*[Block(16, drop=(l / L) * p_drop_last) for l in range(1, L + 1)])

# timm-style DropPath (what ViTs use): rates torch.linspace(0, drop_path_rate, depth)
def drop_path(x, drop_prob=0.0, training=False):
    if drop_prob == 0.0 or not training:
        return x
    keep = 1 - drop_prob
    shape = (x.shape[0],) + (1,) * (x.dim() - 1)        # one draw per example
    return x * x.new_empty(shape).bernoulli_(keep) / keep

# In a transformer block:  x = x + drop_path(attn(norm1(x)), dpr[i], self.training)
#                          x = x + drop_path(mlp(norm2(x)),  dpr[i], self.training)
```

Stochastic depth with torchvision, timm-style DropPath, and a linear schedule

> **Drop probability or survival probability?:** Huang et al.'s $p_l$ and $p_L=0.5$ are **survival** probabilities. `torchvision.ops.stochastic_depth(p=…)`, timm's `drop_path_rate` and EfficientNet's `drop_connect_rate` are **drop** probabilities. A `drop_path_rate` of 0.5 means the last block is skipped half the time; a survival probability of 0.1 typed into a drop-probability argument skips almost every block and the model will not train.

> **Row mode does not save compute:** Only batch mode, where the whole mini-batch skips a block, can actually avoid running $f_l$. That is the mode of the original paper and the source of its speed-up. Row mode (DropPath) still computes the branch for the full batch and multiplies some rows by zero: it regularizes, often better, because each example gets its own sub-network, but it is no faster. Nearly all vision-transformer recipes use row mode.

> **Interactions:** Only drop the residual **branch**, never the shortcut or the whole block output. BatchNorm inside a skipped branch receives no batch statistics that step in batch mode, which is harmless; in row mode BN still sees the full batch. Stochastic depth is usually combined with weight decay, MixUp/CutMix and label smoothing in modern recipes, and the drop rate is tuned upward with model size.

**Strengths**

- Strong regularizer for deep residual networks and ViTs; standard in modern recipes
- Batch mode trains faster: about 25% fewer block evaluations for ResNet-110
- Trains an implicit ensemble of 2^L depths with shared weights
- No cost at test time with inverted scaling; one hyperparameter (the final rate)

**Limitations**

- Needs residual connections: there must be an identity to fall back on
- Row mode (the usual choice) saves no compute
- Drop vs survival conventions differ between the paper and libraries
- Too high a rate hurts small models and short training runs

**Question:** A ViT has 12 blocks and drop_path_rate = 0.1 with timm's linear schedule. What is the drop probability of the first and the last block?

- A. 0.1 for every block
- B. 0 for the first, 0.1 for the last
- C. 0.1 for the first, 0 for the last
- D. 0.0083 for the first, 0.1 for the last

**Answer:** B. 0 for the first, 0.1 for the last — timm uses `torch.linspace(0, drop_path_rate, depth)`, so the 12 rates run from 0 to 0.1 in equal steps of $0.1/11$. Huang et al.'s rule $1-p_l=\frac lL(1-p_L)$ gives the first block a small nonzero drop rate ($0.1/12=0.0083$) instead.

**Question:** With the original (non-inverted) convention, what goes wrong if you run all blocks at test time without scaling them by p_l?

- A. Every residual branch adds more than it did on average during training, so activations drift upward through depth
- B. Nothing: the network is deterministic at test time
- C. The early blocks are skipped
- D. Gradients vanish

**Answer:** A. Every residual branch adds more than it did on average during training, so activations drift upward through depth — During training branch $l$ contributed $p_lf_l$ on average. Without the scaling it contributes $f_l$, so the deviations accumulate block after block. In the four-block example the output is 3.0 instead of 2.375, and in the figure the output is off by 33.2% instead of 4.7%.

**Key points**

- $x_l=x_{l-1}+b_lf_l(x_{l-1})$ with $b_l\sim\mathrm{Bernoulli}(p_l)$: skipped blocks become the identity.
- Linear decay $p_l=1-\frac lL(1-p_L)$ with $p_L=0.5$; expected depth $L-(1-p_L)(L+1)/2$, 40.25 of 54 blocks for ResNet-110.
- Test time: all blocks, branch $\times p_l$ (original) or train-time $\div p_l$ (inverted, torchvision and timm).
- Batch mode (paper) saves compute; row mode (DropPath in ViTs) draws per example and only regularizes.
- Library arguments are drop probabilities; the paper's p_l are survival probabilities.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
