# Dropout

> During training, zero each unit with probability p and scale the survivors by 1/(1−p); at test time use every unit. It trains an ensemble of 2ⁿ weight-sharing sub-networks and stops units from relying on specific partners.

Source: https://www.mlfoundry.in/learn/dropout  
Section: Deep Learning › Regularization & Normalization  
Level: core · 14 min read · updated 2026-10-11

**Dropout:** is a training-time regularizer: on every forward pass each unit of a layer is set to zero independently with probability $p$, and the units that survive are scaled by $1/(1-p)$. At evaluation the layer is the identity. Each step therefore trains a different random **thinned** sub-network, and all of them share one set of weights.

A large network fit to a small dataset can drive training loss to zero by memorising noise. Often it does so through **co-adaptation**: a unit becomes useful only in combination with a few specific other units, which together encode a quirk of the training set. Dropout breaks this. Any partner may be missing on the next step, so each unit has to learn features that are useful on their own and in many different company.

The method came from Hinton, Srivastava, Krizhevsky, Sutskever and Salakhutdinov (2012, "Improving neural networks by preventing co-adaptation of feature detectors"), with the full study in Srivastava et al. (2014, JMLR, "Dropout: a simple way to prevent neural networks from overfitting"). AlexNet (Krizhevsky, Sutskever and Hinton, 2012) used dropout with probability 0.5 in its first two fully connected layers, and noted that it roughly doubled the number of iterations needed to converge. Dropout became the standard regularizer for large fully connected layers and stayed in the Transformer (Vaswani et al., 2017, $P_{drop}=0.1$).

## Formula

$$
m_i\sim\mathrm{Bernoulli}(1-p),\qquad \tilde h_i=\frac{m_i\,h_i}{1-p}\quad\text{(training)},\qquad \tilde h_i=h_i\quad\text{(evaluation)}
$$

- $h_i$: activation of unit i (usually after the nonlinearity)
- $m_i$: keep-mask entry: 1 with probability 1 − p, 0 with probability p. A fresh mask for every example and every step
- $p$: drop probability (PyTorch's `p`). Srivastava et al. write $p$ for the **retain** probability, so their $p=0.8$ is PyTorch's `p=0.2`
- $1/(1-p)$: inverted-dropout scale: it makes $\mathbb E[\tilde h_i]=h_i$, so nothing changes at test time

The backward pass uses the same mask: $\partial\tilde h_i/\partial h_i=m_i/(1-p)$. A dropped unit passes no gradient on that step, and a kept one passes its error scaled by $1/(1-p)$. In expectation the layer is unchanged, $\mathbb E[\tilde h_i]=h_i$, but it now injects multiplicative noise with variance $\operatorname{Var}(\tilde h_i)=\frac{p}{1-p}h_i^2$, which is 1 times $h_i^2$ at $p=0.5$ and $0.11\,h_i^2$ at $p=0.1$.

> Figure: A fresh random mask every training step (p = 0.4): crossed-out units and their connections are absent for that step. The next step trains a different thinned network with the same weights.

**At a glance**

- **Acts on:** activations (units), not weights. Masking weights is DropConnect
- **Parameters:** none learned; one hyperparameter $p$ per dropout layer
- **Typical p:** 0.5 for wide fully connected hidden layers, 0.1 to 0.2 on inputs, 0.1 in Transformers, 0.2 to 0.5 before a CNN's classifier
- **Train vs test:** random masks in `model.train()`, identity in `model.eval()`
- **Cost:** one random draw and a multiply per unit; but training typically needs more steps to converge
- **Interpretations:** an ensemble of $2^n$ sub-networks; adaptive L2 penalty; approximate Bayesian inference (MC dropout)

## Inverted dropout and the weight-scaling rule

The 2012 and 2014 papers did not rescale during training. They dropped units with retention probability $q=1-p$ and, at test time, multiplied each layer's outgoing weights by $q$, so that a unit's expected input matched what it had seen during training. This is the **weight-scaling inference rule**. Modern frameworks do the equivalent thing on the training side: divide the survivors by $q$ at training time, which is **inverted dropout**, and leave the network untouched at test time. The two are identical in expectation; inverted dropout is preferred because the deployed model needs no special code.

$$
\underbrace{\mathbb E_m\Big[\sum_i w_i\,\frac{m_i h_i}{1-p}\Big]}_{\text{train, inverted}}=\sum_i w_i h_i\qquad\Longleftrightarrow\qquad \underbrace{\mathbb E_m\Big[\sum_i w_i\,m_i h_i\Big]}_{\text{train, original}}=\sum_i (1-p)\,w_i\,h_i
$$

- $w_i$: outgoing weight from unit i to one unit of the next layer
- $(1-p)\,w_i$: the original paper's test-time weights: scaled by the retain probability

## Why it regularizes: three views

**An ensemble of $2^n$ networks.** A layer with $n$ droppable units has $2^n$ possible masks, so dropout samples one of $2^n$ thinned networks per example and takes one gradient step on it. Almost none of them is ever trained twice, but they share weights, so every step improves all sub-networks that contain the touched units. Ideally the test-time prediction would average all $2^n$ of them. The weight-scaled (or, with inverted dropout, the plain) network is a one-pass approximation of that average. For a model whose only nonlinearity is the final softmax it is exact: it computes the normalized **geometric** mean of all sub-model predictions (Hinton et al., 2012; Goodfellow, Bengio and Courville, 2016, §7.12). With hidden nonlinearities it is an approximation that works well in practice.

**An adaptive L2 penalty.** For a linear model with dropout on the inputs, averaging the squared error over the mask gives the clean loss plus a ridge term weighted by each feature's magnitude (Srivastava et al., 2014; Wager, Wang and Liang, 2013):

$$
\mathbb E_m\Big[\Big(y-\sum_i w_i\,\frac{m_i x_i}{1-p}\Big)^2\Big]=\big(y-w^\top x\big)^2+\frac{p}{1-p}\sum_i w_i^2\,x_i^2
$$

- $\frac{p}{1-p}$: the penalty strength: 1 at $p=0.5$, $0.25$ at $p=0.2$
- $w_i^2x_i^2$: weights on large, frequently active features are penalized more: an L2 penalty that adapts to the data scale

**Approximate Bayesian inference.** Gal and Ghahramani (2016, "Dropout as a Bayesian approximation") showed that training with dropout before every weight layer approximately minimizes the KL divergence to the posterior of a deep Gaussian process. The consequence is **MC dropout**: keep dropout on at test time, run $T$ stochastic forward passes, and use their mean as the prediction and their spread as an uncertainty estimate.

$$
\hat p(y\mid x)\approx\frac1T\sum_{t=1}^{T}f_{\tilde W_t}(x),\qquad \widehat{\operatorname{Var}}(y\mid x)\approx\frac1T\sum_{t=1}^{T}f_{\tilde W_t}(x)^2-\hat p(y\mid x)^2
$$

- $\tilde W_t$: the network with dropout mask t applied
- $T$: number of stochastic passes, commonly 10 to 100

> Figure: A 2→20→20→1 ReLU network trained by full-batch Adam (800 steps, lr 0.01) on 30 points, 7 of them with flipped labels; validation is 300 fresh points from the same distribution. The dashed curves are the same network without dropout: it reaches 100% training accuracy by carving islands around every flipped label, and its validation loss climbs from 0.578 at step 25 to 7.842 at step 800 (58% accuracy). At the default p = 0.5 the eval-mode training loss stays at 0.376 (80% accuracy) and validation ends at 0.650 with 78% accuracy. Small p helps less (p = 0.2: validation 0.922, 69%). The row of dots shows the masks for one example at that step: about half the units are gone and the rest are scaled ×2. Switch to MC σ for MC dropout's uncertainty (30 passes): it ranges from about 0.01 in the dense core of each class to 0.41, highest in the top-right and bottom-left corners, which hold almost no training data and where the sub-networks disagree about how to extrapolate.

## Worked example: one unit, all 16 masks

Four inputs $h=(2,\,0.5,\,1,\,3)$ feed one unit with weights $w=(0.5,\,-1,\,1,\,0.25)$, and dropout $p=0.5$ is applied to $h$. The eval-mode pre-activation is $z=w^\top h=1-0.5+1+0.75=2.25$. In training, mask $m=(1,0,0,1)$ gives $\tilde h=(4,\,0,\,0,\,6)$ and $z=2+1.5=3.5$. Enumerating all 16 masks (checked in Python):

Statistics over all 2⁴ = 16 equally likely masks, p = 0.5

| quantity | over the 16 masks | eval mode (no mask) | comment |
| --- | --- | --- | --- |
| $z=w^\top\tilde h$ | mean $2.25$ | $2.25$ | exact: z is linear in the mask |
| $\operatorname{Var}(z)$ | $2.8125$ | $0$ | $=\frac{p}{1-p}\sum_i w_i^2h_i^2=1+0.25+1+0.5625$ |
| range of $z$ | $-1.0$ to $5.5$ | $2.25$ | the noise dropout injects |
| $\mathrm{ReLU}(z-2)$ | mean $0.8125$ | $0.25$ | not exact after a nonlinearity |

The last row is why the weight-scaling rule is called an approximation. Before the nonlinearity, eval mode reproduces the ensemble mean exactly. After it, the ensemble average $\mathbb E[\mathrm{ReLU}(z-2)]=0.8125$ differs from $\mathrm{ReLU}(\mathbb E[z]-2)=0.25$ (Jensen's inequality). MC dropout would estimate the 0.8125. In the figure, the average gap between eval-mode and MC-averaged probabilities at p = 0.5 is about 0.06.

## Where to put it

**Placement that works**

1. **MLPs:** after the activation of each wide hidden layer, `Linear → ReLU → Dropout`. Not on the output layer. Optional light dropout (0.1 to 0.2) on the inputs.
2. **CNNs:** element-wise dropout on conv feature maps does little, because neighbouring pixels are correlated (see **Spatial dropout**). Modern CNNs with batch norm use dropout only before the final classifier, e.g. EfficientNet uses 0.2 for B0 up to 0.5 for B7.
3. **Transformers:** on each sub-layer output before it is added to the residual stream, on the attention probabilities, and on the embedding sum, all at 0.1 in the original model. Many large pretraining runs now set it to 0, since each token is seen about once and overfitting is not the bottleneck.
4. **RNNs:** on the non-recurrent connections between layers (Zaremba et al., 2014), or with the same mask at every time step on the recurrent state too ("variational" dropout, Gal and Ghahramani, 2016). Dropping the hidden state with a fresh mask each step destroys memory.
5. **With batch norm:** after the last BN layer, not between a dropout and a later BN (next callout).

> **Dropout before batch norm: variance shift:** During training, dropout inflates the variance of a unit: with $\tilde h=mh/(1-p)$, $\mathbb E[\tilde h^2]=\mathbb E[h^2]/(1-p)$. A batch-norm layer after it records that inflated variance in its running statistics. At test time dropout is off, the real variance is smaller, and BN normalizes with the wrong statistics, so every later layer sees shifted inputs. Li, Chen, Hu and Yang (2019, "Understanding the disharmony between dropout and batch normalization by variance shift") traced accuracy drops to this and recommend placing dropout only after the last BN layer.

```python
import torch, torch.nn as nn, torch.nn.functional as F

net = nn.Sequential(
    nn.Linear(784, 1024), nn.ReLU(), nn.Dropout(p=0.5),   # p = DROP probability
    nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.5),
    nn.Linear(1024, 10),                                  # no dropout on the logits
)

x = torch.ones(1, 8)
drop = nn.Dropout(p=0.5)
drop.train(); print(drop(x))   # each entry is 0.0 or 2.0: survivors scaled by 1/(1-p)
drop.eval();  print(drop(x))   # identity: all ones

# From scratch (what nn.Dropout does in training mode)
def inverted_dropout(h, p):
    mask = (torch.rand_like(h) > p).to(h.dtype)
    return h * mask / (1 - p)

# Inside a custom module, tie it to the module's mode:
#   h = F.dropout(h, p=self.p, training=self.training)

# MC dropout: keep only the dropout layers stochastic at test time
@torch.no_grad()
def mc_predict(model, x, T=50):
    model.eval()                                # BN etc. in eval mode
    for m in model.modules():
        if isinstance(m, nn.Dropout):
            m.train()                           # masks back on
    probs = torch.stack([F.softmax(model(x), dim=-1) for _ in range(T)])
    model.eval()
    return probs.mean(0), probs.std(0)          # prediction, uncertainty
```

Dropout in PyTorch, inverted dropout by hand, and MC dropout

> **Forgetting model.eval():** Leaving the model in training mode at inference keeps the masks on: predictions become random and, on average, worse. Validation curves measured in training mode also look worse than they are, which is one reason a regularized model can show validation loss *below* its (masked) training loss. Call `model.eval()` for evaluation and `model.train()` before the next epoch.

> **Dropout raises training loss on purpose:** The figure shows the trade: at p = 0.5 training loss rises from 0.055 to 0.376 and training accuracy falls to 80%, while validation accuracy rises from 58% to 78%. It also shows the limit: at p = 0.7 training loss is 0.496 and the model starts to underfit. On a model that already underfits, dropout only makes things worse (see **Underfitting & overfitting**).

> **Relatives:** **Gaussian dropout** multiplies by $\mathcal N(1,\,p/(1-p))$ instead of a scaled Bernoulli: same mean and variance, no exact zeros (Srivastava et al., 2014). **AlphaDropout** keeps the mean and variance of SELU networks fixed. **DropConnect** masks weights, **Spatial dropout** masks whole channels and **Stochastic depth** masks whole residual blocks. Srivastava et al. also recommend pairing dropout with a max-norm constraint on each unit's incoming weights and a higher learning rate.

**Strengths**

- Strong, cheap regularizer for large fully connected layers and Transformers
- No change to the deployed model with inverted dropout: just call eval()
- Acts like an ensemble of $2^n$ sub-networks at the cost of one
- MC dropout gives uncertainty estimates from an ordinary trained network

**Limitations**

- Slower convergence: noisy gradients, often needing 2× more iterations
- Little effect on convolutional feature maps with element-wise masks
- Clashes with batch norm placed after it (variance shift)
- Adds a hyperparameter per layer; too large a p underfits

**Question:** A layer uses nn.Dropout(p=0.2). In training mode, a unit with activation 3.0 is kept. What value does the next layer see, and what does it see in eval mode?

- A. 3.75 in training, 3.0 in eval
- B. 3.0 in training, 2.4 in eval
- C. 2.4 in training, 3.0 in eval
- D. 3.0 in both

**Answer:** A. 3.75 in training, 3.0 in eval — PyTorch's `p` is the drop probability, so survivors are divided by $1-p=0.8$: $3.0/0.8=3.75$. In eval mode dropout is the identity, so the next layer sees 3.0, which equals the training-time expectation $0.8\cdot3.75$.

**Question:** Why is the eval-mode network only an approximation of averaging all dropout sub-networks?

- A. Because the mean of a nonlinear function of the masked activations is not the function of the mean
- B. Because inverted dropout scales by the wrong factor
- C. Because the sub-networks do not share weights
- D. It is exact for any network

**Answer:** A. Because the mean of a nonlinear function of the masked activations is not the function of the mean — Pre-activations are linear in the mask, so their mean is reproduced exactly. After a ReLU or sigmoid, Jensen's inequality breaks it: in the worked example the ensemble mean of $\mathrm{ReLU}(z-2)$ is 0.8125 while eval mode gives 0.25. MC dropout estimates the true average by sampling masks.

**Key points**

- Training: $\tilde h=m\odot h/(1-p)$ with $m_i\sim\mathrm{Bernoulli}(1-p)$; evaluation: the identity. PyTorch's `p` is the drop probability.
- Inverted dropout and the original weight-scaling rule (multiply weights by $1-p$ at test) are equivalent in expectation.
- Views: an ensemble of $2^n$ weight-sharing sub-networks, an adaptive L2 penalty $\frac{p}{1-p}\sum w_i^2x_i^2$, and approximate Bayesian inference (MC dropout).
- Typical p: 0.5 for wide FC layers, 0.1 in Transformers, 0.1 to 0.2 on inputs; not on the output layer.
- Place it after the last batch-norm layer, and always call model.eval() before evaluating.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
