Dropout

During training, zero each unit with probability p and scale the survivors by 1/(1−p); at test time use every unit. It trains an ensemble of 2ⁿ weight-sharing sub-networks and stops units from relying on specific partners.

14 min read1243 wordscore0 viewsupdated 2026-10-11

Definition

Dropout is a training-time regularizer: on every forward pass each unit of a layer is set to zero independently with probability pp, and the units that survive are scaled by 1/(1−p)1/(1-p). At evaluation the layer is the identity. Each step therefore trains a different random thinned sub-network, and all of them share one set of weights.

A large network fit to a small dataset can drive training loss to zero by memorising noise. Often it does so through co-adaptation: a unit becomes useful only in combination with a few specific other units, which together encode a quirk of the training set. Dropout breaks this. Any partner may be missing on the next step, so each unit has to learn features that are useful on their own and in many different company.

The method came from Hinton, Srivastava, Krizhevsky, Sutskever and Salakhutdinov (2012, "Improving neural networks by preventing co-adaptation of feature detectors"), with the full study in Srivastava et al. (2014, JMLR, "Dropout: a simple way to prevent neural networks from overfitting"). AlexNet (Krizhevsky, Sutskever and Hinton, 2012) used dropout with probability 0.5 in its first two fully connected layers, and noted that it roughly doubled the number of iterations needed to converge. Dropout became the standard regularizer for large fully connected layers and stayed in the Transformer (Vaswani et al., 2017, Pdrop=0.1P_{drop}=0.1).

Formula

mi∼Bernoulli(1−p),h~i=mi hi1−p(training),h~i=hi(evaluation)m_i\sim\mathrm{Bernoulli}(1-p),\qquad \tilde h_i=\frac{m_i\,h_i}{1-p}\quad\text{(training)},\qquad \tilde h_i=h_i\quad\text{(evaluation)}
hih_i
activation of unit i (usually after the nonlinearity)
mim_i
keep-mask entry: 1 with probability 1 − p, 0 with probability p. A fresh mask for every example and every step
pp
drop probability (PyTorch's p). Srivastava et al. write pp for the retain probability, so their p=0.8p=0.8 is PyTorch's p=0.2
1/(1−p)1/(1-p)
inverted-dropout scale: it makes E[h~i]=hi\mathbb E[\tilde h_i]=h_i, so nothing changes at test time

The backward pass uses the same mask: ∂h~i/∂hi=mi/(1−p)\partial\tilde h_i/\partial h_i=m_i/(1-p). A dropped unit passes no gradient on that step, and a kept one passes its error scaled by 1/(1−p)1/(1-p). In expectation the layer is unchanged, E[h~i]=hi\mathbb E[\tilde h_i]=h_i, but it now injects multiplicative noise with variance Var⁡(h~i)=p1−phi2\operatorname{Var}(\tilde h_i)=\frac{p}{1-p}h_i^2, which is 1 times hi2h_i^2 at p=0.5p=0.5 and 0.11 hi20.11\,h_i^2 at p=0.1p=0.1.

p = 0.4 · step 0
fig.A fresh random mask every training step (p = 0.4): crossed-out units and their connections are absent for that step. The next step trains a different thinned network with the same weights.

At a glance

Acts on
activations (units), not weights. Masking weights is DropConnect
Parameters
none learned; one hyperparameter pp per dropout layer
Typical p
0.5 for wide fully connected hidden layers, 0.1 to 0.2 on inputs, 0.1 in Transformers, 0.2 to 0.5 before a CNN's classifier
Train vs test
random masks in model.train(), identity in model.eval()
Cost
one random draw and a multiply per unit; but training typically needs more steps to converge
Interpretations
an ensemble of 2n2^n sub-networks; adaptive L2 penalty; approximate Bayesian inference (MC dropout)

Inverted dropout and the weight-scaling rule

The 2012 and 2014 papers did not rescale during training. They dropped units with retention probability q=1−pq=1-p and, at test time, multiplied each layer's outgoing weights by qq, so that a unit's expected input matched what it had seen during training. This is the weight-scaling inference rule. Modern frameworks do the equivalent thing on the training side: divide the survivors by qq at training time, which is inverted dropout, and leave the network untouched at test time. The two are identical in expectation; inverted dropout is preferred because the deployed model needs no special code.

Em[∑iwi mihi1−p]⏟train, inverted=∑iwihi⟺Em[∑iwi mihi]⏟train, original=∑i(1−p) wi hi\underbrace{\mathbb E_m\Big[\sum_i w_i\,\frac{m_i h_i}{1-p}\Big]}_{\text{train, inverted}}=\sum_i w_i h_i\qquad\Longleftrightarrow\qquad \underbrace{\mathbb E_m\Big[\sum_i w_i\,m_i h_i\Big]}_{\text{train, original}}=\sum_i (1-p)\,w_i\,h_i
wiw_i
outgoing weight from unit i to one unit of the next layer
(1−p) wi(1-p)\,w_i
the original paper's test-time weights: scaled by the retain probability

Why it regularizes: three views

An ensemble of 2n2^n networks. A layer with nn droppable units has 2n2^n possible masks, so dropout samples one of 2n2^n thinned networks per example and takes one gradient step on it. Almost none of them is ever trained twice, but they share weights, so every step improves all sub-networks that contain the touched units. Ideally the test-time prediction would average all 2n2^n of them. The weight-scaled (or, with inverted dropout, the plain) network is a one-pass approximation of that average. For a model whose only nonlinearity is the final softmax it is exact: it computes the normalized geometric mean of all sub-model predictions (Hinton et al., 2012; Goodfellow, Bengio and Courville, 2016, §7.12). With hidden nonlinearities it is an approximation that works well in practice.

An adaptive L2 penalty. For a linear model with dropout on the inputs, averaging the squared error over the mask gives the clean loss plus a ridge term weighted by each feature's magnitude (Srivastava et al., 2014; Wager, Wang and Liang, 2013):

Em[(y−∑iwi mixi1−p)2]=(y−w⊤x)2+p1−p∑iwi2 xi2\mathbb E_m\Big[\Big(y-\sum_i w_i\,\frac{m_i x_i}{1-p}\Big)^2\Big]=\big(y-w^\top x\big)^2+\frac{p}{1-p}\sum_i w_i^2\,x_i^2
p1−p\frac{p}{1-p}
the penalty strength: 1 at p=0.5p=0.5, 0.250.25 at p=0.2p=0.2
wi2xi2w_i^2x_i^2
weights on large, frequently active features are penalized more: an L2 penalty that adapts to the data scale

Approximate Bayesian inference. Gal and Ghahramani (2016, "Dropout as a Bayesian approximation") showed that training with dropout before every weight layer approximately minimizes the KL divergence to the posterior of a deep Gaussian process. The consequence is MC dropout: keep dropout on at test time, run TT stochastic forward passes, and use their mean as the prediction and their spread as an uncertainty estimate.

p^(y∣x)≈1T∑t=1TfW~t(x),Var⁡^(y∣x)≈1T∑t=1TfW~t(x)2−p^(y∣x)2\hat p(y\mid x)\approx\frac1T\sum_{t=1}^{T}f_{\tilde W_t}(x),\qquad \widehat{\operatorname{Var}}(y\mid x)\approx\frac1T\sum_{t=1}^{T}f_{\tilde W_t}(x)^2-\hat p(y\mid x)^2
W~t\tilde W_t
the network with dropout mask t applied
TT
number of stochastic passes, commonly 10 to 100
eval mode: masks off, P(class 1)30 training points, 7 with flipped labelsmasks for example 1 at step 800 (p = 0.50)h¹h²kept 10/20 · 7/20 · survivors ×2.000.010.1110step 800BCE (log), eval modetrain loss 0.376 · acc 80%val loss 0.650 · acc 78%p = 0, same step: val 7.842 · acc 58%● class 0 ● class 1 ━ train ━ val ┄ same, p = 0 ● kept ○ dropped
view
fig.A 2→20→20→1 ReLU network trained by full-batch Adam (800 steps, lr 0.01) on 30 points, 7 of them with flipped labels; validation is 300 fresh points from the same distribution. The dashed curves are the same network without dropout: it reaches 100% training accuracy by carving islands around every flipped label, and its validation loss climbs from 0.578 at step 25 to 7.842 at step 800 (58% accuracy). At the default p = 0.5 the eval-mode training loss stays at 0.376 (80% accuracy) and validation ends at 0.650 with 78% accuracy. Small p helps less (p = 0.2: validation 0.922, 69%). The row of dots shows the masks for one example at that step: about half the units are gone and the rest are scaled ×2. Switch to MC σ for MC dropout's uncertainty (30 passes): it ranges from about 0.01 in the dense core of each class to 0.41, highest in the top-right and bottom-left corners, which hold almost no training data and where the sub-networks disagree about how to extrapolate.

Worked example: one unit, all 16 masks

Four inputs h=(2, 0.5, 1, 3)h=(2,\,0.5,\,1,\,3) feed one unit with weights w=(0.5, −1, 1, 0.25)w=(0.5,\,-1,\,1,\,0.25), and dropout p=0.5p=0.5 is applied to hh. The eval-mode pre-activation is z=w⊤h=1−0.5+1+0.75=2.25z=w^\top h=1-0.5+1+0.75=2.25. In training, mask m=(1,0,0,1)m=(1,0,0,1) gives h~=(4, 0, 0, 6)\tilde h=(4,\,0,\,0,\,6) and z=2+1.5=3.5z=2+1.5=3.5. Enumerating all 16 masks (checked in Python):

quantityover the 16 maskseval mode (no mask)comment
z=w⊤h~z=w^\top\tilde hmean 2.252.252.252.25exact: z is linear in the mask
Var⁡(z)\operatorname{Var}(z)2.81252.812500=p1−p∑iwi2hi2=1+0.25+1+0.5625=\frac{p}{1-p}\sum_i w_i^2h_i^2=1+0.25+1+0.5625
range of zz−1.0-1.0 to 5.55.52.252.25the noise dropout injects
ReLU(z−2)\mathrm{ReLU}(z-2)mean 0.81250.81250.250.25not exact after a nonlinearity
Statistics over all 2⁴ = 16 equally likely masks, p = 0.5

The last row is why the weight-scaling rule is called an approximation. Before the nonlinearity, eval mode reproduces the ensemble mean exactly. After it, the ensemble average E[ReLU(z−2)]=0.8125\mathbb E[\mathrm{ReLU}(z-2)]=0.8125 differs from ReLU(E[z]−2)=0.25\mathrm{ReLU}(\mathbb E[z]-2)=0.25 (Jensen's inequality). MC dropout would estimate the 0.8125. In the figure, the average gap between eval-mode and MC-averaged probabilities at p = 0.5 is about 0.06.

Where to put it

Placement that works

  1. 1MLPs: after the activation of each wide hidden layer, Linear → ReLU → Dropout. Not on the output layer. Optional light dropout (0.1 to 0.2) on the inputs.
  2. 2CNNs: element-wise dropout on conv feature maps does little, because neighbouring pixels are correlated (see Spatial dropout). Modern CNNs with batch norm use dropout only before the final classifier, e.g. EfficientNet uses 0.2 for B0 up to 0.5 for B7.
  3. 3Transformers: on each sub-layer output before it is added to the residual stream, on the attention probabilities, and on the embedding sum, all at 0.1 in the original model. Many large pretraining runs now set it to 0, since each token is seen about once and overfitting is not the bottleneck.
  4. 4RNNs: on the non-recurrent connections between layers (Zaremba et al., 2014), or with the same mask at every time step on the recurrent state too ("variational" dropout, Gal and Ghahramani, 2016). Dropping the hidden state with a fresh mask each step destroys memory.
  5. 5With batch norm: after the last BN layer, not between a dropout and a later BN (next callout).
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3net = nn.Sequential(
4 nn.Linear(784, 1024), nn.ReLU(), nn.Dropout(p=0.5), # p = DROP probability
5 nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.5),
6 nn.Linear(1024, 10), # no dropout on the logits
7)
8
9x = torch.ones(1, 8)
10drop = nn.Dropout(p=0.5)
11drop.train(); print(drop(x)) # each entry is 0.0 or 2.0: survivors scaled by 1/(1-p)
12drop.eval(); print(drop(x)) # identity: all ones
13
14# From scratch (what nn.Dropout does in training mode)
15def inverted_dropout(h, p):
16 mask = (torch.rand_like(h) > p).to(h.dtype)
17 return h * mask / (1 - p)
18
19# Inside a custom module, tie it to the module's mode:
20# h = F.dropout(h, p=self.p, training=self.training)
21
22# MC dropout: keep only the dropout layers stochastic at test time
23@torch.no_grad()
24def mc_predict(model, x, T=50):
25 model.eval() # BN etc. in eval mode
26 for m in model.modules():
27 if isinstance(m, nn.Dropout):
28 m.train() # masks back on
29 probs = torch.stack([F.softmax(model(x), dim=-1) for _ in range(T)])
30 model.eval()
31 return probs.mean(0), probs.std(0) # prediction, uncertainty
Dropout in PyTorch, inverted dropout by hand, and MC dropout

Use when

  • +Strong, cheap regularizer for large fully connected layers and Transformers
  • +No change to the deployed model with inverted dropout: just call eval()
  • +Acts like an ensemble of 2n2^n sub-networks at the cost of one
  • +MC dropout gives uncertainty estimates from an ordinary trained network

Avoid when

  • −Slower convergence: noisy gradients, often needing 2× more iterations
  • −Little effect on convolutional feature maps with element-wise masks
  • −Clashes with batch norm placed after it (variance shift)
  • −Adds a hyperparameter per layer; too large a p underfits

Check yourself

A layer uses nn.Dropout(p=0.2). In training mode, a unit with activation 3.0 is kept. What value does the next layer see, and what does it see in eval mode?

Check yourself

Why is the eval-mode network only an approximation of averaging all dropout sub-networks?

Remember

  • ✓Training: h~=m⊙h/(1−p)\tilde h=m\odot h/(1-p) with mi∼Bernoulli(1−p)m_i\sim\mathrm{Bernoulli}(1-p); evaluation: the identity. PyTorch's p is the drop probability.
  • ✓Inverted dropout and the original weight-scaling rule (multiply weights by 1−p1-p at test) are equivalent in expectation.
  • ✓Views: an ensemble of 2n2^n weight-sharing sub-networks, an adaptive L2 penalty p1−p∑wi2xi2\frac{p}{1-p}\sum w_i^2x_i^2, and approximate Bayesian inference (MC dropout).
  • ✓Typical p: 0.5 for wide FC layers, 0.1 in Transformers, 0.1 to 0.2 on inputs; not on the output layer.
  • ✓Place it after the last batch-norm layer, and always call model.eval() before evaluating.
← back to Deep Learning roadmap