Definition
Dropout is a training-time regularizer: on every forward pass each unit of a layer is set to zero independently with probability , and the units that survive are scaled by . At evaluation the layer is the identity. Each step therefore trains a different random thinned sub-network, and all of them share one set of weights.
A large network fit to a small dataset can drive training loss to zero by memorising noise. Often it does so through co-adaptation: a unit becomes useful only in combination with a few specific other units, which together encode a quirk of the training set. Dropout breaks this. Any partner may be missing on the next step, so each unit has to learn features that are useful on their own and in many different company.
The method came from Hinton, Srivastava, Krizhevsky, Sutskever and Salakhutdinov (2012, "Improving neural networks by preventing co-adaptation of feature detectors"), with the full study in Srivastava et al. (2014, JMLR, "Dropout: a simple way to prevent neural networks from overfitting"). AlexNet (Krizhevsky, Sutskever and Hinton, 2012) used dropout with probability 0.5 in its first two fully connected layers, and noted that it roughly doubled the number of iterations needed to converge. Dropout became the standard regularizer for large fully connected layers and stayed in the Transformer (Vaswani et al., 2017, ).
Formula
- activation of unit i (usually after the nonlinearity)
- keep-mask entry: 1 with probability 1 − p, 0 with probability p. A fresh mask for every example and every step
- drop probability (PyTorch's
p). Srivastava et al. write for the retain probability, so their is PyTorch'sp=0.2 - inverted-dropout scale: it makes , so nothing changes at test time
The backward pass uses the same mask: . A dropped unit passes no gradient on that step, and a kept one passes its error scaled by . In expectation the layer is unchanged, , but it now injects multiplicative noise with variance , which is 1 times at and at .
At a glance
- Acts on
- activations (units), not weights. Masking weights is DropConnect
- Parameters
- none learned; one hyperparameter per dropout layer
- Typical p
- 0.5 for wide fully connected hidden layers, 0.1 to 0.2 on inputs, 0.1 in Transformers, 0.2 to 0.5 before a CNN's classifier
- Train vs test
- random masks in
model.train(), identity inmodel.eval() - Cost
- one random draw and a multiply per unit; but training typically needs more steps to converge
- Interpretations
- an ensemble of sub-networks; adaptive L2 penalty; approximate Bayesian inference (MC dropout)
Inverted dropout and the weight-scaling rule
The 2012 and 2014 papers did not rescale during training. They dropped units with retention probability and, at test time, multiplied each layer's outgoing weights by , so that a unit's expected input matched what it had seen during training. This is the weight-scaling inference rule. Modern frameworks do the equivalent thing on the training side: divide the survivors by at training time, which is inverted dropout, and leave the network untouched at test time. The two are identical in expectation; inverted dropout is preferred because the deployed model needs no special code.
- outgoing weight from unit i to one unit of the next layer
- the original paper's test-time weights: scaled by the retain probability
Why it regularizes: three views
An ensemble of networks. A layer with droppable units has possible masks, so dropout samples one of thinned networks per example and takes one gradient step on it. Almost none of them is ever trained twice, but they share weights, so every step improves all sub-networks that contain the touched units. Ideally the test-time prediction would average all of them. The weight-scaled (or, with inverted dropout, the plain) network is a one-pass approximation of that average. For a model whose only nonlinearity is the final softmax it is exact: it computes the normalized geometric mean of all sub-model predictions (Hinton et al., 2012; Goodfellow, Bengio and Courville, 2016, §7.12). With hidden nonlinearities it is an approximation that works well in practice.
An adaptive L2 penalty. For a linear model with dropout on the inputs, averaging the squared error over the mask gives the clean loss plus a ridge term weighted by each feature's magnitude (Srivastava et al., 2014; Wager, Wang and Liang, 2013):
- the penalty strength: 1 at , at
- weights on large, frequently active features are penalized more: an L2 penalty that adapts to the data scale
Approximate Bayesian inference. Gal and Ghahramani (2016, "Dropout as a Bayesian approximation") showed that training with dropout before every weight layer approximately minimizes the KL divergence to the posterior of a deep Gaussian process. The consequence is MC dropout: keep dropout on at test time, run stochastic forward passes, and use their mean as the prediction and their spread as an uncertainty estimate.
- the network with dropout mask t applied
- number of stochastic passes, commonly 10 to 100
Worked example: one unit, all 16 masks
Four inputs feed one unit with weights , and dropout is applied to . The eval-mode pre-activation is . In training, mask gives and . Enumerating all 16 masks (checked in Python):
| quantity | over the 16 masks | eval mode (no mask) | comment |
|---|---|---|---|
| mean | exact: z is linear in the mask | ||
| range of | to | the noise dropout injects | |
| mean | not exact after a nonlinearity |
The last row is why the weight-scaling rule is called an approximation. Before the nonlinearity, eval mode reproduces the ensemble mean exactly. After it, the ensemble average differs from (Jensen's inequality). MC dropout would estimate the 0.8125. In the figure, the average gap between eval-mode and MC-averaged probabilities at p = 0.5 is about 0.06.
Where to put it
Placement that works
- 1MLPs: after the activation of each wide hidden layer,
Linear → ReLU → Dropout. Not on the output layer. Optional light dropout (0.1 to 0.2) on the inputs. - 2CNNs: element-wise dropout on conv feature maps does little, because neighbouring pixels are correlated (see Spatial dropout). Modern CNNs with batch norm use dropout only before the final classifier, e.g. EfficientNet uses 0.2 for B0 up to 0.5 for B7.
- 3Transformers: on each sub-layer output before it is added to the residual stream, on the attention probabilities, and on the embedding sum, all at 0.1 in the original model. Many large pretraining runs now set it to 0, since each token is seen about once and overfitting is not the bottleneck.
- 4RNNs: on the non-recurrent connections between layers (Zaremba et al., 2014), or with the same mask at every time step on the recurrent state too ("variational" dropout, Gal and Ghahramani, 2016). Dropping the hidden state with a fresh mask each step destroys memory.
- 5With batch norm: after the last BN layer, not between a dropout and a later BN (next callout).
1import torch, torch.nn as nn, torch.nn.functional as F23net = nn.Sequential(4 nn.Linear(784, 1024), nn.ReLU(), nn.Dropout(p=0.5), # p = DROP probability5 nn.Linear(1024, 1024), nn.ReLU(), nn.Dropout(p=0.5),6 nn.Linear(1024, 10), # no dropout on the logits7)89x = torch.ones(1, 8)10drop = nn.Dropout(p=0.5)11drop.train(); print(drop(x)) # each entry is 0.0 or 2.0: survivors scaled by 1/(1-p)12drop.eval(); print(drop(x)) # identity: all ones1314# From scratch (what nn.Dropout does in training mode)15def inverted_dropout(h, p):16 mask = (torch.rand_like(h) > p).to(h.dtype)17 return h * mask / (1 - p)1819# Inside a custom module, tie it to the module's mode:20# h = F.dropout(h, p=self.p, training=self.training)2122# MC dropout: keep only the dropout layers stochastic at test time23@torch.no_grad()24def mc_predict(model, x, T=50):25 model.eval() # BN etc. in eval mode26 for m in model.modules():27 if isinstance(m, nn.Dropout):28 m.train() # masks back on29 probs = torch.stack([F.softmax(model(x), dim=-1) for _ in range(T)])30 model.eval()31 return probs.mean(0), probs.std(0) # prediction, uncertaintyUse when
- +Strong, cheap regularizer for large fully connected layers and Transformers
- +No change to the deployed model with inverted dropout: just call eval()
- +Acts like an ensemble of sub-networks at the cost of one
- +MC dropout gives uncertainty estimates from an ordinary trained network
Avoid when
- −Slower convergence: noisy gradients, often needing 2× more iterations
- −Little effect on convolutional feature maps with element-wise masks
- −Clashes with batch norm placed after it (variance shift)
- −Adds a hyperparameter per layer; too large a p underfits
Check yourself
A layer uses nn.Dropout(p=0.2). In training mode, a unit with activation 3.0 is kept. What value does the next layer see, and what does it see in eval mode?
Check yourself
Why is the eval-mode network only an approximation of averaging all dropout sub-networks?
Remember
- ✓Training: with ; evaluation: the identity. PyTorch's
pis the drop probability. - ✓Inverted dropout and the original weight-scaling rule (multiply weights by at test) are equivalent in expectation.
- ✓Views: an ensemble of weight-sharing sub-networks, an adaptive L2 penalty , and approximate Bayesian inference (MC dropout).
- ✓Typical p: 0.5 for wide FC layers, 0.1 in Transformers, 0.1 to 0.2 on inputs; not on the output layer.
- ✓Place it after the last batch-norm layer, and always call model.eval() before evaluating.