Definition
Spatial dropout (channel-wise dropout) zeroes entire channels of a convolutional feature map: for each example and channel one Bernoulli variable decides whether the whole map is kept (and scaled by ) or set to zero. In PyTorch this is nn.Dropout2d for images, nn.Dropout1d for sequences and nn.Dropout3d for volumes.
Standard Dropout works well on fully connected layers, where every unit carries its own piece of information. Convolutional feature maps are different: a feature that fires at pixel almost always fires at too, because the receptive fields overlap and images are smooth. Zero one activation and its neighbours still carry nearly the same value, so the next convolution, which looks at a window anyway, barely notices. The noise is there but the information is not removed, so the regularization effect is weak.
Tompson, Goroshin, Jain, LeCun and Bregler (2015, "Efficient object localization using convolutional networks", CVPR) hit exactly this in a fully convolutional human-pose network. They observed that when adjacent pixels are strongly correlated, i.i.d. dropout does not regularize the activations and mainly acts like a lower learning rate. Their fix, SpatialDropout, extends one dropout value across the whole feature map, so either all of a channel is present or none of it is. Keras exposes it as SpatialDropout1D/2D/3D; PyTorch calls it Dropout1d/2d/3d.
Formula
- activation of example , channel , position
- one keep-mask entry per example and channel, shared by every spatial position
- probability of dropping a channel (PyTorch's p)
- inverted-dropout scale, so and evaluation is the identity
Only the shape of the mask changes. Element-wise dropout draws Bernoulli variables per batch; spatial dropout draws and broadcasts each over . In the backward pass , so a dropped channel sends no gradient back for that example: the filter that produced it gets no learning signal from that example. The next layer has to produce its output without that channel, which discourages relying on any single filter.
At a glance
- Mask shape
- broadcast over
- PyTorch
nn.Dropout1don ,nn.Dropout2don ,nn.Dropout3don- Typical p
- 0.1 to 0.3 (a whole channel is a lot of information)
- Channels dropped
- Binomial per example: for , , on average 6.4 channels with standard deviation 2.4
- Spatial cousin
- DropBlock zeroes contiguous squares; is element dropout, feature size is spatial dropout
- Introduced
- Tompson et al., CVPR 2015 (SpatialDropout)
Why correlation defeats element-wise dropout
Suppose neighbouring activations have correlation . If is dropped but is kept, the best linear guess of the missing value from that one neighbour is , and the fraction of its variance that guess explains is . With eight neighbours and near 0.9, almost all of it is recoverable, which is what the figure's R² of 0.93 measures. A convolution in the next layer sums over that window, so its output barely changes when one input pixel disappears. Spatial dropout makes the removed information unrecoverable from the same channel: the only way to compensate is through other channels, which forces redundant, independent features across filters.
| scheme | smooth maps (ℓ = 2): R² | dropped with a kept neighbour | white noise (ℓ = 0): R² |
|---|---|---|---|
element (nn.Dropout) | 0.93 | 99.9% | −0.21 |
| DropBlock 3×3 | 0.64 | 78.2% | −0.28 |
| DropBlock 5×5 | 0.41 | 51.7% | −0.18 |
channel (nn.Dropout2d) | 0.00 | 0% | 0.00 |
Negative R² on white noise means the neighbour-mean guess is worse than guessing 0: there is nothing to recover, and plain dropout already removes the full information. The decisive quantity is therefore the spatial correlation of the feature maps, which is high in most convolutional layers.
Worked example
nn.Dropout2d on one example with 3 channels of 2×2, p = 0.5
- 1Input channels: , , .
- 2Draw one keep bit per channel, say . Three random numbers, not twelve.
- 3Scale the kept channels by : , , .
- 4Backward: the gradient into and is the incoming gradient, the gradient into is zero at every position, so the filter that produced channel 1 learns nothing from this example on this step.
- 5Expected output: each channel is kept with probability 0.5 and doubled, so . In
eval()mode .
DropBlock: between the two extremes
Ghiasi, Lin and Le (2018, "DropBlock: a form of structured dropout", NeurIPS) drop contiguous square regions of each feature map. Block centres are sampled at a rate chosen so that roughly a fraction of activations ends up zeroed, then the survivors are rescaled by (number of activations)/(number kept). With block size 7 on ResNet-50 they report ImageNet top-1 rising from 76.51% to 78.13%, and they found it works best when the drop rate is ramped up linearly during training.
- block size; recovers element dropout
- feature-map side length
- number of valid block centres (blocks must fit inside the map)
The formula ignores overlaps between blocks, so the realized drop rate is a little below : in the figure (, ) the average is 22.2% for () and 21.8% for (). The same idea applied to the input image is Cutout (DeVries and Taylor, 2017), a data-augmentation method.
1import torch, torch.nn as nn2from torchvision.ops import DropBlock2d34x = torch.ones(2, 4, 3, 3) # (N, C, H, W)5sd = nn.Dropout2d(p=0.5).train()6y = sd(x)7print([y[0, c].unique().tolist() for c in range(4)])8# e.g. [[2.0], [0.0], [0.0], [2.0]]: each channel is all 0 or all 1/(1-p) = 2910# Sequences: (N, C, L), e.g. a 1-D CNN or embeddings transposed to (N, D, T)11sd1 = nn.Dropout1d(p=0.2) # drops whole embedding dims for all time steps12seq = sd1(torch.randn(8, 128, 50))13# Passing (N, C, L) to Dropout2d is ambiguous (it could mean an unbatched (C, H, W)):14# PyTorch warns; use Dropout1d for 3-D sequence input.1516# From scratch17def spatial_dropout(x, p, training=True):18 if not training or p == 0:19 return x20 mask = (torch.rand(x.shape[0], x.shape[1], *([1] * (x.dim() - 2)),21 device=x.device) > p).to(x.dtype)22 return x * mask / (1 - p) # mask broadcasts over H, W2324block = nn.Sequential(25 nn.Conv2d(64, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),26 nn.Dropout2d(p=0.1),27)28db = DropBlock2d(p=0.1, block_size=7) # torchvision's DropBlockUse when
- +Actually removes information from correlated feature maps, unlike element-wise dropout
- +Cheap: random numbers instead of
- +Encourages redundancy across filters; no change at evaluation
- +Works for 1-D sequences, images and volumes
Avoid when
- −Coarse: removing a whole channel is a strong perturbation, so only small p work
- −Few channels (early layers) make the per-example drop rate very noisy
- −Interacts with batch-norm statistics like ordinary dropout
- −DropBlock often regularizes CNNs better, at the cost of a block-size hyperparameter
Check yourself
A conv layer outputs (N, C, H, W) = (32, 64, 28, 28). How many Bernoulli draws do nn.Dropout and nn.Dropout2d make per forward pass?
Check yourself
Why does element-wise dropout regularize smooth convolutional feature maps so weakly?
Remember
- ✓Spatial dropout draws one keep bit per (example, channel) and broadcasts it over all positions: .
- ✓Element-wise dropout fails on conv maps because neighbouring activations are highly correlated; a dropped pixel is recoverable from its neighbours.
- ✓PyTorch:
nn.Dropout1dfor ,nn.Dropout2dfor ,nn.Dropout3dfor volumes; identity ineval(). - ✓Use small rates (0.1 to 0.3); a whole channel is a lot of signal.
- ✓DropBlock (contiguous b×b squares) sits between element and channel dropout and often works better in deep CNNs.