Spatial dropout

Drop whole feature-map channels instead of single activations. Neighbouring pixels are so correlated that element-wise dropout removes almost no information; zeroing a channel everywhere does, so later layers cannot lean on any one channel.

10 min read926 wordscore0 viewsupdated 2026-10-11

Definition

Spatial dropout (channel-wise dropout) zeroes entire channels of a convolutional feature map: for each example nn and channel cc one Bernoulli variable decides whether the whole map x[n,c,⋅,⋅]x[n,c,\cdot,\cdot] is kept (and scaled by 1/(1−p)1/(1-p)) or set to zero. In PyTorch this is nn.Dropout2d for images, nn.Dropout1d for sequences and nn.Dropout3d for volumes.

Standard Dropout works well on fully connected layers, where every unit carries its own piece of information. Convolutional feature maps are different: a feature that fires at pixel (i,j)(i,j) almost always fires at (i,j+1)(i,j+1) too, because the receptive fields overlap and images are smooth. Zero one activation and its neighbours still carry nearly the same value, so the next convolution, which looks at a 3×33\times3 window anyway, barely notices. The noise is there but the information is not removed, so the regularization effect is weak.

Tompson, Goroshin, Jain, LeCun and Bregler (2015, "Efficient object localization using convolutional networks", CVPR) hit exactly this in a fully convolutional human-pose network. They observed that when adjacent pixels are strongly correlated, i.i.d. dropout does not regularize the activations and mainly acts like a lower learning rate. Their fix, SpatialDropout, extends one dropout value across the whole feature map, so either all of a channel is present or none of it is. Keras exposes it as SpatialDropout1D/2D/3D; PyTorch calls it Dropout1d/2d/3d.

Formula

mn,c∼Bernoulli(1−p),y[n,c,i,j]=mn,c1−p  x[n,c,i,j]for all i,jm_{n,c}\sim\mathrm{Bernoulli}(1-p),\qquad y[n,c,i,j]=\frac{m_{n,c}}{1-p}\;x[n,c,i,j]\quad\text{for all } i,j
x[n,c,i,j]x[n,c,i,j]
activation of example nn, channel cc, position (i,j)(i,j)
mn,cm_{n,c}
one keep-mask entry per example and channel, shared by every spatial position
pp
probability of dropping a channel (PyTorch's p)
1/(1−p)1/(1-p)
inverted-dropout scale, so E[y]=x\mathbb E[y]=x and evaluation is the identity

Only the shape of the mask changes. Element-wise dropout draws N⋅C⋅H⋅WN\cdot C\cdot H\cdot W Bernoulli variables per batch; spatial dropout draws N⋅CN\cdot C and broadcasts each over H×WH\times W. In the backward pass ∂y/∂x=mn,c/(1−p)\partial y/\partial x=m_{n,c}/(1-p), so a dropped channel sends no gradient back for that example: the filter that produced it gets no learning signal from that example. The next layer has to produce its output without that channel, which discourages relying on any single filter.

At a glance

Mask shape
(N,C,1,1)(N, C, 1, 1) broadcast over (N,C,H,W)(N, C, H, W)
PyTorch
nn.Dropout1d on (N,C,L)(N,C,L), nn.Dropout2d on (N,C,H,W)(N,C,H,W), nn.Dropout3d on (N,C,D,H,W)(N,C,D,H,W)
Typical p
0.1 to 0.3 (a whole channel is a lot of information)
Channels dropped
Binomial(C,p)(C,p) per example: for C=64C=64, p=0.1p=0.1, on average 6.4 channels with standard deviation 2.4
Spatial cousin
DropBlock zeroes contiguous b×bb\times b squares; b=1b=1 is element dropout, b=b= feature size is spatial dropout
Introduced
Tompson et al., CVPR 2015 (SpatialDropout)
feature maps x[c, i, j]: 6 channels of 12×12, correlation length ℓ = 2.00after nn.Dropout (p = 0.25): what the next layer sees, survivors ×1.33dropped this draw25.8%dropped, mean of 100 draws24.9%dropped with a kept neighbour100.0%neighbour correlation ρ0.89R² of guessing a dropped value fromthe mean of its kept neighbours0.94this draw0.93100 drawshigh R² = the drop removed little information■ positive ■ negative activation □ dropped (zero)
fig.Six 12×12 feature maps made by blurring white noise with a Gaussian of width ℓ, so neighbouring activations are correlated (ρ = 0.89 at the default ℓ = 2). Each dropped activation is guessed as the mean of its kept 8-neighbours in the same channel, and the bar is the R² of that guess: how much of the dropped signal the next layer can still recover. At p = 0.25, element-wise dropout leaves a kept neighbour for 99.9% of dropped activations and R² = 0.93 (average of 100 masks): almost nothing is lost. DropBlock gives 0.64 with 3×3 blocks and 0.41 with 5×5. Channel dropout gives exactly 0: the whole map is gone. Slide ℓ to 0 (white noise, ρ = −0.04): now element dropout's R² is −0.21, neighbours carry no information, and element-wise dropout is fine.

Why correlation defeats element-wise dropout

Suppose neighbouring activations have correlation ρ\rho. If xijx_{ij} is dropped but xi,j+1x_{i,j+1} is kept, the best linear guess of the missing value from that one neighbour is ρ xi,j+1\rho\,x_{i,j+1}, and the fraction of its variance that guess explains is ρ2\rho^2. With eight neighbours and ρ\rho near 0.9, almost all of it is recoverable, which is what the figure's R² of 0.93 measures. A 3×33\times3 convolution in the next layer sums over that window, so its output barely changes when one input pixel disappears. Spatial dropout makes the removed information unrecoverable from the same channel: the only way to compensate is through other channels, which forces redundant, independent features across filters.

schemesmooth maps (ℓ = 2): R²dropped with a kept neighbourwhite noise (ℓ = 0): R²
element (nn.Dropout)0.9399.9%−0.21
DropBlock 3×30.6478.2%−0.28
DropBlock 5×50.4151.7%−0.18
channel (nn.Dropout2d)0.000%0.00
The figure's numbers at p = 0.25, averaged over 100 masks (computed in Node from the figure's code)

Negative R² on white noise means the neighbour-mean guess is worse than guessing 0: there is nothing to recover, and plain dropout already removes the full information. The decisive quantity is therefore the spatial correlation of the feature maps, which is high in most convolutional layers.

Worked example

nn.Dropout2d on one example with 3 channels of 2×2, p = 0.5

  1. 1Input channels: x0=[1234]x_0=\begin{bmatrix}1&2\\3&4\end{bmatrix}, x1=[0.50.50.50.5]x_1=\begin{bmatrix}0.5&0.5\\0.5&0.5\end{bmatrix}, x2=[−1012]x_2=\begin{bmatrix}-1&0\\1&2\end{bmatrix}.
  2. 2Draw one keep bit per channel, say m=(1,0,1)m=(1,0,1). Three random numbers, not twelve.
  3. 3Scale the kept channels by 1/(1−p)=21/(1-p)=2: y0=[2468]y_0=\begin{bmatrix}2&4\\6&8\end{bmatrix}, y1=0y_1=\mathbf 0, y2=[−2024]y_2=\begin{bmatrix}-2&0\\2&4\end{bmatrix}.
  4. 4Backward: the gradient into x0x_0 and x2x_2 is 2×2\times the incoming gradient, the gradient into x1x_1 is zero at every position, so the filter that produced channel 1 learns nothing from this example on this step.
  5. 5Expected output: each channel is kept with probability 0.5 and doubled, so E[yc]=xc\mathbb E[y_c]=x_c. In eval() mode y=xy=x.

DropBlock: between the two extremes

Ghiasi, Lin and Le (2018, "DropBlock: a form of structured dropout", NeurIPS) drop contiguous square regions of each feature map. Block centres are sampled at a rate γ\gamma chosen so that roughly a fraction pp of activations ends up zeroed, then the survivors are rescaled by (number of activations)/(number kept). With block size 7 on ResNet-50 they report ImageNet top-1 rising from 76.51% to 78.13%, and they found it works best when the drop rate is ramped up linearly during training.

γ=pb2⋅S2(S−b+1)2\gamma=\frac{p}{b^2}\cdot\frac{S^2}{(S-b+1)^2}
bb
block size; b=1b=1 recovers element dropout
SS
feature-map side length
(S−b+1)2(S-b+1)^2
number of valid block centres (blocks must fit inside the map)

The formula ignores overlaps between blocks, so the realized drop rate is a little below pp: in the figure (S=12S=12, p=0.25p=0.25) the average is 22.2% for b=3b=3 (γ=0.040\gamma=0.040) and 21.8% for b=5b=5 (γ=0.0225\gamma=0.0225). The same idea applied to the input image is Cutout (DeVries and Taylor, 2017), a data-augmentation method.

python
1import torch, torch.nn as nn
2from torchvision.ops import DropBlock2d
3
4x = torch.ones(2, 4, 3, 3) # (N, C, H, W)
5sd = nn.Dropout2d(p=0.5).train()
6y = sd(x)
7print([y[0, c].unique().tolist() for c in range(4)])
8# e.g. [[2.0], [0.0], [0.0], [2.0]]: each channel is all 0 or all 1/(1-p) = 2
9
10# Sequences: (N, C, L), e.g. a 1-D CNN or embeddings transposed to (N, D, T)
11sd1 = nn.Dropout1d(p=0.2) # drops whole embedding dims for all time steps
12seq = sd1(torch.randn(8, 128, 50))
13# Passing (N, C, L) to Dropout2d is ambiguous (it could mean an unbatched (C, H, W)):
14# PyTorch warns; use Dropout1d for 3-D sequence input.
15
16# From scratch
17def spatial_dropout(x, p, training=True):
18 if not training or p == 0:
19 return x
20 mask = (torch.rand(x.shape[0], x.shape[1], *([1] * (x.dim() - 2)),
21 device=x.device) > p).to(x.dtype)
22 return x * mask / (1 - p) # mask broadcasts over H, W
23
24block = nn.Sequential(
25 nn.Conv2d(64, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),
26 nn.Dropout2d(p=0.1),
27)
28db = DropBlock2d(p=0.1, block_size=7) # torchvision's DropBlock
Channel-wise dropout in PyTorch, and a from-scratch version

Use when

  • +Actually removes information from correlated feature maps, unlike element-wise dropout
  • +Cheap: N⋅CN\cdot C random numbers instead of N⋅C⋅H⋅WN\cdot C\cdot H\cdot W
  • +Encourages redundancy across filters; no change at evaluation
  • +Works for 1-D sequences, images and volumes

Avoid when

  • −Coarse: removing a whole channel is a strong perturbation, so only small p work
  • −Few channels (early layers) make the per-example drop rate very noisy
  • −Interacts with batch-norm statistics like ordinary dropout
  • −DropBlock often regularizes CNNs better, at the cost of a block-size hyperparameter

Check yourself

A conv layer outputs (N, C, H, W) = (32, 64, 28, 28). How many Bernoulli draws do nn.Dropout and nn.Dropout2d make per forward pass?

Check yourself

Why does element-wise dropout regularize smooth convolutional feature maps so weakly?

Remember

  • ✓Spatial dropout draws one keep bit per (example, channel) and broadcasts it over all positions: y=mn,c x/(1−p)y=m_{n,c}\,x/(1-p).
  • ✓Element-wise dropout fails on conv maps because neighbouring activations are highly correlated; a dropped pixel is recoverable from its neighbours.
  • ✓PyTorch: nn.Dropout1d for (N,C,L)(N,C,L), nn.Dropout2d for (N,C,H,W)(N,C,H,W), nn.Dropout3d for volumes; identity in eval().
  • ✓Use small rates (0.1 to 0.3); a whole channel is a lot of signal.
  • ✓DropBlock (contiguous b×b squares) sits between element and channel dropout and often works better in deep CNNs.
← back to Deep Learning roadmap