# Spatial dropout

> Drop whole feature-map channels instead of single activations. Neighbouring pixels are so correlated that element-wise dropout removes almost no information; zeroing a channel everywhere does, so later layers cannot lean on any one channel.

Source: https://www.mlfoundry.in/learn/spatial-dropout  
Section: Deep Learning › Regularization & Normalization  
Level: core · 10 min read · updated 2026-10-11

**Spatial dropout:** (channel-wise dropout) zeroes entire channels of a convolutional feature map: for each example $n$ and channel $c$ one Bernoulli variable decides whether the whole map $x[n,c,\cdot,\cdot]$ is kept (and scaled by $1/(1-p)$) or set to zero. In PyTorch this is `nn.Dropout2d` for images, `nn.Dropout1d` for sequences and `nn.Dropout3d` for volumes.

Standard **Dropout** works well on fully connected layers, where every unit carries its own piece of information. Convolutional feature maps are different: a feature that fires at pixel $(i,j)$ almost always fires at $(i,j+1)$ too, because the receptive fields overlap and images are smooth. Zero one activation and its neighbours still carry nearly the same value, so the next convolution, which looks at a $3\times3$ window anyway, barely notices. The noise is there but the information is not removed, so the regularization effect is weak.

Tompson, Goroshin, Jain, LeCun and Bregler (2015, "Efficient object localization using convolutional networks", CVPR) hit exactly this in a fully convolutional human-pose network. They observed that when adjacent pixels are strongly correlated, i.i.d. dropout does not regularize the activations and mainly acts like a lower learning rate. Their fix, **SpatialDropout**, extends one dropout value across the whole feature map, so either all of a channel is present or none of it is. Keras exposes it as `SpatialDropout1D/2D/3D`; PyTorch calls it `Dropout1d/2d/3d`.

## Formula

$$
m_{n,c}\sim\mathrm{Bernoulli}(1-p),\qquad y[n,c,i,j]=\frac{m_{n,c}}{1-p}\;x[n,c,i,j]\quad\text{for all } i,j
$$

- $x[n,c,i,j]$: activation of example $n$, channel $c$, position $(i,j)$
- $m_{n,c}$: one keep-mask entry per example and channel, shared by every spatial position
- $p$: probability of dropping a channel (PyTorch's p)
- $1/(1-p)$: inverted-dropout scale, so $\mathbb E[y]=x$ and evaluation is the identity

Only the shape of the mask changes. Element-wise dropout draws $N\cdot C\cdot H\cdot W$ Bernoulli variables per batch; spatial dropout draws $N\cdot C$ and broadcasts each over $H\times W$. In the backward pass $\partial y/\partial x=m_{n,c}/(1-p)$, so a dropped channel sends no gradient back for that example: the filter that produced it gets no learning signal from that example. The next layer has to produce its output without that channel, which discourages relying on any single filter.

**At a glance**

- **Mask shape:** $(N, C, 1, 1)$ broadcast over $(N, C, H, W)$
- **PyTorch:** `nn.Dropout1d` on $(N,C,L)$, `nn.Dropout2d` on $(N,C,H,W)$, `nn.Dropout3d` on $(N,C,D,H,W)$
- **Typical p:** 0.1 to 0.3 (a whole channel is a lot of information)
- **Channels dropped:** Binomial$(C,p)$ per example: for $C=64$, $p=0.1$, on average 6.4 channels with standard deviation 2.4
- **Spatial cousin:** **DropBlock** zeroes contiguous $b\times b$ squares; $b=1$ is element dropout, $b=$ feature size is spatial dropout
- **Introduced:** Tompson et al., CVPR 2015 (SpatialDropout)

> Figure: Six 12×12 feature maps made by blurring white noise with a Gaussian of width ℓ, so neighbouring activations are correlated (ρ = 0.89 at the default ℓ = 2). Each dropped activation is guessed as the mean of its kept 8-neighbours in the same channel, and the bar is the R² of that guess: how much of the dropped signal the next layer can still recover. At p = 0.25, element-wise dropout leaves a kept neighbour for 99.9% of dropped activations and R² = 0.93 (average of 100 masks): almost nothing is lost. DropBlock gives 0.64 with 3×3 blocks and 0.41 with 5×5. Channel dropout gives exactly 0: the whole map is gone. Slide ℓ to 0 (white noise, ρ = −0.04): now element dropout's R² is −0.21, neighbours carry no information, and element-wise dropout is fine.

## Why correlation defeats element-wise dropout

Suppose neighbouring activations have correlation $\rho$. If $x_{ij}$ is dropped but $x_{i,j+1}$ is kept, the best linear guess of the missing value from that one neighbour is $\rho\,x_{i,j+1}$, and the fraction of its variance that guess explains is $\rho^2$. With eight neighbours and $\rho$ near 0.9, almost all of it is recoverable, which is what the figure's R² of 0.93 measures. A $3\times3$ convolution in the next layer sums over that window, so its output barely changes when one input pixel disappears. Spatial dropout makes the removed information unrecoverable from the same channel: the only way to compensate is through **other** channels, which forces redundant, independent features across filters.

The figure's numbers at p = 0.25, averaged over 100 masks (computed in Node from the figure's code)

| scheme | smooth maps (ℓ = 2): R² | dropped with a kept neighbour | white noise (ℓ = 0): R² |
| --- | --- | --- | --- |
| element (`nn.Dropout`) | 0.93 | 99.9% | −0.21 |
| DropBlock 3×3 | 0.64 | 78.2% | −0.28 |
| DropBlock 5×5 | 0.41 | 51.7% | −0.18 |
| channel (`nn.Dropout2d`) | 0.00 | 0% | 0.00 |

Negative R² on white noise means the neighbour-mean guess is worse than guessing 0: there is nothing to recover, and plain dropout already removes the full information. The decisive quantity is therefore the spatial correlation of the feature maps, which is high in most convolutional layers.

## Worked example

**nn.Dropout2d on one example with 3 channels of 2×2, p = 0.5**

1. Input channels: $x_0=\begin{bmatrix}1&2\\3&4\end{bmatrix}$, $x_1=\begin{bmatrix}0.5&0.5\\0.5&0.5\end{bmatrix}$, $x_2=\begin{bmatrix}-1&0\\1&2\end{bmatrix}$.
2. Draw one keep bit per channel, say $m=(1,0,1)$. Three random numbers, not twelve.
3. Scale the kept channels by $1/(1-p)=2$: $y_0=\begin{bmatrix}2&4\\6&8\end{bmatrix}$, $y_1=\mathbf 0$, $y_2=\begin{bmatrix}-2&0\\2&4\end{bmatrix}$.
4. Backward: the gradient into $x_0$ and $x_2$ is $2\times$ the incoming gradient, the gradient into $x_1$ is zero at every position, so the filter that produced channel 1 learns nothing from this example on this step.
5. Expected output: each channel is kept with probability 0.5 and doubled, so $\mathbb E[y_c]=x_c$. In `eval()` mode $y=x$.

## DropBlock: between the two extremes

Ghiasi, Lin and Le (2018, "DropBlock: a form of structured dropout", NeurIPS) drop contiguous square regions of each feature map. Block centres are sampled at a rate $\gamma$ chosen so that roughly a fraction $p$ of activations ends up zeroed, then the survivors are rescaled by (number of activations)/(number kept). With block size 7 on ResNet-50 they report ImageNet top-1 rising from 76.51% to 78.13%, and they found it works best when the drop rate is ramped up linearly during training.

$$
\gamma=\frac{p}{b^2}\cdot\frac{S^2}{(S-b+1)^2}
$$

- $b$: block size; $b=1$ recovers element dropout
- $S$: feature-map side length
- $(S-b+1)^2$: number of valid block centres (blocks must fit inside the map)

The formula ignores overlaps between blocks, so the realized drop rate is a little below $p$: in the figure ($S=12$, $p=0.25$) the average is 22.2% for $b=3$ ($\gamma=0.040$) and 21.8% for $b=5$ ($\gamma=0.0225$). The same idea applied to the input image is **Cutout** (DeVries and Taylor, 2017), a data-augmentation method.

```python
import torch, torch.nn as nn
from torchvision.ops import DropBlock2d

x = torch.ones(2, 4, 3, 3)                  # (N, C, H, W)
sd = nn.Dropout2d(p=0.5).train()
y = sd(x)
print([y[0, c].unique().tolist() for c in range(4)])
# e.g. [[2.0], [0.0], [0.0], [2.0]]: each channel is all 0 or all 1/(1-p) = 2

# Sequences: (N, C, L), e.g. a 1-D CNN or embeddings transposed to (N, D, T)
sd1 = nn.Dropout1d(p=0.2)                   # drops whole embedding dims for all time steps
seq = sd1(torch.randn(8, 128, 50))
# Passing (N, C, L) to Dropout2d is ambiguous (it could mean an unbatched (C, H, W)):
# PyTorch warns; use Dropout1d for 3-D sequence input.

# From scratch
def spatial_dropout(x, p, training=True):
    if not training or p == 0:
        return x
    mask = (torch.rand(x.shape[0], x.shape[1], *([1] * (x.dim() - 2)),
                       device=x.device) > p).to(x.dtype)
    return x * mask / (1 - p)                # mask broadcasts over H, W

block = nn.Sequential(
    nn.Conv2d(64, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),
    nn.Dropout2d(p=0.1),
)
db = DropBlock2d(p=0.1, block_size=7)        # torchvision's DropBlock
```

Channel-wise dropout in PyTorch, and a from-scratch version

> **Dropout2d on the wrong axis:** Spatial dropout drops along dimension 1. For sequence models with tensors laid out as $(N, T, D)$ (batch, time, features), `nn.Dropout1d` would drop whole **time steps**, not feature channels. Transpose to $(N, D, T)$ first if you mean "drop a feature at every time step". And with 3-D input to `Dropout2d`, PyTorch has to guess between $(N,C,L)$ and unbatched $(C,H,W)$ and warns; pick `Dropout1d` explicitly.

> **Same idea, other places:** Sharing one mask over a dimension along which activations are correlated is a recurring pattern. Variational RNN dropout (Gal and Ghahramani, 2016) shares a mask across time steps; spatial dropout shares it across positions; **Stochastic depth** shares it across an entire residual branch. In every case the shared mask removes information that would otherwise leak back in from correlated copies.

> **With batch norm:** Channel dropout before a BatchNorm layer shifts that layer's running variance just as ordinary dropout does (see **Dropout**), and dropping a whole channel changes the per-channel statistics more abruptly. Modern CNNs with BN mostly rely on weight decay, data augmentation and stochastic depth instead; when spatial dropout is used, small rates (0.1 to 0.2) after the BN-ReLU pair are typical.

**Strengths**

- Actually removes information from correlated feature maps, unlike element-wise dropout
- Cheap: $N\cdot C$ random numbers instead of $N\cdot C\cdot H\cdot W$
- Encourages redundancy across filters; no change at evaluation
- Works for 1-D sequences, images and volumes

**Limitations**

- Coarse: removing a whole channel is a strong perturbation, so only small p work
- Few channels (early layers) make the per-example drop rate very noisy
- Interacts with batch-norm statistics like ordinary dropout
- DropBlock often regularizes CNNs better, at the cost of a block-size hyperparameter

**Question:** A conv layer outputs (N, C, H, W) = (32, 64, 28, 28). How many Bernoulli draws do nn.Dropout and nn.Dropout2d make per forward pass?

- A. 2,048 and 1,605,632
- B. 1,605,632 and 2,048
- C. 64 and 1
- D. 1,605,632 and 64

**Answer:** B. 1,605,632 and 2,048 — Element-wise dropout masks every activation: $32\cdot64\cdot28\cdot28=1{,}605{,}632$. Spatial dropout draws one bit per example and channel, $32\cdot64=2{,}048$, and broadcasts it over the $28\times28$ positions. The mask is per example, so it is not 64.

**Question:** Why does element-wise dropout regularize smooth convolutional feature maps so weakly?

- A. The dropped value can be reconstructed from its strongly correlated neighbours, so little information is removed
- B. Convolutions ignore zeros
- C. The inverted-dropout scale cancels the mask in convolutional layers
- D. Batch norm undoes it

**Answer:** A. The dropped value can be reconstructed from its strongly correlated neighbours, so little information is removed — With neighbour correlation $\rho\approx0.9$, the kept neighbours almost determine the dropped activation (R² = 0.93 in the figure), and the next $3\times3$ convolution sees them anyway. Dropping the whole channel leaves nothing in that channel to reconstruct from.

**Key points**

- Spatial dropout draws one keep bit per (example, channel) and broadcasts it over all positions: $y=m_{n,c}\,x/(1-p)$.
- Element-wise dropout fails on conv maps because neighbouring activations are highly correlated; a dropped pixel is recoverable from its neighbours.
- PyTorch: `nn.Dropout1d` for $(N,C,L)$, `nn.Dropout2d` for $(N,C,H,W)$, `nn.Dropout3d` for volumes; identity in `eval()`.
- Use small rates (0.1 to 0.3); a whole channel is a lot of signal.
- DropBlock (contiguous b×b squares) sits between element and channel dropout and often works better in deep CNNs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
