Definition
Pooling layer A pooling layer slides a window with stride over each channel separately and replaces the window by one summary statistic: its maximum (max pooling) or its mean (average pooling). It has no learnable parameters and does not mix channels, so input maps give smaller output maps. Global average pooling uses one window that covers the whole map.
A convolution says where a pattern is: its feature map is equivariant, so the response moves with the object. For classification we mostly care whether the pattern is there. Pooling throws away some of the "where". After a max pool, a feature that moves by one pixel inside its window produces exactly the same output. Pooling also halves the resolution at no parameter cost, which cuts the compute of every later layer by 4× and doubles their Receptive field growth.
The two flavours have different roots. LeNet-5 (LeCun et al., 1998) used subsampling layers that averaged each block, multiplied by a trainable coefficient and added a bias. The max operation came from models of the visual cortex: Riesenhuber and Poggio's HMAX (1999) used a MAX over afferent units to build position-tolerant "complex cells". Scherer, Müller and Behnke (2010) found max pooling clearly better than subsampling for object recognition, and AlexNet (Krizhevsky et al., 2012) made overlapping , stride-2 max pooling standard. Lin, Chen and Yan's Network in Network (2014) introduced global average pooling as the classifier head, and Springenberg et al. (2015, "Striving for Simplicity: The All Convolutional Net") showed that strided convolutions can replace pooling altogether.
Forward and backward
- the window of output ; output size , exactly as for a convolution (see Stride and padding)
- channel: pooling acts on each channel independently
- window area ; with padding,
AvgPool2ddivides by including padded zeros unlesscount_include_pad=False
- max pooling is a router: the upstream gradient goes unchanged to the single winning input and every other input in the window gets 0
- average pooling spreads the gradient evenly, to every input in the window
Max pooling's backward pass needs to remember where each maximum was (return_indices=True exposes these "switches", which nn.MaxUnpool2d uses to undo the pooling). With non-overlapping windows, 3 of every 4 inputs receive no gradient through that layer on a given example: only the winners learn. With overlapping windows () an input can win several windows and its gradients add up. The max is not differentiable when two inputs tie; frameworks pick one of them (PyTorch's kernels keep the first maximum they scan) and that is a valid subgradient.
Pooling layers at a glance
- Parameters
- none (LeNet's subsampling had one weight and one bias per map)
- Channels
- unchanged:
- Typical setting
- stride 2 (VGG), stride 2 padding 1 (ResNet stem), stride 2 (AlexNet)
- Invariance
- local: to shifts smaller than the window, and only partly; equivariant to shifts by the stride
- Gradient
- max: all to the argmax; average: to each input
- PyTorch
nn.MaxPool2d,nn.AvgPool2d,nn.AdaptiveAvgPool2d,nn.AdaptiveMaxPool2d,nn.LPPool2d
Local invariance, not global
Pooling buys invariance in a precise but limited sense. If a peak stays inside the same window, the max does not change. If it crosses into the next window, the output moves by one cell: still equivariant, just at the coarser resolution. Stack several pooling stages and the tolerated shift grows, which is how a CNN becomes reasonably insensitive to where an object sits. But it is not exact: the figure shows that a 1-pixel shift still changes 56% of a max-pooled map, because some peaks sit on a window boundary. Max pooling also aliases, like any subsampling. Zhang (2019) made it far more shift-consistent by splitting it into a stride-1 max, a blur, and subsampling (MaxBlurPool; see Stride and padding).
Max and average pooling encode different assumptions. Max asks "is the feature present anywhere in the window?" and suits sparse, detector-like activations after ReLU. Average asks "how much of the feature is there?" and keeps more information but dilutes a single strong response with its zero neighbours. Boureau, Ponce and LeCun (2010) analysed this trade-off and found max pooling better when the features are sparse. A family in between is pooling, , which is the mean for and approaches the max as .
Overlapping pooling and strided convolutions
AlexNet used windows with stride 2, so neighbouring windows overlap by one row and column. Krizhevsky et al. reported that this reduced top-1 and top-5 error by 0.4% and 0.3% compared with non-overlapping , stride-2 pooling with the same output size, and that the overlapping models were slightly harder to overfit. ResNet's stem uses the same , stride-2 max pool with padding 1.
Pooling is a fixed downsampler. A stride-2 convolution is a learned one. Springenberg, Dosovitskiy, Brox and Riedmiller (2015) replaced every max-pooling layer of their CIFAR networks with a stride-2 convolution (or with a stride-1 convolution followed by subsampling) and matched or beat the pooled versions. Most later architectures followed: ResNet downsamples inside its residual stages with stride-2 convolutions and keeps only one max pool in the stem; ConvNeXt has no max pooling at all, using strided "patchify" and downsampling convolutions. Pooling survives where a parameter-free, robust summary is wanted: at the end of the network.
Global and adaptive average pooling
Global average pooling (GAP) averages each channel over all positions, giving a -dimensional vector: . Network in Network (Lin et al., 2014) used it to replace the large fully connected layers that dominated AlexNet's and VGG's parameter counts (VGG-16's first FC layer alone has M weights). GoogLeNet, ResNet, DenseNet, MobileNet and EfficientNet all end with GAP followed by one linear layer. Because GAP does not care about and , such networks accept any input resolution. It also makes the last conv layer interpretable: weighting its maps by the classifier weights gives a class activation map (Zhou et al., 2016).
- the requested output size; is global average pooling
- the input size, any value: the bins adapt so the output is always
- when does not divide the bins have unequal sizes and may overlap: gives
Adaptive pooling specifies the output size instead of the window. torchvision's ResNet ends with nn.AdaptiveAvgPool2d((1, 1)) and its VGG with nn.AdaptiveAvgPool2d((7, 7)), so both run on images of any size. The idea goes back to spatial pyramid pooling (He et al., 2014), which pooled to several fixed grids at once to get a fixed-length vector from arbitrary inputs.
Worked example: a 4×4 corner of the figure's map
Take the top-left block of the default map, , and pool it with stride 2 (checked in Node).
| window (rows, cols) | values | max (argmax) | mean |
|---|---|---|---|
| (0–1, 0–1) | 0, 0, 0, 3.6 | 3.6 at (1,1) | 0.9 |
| (0–1, 2–3) | 0, 0.5, 0, 0.6 | 0.6 at (1,3) | 0.275 |
| (2–3, 0–1) | 0, 0.8, 0.5, 0.8 | 0.8 at (2,1), a tie with (3,1) | 0.525 |
| (2–3, 2–3) | 0, 0.5, 0.2, 4.2 | 4.2 at (3,3) | 1.225 |
Backward with upstream gradient G = [[1, 2], [3, 4]]
- 1Max pooling: each entry goes to its window's argmax, so has at , at , at , at and 0 at the other 12 inputs.
- 2The tie in the bottom-left window is broken by scan order: PyTorch keeps the first maximum, , so gets nothing even though it has the same value.
- 3Average pooling: every input in window gets : for the top-left block, , and for the others. All 16 inputs receive gradient.
- 4Global average pooling of is , and each input gets of the upstream gradient.
1import torch, torch.nn as nn, torch.nn.functional as F23X = torch.tensor([[0, 0, 0, .5], [0, 3.6, 0, .6], [0, .8, 0, .5], [.5, .8, .2, 4.2]]).view(1, 1, 4, 4)4X.requires_grad_(True)56y, idx = F.max_pool2d(X, kernel_size=2, stride=2, return_indices=True)7print(y.flatten().tolist(), idx.flatten().tolist()) # ≈ [3.6, 0.6, 0.8, 4.2] [5, 7, 9, 15] (flat argmax positions)8y.backward(torch.tensor([[1., 2], [3, 4]]).view(1, 1, 2, 2))9print(X.grad.view(4, 4)) # 1 at (1,1), 2 at (1,3), 3 at (2,1), 4 at (3,3), zeros elsewhere1011print(F.avg_pool2d(X, 2).flatten().tolist()) # ≈ [0.9, 0.275, 0.525, 1.225]12print(F.adaptive_avg_pool2d(X, 1).item()) # ≈ 0.73125 (global average pooling)1314# Common configurations15alex_pool = nn.MaxPool2d(kernel_size=3, stride=2) # overlapping (AlexNet)16resnet_pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) # 112 -> 56 (ResNet stem)17vgg_pool = nn.MaxPool2d(2) # stride defaults to kernel_size18unpool = nn.MaxUnpool2d(2) # uses the saved indices1920# Strided conv instead of pooling (all-convolutional net): learnable downsampling21down = nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1)2223# GAP head: works for any input resolution24head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(2048, 1000))25for hw in [(7, 7), (10, 13)]:26 print(head(torch.randn(2, 2048, *hw)).shape) # torch.Size([2, 1000]) both timesUse when
- +No parameters and very cheap; cuts the next layers' compute by 4× per 2× downsampling
- +Local tolerance to small shifts and deformations
- +Global average pooling replaces huge FC layers and makes the network resolution-agnostic
- +Max pooling keeps the strongest response of sparse ReLU features
Avoid when
- −Fixed, not learned: strided convolutions are usually as good or better for downsampling
- −Discards precise position, which dense-prediction tasks must recover
- −Subsampling still aliases; invariance is only partial (MaxBlurPool helps)
- −Max pooling gives gradient to one input per window; average pooling dilutes strong, sparse responses
Check yourself
In a 2×2, stride-2 max pooling layer, the window [[0.1, 0.9], [0.4, 0.3]] receives upstream gradient 2. What gradient does each input get?
Check yourself
Why does torchvision's ResNet-50 accept 320×320 images even though it was trained at 224×224?
Remember
- ✓Pooling summarises each k×k window of each channel by its max or mean; no parameters, no channel mixing.
- ✓Backward: max pooling routes the whole gradient to the argmax; average pooling gives 1/k² to every input.
- ✓It gives local, partial shift invariance and cheap downsampling; overlapping 3×3/stride-2 pooling came from AlexNet.
- ✓Strided convolutions can replace pooling (all-convolutional net); modern CNNs keep at most one max pool.
- ✓Global/adaptive average pooling turns any map into a fixed-size vector and replaced large FC heads.