Pooling layers

Pooling summarises each window of a feature map with a fixed function, max or mean, and has no parameters. It downsamples, makes features tolerant to small shifts, and global average pooling turns any map into one vector. Max pooling sends the whole gradient to the argmax.

12 min read1249 wordscore0 viewsupdated 2026-10-11

Definition

Pooling layer A pooling layer slides a k×kk\times k window with stride ss over each channel separately and replaces the window by one summary statistic: its maximum (max pooling) or its mean (average pooling). It has no learnable parameters and does not mix channels, so CC input maps give CC smaller output maps. Global average pooling uses one window that covers the whole map.

A convolution says where a pattern is: its feature map is equivariant, so the response moves with the object. For classification we mostly care whether the pattern is there. Pooling throws away some of the "where". After a 2×22\times2 max pool, a feature that moves by one pixel inside its window produces exactly the same output. Pooling also halves the resolution at no parameter cost, which cuts the compute of every later layer by 4× and doubles their Receptive field growth.

The two flavours have different roots. LeNet-5 (LeCun et al., 1998) used subsampling layers that averaged each 2×22\times2 block, multiplied by a trainable coefficient and added a bias. The max operation came from models of the visual cortex: Riesenhuber and Poggio's HMAX (1999) used a MAX over afferent units to build position-tolerant "complex cells". Scherer, Müller and Behnke (2010) found max pooling clearly better than subsampling for object recognition, and AlexNet (Krizhevsky et al., 2012) made overlapping 3×33\times3, stride-2 max pooling standard. Lin, Chen and Yan's Network in Network (2014) introduced global average pooling as the classifier head, and Springenberg et al. (2015, "Striving for Simplicity: The All Convolutional Net") showed that strided convolutions can replace pooling altogether.

Forward and backward

yc,i,jmax⁡=max⁡(u,v)∈Rijxc,u,vyc,i,javg=1∣Rij∣∑(u,v)∈Rijxc,u,vRij={si≤u<si+k,  sj≤v<sj+k}y^{\max}_{c,i,j}=\max_{(u,v)\in R_{ij}}x_{c,u,v}\qquad y^{\text{avg}}_{c,i,j}=\frac{1}{|R_{ij}|}\sum_{(u,v)\in R_{ij}}x_{c,u,v}\qquad R_{ij}=\{si\le u<si+k,\;sj\le v<sj+k\}
RijR_{ij}
the window of output (i,j)(i,j); output size ⌊(n+2p−k)/s⌋+1\lfloor (n+2p-k)/s\rfloor+1, exactly as for a convolution (see Stride and padding)
cc
channel: pooling acts on each channel independently
∣Rij∣|R_{ij}|
window area k2k^2; with padding, AvgPool2d divides by k2k^2 including padded zeros unless count_include_pad=False
∂yijmax⁡∂xuv=1[(u,v)=arg⁡max⁡Rijx]∂yijavg∂xuv=1[(u,v)∈Rij]∣Rij∣\frac{\partial y^{\max}_{ij}}{\partial x_{uv}}=\mathbb 1\big[(u,v)=\arg\max_{R_{ij}}x\big]\qquad\qquad \frac{\partial y^{\text{avg}}_{ij}}{\partial x_{uv}}=\frac{\mathbb 1[(u,v)\in R_{ij}]}{|R_{ij}|}
arg⁡max⁡\arg\max
max pooling is a router: the upstream gradient goes unchanged to the single winning input and every other input in the window gets 0
1/∣Rij∣1/|R_{ij}|
average pooling spreads the gradient evenly, 1/k21/k^2 to every input in the window

Max pooling's backward pass needs to remember where each maximum was (return_indices=True exposes these "switches", which nn.MaxUnpool2d uses to undo the pooling). With non-overlapping 2×22\times2 windows, 3 of every 4 inputs receive no gradient through that layer on a given example: only the winners learn. With overlapping windows (k>sk>s) an input can win several windows and its gradients add up. The max is not differentiable when two inputs tie; frameworks pick one of them (PyTorch's kernels keep the first maximum they scan) and that is a valid subgradient.

Pooling layers at a glance

Parameters
none (LeNet's subsampling had one weight and one bias per map)
Channels
unchanged: C×H×W→C×Hout×WoutC\times H\times W\to C\times H_{out}\times W_{out}
Typical setting
2×22\times2 stride 2 (VGG), 3×33\times3 stride 2 padding 1 (ResNet stem), 3×33\times3 stride 2 (AlexNet)
Invariance
local: to shifts smaller than the window, and only partly; equivariant to shifts by the stride
Gradient
max: all to the argmax; average: 1/k21/k^2 to each input
PyTorch
nn.MaxPool2d, nn.AvgPool2d, nn.AdaptiveAvgPool2d, nn.AdaptiveMaxPool2d, nn.LPPool2d
feature map 8×80.53.60.60.12.80.80.50.50.80.24.20.40.40.23.42.60.34.50.23.0MaxPool2d(2, stride=2)3.60.60.12.80.84.20.40.02.60.00.33.40.03.04.50.2click a cell∂y[1,1]/∂x1max: all of it to the argmaxΣ ∂ = 1.00 · params: 0output size ⌊(8 − 2)/2⌋ + 1 = 4 → 4×4y[1,1] = max of 4 values = 4.2 at (3,3)global average pool of this map = 0.4621-px shift: map changes 177% (Σ|Δ|/Σ|x|), pooled output 56%strong peaks (≥ 2) that keep their pooled cell exactly: 4 / 7□ window □ argmax ■ gradient reaching the input
presets
kernelstride
fig.An 8×8 post-ReLU feature map (sparse activations with nine strong peaks) pooled by max, average or adaptive average pooling. The middle panel is the pooled output; the right panel is the gradient each input receives. By default (2×2, stride 2) output (1, 1) is 4.2, the maximum of its window at (3, 3), and its whole gradient goes to that one pixel; switch to average and each of the 4 inputs gets 1/4. With 'gradient of Σ outputs', max pooling reaches only 16 of the 64 inputs. The shift slider moves the content left: a 1-pixel shift changes the map itself by 177% (Σ|Δ|/Σ|x|) but the 2×2 max-pooled output by only 56%, and 4 of the 7 strong peaks keep their pooled value exactly. AlexNet's overlapping 3×3/stride-2 pool gives a 3×3 output (66% change, 4 of 5 peaks kept), and global average pooling (0.462 here) changes by just 7%. A 2-pixel shift is a full stride: the 2×2 output then changes by 147%, because every peak moves into the neighbouring cell.

Local invariance, not global

Pooling buys invariance in a precise but limited sense. If a peak stays inside the same window, the max does not change. If it crosses into the next window, the output moves by one cell: still equivariant, just at the coarser resolution. Stack several pooling stages and the tolerated shift grows, which is how a CNN becomes reasonably insensitive to where an object sits. But it is not exact: the figure shows that a 1-pixel shift still changes 56% of a 2×22\times2 max-pooled map, because some peaks sit on a window boundary. Max pooling also aliases, like any subsampling. Zhang (2019) made it far more shift-consistent by splitting it into a stride-1 max, a blur, and subsampling (MaxBlurPool; see Stride and padding).

Max and average pooling encode different assumptions. Max asks "is the feature present anywhere in the window?" and suits sparse, detector-like activations after ReLU. Average asks "how much of the feature is there?" and keeps more information but dilutes a single strong response with its zero neighbours. Boureau, Ponce and LeCun (2010) analysed this trade-off and found max pooling better when the features are sparse. A family in between is LpL_p pooling, (1∣R∣∑xp)1/p(\tfrac1{|R|}\sum x^p)^{1/p}, which is the mean for p=1p=1 and approaches the max as p→∞p\to\infty.

Overlapping pooling and strided convolutions

AlexNet used 3×33\times3 windows with stride 2, so neighbouring windows overlap by one row and column. Krizhevsky et al. reported that this reduced top-1 and top-5 error by 0.4% and 0.3% compared with non-overlapping 2×22\times2, stride-2 pooling with the same output size, and that the overlapping models were slightly harder to overfit. ResNet's stem uses the same 3×33\times3, stride-2 max pool with padding 1.

Pooling is a fixed downsampler. A stride-2 convolution is a learned one. Springenberg, Dosovitskiy, Brox and Riedmiller (2015) replaced every max-pooling layer of their CIFAR networks with a stride-2 convolution (or with a stride-1 convolution followed by subsampling) and matched or beat the pooled versions. Most later architectures followed: ResNet downsamples inside its residual stages with stride-2 convolutions and keeps only one max pool in the stem; ConvNeXt has no max pooling at all, using strided "patchify" and downsampling convolutions. Pooling survives where a parameter-free, robust summary is wanted: at the end of the network.

Global and adaptive average pooling

Global average pooling (GAP) averages each channel over all H×WH\times W positions, giving a CC-dimensional vector: zc=1HW∑u,vxc,u,vz_c=\frac{1}{HW}\sum_{u,v}x_{c,u,v}. Network in Network (Lin et al., 2014) used it to replace the large fully connected layers that dominated AlexNet's and VGG's parameter counts (VGG-16's first FC layer alone has 7⋅7⋅512⋅4096≈102.87\cdot7\cdot512\cdot4096\approx102.8 M weights). GoogLeNet, ResNet, DenseNet, MobileNet and EfficientNet all end with GAP followed by one linear layer. Because GAP does not care about HH and WW, such networks accept any input resolution. It also makes the last conv layer interpretable: weighting its maps by the classifier weights gives a class activation map (Zhou et al., 2016).

AdaptiveAvgPool(o):bin i=[⌊i no⌋,  ⌈(i+1) no⌉),i=0,…,o−1\text{AdaptiveAvgPool}(o):\quad \text{bin }i=\Big[\Big\lfloor\frac{i\,n}{o}\Big\rfloor,\;\Big\lceil\frac{(i+1)\,n}{o}\Big\rceil\Big),\qquad i=0,\dots,o-1
oo
the requested output size; o=1o=1 is global average pooling
nn
the input size, any value: the bins adapt so the output is always o×oo\times o
⌈⋅⌉\lceil\cdot\rceil
when oo does not divide nn the bins have unequal sizes and may overlap: n=8, o=3n=8,\ o=3 gives [0,3), [2,6), [5,8)[0,3),\ [2,6),\ [5,8)

Adaptive pooling specifies the output size instead of the window. torchvision's ResNet ends with nn.AdaptiveAvgPool2d((1, 1)) and its VGG with nn.AdaptiveAvgPool2d((7, 7)), so both run on images of any size. The idea goes back to spatial pyramid pooling (He et al., 2014), which pooled to several fixed grids at once to get a fixed-length vector from arbitrary inputs.

Worked example: a 4×4 corner of the figure's map

Take the top-left 4×44\times4 block of the default map, X=(0000.503.600.600.800.50.50.80.24.2)X=\begin{pmatrix}0&0&0&0.5\\0&3.6&0&0.6\\0&0.8&0&0.5\\0.5&0.8&0.2&4.2\end{pmatrix}, and pool it 2×22\times2 with stride 2 (checked in Node).

window (rows, cols)valuesmax (argmax)mean
(0–1, 0–1)0, 0, 0, 3.63.6 at (1,1)0.9
(0–1, 2–3)0, 0.5, 0, 0.60.6 at (1,3)0.275
(2–3, 0–1)0, 0.8, 0.5, 0.80.8 at (2,1), a tie with (3,1)0.525
(2–3, 2–3)0, 0.5, 0.2, 4.24.2 at (3,3)1.225
2×2 / stride-2 pooling of X, window by window

Backward with upstream gradient G = [[1, 2], [3, 4]]

  1. 1Max pooling: each GG entry goes to its window's argmax, so ∂L/∂X\partial L/\partial X has 11 at (1,1)(1,1), 22 at (1,3)(1,3), 33 at (2,1)(2,1), 44 at (3,3)(3,3) and 0 at the other 12 inputs.
  2. 2The tie in the bottom-left window is broken by scan order: PyTorch keeps the first maximum, (2,1)(2,1), so (3,1)(3,1) gets nothing even though it has the same value.
  3. 3Average pooling: every input in window (a,b)(a,b) gets Gab/4G_{ab}/4: 0.250.25 for the top-left block, 0.50.5, 0.750.75 and 11 for the others. All 16 inputs receive gradient.
  4. 4Global average pooling of XX is 11.7/16=0.73111.7/16=0.731, and each input gets 1/161/16 of the upstream gradient.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3X = torch.tensor([[0, 0, 0, .5], [0, 3.6, 0, .6], [0, .8, 0, .5], [.5, .8, .2, 4.2]]).view(1, 1, 4, 4)
4X.requires_grad_(True)
5
6y, idx = F.max_pool2d(X, kernel_size=2, stride=2, return_indices=True)
7print(y.flatten().tolist(), idx.flatten().tolist()) # ≈ [3.6, 0.6, 0.8, 4.2] [5, 7, 9, 15] (flat argmax positions)
8y.backward(torch.tensor([[1., 2], [3, 4]]).view(1, 1, 2, 2))
9print(X.grad.view(4, 4)) # 1 at (1,1), 2 at (1,3), 3 at (2,1), 4 at (3,3), zeros elsewhere
10
11print(F.avg_pool2d(X, 2).flatten().tolist()) # ≈ [0.9, 0.275, 0.525, 1.225]
12print(F.adaptive_avg_pool2d(X, 1).item()) # ≈ 0.73125 (global average pooling)
13
14# Common configurations
15alex_pool = nn.MaxPool2d(kernel_size=3, stride=2) # overlapping (AlexNet)
16resnet_pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) # 112 -> 56 (ResNet stem)
17vgg_pool = nn.MaxPool2d(2) # stride defaults to kernel_size
18unpool = nn.MaxUnpool2d(2) # uses the saved indices
19
20# Strided conv instead of pooling (all-convolutional net): learnable downsampling
21down = nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1)
22
23# GAP head: works for any input resolution
24head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(2048, 1000))
25for hw in [(7, 7), (10, 13)]:
26 print(head(torch.randn(2, 2048, *hw)).shape) # torch.Size([2, 1000]) both times
Max, average and adaptive pooling, the routed gradient, and a GAP classifier head

Use when

  • +No parameters and very cheap; cuts the next layers' compute by 4× per 2× downsampling
  • +Local tolerance to small shifts and deformations
  • +Global average pooling replaces huge FC layers and makes the network resolution-agnostic
  • +Max pooling keeps the strongest response of sparse ReLU features

Avoid when

  • −Fixed, not learned: strided convolutions are usually as good or better for downsampling
  • −Discards precise position, which dense-prediction tasks must recover
  • −Subsampling still aliases; invariance is only partial (MaxBlurPool helps)
  • −Max pooling gives gradient to one input per window; average pooling dilutes strong, sparse responses

Check yourself

In a 2×2, stride-2 max pooling layer, the window [[0.1, 0.9], [0.4, 0.3]] receives upstream gradient 2. What gradient does each input get?

Check yourself

Why does torchvision's ResNet-50 accept 320×320 images even though it was trained at 224×224?

Remember

  • ✓Pooling summarises each k×k window of each channel by its max or mean; no parameters, no channel mixing.
  • ✓Backward: max pooling routes the whole gradient to the argmax; average pooling gives 1/k² to every input.
  • ✓It gives local, partial shift invariance and cheap downsampling; overlapping 3×3/stride-2 pooling came from AlexNet.
  • ✓Strided convolutions can replace pooling (all-convolutional net); modern CNNs keep at most one max pool.
  • ✓Global/adaptive average pooling turns any map into a fixed-size vector and replaced large FC heads.
← back to Deep Learning roadmap