# Pooling layers

> Pooling summarises each window of a feature map with a fixed function, max or mean, and has no parameters. It downsamples, makes features tolerant to small shifts, and global average pooling turns any map into one vector. Max pooling sends the whole gradient to the argmax.

Source: https://www.mlfoundry.in/learn/pooling-layers  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 12 min read · updated 2026-10-11

**Pooling layer:** A pooling layer slides a $k\times k$ window with stride $s$ over each channel **separately** and replaces the window by one summary statistic: its maximum (**max pooling**) or its mean (**average pooling**). It has no learnable parameters and does not mix channels, so $C$ input maps give $C$ smaller output maps. **Global average pooling** uses one window that covers the whole map.

A convolution says *where* a pattern is: its feature map is equivariant, so the response moves with the object. For classification we mostly care *whether* the pattern is there. Pooling throws away some of the "where". After a $2\times2$ max pool, a feature that moves by one pixel inside its window produces exactly the same output. Pooling also halves the resolution at no parameter cost, which cuts the compute of every later layer by 4× and doubles their **Receptive field** growth.

The two flavours have different roots. LeNet-5 (LeCun et al., 1998) used **subsampling** layers that averaged each $2\times2$ block, multiplied by a trainable coefficient and added a bias. The max operation came from models of the visual cortex: Riesenhuber and Poggio's HMAX (1999) used a MAX over afferent units to build position-tolerant "complex cells". Scherer, Müller and Behnke (2010) found max pooling clearly better than subsampling for object recognition, and AlexNet (Krizhevsky et al., 2012) made overlapping $3\times3$, stride-2 max pooling standard. Lin, Chen and Yan's **Network in Network** (2014) introduced global average pooling as the classifier head, and Springenberg et al. (2015, "Striving for Simplicity: The All Convolutional Net") showed that strided convolutions can replace pooling altogether.

## Forward and backward

$$
y^{\max}_{c,i,j}=\max_{(u,v)\in R_{ij}}x_{c,u,v}\qquad y^{\text{avg}}_{c,i,j}=\frac{1}{|R_{ij}|}\sum_{(u,v)\in R_{ij}}x_{c,u,v}\qquad R_{ij}=\{si\le u<si+k,\;sj\le v<sj+k\}
$$

- $R_{ij}$: the window of output $(i,j)$; output size $\lfloor (n+2p-k)/s\rfloor+1$, exactly as for a convolution (see **Stride and padding**)
- $c$: channel: pooling acts on each channel independently
- $|R_{ij}|$: window area $k^2$; with padding, `AvgPool2d` divides by $k^2$ including padded zeros unless `count_include_pad=False`

$$
\frac{\partial y^{\max}_{ij}}{\partial x_{uv}}=\mathbb 1\big[(u,v)=\arg\max_{R_{ij}}x\big]\qquad\qquad \frac{\partial y^{\text{avg}}_{ij}}{\partial x_{uv}}=\frac{\mathbb 1[(u,v)\in R_{ij}]}{|R_{ij}|}
$$

- $\arg\max$: max pooling is a **router**: the upstream gradient goes unchanged to the single winning input and every other input in the window gets 0
- $1/|R_{ij}|$: average pooling spreads the gradient evenly, $1/k^2$ to every input in the window

Max pooling's backward pass needs to remember where each maximum was (`return_indices=True` exposes these "switches", which `nn.MaxUnpool2d` uses to undo the pooling). With non-overlapping $2\times2$ windows, 3 of every 4 inputs receive no gradient through that layer on a given example: only the winners learn. With overlapping windows ($k>s$) an input can win several windows and its gradients add up. The max is not differentiable when two inputs tie; frameworks pick one of them (PyTorch's kernels keep the first maximum they scan) and that is a valid subgradient.

**Pooling layers at a glance**

- **Parameters:** none (LeNet's subsampling had one weight and one bias per map)
- **Channels:** unchanged: $C\times H\times W\to C\times H_{out}\times W_{out}$
- **Typical setting:** $2\times2$ stride 2 (VGG), $3\times3$ stride 2 padding 1 (ResNet stem), $3\times3$ stride 2 (AlexNet)
- **Invariance:** local: to shifts smaller than the window, and only partly; equivariant to shifts by the stride
- **Gradient:** max: all to the argmax; average: $1/k^2$ to each input
- **PyTorch:** `nn.MaxPool2d`, `nn.AvgPool2d`, `nn.AdaptiveAvgPool2d`, `nn.AdaptiveMaxPool2d`, `nn.LPPool2d`

> Figure: An 8×8 post-ReLU feature map (sparse activations with nine strong peaks) pooled by max, average or adaptive average pooling. The middle panel is the pooled output; the right panel is the gradient each input receives. By default (2×2, stride 2) output (1, 1) is 4.2, the maximum of its window at (3, 3), and its whole gradient goes to that one pixel; switch to average and each of the 4 inputs gets 1/4. With 'gradient of Σ outputs', max pooling reaches only 16 of the 64 inputs. The shift slider moves the content left: a 1-pixel shift changes the map itself by 177% (Σ|Δ|/Σ|x|) but the 2×2 max-pooled output by only 56%, and 4 of the 7 strong peaks keep their pooled value exactly. AlexNet's overlapping 3×3/stride-2 pool gives a 3×3 output (66% change, 4 of 5 peaks kept), and global average pooling (0.462 here) changes by just 7%. A 2-pixel shift is a full stride: the 2×2 output then changes by 147%, because every peak moves into the neighbouring cell.

## Local invariance, not global

Pooling buys invariance in a precise but limited sense. If a peak stays inside the same window, the max does not change. If it crosses into the next window, the output moves by one cell: still equivariant, just at the coarser resolution. Stack several pooling stages and the tolerated shift grows, which is how a CNN becomes reasonably insensitive to where an object sits. But it is not exact: the figure shows that a 1-pixel shift still changes 56% of a $2\times2$ max-pooled map, because some peaks sit on a window boundary. Max pooling also aliases, like any subsampling. Zhang (2019) made it far more shift-consistent by splitting it into a stride-1 max, a blur, and subsampling (MaxBlurPool; see **Stride and padding**).

Max and average pooling encode different assumptions. Max asks "is the feature present anywhere in the window?" and suits sparse, detector-like activations after ReLU. Average asks "how much of the feature is there?" and keeps more information but dilutes a single strong response with its zero neighbours. Boureau, Ponce and LeCun (2010) analysed this trade-off and found max pooling better when the features are sparse. A family in between is $L_p$ pooling, $(\tfrac1{|R|}\sum x^p)^{1/p}$, which is the mean for $p=1$ and approaches the max as $p\to\infty$.

## Overlapping pooling and strided convolutions

AlexNet used $3\times3$ windows with stride 2, so neighbouring windows overlap by one row and column. Krizhevsky et al. reported that this reduced top-1 and top-5 error by 0.4% and 0.3% compared with non-overlapping $2\times2$, stride-2 pooling with the same output size, and that the overlapping models were slightly harder to overfit. ResNet's stem uses the same $3\times3$, stride-2 max pool with padding 1.

Pooling is a fixed downsampler. A stride-2 convolution is a **learned** one. Springenberg, Dosovitskiy, Brox and Riedmiller (2015) replaced every max-pooling layer of their CIFAR networks with a stride-2 convolution (or with a stride-1 convolution followed by subsampling) and matched or beat the pooled versions. Most later architectures followed: ResNet downsamples inside its residual stages with stride-2 convolutions and keeps only one max pool in the stem; ConvNeXt has no max pooling at all, using strided "patchify" and downsampling convolutions. Pooling survives where a parameter-free, robust summary is wanted: at the end of the network.

## Global and adaptive average pooling

**Global average pooling** (GAP) averages each channel over all $H\times W$ positions, giving a $C$-dimensional vector: $z_c=\frac{1}{HW}\sum_{u,v}x_{c,u,v}$. Network in Network (Lin et al., 2014) used it to replace the large fully connected layers that dominated AlexNet's and VGG's parameter counts (VGG-16's first FC layer alone has $7\cdot7\cdot512\cdot4096\approx102.8$ M weights). GoogLeNet, ResNet, DenseNet, MobileNet and EfficientNet all end with GAP followed by one linear layer. Because GAP does not care about $H$ and $W$, such networks accept any input resolution. It also makes the last conv layer interpretable: weighting its maps by the classifier weights gives a **class activation map** (Zhou et al., 2016).

$$
\text{AdaptiveAvgPool}(o):\quad \text{bin }i=\Big[\Big\lfloor\frac{i\,n}{o}\Big\rfloor,\;\Big\lceil\frac{(i+1)\,n}{o}\Big\rceil\Big),\qquad i=0,\dots,o-1
$$

- $o$: the requested output size; $o=1$ is global average pooling
- $n$: the input size, any value: the bins adapt so the output is always $o\times o$
- $\lceil\cdot\rceil$: when $o$ does not divide $n$ the bins have unequal sizes and may overlap: $n=8,\ o=3$ gives $[0,3),\ [2,6),\ [5,8)$

Adaptive pooling specifies the **output** size instead of the window. torchvision's ResNet ends with `nn.AdaptiveAvgPool2d((1, 1))` and its VGG with `nn.AdaptiveAvgPool2d((7, 7))`, so both run on images of any size. The idea goes back to spatial pyramid pooling (He et al., 2014), which pooled to several fixed grids at once to get a fixed-length vector from arbitrary inputs.

## Worked example: a 4×4 corner of the figure's map

Take the top-left $4\times4$ block of the default map, $X=\begin{pmatrix}0&0&0&0.5\\0&3.6&0&0.6\\0&0.8&0&0.5\\0.5&0.8&0.2&4.2\end{pmatrix}$, and pool it $2\times2$ with stride 2 (checked in Node).

2×2 / stride-2 pooling of X, window by window

| window (rows, cols) | values | max (argmax) | mean |
| --- | --- | --- | --- |
| (0–1, 0–1) | 0, 0, 0, 3.6 | 3.6 at (1,1) | 0.9 |
| (0–1, 2–3) | 0, 0.5, 0, 0.6 | 0.6 at (1,3) | 0.275 |
| (2–3, 0–1) | 0, 0.8, 0.5, 0.8 | 0.8 at (2,1), a tie with (3,1) | 0.525 |
| (2–3, 2–3) | 0, 0.5, 0.2, 4.2 | 4.2 at (3,3) | 1.225 |

**Backward with upstream gradient G = [[1, 2], [3, 4]]**

1. Max pooling: each $G$ entry goes to its window's argmax, so $\partial L/\partial X$ has $1$ at $(1,1)$, $2$ at $(1,3)$, $3$ at $(2,1)$, $4$ at $(3,3)$ and 0 at the other 12 inputs.
2. The tie in the bottom-left window is broken by scan order: PyTorch keeps the first maximum, $(2,1)$, so $(3,1)$ gets nothing even though it has the same value.
3. Average pooling: every input in window $(a,b)$ gets $G_{ab}/4$: $0.25$ for the top-left block, $0.5$, $0.75$ and $1$ for the others. All 16 inputs receive gradient.
4. Global average pooling of $X$ is $11.7/16=0.731$, and each input gets $1/16$ of the upstream gradient.

```python
import torch, torch.nn as nn, torch.nn.functional as F

X = torch.tensor([[0, 0, 0, .5], [0, 3.6, 0, .6], [0, .8, 0, .5], [.5, .8, .2, 4.2]]).view(1, 1, 4, 4)
X.requires_grad_(True)

y, idx = F.max_pool2d(X, kernel_size=2, stride=2, return_indices=True)
print(y.flatten().tolist(), idx.flatten().tolist())   # ≈ [3.6, 0.6, 0.8, 4.2] [5, 7, 9, 15]  (flat argmax positions)
y.backward(torch.tensor([[1., 2], [3, 4]]).view(1, 1, 2, 2))
print(X.grad.view(4, 4))                              # 1 at (1,1), 2 at (1,3), 3 at (2,1), 4 at (3,3), zeros elsewhere

print(F.avg_pool2d(X, 2).flatten().tolist())          # ≈ [0.9, 0.275, 0.525, 1.225]
print(F.adaptive_avg_pool2d(X, 1).item())             # ≈ 0.73125  (global average pooling)

# Common configurations
alex_pool = nn.MaxPool2d(kernel_size=3, stride=2)              # overlapping (AlexNet)
resnet_pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) # 112 -> 56 (ResNet stem)
vgg_pool = nn.MaxPool2d(2)                                     # stride defaults to kernel_size
unpool = nn.MaxUnpool2d(2)                                     # uses the saved indices

# Strided conv instead of pooling (all-convolutional net): learnable downsampling
down = nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1)

# GAP head: works for any input resolution
head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(2048, 1000))
for hw in [(7, 7), (10, 13)]:
    print(head(torch.randn(2, 2048, *hw)).shape)      # torch.Size([2, 1000]) both times
```

Max, average and adaptive pooling, the routed gradient, and a GAP classifier head

> **Pooling away what you need:** Every pooling stage discards position. That is fine for classification, but detection, segmentation and pose estimation need it back, which is why their decoders use skip connections, unpooling with saved indices (SegNet) or **Transposed convolution**. Pooling too early is the classic mistake: a $2\times2$ pool directly on a small $32\times32$ input throws away three quarters of the pixels before any feature has been computed.

> **Average pooling with padding:** `nn.AvgPool2d(3, stride=1, padding=1)` counts the padded zeros in the denominator by default (`count_include_pad=True`), so border outputs are biased toward 0. Set `count_include_pad=False` to average only real pixels. Max pooling pads with $-\infty$ instead, so padding never wins.

> **Why max pooling regularises a little:** Because only the winning input of each window gets gradient, max pooling acts like a data-dependent sparse gate, similar to ReLU. Each example trains a different subset of positions, and small shifts of the input change which subset. Variants make this explicit: stochastic pooling (Zeiler and Fergus, 2013) samples the winner in proportion to the activations, and fractional max pooling (Graham, 2014) uses random, non-integer pooling ratios.

**Strengths**

- No parameters and very cheap; cuts the next layers' compute by 4× per 2× downsampling
- Local tolerance to small shifts and deformations
- Global average pooling replaces huge FC layers and makes the network resolution-agnostic
- Max pooling keeps the strongest response of sparse ReLU features

**Limitations**

- Fixed, not learned: strided convolutions are usually as good or better for downsampling
- Discards precise position, which dense-prediction tasks must recover
- Subsampling still aliases; invariance is only partial (MaxBlurPool helps)
- Max pooling gives gradient to one input per window; average pooling dilutes strong, sparse responses

**Question:** In a 2×2, stride-2 max pooling layer, the window [[0.1, 0.9], [0.4, 0.3]] receives upstream gradient 2. What gradient does each input get?

- A. 0, 2, 0, 0
- B. 0.5 each
- C. 0.2, 1.8, 0.8, 0.6
- D. 2 each

**Answer:** A. 0, 2, 0, 0 — The output is $\max=0.9$, and $\partial y/\partial x$ is 1 for the argmax and 0 elsewhere. The whole upstream gradient, 2, goes to the 0.9 input. Average pooling would give $2/4=0.5$ to each.

**Question:** Why does torchvision's ResNet-50 accept 320×320 images even though it was trained at 224×224?

- A. Its AdaptiveAvgPool2d((1, 1)) reduces any final map size to one value per channel before the linear layer
- B. Its convolutions resize the input to 224×224 internally
- C. Max pooling layers crop the image to a fixed size
- D. It does not: the linear layer would get the wrong input size

**Answer:** A. Its AdaptiveAvgPool2d((1, 1)) reduces any final map size to one value per channel before the linear layer — Convolutions and pooling work at any resolution; only the final linear layer needs a fixed input size. Global (adaptive) average pooling turns a $2048\times10\times10$ map into the same 2048-vector as a $2048\times7\times7$ one. The accuracy may change with resolution, but the shapes always fit.

**Key points**

- Pooling summarises each k×k window of each channel by its max or mean; no parameters, no channel mixing.
- Backward: max pooling routes the whole gradient to the argmax; average pooling gives 1/k² to every input.
- It gives local, partial shift invariance and cheap downsampling; overlapping 3×3/stride-2 pooling came from AlexNet.
- Strided convolutions can replace pooling (all-convolutional net); modern CNNs keep at most one max pool.
- Global/adaptive average pooling turns any map into a fixed-size vector and replaced large FC heads.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
