Stride and padding

Padding adds a border so the kernel can be centred on edge pixels; stride moves the kernel s pixels at a time. Together they fix the output size ⌊(n+2p−k)/s⌋+1. Stride is cheap downsampling, but without a blur first it aliases and breaks shift equivariance.

12 min read1086 wordscore0 viewsupdated 2026-10-11

Definition

Stride and padding Padding pp extends the input by pp rows and columns on every side before the kernel slides over it, using zeros or values copied from the input. Stride ss is the step between consecutive kernel positions: s=1s=1 visits every position, s=2s=2 every other one. With kernel size kk, an nn-pixel axis gives ⌊(n+2p−k)/s⌋+1\lfloor (n+2p-k)/s\rfloor+1 outputs.

Without padding, every k×kk\times k convolution shrinks the map by k−1k-1 pixels, and border pixels take part in fewer windows than central ones. A 20-layer stack of 3×33\times3 convolutions would eat 40 pixels off each axis. Padding with p=(k−1)/2p=(k-1)/2 keeps the size fixed, which is why VGG and ResNet can stack dozens of layers and still add skip connections whose shapes match. Stride is the opposite tool. It shrinks the map on purpose, by a factor of ss per axis, so later layers are cheaper and each output sees a larger part of the image.

Both ideas are as old as CNNs. LeNet-5 (LeCun et al., 1998) padded its 28×2828\times28 digits to 32×3232\times32 so that stroke end-points and corners near the border could still fall at the centre of the highest-level feature detectors' receptive fields. AlexNet (Krizhevsky et al., 2012) used an 11×1111\times11 kernel with stride 4 in its first layer. Dumoulin and Visin's "A guide to convolution arithmetic for deep learning" (2016) is the standard reference for the shape arithmetic. The downside of striding was made precise later: Azulay and Weiss (2019) showed that a one-pixel shift can change a CNN's prediction, and Zhang (2019, "Making Convolutional Networks Shift-Invariant Again") traced this to aliasing and fixed it with a blur before each subsampling step.

The output-size formula

nout=⌊n+2p−d (k−1)−1s⌋+1  =d=1  ⌊n+2p−ks⌋+1n_{out}=\left\lfloor\frac{n+2p-d\,(k-1)-1}{s}\right\rfloor+1\;\overset{d=1}{=}\;\left\lfloor\frac{n+2p-k}{s}\right\rfloor+1
nn
input size along one axis (height or width; each axis is computed separately)
pp
padding added to each side, so the padded axis has n+2pn+2p entries
kk
kernel size along that axis
ss
stride: the step between window starts
dd
dilation (1 for an ordinary convolution); the kernel then spans d(k−1)+1d(k-1)+1 pixels, see Dilated convolution
⌊⋅⌋\lfloor\cdot\rfloor
windows that would run past the padded edge are dropped: the last (n+2p−k) mod s(n+2p-k)\bmod s rows/columns are never used

The formula is just counting. The first window starts at 0, the last legal start is n+2p−kn+2p-k, and starts are ss apart, so there are ⌊(n+2p−k)/s⌋+1\lfloor (n+2p-k)/s\rfloor+1 of them. When ss does not divide n+2p−kn+2p-k, the leftover rows at the bottom and right are silently ignored. That is harmless for one layer and occasionally a bug: information on the last row may never reach the output.

Named padding schemes

valid
p=0p=0: output n−k+1n-k+1 at stride 1. Only windows that fit entirely inside the input
same
p=(k−1)/2p=(k-1)/2 for odd kk at stride 1: output size = input size. PyTorch's padding="same" only allows stride 1 and pads the extra row on the bottom/right for even kk
full
p=k−1p=k-1: every partial overlap, output n+k−1n+k-1; this is what the input gradient of a valid convolution uses
TensorFlow SAME
output ⌈n/s⌉\lceil n/s\rceil for any stride, padding asymmetrically (extra on the bottom/right). Porting such weights to PyTorch needs the same asymmetric padding (timm's Conv2dSame)

What goes in the border: padding modes

The padded values are a modelling choice. PyTorch's nn.Conv2d offers four via padding_mode. On the row 1,8,6,4,21,8,6,4,2 with p=2p=2 they give:

padding_modepadded rowruletypical use
zeros (default)0 0 | 1 8 6 4 2 | 0 0constant 0almost all classification CNNs
reflect6 8 | 1 8 6 4 2 | 4 6mirror, edge not repeated; needs p < nimage-to-image nets, style transfer, GAN generators
replicate1 1 | 1 8 6 4 2 | 2 2repeat the edge valuesmooth fields, depth/flow maps
circular4 2 | 1 8 6 4 2 | 1 8wrap aroundperiodic data: panoramas, physics on a torus
PyTorch padding modes on one row, p = 2 (the figure's input row 0; checked against the lib's padded())

Zero padding makes the border look like a dark frame. For a smoothing kernel that pulls border outputs toward 0: in the figure's 5×55\times5 input (mean 5.00), a 3×33\times3 box filter with p=1p=1 gives 1.89 at the top-left corner with zeros, versus 4.33 (reflect), 3.67 (replicate) and 5.33 (circular), while the interior stays at 5.00 for every mode. That frame is not just a nuisance. Islam, Jia and Bruce (2020, "How much position information do convolutional neural networks encode?") showed that zero padding lets CNNs learn absolute position, which partly breaks the translation equivariance of the Convolution operation.

padded input 7×7 (7×7 + p=0, zeros)1864297531864297531864297531864297531864297531864output 3×3 (box mean; click)5.04.05.06.05.05.05.05.06.0⌊(n + 2p − k)/s⌋ + 1= ⌊(7 + 0 − 3)/2⌋ + 1 = 3every padded row/col covered'valid': no paddingwindow (0,0) starts at (0,0)box mean there = 5.00torch: padding='same' needs s = 1■ input ┄ padding ■ window □ skipped by stride
padding_mode
k
fig.Geometry view: an n×n input of digits (blue), its padding ring (dashed), one k×k window (orange) and the output of a k×k box filter. The default n = 7, k = 3, s = 2, p = 0 gives ⌊(7 − 3)/2⌋ + 1 = 3, with every row covered. Set n = 8: still 3×3, but one row and column (red outline) are never visited. Set n = 5, k = 3, s = 1, p = 1 ('same') and switch padding_mode: the top-left output is 1.89 with zeros, 4.33 reflect, 3.67 replicate and 5.33 circular, while the interior stays 5.00. Aliasing view: a smooth bump plus an alternating ±a texture, subsampled with stride 2. At a = 0.3 a one-sample input shift changes the output by up to 0.717. With a [1, 2, 1]/4 blur before the stride (BlurPool) the change is 0.115, close to the 0.117 a texture-free signal gives.

Stride: downsampling inside the convolution

A stride-ss convolution computes exactly the stride-1 output and keeps every ss-th row and column, but without wasting work on the discarded positions. Halving each axis cuts the next layer's compute by 4×. That is why CNNs follow the pattern "stride 2, double the channels": the per-layer compute stays roughly constant while the map shrinks from 224224 to 77 over five stages. ResNet does its downsampling with a stride-2 7×77\times7 stem, a stride-2 max pool and a stride-2 convolution at the start of stages 3, 4 and 5, for a total factor of 32. Strided convolution can also replace pooling entirely (Springenberg et al., 2015; see Pooling layers).

Each stride also multiplies the Receptive field growth of every later layer: after a stride-2 layer, one step in the output corresponds to two input pixels, so a following 3×33\times3 kernel covers twice as much of the original image.

Aliasing: why stride breaks shift equivariance

Subsampling with stride ss is sampling. By the Nyquist–Shannon theorem, any frequency above 1/(2s)1/(2s) cycles per pixel cannot be represented after subsampling and aliases: it shows up as a lower frequency that was not in the signal. The extreme case is the alternating pattern +a,−a,+a,…+a,-a,+a,\dots in the figure. Stride 2 keeps only the +a+a samples, so the output gains a constant offset +a+a. Shift the input by one pixel and the kept samples are all −a-a: the output jumps by 2a2a everywhere. A stride-1 convolution would simply have shifted its output; a strided one changes it.

(blur ⋆ x)[2m]⏟BlurPoolblur=14(1,2,1)  ⇒  14(−a+2a−a)=0 for the alternating texture\underbrace{\big(\text{blur}\,\star\,x\big)[2m]}_{\text{BlurPool}}\qquad \text{blur}=\tfrac14(1,2,1)\;\Rightarrow\;\tfrac14(-a+2a-a)=0\ \text{for the alternating texture}
[2m][2m]
keep every second sample (stride 2)
14(1,2,1)\tfrac14(1,2,1)
a binomial low-pass filter; it exactly cancels the highest (Nyquist) frequency

Zhang (2019) applied this classic signal-processing fix inside CNNs. Max pooling becomes MaxBlurPool: a stride-1 max, then a blur, then subsampling. A stride-2 convolution becomes a stride-1 convolution, ReLU, then blur and subsample. On ImageNet the anti-aliased VGG, ResNet, DenseNet and MobileNetV2 models were both more consistent under small shifts and more accurate than their baselines, at a small extra cost. Zeiler and Fergus (2014) had already seen the symptom: aliasing artifacts in AlexNet's second layer, caused by its stride-4 first layer, which they reduced to stride 2.

Worked example: shapes through real layers

layernkspoutputnote
valid 5×5 conv3251028LeNet-5 C1
'same' 3×3 conv224311224VGG, ResNet body
ResNet stem224723112stride 2 halves the map
ResNet max pool11232156total stride 4 after the stem
AlexNet conv1 (as described)224114054⌊213/4⌋ + 1: not the 55 the paper reports
AlexNet conv1 (227 input)227114055(227 − 11)/4 + 1 = 55 exactly
torchvision AlexNet conv1224114255padding 2 fixes it
figure, n = 883203(8 − 3) mod 2 = 1 row never used
Output sizes from ⌊(n + 2p − k)/s⌋ + 1 (computed with the lib's outSize())

Designing a 'same' stride-2 layer

  1. 1Goal: halve a 56×5656\times56 map with a 3×33\times3 kernel. We want ⌊(56+2p−3)/2⌋+1=28\lfloor (56+2p-3)/2\rfloor+1=28.
  2. 2p=0p=0 gives ⌊53/2⌋+1=27\lfloor 53/2\rfloor+1=27. One pixel short.
  3. 3p=1p=1 gives ⌊55/2⌋+1=28\lfloor 55/2\rfloor+1=28. This is ResNet's choice: p=(k−1)/2p=(k-1)/2 with stride 2 gives ⌈n/2⌉\lceil n/2\rceil.
  4. 4padding="same" would raise an error here, because PyTorch only supports it for stride 1. Write padding=1 explicitly.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def out_size(n, k, s=1, p=0, d=1):
4 return (n + 2 * p - d * (k - 1) - 1) // s + 1
5
6x = torch.randn(1, 3, 224, 224)
7stem = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
8print(stem(x).shape, out_size(224, 7, 2, 3)) # (1, 64, 112, 112) 112
9
10same = nn.Conv2d(64, 64, 3, padding="same") # stride 1 only
11# nn.Conv2d(64, 64, 3, stride=2, padding="same") # ValueError: not supported for strided convs
12down = nn.Conv2d(64, 128, 3, stride=2, padding=1) # 56 -> 28
13
14row = torch.tensor([[[1., 8, 6, 4, 2]]]) # (N, C, W)
15for mode in ["constant", "reflect", "replicate", "circular"]:
16 print(mode, F.pad(row, (2, 2), mode=mode).flatten().tolist())
17# constant [0, 0, 1, 8, 6, 4, 2, 0, 0]
18# reflect [6, 8, 1, 8, 6, 4, 2, 4, 6]
19# replicate [1, 1, 1, 8, 6, 4, 2, 2, 2]
20# circular [4, 2, 1, 8, 6, 4, 2, 1, 8]
21conv_reflect = nn.Conv2d(3, 16, 3, padding=1, padding_mode="reflect")
22
23class BlurPool2d(nn.Module):
24 """Anti-aliased stride-2 downsampling (Zhang, 2019): blur with [1,2,1]x[1,2,1]/16, then subsample."""
25 def __init__(self, channels):
26 super().__init__()
27 f = torch.tensor([1., 2, 1])
28 k = (f[:, None] * f[None, :]) / 16
29 self.register_buffer("k", k.expand(channels, 1, 3, 3).clone())
30 self.channels = channels
31 def forward(self, x):
32 x = F.pad(x, (1, 1, 1, 1), mode="reflect")
33 return F.conv2d(x, self.k, stride=2, groups=self.channels)
34
35# MaxBlurPool = dense max, then blur + subsample
36max_blur_pool = nn.Sequential(nn.MaxPool2d(kernel_size=2, stride=1), BlurPool2d(64))
Shape arithmetic, padding modes, the 'same' restriction and a minimal BlurPool

Use when

  • +'Same' padding keeps shapes fixed, so layers stack and residual additions line up
  • +Stride 2 cuts the next layers' compute by 4× and grows the receptive field quickly
  • +Reflect, replicate and circular modes avoid the dark-frame artifact for image-to-image tasks

Avoid when

  • −Zero padding leaks absolute position and distorts border outputs
  • −Stride without a low-pass filter aliases: small input shifts can change outputs and predictions
  • −The floor in the formula silently drops rows and causes off-by-one mismatches
  • −PyTorch's padding="same" is unavailable for strided layers; TF-style 'SAME' needs asymmetric padding

Check yourself

Input 32×32, kernel 5×5, stride 2, padding 2. What is the output size?

Check yourself

A feature map contains a fine checkerboard texture (alternating +a, −a). It goes through a stride-2 convolution with no blur. What happens when the input is shifted by one pixel?

Remember

  • ✓Output size ⌊(n+2p−k)/s⌋+1\lfloor (n+2p-k)/s\rfloor+1 per axis; the last (n+2p−k) mod s(n+2p-k)\bmod s rows are dropped.
  • ✓'valid' is p=0p=0, 'same' is p=(k−1)/2p=(k-1)/2 at stride 1; PyTorch's padding="same" refuses stride >1>1.
  • ✓padding_mode: zeros (default), reflect (p < n), replicate, circular; zeros leak absolute position.
  • ✓Stride s is downsampling inside the conv: s² less compute downstream and faster receptive-field growth.
  • ✓Subsampling without a low-pass filter aliases; blur before stride (BlurPool) restores near shift-equivariance.
← back to Deep Learning roadmap