Definition
Stride and padding Padding extends the input by rows and columns on every side before the kernel slides over it, using zeros or values copied from the input. Stride is the step between consecutive kernel positions: visits every position, every other one. With kernel size , an -pixel axis gives outputs.
Without padding, every convolution shrinks the map by pixels, and border pixels take part in fewer windows than central ones. A 20-layer stack of convolutions would eat 40 pixels off each axis. Padding with keeps the size fixed, which is why VGG and ResNet can stack dozens of layers and still add skip connections whose shapes match. Stride is the opposite tool. It shrinks the map on purpose, by a factor of per axis, so later layers are cheaper and each output sees a larger part of the image.
Both ideas are as old as CNNs. LeNet-5 (LeCun et al., 1998) padded its digits to so that stroke end-points and corners near the border could still fall at the centre of the highest-level feature detectors' receptive fields. AlexNet (Krizhevsky et al., 2012) used an kernel with stride 4 in its first layer. Dumoulin and Visin's "A guide to convolution arithmetic for deep learning" (2016) is the standard reference for the shape arithmetic. The downside of striding was made precise later: Azulay and Weiss (2019) showed that a one-pixel shift can change a CNN's prediction, and Zhang (2019, "Making Convolutional Networks Shift-Invariant Again") traced this to aliasing and fixed it with a blur before each subsampling step.
The output-size formula
- input size along one axis (height or width; each axis is computed separately)
- padding added to each side, so the padded axis has entries
- kernel size along that axis
- stride: the step between window starts
- dilation (1 for an ordinary convolution); the kernel then spans pixels, see Dilated convolution
- windows that would run past the padded edge are dropped: the last rows/columns are never used
The formula is just counting. The first window starts at 0, the last legal start is , and starts are apart, so there are of them. When does not divide , the leftover rows at the bottom and right are silently ignored. That is harmless for one layer and occasionally a bug: information on the last row may never reach the output.
Named padding schemes
- valid
- : output at stride 1. Only windows that fit entirely inside the input
- same
- for odd at stride 1: output size = input size. PyTorch's
padding="same"only allows stride 1 and pads the extra row on the bottom/right for even - full
- : every partial overlap, output ; this is what the input gradient of a valid convolution uses
- TensorFlow SAME
- output for any stride, padding asymmetrically (extra on the bottom/right). Porting such weights to PyTorch needs the same asymmetric padding (timm's
Conv2dSame)
What goes in the border: padding modes
The padded values are a modelling choice. PyTorch's nn.Conv2d offers four via padding_mode. On the row with they give:
| padding_mode | padded row | rule | typical use |
|---|---|---|---|
| zeros (default) | 0 0 | 1 8 6 4 2 | 0 0 | constant 0 | almost all classification CNNs |
| reflect | 6 8 | 1 8 6 4 2 | 4 6 | mirror, edge not repeated; needs p < n | image-to-image nets, style transfer, GAN generators |
| replicate | 1 1 | 1 8 6 4 2 | 2 2 | repeat the edge value | smooth fields, depth/flow maps |
| circular | 4 2 | 1 8 6 4 2 | 1 8 | wrap around | periodic data: panoramas, physics on a torus |
Zero padding makes the border look like a dark frame. For a smoothing kernel that pulls border outputs toward 0: in the figure's input (mean 5.00), a box filter with gives 1.89 at the top-left corner with zeros, versus 4.33 (reflect), 3.67 (replicate) and 5.33 (circular), while the interior stays at 5.00 for every mode. That frame is not just a nuisance. Islam, Jia and Bruce (2020, "How much position information do convolutional neural networks encode?") showed that zero padding lets CNNs learn absolute position, which partly breaks the translation equivariance of the Convolution operation.
Stride: downsampling inside the convolution
A stride- convolution computes exactly the stride-1 output and keeps every -th row and column, but without wasting work on the discarded positions. Halving each axis cuts the next layer's compute by 4×. That is why CNNs follow the pattern "stride 2, double the channels": the per-layer compute stays roughly constant while the map shrinks from to over five stages. ResNet does its downsampling with a stride-2 stem, a stride-2 max pool and a stride-2 convolution at the start of stages 3, 4 and 5, for a total factor of 32. Strided convolution can also replace pooling entirely (Springenberg et al., 2015; see Pooling layers).
Each stride also multiplies the Receptive field growth of every later layer: after a stride-2 layer, one step in the output corresponds to two input pixels, so a following kernel covers twice as much of the original image.
Aliasing: why stride breaks shift equivariance
Subsampling with stride is sampling. By the Nyquist–Shannon theorem, any frequency above cycles per pixel cannot be represented after subsampling and aliases: it shows up as a lower frequency that was not in the signal. The extreme case is the alternating pattern in the figure. Stride 2 keeps only the samples, so the output gains a constant offset . Shift the input by one pixel and the kept samples are all : the output jumps by everywhere. A stride-1 convolution would simply have shifted its output; a strided one changes it.
- keep every second sample (stride 2)
- a binomial low-pass filter; it exactly cancels the highest (Nyquist) frequency
Zhang (2019) applied this classic signal-processing fix inside CNNs. Max pooling becomes MaxBlurPool: a stride-1 max, then a blur, then subsampling. A stride-2 convolution becomes a stride-1 convolution, ReLU, then blur and subsample. On ImageNet the anti-aliased VGG, ResNet, DenseNet and MobileNetV2 models were both more consistent under small shifts and more accurate than their baselines, at a small extra cost. Zeiler and Fergus (2014) had already seen the symptom: aliasing artifacts in AlexNet's second layer, caused by its stride-4 first layer, which they reduced to stride 2.
Worked example: shapes through real layers
| layer | n | k | s | p | output | note |
|---|---|---|---|---|---|---|
| valid 5×5 conv | 32 | 5 | 1 | 0 | 28 | LeNet-5 C1 |
| 'same' 3×3 conv | 224 | 3 | 1 | 1 | 224 | VGG, ResNet body |
| ResNet stem | 224 | 7 | 2 | 3 | 112 | stride 2 halves the map |
| ResNet max pool | 112 | 3 | 2 | 1 | 56 | total stride 4 after the stem |
| AlexNet conv1 (as described) | 224 | 11 | 4 | 0 | 54 | ⌊213/4⌋ + 1: not the 55 the paper reports |
| AlexNet conv1 (227 input) | 227 | 11 | 4 | 0 | 55 | (227 − 11)/4 + 1 = 55 exactly |
| torchvision AlexNet conv1 | 224 | 11 | 4 | 2 | 55 | padding 2 fixes it |
| figure, n = 8 | 8 | 3 | 2 | 0 | 3 | (8 − 3) mod 2 = 1 row never used |
Designing a 'same' stride-2 layer
- 1Goal: halve a map with a kernel. We want .
- 2 gives . One pixel short.
- 3 gives . This is ResNet's choice: with stride 2 gives .
- 4
padding="same"would raise an error here, because PyTorch only supports it for stride 1. Writepadding=1explicitly.
1import torch, torch.nn as nn, torch.nn.functional as F23def out_size(n, k, s=1, p=0, d=1):4 return (n + 2 * p - d * (k - 1) - 1) // s + 156x = torch.randn(1, 3, 224, 224)7stem = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)8print(stem(x).shape, out_size(224, 7, 2, 3)) # (1, 64, 112, 112) 112910same = nn.Conv2d(64, 64, 3, padding="same") # stride 1 only11# nn.Conv2d(64, 64, 3, stride=2, padding="same") # ValueError: not supported for strided convs12down = nn.Conv2d(64, 128, 3, stride=2, padding=1) # 56 -> 281314row = torch.tensor([[[1., 8, 6, 4, 2]]]) # (N, C, W)15for mode in ["constant", "reflect", "replicate", "circular"]:16 print(mode, F.pad(row, (2, 2), mode=mode).flatten().tolist())17# constant [0, 0, 1, 8, 6, 4, 2, 0, 0]18# reflect [6, 8, 1, 8, 6, 4, 2, 4, 6]19# replicate [1, 1, 1, 8, 6, 4, 2, 2, 2]20# circular [4, 2, 1, 8, 6, 4, 2, 1, 8]21conv_reflect = nn.Conv2d(3, 16, 3, padding=1, padding_mode="reflect")2223class BlurPool2d(nn.Module):24 """Anti-aliased stride-2 downsampling (Zhang, 2019): blur with [1,2,1]x[1,2,1]/16, then subsample."""25 def __init__(self, channels):26 super().__init__()27 f = torch.tensor([1., 2, 1])28 k = (f[:, None] * f[None, :]) / 1629 self.register_buffer("k", k.expand(channels, 1, 3, 3).clone())30 self.channels = channels31 def forward(self, x):32 x = F.pad(x, (1, 1, 1, 1), mode="reflect")33 return F.conv2d(x, self.k, stride=2, groups=self.channels)3435# MaxBlurPool = dense max, then blur + subsample36max_blur_pool = nn.Sequential(nn.MaxPool2d(kernel_size=2, stride=1), BlurPool2d(64))Use when
- +'Same' padding keeps shapes fixed, so layers stack and residual additions line up
- +Stride 2 cuts the next layers' compute by 4× and grows the receptive field quickly
- +Reflect, replicate and circular modes avoid the dark-frame artifact for image-to-image tasks
Avoid when
- −Zero padding leaks absolute position and distorts border outputs
- −Stride without a low-pass filter aliases: small input shifts can change outputs and predictions
- −The floor in the formula silently drops rows and causes off-by-one mismatches
- −PyTorch's
padding="same"is unavailable for strided layers; TF-style 'SAME' needs asymmetric padding
Check yourself
Input 32×32, kernel 5×5, stride 2, padding 2. What is the output size?
Check yourself
A feature map contains a fine checkerboard texture (alternating +a, −a). It goes through a stride-2 convolution with no blur. What happens when the input is shifted by one pixel?
Remember
- ✓Output size per axis; the last rows are dropped.
- ✓'valid' is , 'same' is at stride 1; PyTorch's
padding="same"refuses stride . - ✓padding_mode: zeros (default), reflect (p < n), replicate, circular; zeros leak absolute position.
- ✓Stride s is downsampling inside the conv: s² less compute downstream and faster receptive-field growth.
- ✓Subsampling without a low-pass filter aliases; blur before stride (BlurPool) restores near shift-equivariance.