# Stride and padding

> Padding adds a border so the kernel can be centred on edge pixels; stride moves the kernel s pixels at a time. Together they fix the output size ⌊(n+2p−k)/s⌋+1. Stride is cheap downsampling, but without a blur first it aliases and breaks shift equivariance.

Source: https://www.mlfoundry.in/learn/stride-and-padding  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 12 min read · updated 2026-10-11

**Stride and padding:** **Padding** $p$ extends the input by $p$ rows and columns on every side before the kernel slides over it, using zeros or values copied from the input. **Stride** $s$ is the step between consecutive kernel positions: $s=1$ visits every position, $s=2$ every other one. With kernel size $k$, an $n$-pixel axis gives $\lfloor (n+2p-k)/s\rfloor+1$ outputs.

Without padding, every $k\times k$ convolution shrinks the map by $k-1$ pixels, and border pixels take part in fewer windows than central ones. A 20-layer stack of $3\times3$ convolutions would eat 40 pixels off each axis. Padding with $p=(k-1)/2$ keeps the size fixed, which is why VGG and ResNet can stack dozens of layers and still add skip connections whose shapes match. Stride is the opposite tool. It shrinks the map on purpose, by a factor of $s$ per axis, so later layers are cheaper and each output sees a larger part of the image.

Both ideas are as old as CNNs. LeNet-5 (LeCun et al., 1998) padded its $28\times28$ digits to $32\times32$ so that stroke end-points and corners near the border could still fall at the centre of the highest-level feature detectors' receptive fields. AlexNet (Krizhevsky et al., 2012) used an $11\times11$ kernel with stride 4 in its first layer. Dumoulin and Visin's "A guide to convolution arithmetic for deep learning" (2016) is the standard reference for the shape arithmetic. The downside of striding was made precise later: Azulay and Weiss (2019) showed that a one-pixel shift can change a CNN's prediction, and Zhang (2019, "Making Convolutional Networks Shift-Invariant Again") traced this to aliasing and fixed it with a blur before each subsampling step.

## The output-size formula

$$
n_{out}=\left\lfloor\frac{n+2p-d\,(k-1)-1}{s}\right\rfloor+1\;\overset{d=1}{=}\;\left\lfloor\frac{n+2p-k}{s}\right\rfloor+1
$$

- $n$: input size along one axis (height or width; each axis is computed separately)
- $p$: padding added to **each** side, so the padded axis has $n+2p$ entries
- $k$: kernel size along that axis
- $s$: stride: the step between window starts
- $d$: dilation (1 for an ordinary convolution); the kernel then spans $d(k-1)+1$ pixels, see **Dilated convolution**
- $\lfloor\cdot\rfloor$: windows that would run past the padded edge are dropped: the last $(n+2p-k)\bmod s$ rows/columns are never used

The formula is just counting. The first window starts at 0, the last legal start is $n+2p-k$, and starts are $s$ apart, so there are $\lfloor (n+2p-k)/s\rfloor+1$ of them. When $s$ does not divide $n+2p-k$, the leftover rows at the bottom and right are silently ignored. That is harmless for one layer and occasionally a bug: information on the last row may never reach the output.

**Named padding schemes**

- **valid:** $p=0$: output $n-k+1$ at stride 1. Only windows that fit entirely inside the input
- **same:** $p=(k-1)/2$ for odd $k$ at stride 1: output size = input size. PyTorch's `padding="same"` only allows stride 1 and pads the extra row on the bottom/right for even $k$
- **full:** $p=k-1$: every partial overlap, output $n+k-1$; this is what the input gradient of a valid convolution uses
- **TensorFlow SAME:** output $\lceil n/s\rceil$ for any stride, padding asymmetrically (extra on the bottom/right). Porting such weights to PyTorch needs the same asymmetric padding (timm's `Conv2dSame`)

## What goes in the border: padding modes

The padded values are a modelling choice. PyTorch's `nn.Conv2d` offers four via `padding_mode`. On the row $1,8,6,4,2$ with $p=2$ they give:

PyTorch padding modes on one row, p = 2 (the figure's input row 0; checked against the lib's padded())

| padding_mode | padded row | rule | typical use |
| --- | --- | --- | --- |
| zeros (default) | 0 0 \| 1 8 6 4 2 \| 0 0 | constant 0 | almost all classification CNNs |
| reflect | 6 8 \| 1 8 6 4 2 \| 4 6 | mirror, edge not repeated; needs p < n | image-to-image nets, style transfer, GAN generators |
| replicate | 1 1 \| 1 8 6 4 2 \| 2 2 | repeat the edge value | smooth fields, depth/flow maps |
| circular | 4 2 \| 1 8 6 4 2 \| 1 8 | wrap around | periodic data: panoramas, physics on a torus |

Zero padding makes the border look like a dark frame. For a smoothing kernel that pulls border outputs toward 0: in the figure's $5\times5$ input (mean 5.00), a $3\times3$ box filter with $p=1$ gives 1.89 at the top-left corner with zeros, versus 4.33 (reflect), 3.67 (replicate) and 5.33 (circular), while the interior stays at 5.00 for every mode. That frame is not just a nuisance. Islam, Jia and Bruce (2020, "How much position information do convolutional neural networks encode?") showed that zero padding lets CNNs learn **absolute position**, which partly breaks the translation equivariance of the **Convolution operation**.

> Figure: Geometry view: an n×n input of digits (blue), its padding ring (dashed), one k×k window (orange) and the output of a k×k box filter. The default n = 7, k = 3, s = 2, p = 0 gives ⌊(7 − 3)/2⌋ + 1 = 3, with every row covered. Set n = 8: still 3×3, but one row and column (red outline) are never visited. Set n = 5, k = 3, s = 1, p = 1 ('same') and switch padding_mode: the top-left output is 1.89 with zeros, 4.33 reflect, 3.67 replicate and 5.33 circular, while the interior stays 5.00. Aliasing view: a smooth bump plus an alternating ±a texture, subsampled with stride 2. At a = 0.3 a one-sample input shift changes the output by up to 0.717. With a [1, 2, 1]/4 blur before the stride (BlurPool) the change is 0.115, close to the 0.117 a texture-free signal gives.

## Stride: downsampling inside the convolution

A stride-$s$ convolution computes exactly the stride-1 output and keeps every $s$-th row and column, but without wasting work on the discarded positions. Halving each axis cuts the next layer's compute by 4×. That is why CNNs follow the pattern "stride 2, double the channels": the per-layer compute stays roughly constant while the map shrinks from $224$ to $7$ over five stages. ResNet does its downsampling with a stride-2 $7\times7$ stem, a stride-2 max pool and a stride-2 convolution at the start of stages 3, 4 and 5, for a total factor of 32. Strided convolution can also replace pooling entirely (Springenberg et al., 2015; see **Pooling layers**).

Each stride also multiplies the **Receptive field** growth of every later layer: after a stride-2 layer, one step in the output corresponds to two input pixels, so a following $3\times3$ kernel covers twice as much of the original image.

## Aliasing: why stride breaks shift equivariance

Subsampling with stride $s$ is sampling. By the Nyquist–Shannon theorem, any frequency above $1/(2s)$ cycles per pixel cannot be represented after subsampling and **aliases**: it shows up as a lower frequency that was not in the signal. The extreme case is the alternating pattern $+a,-a,+a,\dots$ in the figure. Stride 2 keeps only the $+a$ samples, so the output gains a constant offset $+a$. Shift the input by one pixel and the kept samples are all $-a$: the output jumps by $2a$ everywhere. A stride-1 convolution would simply have shifted its output; a strided one changes it.

$$
\underbrace{\big(\text{blur}\,\star\,x\big)[2m]}_{\text{BlurPool}}\qquad \text{blur}=\tfrac14(1,2,1)\;\Rightarrow\;\tfrac14(-a+2a-a)=0\ \text{for the alternating texture}
$$

- $[2m]$: keep every second sample (stride 2)
- $\tfrac14(1,2,1)$: a binomial low-pass filter; it exactly cancels the highest (Nyquist) frequency

Zhang (2019) applied this classic signal-processing fix inside CNNs. Max pooling becomes **MaxBlurPool**: a stride-1 max, then a blur, then subsampling. A stride-2 convolution becomes a stride-1 convolution, ReLU, then blur and subsample. On ImageNet the anti-aliased VGG, ResNet, DenseNet and MobileNetV2 models were both more consistent under small shifts and more accurate than their baselines, at a small extra cost. Zeiler and Fergus (2014) had already seen the symptom: aliasing artifacts in AlexNet's second layer, caused by its stride-4 first layer, which they reduced to stride 2.

> **Equivariance holds only up to the stride:** A network with total stride 32 is equivariant to shifts of 32 pixels, not 1. In between, the output depends on where the object falls relative to the sampling grid. Pooling and global averaging give some robustness, but blurring before every subsampling is what actually removes the high frequencies that cause the jumps.

## Worked example: shapes through real layers

Output sizes from ⌊(n + 2p − k)/s⌋ + 1 (computed with the lib's outSize())

| layer | n | k | s | p | output | note |
| --- | --- | --- | --- | --- | --- | --- |
| valid 5×5 conv | 32 | 5 | 1 | 0 | 28 | LeNet-5 C1 |
| 'same' 3×3 conv | 224 | 3 | 1 | 1 | 224 | VGG, ResNet body |
| ResNet stem | 224 | 7 | 2 | 3 | 112 | stride 2 halves the map |
| ResNet max pool | 112 | 3 | 2 | 1 | 56 | total stride 4 after the stem |
| AlexNet conv1 (as described) | 224 | 11 | 4 | 0 | 54 | ⌊213/4⌋ + 1: not the 55 the paper reports |
| AlexNet conv1 (227 input) | 227 | 11 | 4 | 0 | 55 | (227 − 11)/4 + 1 = 55 exactly |
| torchvision AlexNet conv1 | 224 | 11 | 4 | 2 | 55 | padding 2 fixes it |
| figure, n = 8 | 8 | 3 | 2 | 0 | 3 | (8 − 3) mod 2 = 1 row never used |

**Designing a 'same' stride-2 layer**

1. Goal: halve a $56\times56$ map with a $3\times3$ kernel. We want $\lfloor (56+2p-3)/2\rfloor+1=28$.
2. $p=0$ gives $\lfloor 53/2\rfloor+1=27$. One pixel short.
3. $p=1$ gives $\lfloor 55/2\rfloor+1=28$. This is ResNet's choice: $p=(k-1)/2$ with stride 2 gives $\lceil n/2\rceil$.
4. `padding="same"` would raise an error here, because PyTorch only supports it for stride 1. Write `padding=1` explicitly.

```python
import torch, torch.nn as nn, torch.nn.functional as F

def out_size(n, k, s=1, p=0, d=1):
    return (n + 2 * p - d * (k - 1) - 1) // s + 1

x = torch.randn(1, 3, 224, 224)
stem = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
print(stem(x).shape, out_size(224, 7, 2, 3))                 # (1, 64, 112, 112) 112

same = nn.Conv2d(64, 64, 3, padding="same")                  # stride 1 only
# nn.Conv2d(64, 64, 3, stride=2, padding="same")            # ValueError: not supported for strided convs
down = nn.Conv2d(64, 128, 3, stride=2, padding=1)            # 56 -> 28

row = torch.tensor([[[1., 8, 6, 4, 2]]])                     # (N, C, W)
for mode in ["constant", "reflect", "replicate", "circular"]:
    print(mode, F.pad(row, (2, 2), mode=mode).flatten().tolist())
# constant  [0, 0, 1, 8, 6, 4, 2, 0, 0]
# reflect   [6, 8, 1, 8, 6, 4, 2, 4, 6]
# replicate [1, 1, 1, 8, 6, 4, 2, 2, 2]
# circular  [4, 2, 1, 8, 6, 4, 2, 1, 8]
conv_reflect = nn.Conv2d(3, 16, 3, padding=1, padding_mode="reflect")

class BlurPool2d(nn.Module):
    """Anti-aliased stride-2 downsampling (Zhang, 2019): blur with [1,2,1]x[1,2,1]/16, then subsample."""
    def __init__(self, channels):
        super().__init__()
        f = torch.tensor([1., 2, 1])
        k = (f[:, None] * f[None, :]) / 16
        self.register_buffer("k", k.expand(channels, 1, 3, 3).clone())
        self.channels = channels
    def forward(self, x):
        x = F.pad(x, (1, 1, 1, 1), mode="reflect")
        return F.conv2d(x, self.k, stride=2, groups=self.channels)

# MaxBlurPool = dense max, then blur + subsample
max_blur_pool = nn.Sequential(nn.MaxPool2d(kernel_size=2, stride=1), BlurPool2d(64))
```

Shape arithmetic, padding modes, the 'same' restriction and a minimal BlurPool

> **Off-by-one shapes:** Most "size mismatch" errors in U-Nets and FPNs come from the floor in the formula. A $57\times57$ map downsampled with stride 2 gives 29, and upsampling 29 by 2 gives 58, which no longer matches the skip connection. Use input sizes divisible by the network's total stride (32 for most classifiers), or crop/pad explicitly before concatenating. Also note that `padding_mode="reflect"` requires $p<n$ on every axis.

> **Gradients through stride and padding:** The input gradient of a stride-$s$ convolution is a transposed convolution with stride $s$: the upstream gradient is spread out with $s-1$ zeros between entries before being convolved with the flipped kernel (see **Transposed convolution**). Gradients that reach zero-padding cells are discarded. With reflect, replicate or circular padding they are added back onto the input pixels that were copied into the border.

**Strengths**

- 'Same' padding keeps shapes fixed, so layers stack and residual additions line up
- Stride 2 cuts the next layers' compute by 4× and grows the receptive field quickly
- Reflect, replicate and circular modes avoid the dark-frame artifact for image-to-image tasks

**Limitations**

- Zero padding leaks absolute position and distorts border outputs
- Stride without a low-pass filter aliases: small input shifts can change outputs and predictions
- The floor in the formula silently drops rows and causes off-by-one mismatches
- PyTorch's `padding="same"` is unavailable for strided layers; TF-style 'SAME' needs asymmetric padding

**Question:** Input 32×32, kernel 5×5, stride 2, padding 2. What is the output size?

- A. 16×16
- B. 15×15
- C. 14×14
- D. 32×32

**Answer:** A. 16×16 — $\lfloor (32+4-5)/2\rfloor+1=\lfloor 31/2\rfloor+1=15+1=16$. With $p=(k-1)/2$ and stride 2, the output is $\lceil n/2\rceil$.

**Question:** A feature map contains a fine checkerboard texture (alternating +a, −a). It goes through a stride-2 convolution with no blur. What happens when the input is shifted by one pixel?

- A. The texture aliases to a constant offset, and the shift flips its sign: the output changes by about 2a
- B. The output shifts by half a pixel
- C. Nothing: convolutions are shift-equivariant at any stride
- D. The texture is removed automatically by the stride

**Answer:** A. The texture aliases to a constant offset, and the shift flips its sign: the output changes by about 2a — Stride 2 keeps one phase of the alternating pattern, so it looks like a constant $+a$. After a one-pixel shift it keeps the other phase, $-a$. That is aliasing: frequencies above the new Nyquist limit $1/(2s)$ fold into low frequencies. A $\tfrac14(1,2,1)$ blur before subsampling cancels the pattern (BlurPool).

**Key points**

- Output size $\lfloor (n+2p-k)/s\rfloor+1$ per axis; the last $(n+2p-k)\bmod s$ rows are dropped.
- 'valid' is $p=0$, 'same' is $p=(k-1)/2$ at stride 1; PyTorch's `padding="same"` refuses stride $>1$.
- padding_mode: zeros (default), reflect (p < n), replicate, circular; zeros leak absolute position.
- Stride s is downsampling inside the conv: s² less compute downstream and faster receptive-field growth.
- Subsampling without a low-pass filter aliases; blur before stride (BlurPool) restores near shift-equivariance.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
