# Convolution operation

> Slide one small kernel over every position of the input and take a weighted sum over its window and all input channels. The same weights are reused everywhere, so the layer is local, has few parameters, and shifts its output when the input shifts.

Source: https://www.mlfoundry.in/learn/convolution-operation  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 14 min read · updated 2026-10-11

**Convolution layer:** A convolution layer maps an input with $C_{in}$ channels to $C_{out}$ output maps. Each output value is a bias plus a weighted sum of the $k\times k$ window under it, taken across **all** input channels: $y_{o,i,j}=b_o+\sum_{c}\sum_{m,n}K_{o,c,m,n}\,x_{c,\,i+m,\,j+n}$. The same kernel $K_{o}$ is used at every position $(i,j)$.

A fully connected layer from a $224\times224\times3$ image to 1000 units needs $150{,}528\times1000\approx150$ million weights. It also has to learn the same edge detector separately at every location. A convolution makes two assumptions about images instead. **Locality**: an output depends only on a small neighbourhood. **Stationarity**: a pattern worth detecting in one place is worth detecting everywhere. Under those two assumptions a layer of 64 detectors of size $3\times3\times3$ needs $64\cdot27+64=1{,}792$ parameters, whatever the image size.

The idea comes from vision science and early neural networks. Hubel and Wiesel (1959, 1962) found cells in the cat visual cortex that respond to oriented edges inside a small receptive field. Fukushima's **neocognitron** (1980) stacked layers of such local, weight-shared detectors. LeCun et al. (1989, "Backpropagation applied to handwritten zip code recognition") trained the shared kernels end to end with backpropagation, which led to **LeNet-5** (LeCun, Bottou, Bengio and Haffner, 1998). Every CNN since, from **AlexNet** to **ConvNeXt**, is built from this operation.

## Formula: cross-correlation, not convolution

$$
y_{o,i,j}=b_o+\sum_{c=0}^{C_{in}-1}\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}K_{o,c,m,n}\;x_{c,\;i+m,\;j+n}
$$

- $x_{c,i,j}$: input: channel $c$, row $i$, column $j$ (shape $C_{in}\times H\times W$)
- $K_{o,c,m,n}$: kernel weights, tensor of shape $(C_{out},C_{in},k,k)$: one $k\times k$ slice per (output, input) channel pair
- $b_o$: one bias per output channel, shared by all positions
- $y_{o,i,j}$: output map $o$ at position $(i,j)$; with no padding and stride 1 it has size $(H-k+1)\times(W-k+1)$

This is a **cross-correlation**: the window and the kernel are aligned index by index. The convolution of signal processing flips the kernel first, $(x*K)_{i,j}=\sum_{m,n}K_{m,n}\,x_{i-m,\,j-n}$, which is the same as cross-correlating with the kernel rotated by 180°. Deep learning libraries (`torch.nn.functional.conv2d`, cuDNN, `jax.lax.conv`) compute the unflipped version and still call it convolution. For learned kernels the difference does not matter, because the network simply learns the rotated kernel. It does matter when you copy a kernel from a signal-processing text, or compare against `scipy.signal.convolve2d`.

The sum over $c$ is easy to forget. A "$3\times3$ convolution" on a 64-channel input is really a $3\times3\times64$ dot product per output value. Each output channel gets its own $C_{in}\times k\times k$ filter. Only the spatial extent is small; the depth always covers every input channel (unless the layer is grouped, as in **Depthwise separable convolution** or **ResNeXt**).

**One Conv2d layer at a glance**

- **Weight shape:** $(C_{out},\,C_{in},\,k_h,\,k_w)$; bias shape $(C_{out})$
- **Parameters:** $C_{out}C_{in}k^2+C_{out}$, independent of $H$ and $W$
- **Multiply-adds:** $H_{out}W_{out}\cdot C_{out}C_{in}k^2$ (papers usually call this number "FLOPs")
- **Output size:** $\lfloor (n+2p-k)/s\rfloor+1$ per spatial axis (see **Stride and padding**)
- **Equivariant to:** integer translations (exactly, away from borders, at stride 1)
- **Not equivariant to:** rotation, scaling, reflection; stride $>1$ breaks exact shift equivariance
- **Implemented as:** im2col + matrix multiply, Winograd for $3\times3$, FFT for large kernels, or direct kernels

> Figure: A 2-channel 8×8 input (channel 0: a block of 2s and a block of 1s; channel 1: an anti-diagonal line) convolved with a 2×3×3 kernel into one 6×6 map. Click an output cell to see its window and the products. By default (vertical-edge kernel on channel 0, anti-diagonal kernel on channel 1, bias −1) y[1,4] = 6 + 3 − 1 = 8. Flipping the kernel (true convolution) changes the output by up to 12, except for the 180°-symmetric anti-diagonal preset, where it changes nothing. Shifting the input one pixel right shifts the output one pixel right: the shift-test gap is 0 for every kernel. The layer has 1·2·9 + 1 = 19 parameters and 648 multiply-adds, and im2col turns it into an 18×36 patch matrix.

## Three properties that come for free

**Sparse connectivity.** Each output looks at $k^2C_{in}$ inputs, not all $HWC_{in}$ of them. Context grows with depth instead: two stacked $3\times3$ layers see a $5\times5$ region, three see $7\times7$ (see **Receptive field**).

**Parameter sharing.** The same weights are used at all $H_{out}W_{out}$ positions. In the figure, 18 weights produce 36 outputs; in a real layer one $3\times3\times64$ kernel is reused at 3,136 positions of a $56\times56$ map. The gradient of each weight is therefore a sum over all positions, which is a lot of signal per parameter.

**Translation equivariance.** If $T$ shifts an image, then $\mathrm{conv}(Tx)=T\,\mathrm{conv}(x)$: move the cat and its feature map moves with it. This holds exactly for stride 1, away from the borders, and it is what the shift test in the figure checks. Equivariance is not **invariance**: the output moves, it does not stay the same. Invariance comes later, from pooling and global averaging (see **Pooling layers**). Zero padding and stride both break the property in practice; **Stride and padding** shows how much.

## im2col: a convolution is one big matrix multiply

Copy every $C_{in}\times k\times k$ window into a column of a matrix $X_{col}$ of size $C_{in}k^2\times H_{out}W_{out}$. Reshape the kernels into $W_{row}$ of size $C_{out}\times C_{in}k^2$. Then the whole layer is a single GEMM, $Y=W_{row}X_{col}$, which reshapes to $C_{out}\times H_{out}\times W_{out}$. Chellapilla, Puri and Simard (2006) used this lowering to run CNNs fast on GPUs, and cuDNN (Chetlur et al., 2014) builds on the same idea. The cost is memory: each input value is copied up to $k^2$ times.

$$
\underbrace{Y}_{C_{out}\times H_{out}W_{out}}=\underbrace{W_{row}}_{C_{out}\times C_{in}k^2}\;\underbrace{X_{col}}_{C_{in}k^2\times H_{out}W_{out}}\qquad \text{MACs}=H_{out}W_{out}\,C_{out}\,C_{in}\,k^2
$$

- $X_{col}$: one flattened input window per column (`torch.nn.functional.unfold`)
- $W_{row}$: one flattened kernel per row: `weight.view(C_out, -1)`
- $\text{MACs}$: multiply-accumulates, the inner dimension times the output size; one MAC is two floating-point operations

Parameters and multiply-adds of some real layers (bias included where the layer has one; computed with the lib's cost())

| layer | C_in → C_out, k | output | parameters | MACs |
| --- | --- | --- | --- | --- |
| figure | 2 → 1, 3×3 | 6×6 | 19 | 648 |
| ResNet-50 stem (no bias) | 3 → 64, 7×7, stride 2 | 112×112 | 9,408 | 118.0 M |
| ResNet stage-2 3×3 | 64 → 128, 3×3 | 56×56 | 73,856 | 231.2 M |
| VGG conv5 | 512 → 512, 3×3 | 14×14 | 2,359,808 | 462.4 M |

> **Parameters and compute scale differently:** Parameters depend on $C_{in}C_{out}k^2$ only. Compute is that times $H_{out}W_{out}$. The VGG conv5 layer has 32 times the parameters of the ResNet $64\to128$ layer but only twice the compute, because its map is 16 times smaller. Early layers are cheap in parameters and expensive in compute; late layers are the opposite.

## The backward pass is also a convolution

Write $G=\partial L/\partial y$ for the upstream gradient (same shape as $y$). Every weight touches every position, and every input pixel touches up to $k^2$ outputs, so both gradients are sums of products of shifted maps, which is exactly what a convolution computes:

$$
\frac{\partial L}{\partial K_{o,c,m,n}}=\sum_{i,j}G_{o,i,j}\,x_{c,\,i+m,\,j+n}\qquad \frac{\partial L}{\partial x_{c,u,v}}=\sum_{o}\sum_{m,n}G_{o,\,u-m,\,v-n}\,K_{o,c,m,n}\qquad \frac{\partial L}{\partial b_o}=\sum_{i,j}G_{o,i,j}
$$

- $\partial L/\partial K$: a valid cross-correlation of the input with the upstream gradient (the gradient acts as the kernel)
- $\partial L/\partial x$: a "full" cross-correlation of the zero-padded $G$ with the 180°-rotated kernel, i.e. a true convolution of $G$ with $K$; summed over output channels, this is a **Transposed convolution**
- $\partial L/\partial b$: the sum of the gradient over all positions, since the bias is shared

So forward cross-correlation needs a flipped kernel on the way back, and a stride-$s$ convolution has a stride-$s$ transposed convolution as its input gradient. This is why `nn.ConvTranspose2d` is sometimes described as "the backward pass of Conv2d, used forwards".

## Worked example: 4×4 input, 2×2 kernel

Take one channel, no padding, stride 1, bias 0, $X=\begin{pmatrix}1&2&0&1\\0&1&3&1\\2&1&0&2\\1&0&1&3\end{pmatrix}$ and $K=\begin{pmatrix}1&0\\2&-1\end{pmatrix}$. The output is $3\times3$.

Cross-correlation (what Conv2d computes) versus true convolution (kernel rotated 180° to [[−1, 2], [0, 1]]). Checked in Node.

| operation | row 0 | row 1 | row 2 |
| --- | --- | --- | --- |
| cross-correlation | 0, 1, 5 | 3, 3, 1 | 4, 0, −1 |
| true convolution | 4, 1, 3 | 3, 5, 1 | 0, 0, 7 |

**Forward and backward by hand**

1. $y_{0,0}=1\cdot1+0\cdot2+2\cdot0+(-1)\cdot1=0$. The window is $\begin{pmatrix}1&2\\0&1\end{pmatrix}$; the products are summed index by index.
2. $y_{0,2}=1\cdot0+0\cdot1+2\cdot3+(-1)\cdot1=5$: the large $X_{1,2}=3$ sits under the weight 2.
3. Upstream gradient $G=\begin{pmatrix}1&0&0\\0&0&0\\0&0&-1\end{pmatrix}$, i.e. $L=y_{0,0}-y_{2,2}$. Kernel gradient $\partial L/\partial K_{m,n}=x_{m,n}-x_{2+m,2+n}$, so $\partial L/\partial K=\begin{pmatrix}1-0&2-2\\0-1&1-3\end{pmatrix}=\begin{pmatrix}1&0\\-1&-2\end{pmatrix}$.
4. Input gradient: each nonzero $G$ entry stamps a copy of $K$ onto the input, $+K$ at the top-left and $-K$ at the bottom-right: $\partial L/\partial X=\begin{pmatrix}1&0&0&0\\2&-1&0&0\\0&0&-1&0\\0&0&-2&1\end{pmatrix}$. The same result comes from cross-correlating the zero-padded $G$ with the rotated kernel, and from finite differences.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([[1., 2, 0, 1], [0, 1, 3, 1], [2, 1, 0, 2], [1, 0, 1, 3]]).view(1, 1, 4, 4)
k = torch.tensor([[1., 0], [2, -1]]).view(1, 1, 2, 2)      # weight shape (C_out, C_in, kH, kW)

y = F.conv2d(x, k)                     # cross-correlation: [[0,1,5],[3,3,1],[4,0,-1]]
y_true = F.conv2d(x, k.flip(-1, -2))   # true convolution:  [[4,1,3],[3,5,1],[0,0,7]]

# A layer: 64 filters of shape 3x3x3, shared over all positions
conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)
print(conv.weight.shape, sum(p.numel() for p in conv.parameters()))   # (64, 3, 3, 3) 1792

# im2col: unfold copies every C_in*k*k window into a column, then one matmul does the layer
img = torch.randn(1, 3, 32, 32)
cols = F.unfold(img, kernel_size=3, padding=1)            # (1, 27, 1024)
y_gemm = (conv.weight.view(64, -1) @ cols + conv.bias.view(64, 1)).view(1, 64, 32, 32)
print(torch.allclose(y_gemm, conv(img), atol=1e-5))       # True

# Backward: both gradients are convolutions
xg = x.clone().requires_grad_(True)
kg = k.clone().requires_grad_(True)
G = torch.tensor([[1., 0, 0], [0, 0, 0], [0, 0, -1]]).view(1, 1, 3, 3)
F.conv2d(xg, kg).backward(G)
dx = torch.nn.grad.conv2d_input(x.shape, k, G)            # = conv_transpose2d(G, k)
dk = torch.nn.grad.conv2d_weight(x, k.shape, G)           # = conv2d(x, G) for one channel
print(torch.equal(xg.grad, dx), torch.equal(kg.grad, dk)) # True True
```

Conv2d, the same layer as unfold + matmul, and the gradients as convolutions

> **Copying kernels between libraries:** `F.conv2d` does not flip the kernel; `scipy.signal.convolve2d` and the textbook definition do. A Sobel or blur kernel copied from one to the other is applied rotated by 180°: symmetric kernels survive, but derivative kernels change sign. Use `scipy.signal.correlate2d` to match PyTorch, or flip with `k.flip(-1, -2)`.

> **Channels-first and the hidden sum:** PyTorch expects inputs as $(N, C, H, W)$ and weights as $(C_{out}, C_{in}, k_h, k_w)$. Passing an $(N, H, W, C)$ image gives a shape error at best and a silently wrong result at worst if $H$ happens to equal $C_{in}$. And remember the sum over $C_{in}$: a $3\times3$ conv from 256 to 256 channels has $3\cdot3\cdot256\cdot256=589{,}824$ weights, not 9.

> **How libraries actually run it:** For $3\times3$ stride-1 layers cuDNN often uses **Winograd** minimal filtering (Lavin and Gray, 2016), which cuts the multiplications by about 2.25× for the $F(2\times2,3\times3)$ variant. FFT-based convolution (Mathieu, Henaff and LeCun, 2014) pays off for large kernels. `torch.backends.cudnn.benchmark = True` lets cuDNN time the options for your shapes and pick the fastest one.

**Strengths**

- Few parameters: $C_{out}C_{in}k^2$, independent of image size
- Translation equivariance built in: a detector learned at one location works at all of them
- Local connectivity matches the structure of images, audio and other grid data
- Maps onto highly optimised GEMM, Winograd and FFT kernels

**Limitations**

- Only local context per layer: global structure needs depth, pooling, dilation or attention
- Not equivariant to rotation or scale; the network must learn those from data or augmentation
- Compute grows with $H_{out}W_{out}$, so high-resolution early layers are expensive
- Borders and stride break exact equivariance (see Stride and padding)

**Question:** A Conv2d layer maps 64 channels to 128 channels with a 3×3 kernel, padding 1, stride 1, on a 56×56 map. How many weights (excluding bias) and multiply-adds does it have?

- A. 73,728 weights, about 231 M multiply-adds
- B. 1,152 weights, about 3.6 M multiply-adds
- C. 73,728 weights, about 4.1 M multiply-adds
- D. 9 weights, about 28 K multiply-adds

**Answer:** A. 73,728 weights, about 231 M multiply-adds — Weights $=C_{out}C_{in}k^2=128\cdot64\cdot9=73{,}728$ (73,856 with the 128 biases). Every one of the $56\cdot56=3{,}136$ output positions uses all of them once: $3{,}136\cdot73{,}728\approx231.2$ M multiply-adds.

**Question:** You shift an input image 3 pixels to the right and pass it through a stride-1 convolution. What happens to the output feature map (ignoring the borders)?

- A. It shifts 3 pixels to the right and is otherwise identical
- B. It stays the same: convolution is shift-invariant
- C. It changes unpredictably
- D. It shifts 3 pixels to the left, because the kernel is flipped

**Answer:** A. It shifts 3 pixels to the right and is otherwise identical — Convolution is translation **equivariant**: $\mathrm{conv}(Tx)=T\,\mathrm{conv}(x)$, because the same weights are applied at every position. It is not invariant; invariance to small shifts comes from pooling. Whether the kernel is flipped does not affect the direction of the shift.

**Key points**

- $y_{o,i,j}=b_o+\sum_{c,m,n}K_{o,c,m,n}\,x_{c,i+m,j+n}$: a dot product of the kernel with each $C_{in}\times k\times k$ window.
- Deep learning libraries compute cross-correlation; true convolution rotates the kernel 180°. Learned kernels absorb the difference.
- Parameters $C_{out}C_{in}k^2+C_{out}$; multiply-adds $H_{out}W_{out}C_{out}C_{in}k^2$.
- Locality, parameter sharing and translation equivariance are the three built-in assumptions.
- im2col turns the layer into one matrix multiply; the backward pass is a convolution with the flipped kernel (a transposed convolution) and a correlation of input with gradient.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
