Convolution operation

Slide one small kernel over every position of the input and take a weighted sum over its window and all input channels. The same weights are reused everywhere, so the layer is local, has few parameters, and shifts its output when the input shifts.

14 min read1052 wordscore2 viewsupdated 2026-10-11

Definition

Convolution layer A convolution layer maps an input with CinC_{in} channels to CoutC_{out} output maps. Each output value is a bias plus a weighted sum of the k×kk\times k window under it, taken across all input channels: yo,i,j=bo+∑c∑m,nKo,c,m,n xc, i+m, j+ny_{o,i,j}=b_o+\sum_{c}\sum_{m,n}K_{o,c,m,n}\,x_{c,\,i+m,\,j+n}. The same kernel KoK_{o} is used at every position (i,j)(i,j).

A fully connected layer from a 224×224×3224\times224\times3 image to 1000 units needs 150,528×1000≈150150{,}528\times1000\approx150 million weights. It also has to learn the same edge detector separately at every location. A convolution makes two assumptions about images instead. Locality: an output depends only on a small neighbourhood. Stationarity: a pattern worth detecting in one place is worth detecting everywhere. Under those two assumptions a layer of 64 detectors of size 3×3×33\times3\times3 needs 64⋅27+64=1,79264\cdot27+64=1{,}792 parameters, whatever the image size.

The idea comes from vision science and early neural networks. Hubel and Wiesel (1959, 1962) found cells in the cat visual cortex that respond to oriented edges inside a small receptive field. Fukushima's neocognitron (1980) stacked layers of such local, weight-shared detectors. LeCun et al. (1989, "Backpropagation applied to handwritten zip code recognition") trained the shared kernels end to end with backpropagation, which led to LeNet-5 (LeCun, Bottou, Bengio and Haffner, 1998). Every CNN since, from AlexNet to ConvNeXt, is built from this operation.

Formula: cross-correlation, not convolution

yo,i,j=bo+∑c=0Cin−1∑m=0k−1∑n=0k−1Ko,c,m,n  xc,  i+m,  j+ny_{o,i,j}=b_o+\sum_{c=0}^{C_{in}-1}\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}K_{o,c,m,n}\;x_{c,\;i+m,\;j+n}
xc,i,jx_{c,i,j}
input: channel cc, row ii, column jj (shape Cin×H×WC_{in}\times H\times W)
Ko,c,m,nK_{o,c,m,n}
kernel weights, tensor of shape (Cout,Cin,k,k)(C_{out},C_{in},k,k): one k×kk\times k slice per (output, input) channel pair
bob_o
one bias per output channel, shared by all positions
yo,i,jy_{o,i,j}
output map oo at position (i,j)(i,j); with no padding and stride 1 it has size (H−k+1)×(W−k+1)(H-k+1)\times(W-k+1)

This is a cross-correlation: the window and the kernel are aligned index by index. The convolution of signal processing flips the kernel first, (x∗K)i,j=∑m,nKm,n xi−m, j−n(x*K)_{i,j}=\sum_{m,n}K_{m,n}\,x_{i-m,\,j-n}, which is the same as cross-correlating with the kernel rotated by 180°. Deep learning libraries (torch.nn.functional.conv2d, cuDNN, jax.lax.conv) compute the unflipped version and still call it convolution. For learned kernels the difference does not matter, because the network simply learns the rotated kernel. It does matter when you copy a kernel from a signal-processing text, or compare against scipy.signal.convolve2d.

The sum over cc is easy to forget. A "3×33\times3 convolution" on a 64-channel input is really a 3×3×643\times3\times64 dot product per output value. Each output channel gets its own Cin×k×kC_{in}\times k\times k filter. Only the spatial extent is small; the depth always covers every input channel (unless the layer is grouped, as in Depthwise separable convolution or ResNeXt).

One Conv2d layer at a glance

Weight shape
(Cout, Cin, kh, kw)(C_{out},\,C_{in},\,k_h,\,k_w); bias shape (Cout)(C_{out})
Parameters
CoutCink2+CoutC_{out}C_{in}k^2+C_{out}, independent of HH and WW
Multiply-adds
HoutWout⋅CoutCink2H_{out}W_{out}\cdot C_{out}C_{in}k^2 (papers usually call this number "FLOPs")
Output size
⌊(n+2p−k)/s⌋+1\lfloor (n+2p-k)/s\rfloor+1 per spatial axis (see Stride and padding)
Equivariant to
integer translations (exactly, away from borders, at stride 1)
Not equivariant to
rotation, scaling, reflection; stride >1>1 breaks exact shift equivariance
Implemented as
im2col + matrix multiply, Winograd for 3×33\times3, FFT for large kernels, or direct kernels
input channel 0 (8×8)0000000000222000002220000022200000000000000001100000011000000000kernel K[0]10−110−110−1Σ = 6input channel 1 (8×8)0000000100000010000001000000100000010000001000000100000010000000kernel K[1]001010100Σ = 3+output y (6×6) · click a cell−5−5−1332−7−7−158−1−5−5−163−1−3−32000−12−1−3−312−1−1−3−31y[1,4] = 6 + 3 + b(−1) = 8cross-correlation (nn.Conv2d)max |corr − conv| = 12shift test: offch 0: 1·2 + 1·2 + 1·2ch 1: 1·1 + 1·1 + 1·1params C_out·C_in·k² + C_out = 1·2·9 + 1 = 19 · MACs H_out·W_out·C_out·C_in·k² = 648im2col: patches matrix 18×36, weight row 1×18 → one GEMM gives all 36 outputs■ positive ■ negative · same 18 weights at every one of the 36 positions
kernel· click a kernel cell to cycle −2…2
fig.A 2-channel 8×8 input (channel 0: a block of 2s and a block of 1s; channel 1: an anti-diagonal line) convolved with a 2×3×3 kernel into one 6×6 map. Click an output cell to see its window and the products. By default (vertical-edge kernel on channel 0, anti-diagonal kernel on channel 1, bias −1) y[1,4] = 6 + 3 − 1 = 8. Flipping the kernel (true convolution) changes the output by up to 12, except for the 180°-symmetric anti-diagonal preset, where it changes nothing. Shifting the input one pixel right shifts the output one pixel right: the shift-test gap is 0 for every kernel. The layer has 1·2·9 + 1 = 19 parameters and 648 multiply-adds, and im2col turns it into an 18×36 patch matrix.

Three properties that come for free

Sparse connectivity. Each output looks at k2Cink^2C_{in} inputs, not all HWCinHWC_{in} of them. Context grows with depth instead: two stacked 3×33\times3 layers see a 5×55\times5 region, three see 7×77\times7 (see Receptive field).

Parameter sharing. The same weights are used at all HoutWoutH_{out}W_{out} positions. In the figure, 18 weights produce 36 outputs; in a real layer one 3×3×643\times3\times64 kernel is reused at 3,136 positions of a 56×5656\times56 map. The gradient of each weight is therefore a sum over all positions, which is a lot of signal per parameter.

Translation equivariance. If TT shifts an image, then conv(Tx)=T conv(x)\mathrm{conv}(Tx)=T\,\mathrm{conv}(x): move the cat and its feature map moves with it. This holds exactly for stride 1, away from the borders, and it is what the shift test in the figure checks. Equivariance is not invariance: the output moves, it does not stay the same. Invariance comes later, from pooling and global averaging (see Pooling layers). Zero padding and stride both break the property in practice; Stride and padding shows how much.

im2col: a convolution is one big matrix multiply

Copy every Cin×k×kC_{in}\times k\times k window into a column of a matrix XcolX_{col} of size Cink2×HoutWoutC_{in}k^2\times H_{out}W_{out}. Reshape the kernels into WrowW_{row} of size Cout×Cink2C_{out}\times C_{in}k^2. Then the whole layer is a single GEMM, Y=WrowXcolY=W_{row}X_{col}, which reshapes to Cout×Hout×WoutC_{out}\times H_{out}\times W_{out}. Chellapilla, Puri and Simard (2006) used this lowering to run CNNs fast on GPUs, and cuDNN (Chetlur et al., 2014) builds on the same idea. The cost is memory: each input value is copied up to k2k^2 times.

Y⏟Cout×HoutWout=Wrow⏟Cout×Cink2  Xcol⏟Cink2×HoutWoutMACs=HoutWout Cout Cin k2\underbrace{Y}_{C_{out}\times H_{out}W_{out}}=\underbrace{W_{row}}_{C_{out}\times C_{in}k^2}\;\underbrace{X_{col}}_{C_{in}k^2\times H_{out}W_{out}}\qquad \text{MACs}=H_{out}W_{out}\,C_{out}\,C_{in}\,k^2
XcolX_{col}
one flattened input window per column (torch.nn.functional.unfold)
WrowW_{row}
one flattened kernel per row: weight.view(C_out, -1)
MACs\text{MACs}
multiply-accumulates, the inner dimension times the output size; one MAC is two floating-point operations
layerC_in → C_out, koutputparametersMACs
figure2 → 1, 3×36×619648
ResNet-50 stem (no bias)3 → 64, 7×7, stride 2112×1129,408118.0 M
ResNet stage-2 3×364 → 128, 3×356×5673,856231.2 M
VGG conv5512 → 512, 3×314×142,359,808462.4 M
Parameters and multiply-adds of some real layers (bias included where the layer has one; computed with the lib's cost())

The backward pass is also a convolution

Write G=∂L/∂yG=\partial L/\partial y for the upstream gradient (same shape as yy). Every weight touches every position, and every input pixel touches up to k2k^2 outputs, so both gradients are sums of products of shifted maps, which is exactly what a convolution computes:

∂L∂Ko,c,m,n=∑i,jGo,i,j xc, i+m, j+n∂L∂xc,u,v=∑o∑m,nGo, u−m, v−n Ko,c,m,n∂L∂bo=∑i,jGo,i,j\frac{\partial L}{\partial K_{o,c,m,n}}=\sum_{i,j}G_{o,i,j}\,x_{c,\,i+m,\,j+n}\qquad \frac{\partial L}{\partial x_{c,u,v}}=\sum_{o}\sum_{m,n}G_{o,\,u-m,\,v-n}\,K_{o,c,m,n}\qquad \frac{\partial L}{\partial b_o}=\sum_{i,j}G_{o,i,j}
∂L/∂K\partial L/\partial K
a valid cross-correlation of the input with the upstream gradient (the gradient acts as the kernel)
∂L/∂x\partial L/\partial x
a "full" cross-correlation of the zero-padded GG with the 180°-rotated kernel, i.e. a true convolution of GG with KK; summed over output channels, this is a Transposed convolution
∂L/∂b\partial L/\partial b
the sum of the gradient over all positions, since the bias is shared

So forward cross-correlation needs a flipped kernel on the way back, and a stride-ss convolution has a stride-ss transposed convolution as its input gradient. This is why nn.ConvTranspose2d is sometimes described as "the backward pass of Conv2d, used forwards".

Worked example: 4×4 input, 2×2 kernel

Take one channel, no padding, stride 1, bias 0, X=(1201013121021013)X=\begin{pmatrix}1&2&0&1\\0&1&3&1\\2&1&0&2\\1&0&1&3\end{pmatrix} and K=(102−1)K=\begin{pmatrix}1&0\\2&-1\end{pmatrix}. The output is 3×33\times3.

operationrow 0row 1row 2
cross-correlation0, 1, 53, 3, 14, 0, −1
true convolution4, 1, 33, 5, 10, 0, 7
Cross-correlation (what Conv2d computes) versus true convolution (kernel rotated 180° to [[−1, 2], [0, 1]]). Checked in Node.

Forward and backward by hand

  1. 1y0,0=1⋅1+0⋅2+2⋅0+(−1)⋅1=0y_{0,0}=1\cdot1+0\cdot2+2\cdot0+(-1)\cdot1=0. The window is (1201)\begin{pmatrix}1&2\\0&1\end{pmatrix}; the products are summed index by index.
  2. 2y0,2=1⋅0+0⋅1+2⋅3+(−1)⋅1=5y_{0,2}=1\cdot0+0\cdot1+2\cdot3+(-1)\cdot1=5: the large X1,2=3X_{1,2}=3 sits under the weight 2.
  3. 3Upstream gradient G=(10000000−1)G=\begin{pmatrix}1&0&0\\0&0&0\\0&0&-1\end{pmatrix}, i.e. L=y0,0−y2,2L=y_{0,0}-y_{2,2}. Kernel gradient ∂L/∂Km,n=xm,n−x2+m,2+n\partial L/\partial K_{m,n}=x_{m,n}-x_{2+m,2+n}, so ∂L/∂K=(1−02−20−11−3)=(10−1−2)\partial L/\partial K=\begin{pmatrix}1-0&2-2\\0-1&1-3\end{pmatrix}=\begin{pmatrix}1&0\\-1&-2\end{pmatrix}.
  4. 4Input gradient: each nonzero GG entry stamps a copy of KK onto the input, +K+K at the top-left and −K-K at the bottom-right: ∂L/∂X=(10002−10000−1000−21)\partial L/\partial X=\begin{pmatrix}1&0&0&0\\2&-1&0&0\\0&0&-1&0\\0&0&-2&1\end{pmatrix}. The same result comes from cross-correlating the zero-padded GG with the rotated kernel, and from finite differences.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([[1., 2, 0, 1], [0, 1, 3, 1], [2, 1, 0, 2], [1, 0, 1, 3]]).view(1, 1, 4, 4)
4k = torch.tensor([[1., 0], [2, -1]]).view(1, 1, 2, 2) # weight shape (C_out, C_in, kH, kW)
5
6y = F.conv2d(x, k) # cross-correlation: [[0,1,5],[3,3,1],[4,0,-1]]
7y_true = F.conv2d(x, k.flip(-1, -2)) # true convolution: [[4,1,3],[3,5,1],[0,0,7]]
8
9# A layer: 64 filters of shape 3x3x3, shared over all positions
10conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)
11print(conv.weight.shape, sum(p.numel() for p in conv.parameters())) # (64, 3, 3, 3) 1792
12
13# im2col: unfold copies every C_in*k*k window into a column, then one matmul does the layer
14img = torch.randn(1, 3, 32, 32)
15cols = F.unfold(img, kernel_size=3, padding=1) # (1, 27, 1024)
16y_gemm = (conv.weight.view(64, -1) @ cols + conv.bias.view(64, 1)).view(1, 64, 32, 32)
17print(torch.allclose(y_gemm, conv(img), atol=1e-5)) # True
18
19# Backward: both gradients are convolutions
20xg = x.clone().requires_grad_(True)
21kg = k.clone().requires_grad_(True)
22G = torch.tensor([[1., 0, 0], [0, 0, 0], [0, 0, -1]]).view(1, 1, 3, 3)
23F.conv2d(xg, kg).backward(G)
24dx = torch.nn.grad.conv2d_input(x.shape, k, G) # = conv_transpose2d(G, k)
25dk = torch.nn.grad.conv2d_weight(x, k.shape, G) # = conv2d(x, G) for one channel
26print(torch.equal(xg.grad, dx), torch.equal(kg.grad, dk)) # True True
Conv2d, the same layer as unfold + matmul, and the gradients as convolutions

Use when

  • +Few parameters: CoutCink2C_{out}C_{in}k^2, independent of image size
  • +Translation equivariance built in: a detector learned at one location works at all of them
  • +Local connectivity matches the structure of images, audio and other grid data
  • +Maps onto highly optimised GEMM, Winograd and FFT kernels

Avoid when

  • −Only local context per layer: global structure needs depth, pooling, dilation or attention
  • −Not equivariant to rotation or scale; the network must learn those from data or augmentation
  • −Compute grows with HoutWoutH_{out}W_{out}, so high-resolution early layers are expensive
  • −Borders and stride break exact equivariance (see Stride and padding)

Check yourself

A Conv2d layer maps 64 channels to 128 channels with a 3×3 kernel, padding 1, stride 1, on a 56×56 map. How many weights (excluding bias) and multiply-adds does it have?

Check yourself

You shift an input image 3 pixels to the right and pass it through a stride-1 convolution. What happens to the output feature map (ignoring the borders)?

Remember

  • ✓yo,i,j=bo+∑c,m,nKo,c,m,n xc,i+m,j+ny_{o,i,j}=b_o+\sum_{c,m,n}K_{o,c,m,n}\,x_{c,i+m,j+n}: a dot product of the kernel with each Cin×k×kC_{in}\times k\times k window.
  • ✓Deep learning libraries compute cross-correlation; true convolution rotates the kernel 180°. Learned kernels absorb the difference.
  • ✓Parameters CoutCink2+CoutC_{out}C_{in}k^2+C_{out}; multiply-adds HoutWoutCoutCink2H_{out}W_{out}C_{out}C_{in}k^2.
  • ✓Locality, parameter sharing and translation equivariance are the three built-in assumptions.
  • ✓im2col turns the layer into one matrix multiply; the backward pass is a convolution with the flipped kernel (a transposed convolution) and a correlation of input with gradient.
← back to Deep Learning roadmap