Definition
Convolution layer A convolution layer maps an input with channels to output maps. Each output value is a bias plus a weighted sum of the window under it, taken across all input channels: . The same kernel is used at every position .
A fully connected layer from a image to 1000 units needs million weights. It also has to learn the same edge detector separately at every location. A convolution makes two assumptions about images instead. Locality: an output depends only on a small neighbourhood. Stationarity: a pattern worth detecting in one place is worth detecting everywhere. Under those two assumptions a layer of 64 detectors of size needs parameters, whatever the image size.
The idea comes from vision science and early neural networks. Hubel and Wiesel (1959, 1962) found cells in the cat visual cortex that respond to oriented edges inside a small receptive field. Fukushima's neocognitron (1980) stacked layers of such local, weight-shared detectors. LeCun et al. (1989, "Backpropagation applied to handwritten zip code recognition") trained the shared kernels end to end with backpropagation, which led to LeNet-5 (LeCun, Bottou, Bengio and Haffner, 1998). Every CNN since, from AlexNet to ConvNeXt, is built from this operation.
Formula: cross-correlation, not convolution
- input: channel , row , column (shape )
- kernel weights, tensor of shape : one slice per (output, input) channel pair
- one bias per output channel, shared by all positions
- output map at position ; with no padding and stride 1 it has size
This is a cross-correlation: the window and the kernel are aligned index by index. The convolution of signal processing flips the kernel first, , which is the same as cross-correlating with the kernel rotated by 180°. Deep learning libraries (torch.nn.functional.conv2d, cuDNN, jax.lax.conv) compute the unflipped version and still call it convolution. For learned kernels the difference does not matter, because the network simply learns the rotated kernel. It does matter when you copy a kernel from a signal-processing text, or compare against scipy.signal.convolve2d.
The sum over is easy to forget. A " convolution" on a 64-channel input is really a dot product per output value. Each output channel gets its own filter. Only the spatial extent is small; the depth always covers every input channel (unless the layer is grouped, as in Depthwise separable convolution or ResNeXt).
One Conv2d layer at a glance
- Weight shape
- ; bias shape
- Parameters
- , independent of and
- Multiply-adds
- (papers usually call this number "FLOPs")
- Output size
- per spatial axis (see Stride and padding)
- Equivariant to
- integer translations (exactly, away from borders, at stride 1)
- Not equivariant to
- rotation, scaling, reflection; stride breaks exact shift equivariance
- Implemented as
- im2col + matrix multiply, Winograd for , FFT for large kernels, or direct kernels
Three properties that come for free
Sparse connectivity. Each output looks at inputs, not all of them. Context grows with depth instead: two stacked layers see a region, three see (see Receptive field).
Parameter sharing. The same weights are used at all positions. In the figure, 18 weights produce 36 outputs; in a real layer one kernel is reused at 3,136 positions of a map. The gradient of each weight is therefore a sum over all positions, which is a lot of signal per parameter.
Translation equivariance. If shifts an image, then : move the cat and its feature map moves with it. This holds exactly for stride 1, away from the borders, and it is what the shift test in the figure checks. Equivariance is not invariance: the output moves, it does not stay the same. Invariance comes later, from pooling and global averaging (see Pooling layers). Zero padding and stride both break the property in practice; Stride and padding shows how much.
im2col: a convolution is one big matrix multiply
Copy every window into a column of a matrix of size . Reshape the kernels into of size . Then the whole layer is a single GEMM, , which reshapes to . Chellapilla, Puri and Simard (2006) used this lowering to run CNNs fast on GPUs, and cuDNN (Chetlur et al., 2014) builds on the same idea. The cost is memory: each input value is copied up to times.
- one flattened input window per column (
torch.nn.functional.unfold) - one flattened kernel per row:
weight.view(C_out, -1) - multiply-accumulates, the inner dimension times the output size; one MAC is two floating-point operations
| layer | C_in → C_out, k | output | parameters | MACs |
|---|---|---|---|---|
| figure | 2 → 1, 3×3 | 6×6 | 19 | 648 |
| ResNet-50 stem (no bias) | 3 → 64, 7×7, stride 2 | 112×112 | 9,408 | 118.0 M |
| ResNet stage-2 3×3 | 64 → 128, 3×3 | 56×56 | 73,856 | 231.2 M |
| VGG conv5 | 512 → 512, 3×3 | 14×14 | 2,359,808 | 462.4 M |
The backward pass is also a convolution
Write for the upstream gradient (same shape as ). Every weight touches every position, and every input pixel touches up to outputs, so both gradients are sums of products of shifted maps, which is exactly what a convolution computes:
- a valid cross-correlation of the input with the upstream gradient (the gradient acts as the kernel)
- a "full" cross-correlation of the zero-padded with the 180°-rotated kernel, i.e. a true convolution of with ; summed over output channels, this is a Transposed convolution
- the sum of the gradient over all positions, since the bias is shared
So forward cross-correlation needs a flipped kernel on the way back, and a stride- convolution has a stride- transposed convolution as its input gradient. This is why nn.ConvTranspose2d is sometimes described as "the backward pass of Conv2d, used forwards".
Worked example: 4×4 input, 2×2 kernel
Take one channel, no padding, stride 1, bias 0, and . The output is .
| operation | row 0 | row 1 | row 2 |
|---|---|---|---|
| cross-correlation | 0, 1, 5 | 3, 3, 1 | 4, 0, −1 |
| true convolution | 4, 1, 3 | 3, 5, 1 | 0, 0, 7 |
Forward and backward by hand
- 1. The window is ; the products are summed index by index.
- 2: the large sits under the weight 2.
- 3Upstream gradient , i.e. . Kernel gradient , so .
- 4Input gradient: each nonzero entry stamps a copy of onto the input, at the top-left and at the bottom-right: . The same result comes from cross-correlating the zero-padded with the rotated kernel, and from finite differences.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([[1., 2, 0, 1], [0, 1, 3, 1], [2, 1, 0, 2], [1, 0, 1, 3]]).view(1, 1, 4, 4)4k = torch.tensor([[1., 0], [2, -1]]).view(1, 1, 2, 2) # weight shape (C_out, C_in, kH, kW)56y = F.conv2d(x, k) # cross-correlation: [[0,1,5],[3,3,1],[4,0,-1]]7y_true = F.conv2d(x, k.flip(-1, -2)) # true convolution: [[4,1,3],[3,5,1],[0,0,7]]89# A layer: 64 filters of shape 3x3x3, shared over all positions10conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)11print(conv.weight.shape, sum(p.numel() for p in conv.parameters())) # (64, 3, 3, 3) 17921213# im2col: unfold copies every C_in*k*k window into a column, then one matmul does the layer14img = torch.randn(1, 3, 32, 32)15cols = F.unfold(img, kernel_size=3, padding=1) # (1, 27, 1024)16y_gemm = (conv.weight.view(64, -1) @ cols + conv.bias.view(64, 1)).view(1, 64, 32, 32)17print(torch.allclose(y_gemm, conv(img), atol=1e-5)) # True1819# Backward: both gradients are convolutions20xg = x.clone().requires_grad_(True)21kg = k.clone().requires_grad_(True)22G = torch.tensor([[1., 0, 0], [0, 0, 0], [0, 0, -1]]).view(1, 1, 3, 3)23F.conv2d(xg, kg).backward(G)24dx = torch.nn.grad.conv2d_input(x.shape, k, G) # = conv_transpose2d(G, k)25dk = torch.nn.grad.conv2d_weight(x, k.shape, G) # = conv2d(x, G) for one channel26print(torch.equal(xg.grad, dx), torch.equal(kg.grad, dk)) # True TrueUse when
- +Few parameters: , independent of image size
- +Translation equivariance built in: a detector learned at one location works at all of them
- +Local connectivity matches the structure of images, audio and other grid data
- +Maps onto highly optimised GEMM, Winograd and FFT kernels
Avoid when
- −Only local context per layer: global structure needs depth, pooling, dilation or attention
- −Not equivariant to rotation or scale; the network must learn those from data or augmentation
- −Compute grows with , so high-resolution early layers are expensive
- −Borders and stride break exact equivariance (see Stride and padding)
Check yourself
A Conv2d layer maps 64 channels to 128 channels with a 3×3 kernel, padding 1, stride 1, on a 56×56 map. How many weights (excluding bias) and multiply-adds does it have?
Check yourself
You shift an input image 3 pixels to the right and pass it through a stride-1 convolution. What happens to the output feature map (ignoring the borders)?
Remember
- ✓: a dot product of the kernel with each window.
- ✓Deep learning libraries compute cross-correlation; true convolution rotates the kernel 180°. Learned kernels absorb the difference.
- ✓Parameters ; multiply-adds .
- ✓Locality, parameter sharing and translation equivariance are the three built-in assumptions.
- ✓im2col turns the layer into one matrix multiply; the backward pass is a convolution with the flipped kernel (a transposed convolution) and a correlation of input with gradient.