Definition
Kernel (filter) A kernel is the small array of weights that a convolution slides over its input. Its response at each position is the dot product , so it is largest where the window looks most like . In a CNN layer, a filter is the full stack that produces one output channel, and the layer holds of them in a weight tensor of shape . The two words are often used interchangeably.
Before deep learning, kernels were designed by hand for specific jobs: average to remove noise, difference to find edges, take a second difference to find blobs and corners. A CNN replaces the design with learning. Its kernels start as random numbers and are shaped by gradient descent into whatever detectors reduce the loss. The surprising empirical result is that the first layer of almost every image CNN learns roughly the same kernels: oriented edge detectors that look like Gabor filters, plus colour-opponent blobs.
Hand-designed kernels have a long history in image processing. The Sobel operator (Sobel and Feldman, 1968) estimates the image gradient. The Laplacian of Gaussian (Marr and Hildreth, 1980) finds edges as zero crossings. Gabor (1946) introduced the Gaussian-windowed sinusoid in signal analysis, and Daugman (1985) showed that 2-D Gabor functions fit the receptive fields of simple cells in the visual cortex. The learned version appears in Krizhevsky, Sutskever and Hinton's AlexNet (2012), whose Figure 3 shows the 96 first-layer kernels of size , and in Zeiler and Fergus (2014, "Visualizing and Understanding Convolutional Networks"), who used these visualizations to fix AlexNet's first layer (see ZFNet).
A kernel is a template: response = dot product
- the kernel, , indexed from its centre
- sum of elementwise products; by Cauchy–Schwarz it is largest, for fixed window energy, when the window is proportional to
- the DC gain: the response to a constant image. Smoothers have (brightness kept), detectors have (flat regions give 0)
- the window itself: a learned kernel moves toward the patches that the loss wants it to respond to
The derivative explains what training does. With upstream gradient , the update is : a weighted average of the input patches. A kernel literally becomes a blend of the patches it should fire on, minus the ones it should not. That is why first-layer kernels can be displayed as images and still make sense.
The classic hand-designed kernels
| kernel | weights (rows) | DC gain | separable | what it does |
|---|---|---|---|---|
| box blur | all 1/9 | 1 | yes: | local mean; removes noise and fine detail |
| Gaussian (binomial) | (1,2,1; 2,4,2; 1,2,1)/16 | 1 | yes: | smooth blur without the box's ringing; σ sets the scale |
| Sobel x | (−1,0,1; −2,0,2; −1,0,1) | 0 | yes: | horizontal derivative: responds to vertical edges |
| Sobel y | (−1,−2,−1; 0,0,0; 1,2,1) | 0 | yes: | vertical derivative: responds to horizontal edges |
| Laplacian | (0,1,0; 1,−4,1; 0,1,0) | 0 | no (sum of two separable terms) | second derivative: blobs, corners, edge zero-crossings |
| sharpen | (0,−1,0; −1,5,−1; 0,−1,0) | 1 | no | identity minus Laplacian: boosts edges, keeps brightness |
Sobel x is a smoothing down the columns times a central difference along the rows. The smoothing makes it less noise-sensitive than a bare difference. Sharpening is : the identity kernel minus the Laplacian, so it adds back the second derivative and exaggerates every edge.
What a CNN learns instead
A conv layer's weight is a 4-D tensor . For the first layer of an RGB network, each filter is a colour image and can be displayed as one. AlexNet's first layer has 96 such filters of size ( weights). Krizhevsky et al. observed that the network, split across two GPUs, specialised them: the filters on one GPU were largely colour-agnostic oriented edges and gratings, the ones on the other were colour-specific blobs. ResNet's stem holds 64 filters of size (9,408 weights) and learns the same kinds of pattern.
- orientation; the stripes run perpendicular to the axis
- wavelength of the carrier (4 px in the figure), so the spatial frequency the filter is tuned to
- envelope width (1.6 px) and aspect ratio (0.6): how local and how elongated the filter is
Why Gabors? Natural images are dominated by edges at all orientations and scales, and a Gaussian-windowed sinusoid is the filter with the best joint localisation in space and frequency. Olshausen and Field (1996) found that learning a sparse code for natural image patches produces Gabor-like filters without any supervision, and Yosinski et al. (2014, "How transferable are features in deep neural networks?") showed that such first-layer features are general: they transfer almost perfectly between tasks. Deeper layers are task-specific and do not look like anything simple.
Visualising filters only works directly for the first layer, where the weights live in pixel space. A deeper filter has input channels that are themselves features, so plotting its weights says little. For those layers people visualise what the filter responds to instead: the top-activating image patches, Zeiler and Fergus's deconvnet projections, or images optimised to maximise a unit (Erhan et al., 2009; Olah, Mordvintsev and Schubert, "Feature Visualization", Distill 2017). See Feature maps.
Separable kernels: k² becomes 2k
A kernel is separable when it is an outer product of a column and a row, (rank 1). Then filtering with equals filtering with the column and then with the row , and the cost per output drops from to multiply-adds. Every Gaussian is separable, which is why large blurs are cheap. For any kernel, the singular value decomposition gives the best rank- approximation, a sum of separable passes.
- a column filter and a row filter
- cross-correlation; associativity of the two 1-D passes is what makes the split exact
CNNs use the same trick with learned kernels. Inception-v3 (Szegedy et al., 2016) replaces a convolution by followed by , and Jaderberg, Vedaldi and Zisserman (2014) sped up trained networks by approximating their filters with low-rank, separable ones. Do not confuse this spatial separability with Depthwise separable convolution, which separates the channel mixing from the spatial filtering.
Worked example: one 3×3 patch, six kernels
Take the patch : dark on the left, bright on the right, and brighter toward the bottom-right corner. Each kernel's response is (checked in Node):
| kernel | computation | response |
|---|---|---|
| Sobel x | 24 | |
| Sobel y | 12 | |
| gradient magnitude | 26.8, direction | |
| Laplacian | 12 | |
| sharpen | −10 | |
| box blur | 4.67 |
Sobel x as two 1-D passes
- 1Column pass with : each column of becomes , giving .
- 2Row pass with : , the same as the full kernel.
- 3Cost: 3 + 3 = 6 multiply-adds instead of 9. For a separable kernel it would be 14 instead of 49.
1import torch, torch.nn as nn, torch.nn.functional as F2import torchvision34# Hand-designed: put Sobel x and Sobel y into a frozen 1 -> 2 channel conv5sobel_x = torch.tensor([[-1., 0, 1], [-2, 0, 2], [-1, 0, 1]])6edges = nn.Conv2d(1, 2, kernel_size=3, padding=1, bias=False, padding_mode="replicate")7with torch.no_grad():8 edges.weight.copy_(torch.stack([sobel_x, sobel_x.T]).unsqueeze(1)) # shape (C_out=2, C_in=1, 3, 3)9edges.requires_grad_(False)1011P = torch.tensor([[2., 2, 8], [2, 2, 8], [2, 8, 8]]).view(1, 1, 3, 3)12print(F.conv2d(P, edges.weight).flatten()) # tensor([24., 12.])1314# Separable: a column pass then a row pass gives the same result15u = torch.tensor([1., 2, 1]).view(1, 1, 3, 1)16v = torch.tensor([-1., 0, 1]).view(1, 1, 1, 3)17print(F.conv2d(F.conv2d(P, u), v).item()) # 24.01819# Learned: the 64 first-layer filters of an ImageNet ResNet-1820m = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1)21w = m.conv1.weight.detach() # (64, 3, 7, 7)22w = (w - w.amin(dim=(1, 2, 3), keepdim=True)) / (w.amax(dim=(1, 2, 3), keepdim=True) - w.amin(dim=(1, 2, 3), keepdim=True))23grid = torchvision.utils.make_grid(w, nrow=8, padding=1) # (3, H, W) image: oriented edges and colour blobs24torchvision.utils.save_image(grid, "conv1_filters.png")2526# How close is each learned filter to rank 1 (separable)?27s = torch.linalg.svdvals(w.mean(1)) # singular values of each 7x7 (channel-averaged)28print((s[:, 0] ** 2 / (s ** 2).sum(1)).mean()) # share of energy in the first separable termUse when
- +Hand-designed kernels are interpretable, need no data and have known properties (DC gain, separability, frequency response)
- +Learned kernels adapt to the task and the data, and the first layer learns general, reusable edge and colour detectors
- +Separable kernels cut the cost from to per output
Avoid when
- −Hand-designed banks cannot match learned features on recognition tasks
- −Learned kernels beyond the first layer are hard to interpret from their weights alone
- −Large kernels are expensive unless separable, depthwise or factorised
Check yourself
A kernel's weights sum to 0. What is its response to a perfectly flat (constant) region of the image?
Check yourself
A conv layer has weight shape (128, 64, 3, 3). Which statement is correct?
Remember
- ✓A kernel's response is a dot product with the window: it fires where the window looks like the kernel.
- ✓DC gain : 1 for smoothers (box, Gaussian, sharpen), 0 for detectors (Sobel, Laplacian, Gabor).
- ✓A layer's filter bank has shape ; each filter spans all input channels.
- ✓First layers learn Gabor-like oriented edges and colour blobs (AlexNet, Zeiler and Fergus); deeper filters are visualised by what activates them.
- ✓Separable (rank-1) kernels cost instead of per output.