Filters and kernels

A kernel is a small grid of weights whose dot product with each window measures how much the window looks like it. Image processing designs them by hand (blur, Sobel, Laplacian); a CNN learns a (C_out, C_in, k, k) bank of them, and its first layer ends up with Gabor-like edge and colour filters.

13 min read1091 wordscore0 viewsupdated 2026-10-11

Definition

Kernel (filter) A kernel is the small array of weights K∈Rk×kK\in\mathbb R^{k\times k} that a convolution slides over its input. Its response at each position is the dot product ⟨K,window⟩\langle K,\text{window}\rangle, so it is largest where the window looks most like KK. In a CNN layer, a filter is the full Cin×k×kC_{in}\times k\times k stack that produces one output channel, and the layer holds CoutC_{out} of them in a weight tensor of shape (Cout,Cin,k,k)(C_{out},C_{in},k,k). The two words are often used interchangeably.

Before deep learning, kernels were designed by hand for specific jobs: average to remove noise, difference to find edges, take a second difference to find blobs and corners. A CNN replaces the design with learning. Its kernels start as random numbers and are shaped by gradient descent into whatever detectors reduce the loss. The surprising empirical result is that the first layer of almost every image CNN learns roughly the same kernels: oriented edge detectors that look like Gabor filters, plus colour-opponent blobs.

Hand-designed kernels have a long history in image processing. The Sobel operator (Sobel and Feldman, 1968) estimates the image gradient. The Laplacian of Gaussian (Marr and Hildreth, 1980) finds edges as zero crossings. Gabor (1946) introduced the Gaussian-windowed sinusoid in signal analysis, and Daugman (1985) showed that 2-D Gabor functions fit the receptive fields of simple cells in the visual cortex. The learned version appears in Krizhevsky, Sutskever and Hinton's AlexNet (2012), whose Figure 3 shows the 96 first-layer kernels of size 11×11×311\times11\times3, and in Zeiler and Fergus (2014, "Visualizing and Understanding Convolutional Networks"), who used these visualizations to fix AlexNet's first layer (see ZFNet).

A kernel is a template: response = dot product

yi,j=∑m=−hh∑n=−hhKm,n xi+m, j+n=⟨K,  x[i−h:i+h], [j−h:j+h]⟩,∂yi,j∂Km,n=xi+m, j+ny_{i,j}=\sum_{m=-h}^{h}\sum_{n=-h}^{h}K_{m,n}\,x_{i+m,\,j+n}=\big\langle K,\;x_{[i-h:i+h],\,[j-h:j+h]}\big\rangle,\qquad \frac{\partial y_{i,j}}{\partial K_{m,n}}=x_{i+m,\,j+n}
KK
the k×kk\times k kernel, k=2h+1k=2h+1, indexed from its centre
⟨⋅,⋅⟩\langle\cdot,\cdot\rangle
sum of elementwise products; by Cauchy–Schwarz it is largest, for fixed window energy, when the window is proportional to KK
∑K\sum K
the DC gain: the response to a constant image. Smoothers have ∑K=1\sum K=1 (brightness kept), detectors have ∑K=0\sum K=0 (flat regions give 0)
∂y/∂K\partial y/\partial K
the window itself: a learned kernel moves toward the patches that the loss wants it to respond to

The derivative explains what training does. With upstream gradient GG, the update is ΔK∝−∑i,jGi,j windowi,j\Delta K\propto-\sum_{i,j}G_{i,j}\,\text{window}_{i,j}: a weighted average of the input patches. A kernel literally becomes a blend of the patches it should fire on, minus the ones it should not. That is why first-layer kernels can be displayed as images and still make sense.

The classic hand-designed kernels

kernelweights (rows)DC gainseparablewhat it does
box blurall 1/91yes: 13(1,1,1)⊤⋅13(1,1,1)\tfrac13(1,1,1)^\top\cdot\tfrac13(1,1,1)local mean; removes noise and fine detail
Gaussian (binomial)(1,2,1; 2,4,2; 1,2,1)/161yes: 14(1,2,1)⊤⋅14(1,2,1)\tfrac14(1,2,1)^\top\cdot\tfrac14(1,2,1)smooth blur without the box's ringing; σ sets the scale
Sobel x(−1,0,1; −2,0,2; −1,0,1)0yes: (1,2,1)⊤⋅(−1,0,1)(1,2,1)^\top\cdot(-1,0,1)horizontal derivative: responds to vertical edges
Sobel y(−1,−2,−1; 0,0,0; 1,2,1)0yes: (−1,0,1)⊤⋅(1,2,1)(-1,0,1)^\top\cdot(1,2,1)vertical derivative: responds to horizontal edges
Laplacian(0,1,0; 1,−4,1; 0,1,0)0no (sum of two separable terms)second derivative: blobs, corners, edge zero-crossings
sharpen(0,−1,0; −1,5,−1; 0,−1,0)1noidentity minus Laplacian: boosts edges, keeps brightness
Standard 3×3 kernels (Gaussian shown as its 3×3 binomial approximation). DC gain = sum of the weights.

Sobel x is a smoothing (1,2,1)(1,2,1) down the columns times a central difference (−1,0,1)(-1,0,1) along the rows. The smoothing makes it less noise-sensitive than a bare difference. Sharpening is δ−∇2\delta-\nabla^2: the identity kernel minus the Laplacian, so it adds back the second derivative and exaggerates every edge.

input 32×32 (click to probe)kernel Sobel x 3×3−101−202−101output (same size)DC gain Σ K = 0.000flat regions → 0: a detectorseparable: yesK = u vᵀ, u = (1, 2, 1), v = (−1, 0, 1)MACs / pixel: 9as two 1-D passes: 6 (1.5× fewer)output range [−2.83, 3.29]mean |out| = 0.303probe (9,3): Σ K ⊙ patch = 3.074 (patch centre 0.13)■ positive ■ negative · cross-correlation, replicate padding
fig.A procedurally generated 32×32 image (a brightness ramp, a bright disk, a grey square and a thin diagonal bar) filtered with the selected kernel ('same' size, replicate padding). Blue is positive, red negative. With Sobel x the left edge of the disk gives +3.07 at pixel (9, 3) and the right edge −2.74 at (9, 15); flat regions give about 0 because the DC gain is 0. Box, Gaussian and sharpen have DC gain 1, and Sobel, Laplacian and Gabor have 0. Sobel x factors as (1,2,1)ᵀ(−1,0,1), so it costs 6 instead of 9 multiply-adds per pixel; the σ = 2 Gaussian is 9×9 and drops from 81 to 18 (4.5× fewer). Gabor filters are 7×7, have DC gain 0 and are not separable. Rotate θ to pick out the bar and the edges at each orientation.

What a CNN learns instead

A conv layer's weight is a 4-D tensor (Cout,Cin,k,k)(C_{out},C_{in},k,k). For the first layer of an RGB network, each filter is a 3×k×k3\times k\times k colour image and can be displayed as one. AlexNet's first layer has 96 such filters of size 11×11×311\times11\times3 (96⋅3⋅121=34,84896\cdot3\cdot121=34{,}848 weights). Krizhevsky et al. observed that the network, split across two GPUs, specialised them: the filters on one GPU were largely colour-agnostic oriented edges and gratings, the ones on the other were colour-specific blobs. ResNet's stem holds 64 filters of size 7×7×37\times7\times3 (9,408 weights) and learns the same kinds of pattern.

gθ,λ,σ,γ(x,y)=exp⁡ ⁣(−x′2+γ2y′22σ2)cos⁡ ⁣(2πx′λ),x′=xcos⁡θ+ysin⁡θ,  y′=−xsin⁡θ+ycos⁡θg_{\theta,\lambda,\sigma,\gamma}(x,y)=\exp\!\Big(-\frac{x'^2+\gamma^2y'^2}{2\sigma^2}\Big)\cos\!\Big(\frac{2\pi x'}{\lambda}\Big),\qquad x'=x\cos\theta+y\sin\theta,\; y'=-x\sin\theta+y\cos\theta
θ\theta
orientation; the stripes run perpendicular to the x′x' axis
λ\lambda
wavelength of the carrier (4 px in the figure), so the spatial frequency the filter is tuned to
σ, γ\sigma,\ \gamma
envelope width (1.6 px) and aspect ratio (0.6): how local and how elongated the filter is

Why Gabors? Natural images are dominated by edges at all orientations and scales, and a Gaussian-windowed sinusoid is the filter with the best joint localisation in space and frequency. Olshausen and Field (1996) found that learning a sparse code for natural image patches produces Gabor-like filters without any supervision, and Yosinski et al. (2014, "How transferable are features in deep neural networks?") showed that such first-layer features are general: they transfer almost perfectly between tasks. Deeper layers are task-specific and do not look like anything simple.

Visualising filters only works directly for the first layer, where the weights live in pixel space. A deeper filter has Cin=256C_{in}=256 input channels that are themselves features, so plotting its weights says little. For those layers people visualise what the filter responds to instead: the top-activating image patches, Zeiler and Fergus's deconvnet projections, or images optimised to maximise a unit (Erhan et al., 2009; Olah, Mordvintsev and Schubert, "Feature Visualization", Distill 2017). See Feature maps.

Separable kernels: k² becomes 2k

A kernel is separable when it is an outer product of a column and a row, K=u v⊤K=u\,v^\top (rank 1). Then filtering with KK equals filtering with the column uu and then with the row vv, and the cost per output drops from k2k^2 to 2k2k multiply-adds. Every Gaussian is separable, which is why large blurs are cheap. For any kernel, the singular value decomposition K=∑rsrurvr⊤K=\sum_r s_r u_rv_r^\top gives the best rank-rr approximation, a sum of rr separable passes.

K=u v⊤  ⇒  x⋆K=(x⋆u)⋆v⊤,cost per pixel: k2  →  2kK=u\,v^\top\;\Rightarrow\;x\star K=(x\star u)\star v^\top,\qquad \text{cost per pixel: }k^2\;\to\;2k
u, vu,\ v
a k×1k\times1 column filter and a 1×k1\times k row filter
⋆\star
cross-correlation; associativity of the two 1-D passes is what makes the split exact

CNNs use the same trick with learned kernels. Inception-v3 (Szegedy et al., 2016) replaces a 7×77\times7 convolution by 1×71\times7 followed by 7×17\times1, and Jaderberg, Vedaldi and Zisserman (2014) sped up trained networks by approximating their filters with low-rank, separable ones. Do not confuse this spatial separability with Depthwise separable convolution, which separates the channel mixing from the spatial filtering.

Worked example: one 3×3 patch, six kernels

Take the patch P=(228228288)P=\begin{pmatrix}2&2&8\\2&2&8\\2&8&8\end{pmatrix}: dark on the left, bright on the right, and brighter toward the bottom-right corner. Each kernel's response is ⟨K,P⟩\langle K,P\rangle (checked in Node):

kernelcomputationresponse
Sobel x(−2+8)+2(−2+8)+(−2+8)(-2+8)+2(-2+8)+(-2+8)24
Sobel y−(2+4+8)+(2+16+8)-(2+4+8)+(2+16+8)12
gradient magnitude242+122\sqrt{24^2+12^2}26.8, direction arctan⁡(12/24)=26.6°\arctan(12/24)=26.6°
Laplacian2+2+8+8−4⋅22+2+8+8-4\cdot212
sharpen5⋅2−(2+2+8+8)5\cdot2-(2+2+8+8)−10
box blur42/942/94.67
Responses of the classic kernels to the patch P

Sobel x as two 1-D passes

  1. 1Column pass with u=(1,2,1)⊤u=(1,2,1)^\top: each column of PP becomes p0+2p1+p2p_0+2p_1+p_2, giving (2+4+2,  2+4+8,  8+16+8)=(8, 14, 32)(2+4+2,\;2+4+8,\;8+16+8)=(8,\,14,\,32).
  2. 2Row pass with v=(−1,0,1)v=(-1,0,1): −8+0⋅14+32=24-8+0\cdot14+32=24, the same as the full 3×33\times3 kernel.
  3. 3Cost: 3 + 3 = 6 multiply-adds instead of 9. For a 7×77\times7 separable kernel it would be 14 instead of 49.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2import torchvision
3
4# Hand-designed: put Sobel x and Sobel y into a frozen 1 -> 2 channel conv
5sobel_x = torch.tensor([[-1., 0, 1], [-2, 0, 2], [-1, 0, 1]])
6edges = nn.Conv2d(1, 2, kernel_size=3, padding=1, bias=False, padding_mode="replicate")
7with torch.no_grad():
8 edges.weight.copy_(torch.stack([sobel_x, sobel_x.T]).unsqueeze(1)) # shape (C_out=2, C_in=1, 3, 3)
9edges.requires_grad_(False)
10
11P = torch.tensor([[2., 2, 8], [2, 2, 8], [2, 8, 8]]).view(1, 1, 3, 3)
12print(F.conv2d(P, edges.weight).flatten()) # tensor([24., 12.])
13
14# Separable: a column pass then a row pass gives the same result
15u = torch.tensor([1., 2, 1]).view(1, 1, 3, 1)
16v = torch.tensor([-1., 0, 1]).view(1, 1, 1, 3)
17print(F.conv2d(F.conv2d(P, u), v).item()) # 24.0
18
19# Learned: the 64 first-layer filters of an ImageNet ResNet-18
20m = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1)
21w = m.conv1.weight.detach() # (64, 3, 7, 7)
22w = (w - w.amin(dim=(1, 2, 3), keepdim=True)) / (w.amax(dim=(1, 2, 3), keepdim=True) - w.amin(dim=(1, 2, 3), keepdim=True))
23grid = torchvision.utils.make_grid(w, nrow=8, padding=1) # (3, H, W) image: oriented edges and colour blobs
24torchvision.utils.save_image(grid, "conv1_filters.png")
25
26# How close is each learned filter to rank 1 (separable)?
27s = torch.linalg.svdvals(w.mean(1)) # singular values of each 7x7 (channel-averaged)
28print((s[:, 0] ** 2 / (s ** 2).sum(1)).mean()) # share of energy in the first separable term
Fixed kernels as Conv2d weights, a separable pass, and looking at learned filters

Use when

  • +Hand-designed kernels are interpretable, need no data and have known properties (DC gain, separability, frequency response)
  • +Learned kernels adapt to the task and the data, and the first layer learns general, reusable edge and colour detectors
  • +Separable kernels cut the cost from k2k^2 to 2k2k per output

Avoid when

  • −Hand-designed banks cannot match learned features on recognition tasks
  • −Learned kernels beyond the first layer are hard to interpret from their weights alone
  • −Large kernels are expensive unless separable, depthwise or factorised

Check yourself

A kernel's weights sum to 0. What is its response to a perfectly flat (constant) region of the image?

Check yourself

A conv layer has weight shape (128, 64, 3, 3). Which statement is correct?

Remember

  • ✓A kernel's response is a dot product with the window: it fires where the window looks like the kernel.
  • ✓DC gain ∑K\sum K: 1 for smoothers (box, Gaussian, sharpen), 0 for detectors (Sobel, Laplacian, Gabor).
  • ✓A layer's filter bank has shape (Cout,Cin,k,k)(C_{out},C_{in},k,k); each filter spans all input channels.
  • ✓First layers learn Gabor-like oriented edges and colour blobs (AlexNet, Zeiler and Fergus); deeper filters are visualised by what activates them.
  • ✓Separable (rank-1) kernels K=uv⊤K=uv^\top cost 2k2k instead of k2k^2 per output.
← back to Deep Learning roadmap