# Filters and kernels

> A kernel is a small grid of weights whose dot product with each window measures how much the window looks like it. Image processing designs them by hand (blur, Sobel, Laplacian); a CNN learns a (C_out, C_in, k, k) bank of them, and its first layer ends up with Gabor-like edge and colour filters.

Source: https://www.mlfoundry.in/learn/filters-and-kernels  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 13 min read · updated 2026-10-11

**Kernel (filter):** A kernel is the small array of weights $K\in\mathbb R^{k\times k}$ that a convolution slides over its input. Its response at each position is the dot product $\langle K,\text{window}\rangle$, so it is largest where the window looks most like $K$. In a CNN layer, a **filter** is the full $C_{in}\times k\times k$ stack that produces one output channel, and the layer holds $C_{out}$ of them in a weight tensor of shape $(C_{out},C_{in},k,k)$. The two words are often used interchangeably.

Before deep learning, kernels were designed by hand for specific jobs: average to remove noise, difference to find edges, take a second difference to find blobs and corners. A CNN replaces the design with learning. Its kernels start as random numbers and are shaped by gradient descent into whatever detectors reduce the loss. The surprising empirical result is that the first layer of almost every image CNN learns roughly the same kernels: oriented edge detectors that look like **Gabor** filters, plus colour-opponent blobs.

Hand-designed kernels have a long history in image processing. The Sobel operator (Sobel and Feldman, 1968) estimates the image gradient. The Laplacian of Gaussian (Marr and Hildreth, 1980) finds edges as zero crossings. Gabor (1946) introduced the Gaussian-windowed sinusoid in signal analysis, and Daugman (1985) showed that 2-D Gabor functions fit the receptive fields of simple cells in the visual cortex. The learned version appears in Krizhevsky, Sutskever and Hinton's **AlexNet** (2012), whose Figure 3 shows the 96 first-layer kernels of size $11\times11\times3$, and in Zeiler and Fergus (2014, "Visualizing and Understanding Convolutional Networks"), who used these visualizations to fix AlexNet's first layer (see **ZFNet**).

## A kernel is a template: response = dot product

$$
y_{i,j}=\sum_{m=-h}^{h}\sum_{n=-h}^{h}K_{m,n}\,x_{i+m,\,j+n}=\big\langle K,\;x_{[i-h:i+h],\,[j-h:j+h]}\big\rangle,\qquad \frac{\partial y_{i,j}}{\partial K_{m,n}}=x_{i+m,\,j+n}
$$

- $K$: the $k\times k$ kernel, $k=2h+1$, indexed from its centre
- $\langle\cdot,\cdot\rangle$: sum of elementwise products; by Cauchy–Schwarz it is largest, for fixed window energy, when the window is proportional to $K$
- $\sum K$: the **DC gain**: the response to a constant image. Smoothers have $\sum K=1$ (brightness kept), detectors have $\sum K=0$ (flat regions give 0)
- $\partial y/\partial K$: the window itself: a learned kernel moves toward the patches that the loss wants it to respond to

The derivative explains what training does. With upstream gradient $G$, the update is $\Delta K\propto-\sum_{i,j}G_{i,j}\,\text{window}_{i,j}$: a weighted average of the input patches. A kernel literally becomes a blend of the patches it should fire on, minus the ones it should not. That is why first-layer kernels can be displayed as images and still make sense.

## The classic hand-designed kernels

Standard 3×3 kernels (Gaussian shown as its 3×3 binomial approximation). DC gain = sum of the weights.

| kernel | weights (rows) | DC gain | separable | what it does |
| --- | --- | --- | --- | --- |
| box blur | all 1/9 | 1 | yes: $\tfrac13(1,1,1)^\top\cdot\tfrac13(1,1,1)$ | local mean; removes noise and fine detail |
| Gaussian (binomial) | (1,2,1; 2,4,2; 1,2,1)/16 | 1 | yes: $\tfrac14(1,2,1)^\top\cdot\tfrac14(1,2,1)$ | smooth blur without the box's ringing; σ sets the scale |
| Sobel x | (−1,0,1; −2,0,2; −1,0,1) | 0 | yes: $(1,2,1)^\top\cdot(-1,0,1)$ | horizontal derivative: responds to vertical edges |
| Sobel y | (−1,−2,−1; 0,0,0; 1,2,1) | 0 | yes: $(-1,0,1)^\top\cdot(1,2,1)$ | vertical derivative: responds to horizontal edges |
| Laplacian | (0,1,0; 1,−4,1; 0,1,0) | 0 | no (sum of two separable terms) | second derivative: blobs, corners, edge zero-crossings |
| sharpen | (0,−1,0; −1,5,−1; 0,−1,0) | 1 | no | identity minus Laplacian: boosts edges, keeps brightness |

Sobel x is a smoothing $(1,2,1)$ down the columns times a central difference $(-1,0,1)$ along the rows. The smoothing makes it less noise-sensitive than a bare difference. Sharpening is $\delta-\nabla^2$: the identity kernel minus the Laplacian, so it adds back the second derivative and exaggerates every edge.

> Figure: A procedurally generated 32×32 image (a brightness ramp, a bright disk, a grey square and a thin diagonal bar) filtered with the selected kernel ('same' size, replicate padding). Blue is positive, red negative. With Sobel x the left edge of the disk gives +3.07 at pixel (9, 3) and the right edge −2.74 at (9, 15); flat regions give about 0 because the DC gain is 0. Box, Gaussian and sharpen have DC gain 1, and Sobel, Laplacian and Gabor have 0. Sobel x factors as (1,2,1)ᵀ(−1,0,1), so it costs 6 instead of 9 multiply-adds per pixel; the σ = 2 Gaussian is 9×9 and drops from 81 to 18 (4.5× fewer). Gabor filters are 7×7, have DC gain 0 and are not separable. Rotate θ to pick out the bar and the edges at each orientation.

## What a CNN learns instead

A conv layer's weight is a 4-D tensor $(C_{out},C_{in},k,k)$. For the first layer of an RGB network, each filter is a $3\times k\times k$ colour image and can be displayed as one. AlexNet's first layer has 96 such filters of size $11\times11\times3$ ($96\cdot3\cdot121=34{,}848$ weights). Krizhevsky et al. observed that the network, split across two GPUs, specialised them: the filters on one GPU were largely colour-agnostic oriented edges and gratings, the ones on the other were colour-specific blobs. ResNet's stem holds 64 filters of size $7\times7\times3$ (9,408 weights) and learns the same kinds of pattern.

$$
g_{\theta,\lambda,\sigma,\gamma}(x,y)=\exp\!\Big(-\frac{x'^2+\gamma^2y'^2}{2\sigma^2}\Big)\cos\!\Big(\frac{2\pi x'}{\lambda}\Big),\qquad x'=x\cos\theta+y\sin\theta,\; y'=-x\sin\theta+y\cos\theta
$$

- $\theta$: orientation; the stripes run perpendicular to the $x'$ axis
- $\lambda$: wavelength of the carrier (4 px in the figure), so the spatial frequency the filter is tuned to
- $\sigma,\ \gamma$: envelope width (1.6 px) and aspect ratio (0.6): how local and how elongated the filter is

Why Gabors? Natural images are dominated by edges at all orientations and scales, and a Gaussian-windowed sinusoid is the filter with the best joint localisation in space and frequency. Olshausen and Field (1996) found that learning a **sparse** code for natural image patches produces Gabor-like filters without any supervision, and Yosinski et al. (2014, "How transferable are features in deep neural networks?") showed that such first-layer features are general: they transfer almost perfectly between tasks. Deeper layers are task-specific and do not look like anything simple.

Visualising filters only works directly for the first layer, where the weights live in pixel space. A deeper filter has $C_{in}=256$ input channels that are themselves features, so plotting its weights says little. For those layers people visualise what the filter responds to instead: the top-activating image patches, Zeiler and Fergus's deconvnet projections, or images optimised to maximise a unit (Erhan et al., 2009; Olah, Mordvintsev and Schubert, "Feature Visualization", Distill 2017). See **Feature maps**.

> **Filter visualisations are a debugging tool:** Zeiler and Fergus noticed that AlexNet's first-layer filters were a mix of very high and very low frequencies with little in between, and that the second layer showed aliasing artifacts from the stride-4 first layer. Shrinking the first layer to $7\times7$ with stride 2 gave cleaner filters and better accuracy. Noisy, unstructured first-layer filters are a common sign of a network that has not converged or has too high a learning rate.

## Separable kernels: k² becomes 2k

A kernel is **separable** when it is an outer product of a column and a row, $K=u\,v^\top$ (rank 1). Then filtering with $K$ equals filtering with the column $u$ and then with the row $v$, and the cost per output drops from $k^2$ to $2k$ multiply-adds. Every Gaussian is separable, which is why large blurs are cheap. For any kernel, the singular value decomposition $K=\sum_r s_r u_rv_r^\top$ gives the best rank-$r$ approximation, a sum of $r$ separable passes.

$$
K=u\,v^\top\;\Rightarrow\;x\star K=(x\star u)\star v^\top,\qquad \text{cost per pixel: }k^2\;\to\;2k
$$

- $u,\ v$: a $k\times1$ column filter and a $1\times k$ row filter
- $\star$: cross-correlation; associativity of the two 1-D passes is what makes the split exact

CNNs use the same trick with learned kernels. Inception-v3 (Szegedy et al., 2016) replaces a $7\times7$ convolution by $1\times7$ followed by $7\times1$, and Jaderberg, Vedaldi and Zisserman (2014) sped up trained networks by approximating their filters with low-rank, separable ones. Do not confuse this **spatial** separability with **Depthwise separable convolution**, which separates the channel mixing from the spatial filtering.

## Worked example: one 3×3 patch, six kernels

Take the patch $P=\begin{pmatrix}2&2&8\\2&2&8\\2&8&8\end{pmatrix}$: dark on the left, bright on the right, and brighter toward the bottom-right corner. Each kernel's response is $\langle K,P\rangle$ (checked in Node):

Responses of the classic kernels to the patch P

| kernel | computation | response |
| --- | --- | --- |
| Sobel x | $(-2+8)+2(-2+8)+(-2+8)$ | 24 |
| Sobel y | $-(2+4+8)+(2+16+8)$ | 12 |
| gradient magnitude | $\sqrt{24^2+12^2}$ | 26.8, direction $\arctan(12/24)=26.6°$ |
| Laplacian | $2+2+8+8-4\cdot2$ | 12 |
| sharpen | $5\cdot2-(2+2+8+8)$ | −10 |
| box blur | $42/9$ | 4.67 |

**Sobel x as two 1-D passes**

1. Column pass with $u=(1,2,1)^\top$: each column of $P$ becomes $p_0+2p_1+p_2$, giving $(2+4+2,\;2+4+8,\;8+16+8)=(8,\,14,\,32)$.
2. Row pass with $v=(-1,0,1)$: $-8+0\cdot14+32=24$, the same as the full $3\times3$ kernel.
3. Cost: 3 + 3 = 6 multiply-adds instead of 9. For a $7\times7$ separable kernel it would be 14 instead of 49.

```python
import torch, torch.nn as nn, torch.nn.functional as F
import torchvision

# Hand-designed: put Sobel x and Sobel y into a frozen 1 -> 2 channel conv
sobel_x = torch.tensor([[-1., 0, 1], [-2, 0, 2], [-1, 0, 1]])
edges = nn.Conv2d(1, 2, kernel_size=3, padding=1, bias=False, padding_mode="replicate")
with torch.no_grad():
    edges.weight.copy_(torch.stack([sobel_x, sobel_x.T]).unsqueeze(1))   # shape (C_out=2, C_in=1, 3, 3)
edges.requires_grad_(False)

P = torch.tensor([[2., 2, 8], [2, 2, 8], [2, 8, 8]]).view(1, 1, 3, 3)
print(F.conv2d(P, edges.weight).flatten())          # tensor([24., 12.])

# Separable: a column pass then a row pass gives the same result
u = torch.tensor([1., 2, 1]).view(1, 1, 3, 1)
v = torch.tensor([-1., 0, 1]).view(1, 1, 1, 3)
print(F.conv2d(F.conv2d(P, u), v).item())           # 24.0

# Learned: the 64 first-layer filters of an ImageNet ResNet-18
m = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1)
w = m.conv1.weight.detach()                          # (64, 3, 7, 7)
w = (w - w.amin(dim=(1, 2, 3), keepdim=True)) / (w.amax(dim=(1, 2, 3), keepdim=True) - w.amin(dim=(1, 2, 3), keepdim=True))
grid = torchvision.utils.make_grid(w, nrow=8, padding=1)   # (3, H, W) image: oriented edges and colour blobs
torchvision.utils.save_image(grid, "conv1_filters.png")

# How close is each learned filter to rank 1 (separable)?
s = torch.linalg.svdvals(w.mean(1))                  # singular values of each 7x7 (channel-averaged)
print((s[:, 0] ** 2 / (s ** 2).sum(1)).mean())       # share of energy in the first separable term
```

Fixed kernels as Conv2d weights, a separable pass, and looking at learned filters

> **Kernel orientation and sign conventions:** Whether "Sobel x" responds positively to dark-to-bright or bright-to-dark edges depends on whether the library flips the kernel. PyTorch cross-correlates, so $(-1,0,1)$ gives a positive response where brightness increases to the right. SciPy's `convolve2d` flips it and gives the opposite sign. Check one edge numerically before trusting a hand-copied kernel.

> **Kernel size in modern CNNs:** Since VGG, most CNNs use $3\times3$ kernels everywhere: two stacked $3\times3$ layers cover a $5\times5$ region with $18$ instead of $25$ weights per channel pair and an extra nonlinearity (see **VGGNet**). The stem may be larger ($7\times7$ in ResNet). **ConvNeXt** went back to $7\times7$ depthwise kernels, which are cheap because each touches a single channel.

**Strengths**

- Hand-designed kernels are interpretable, need no data and have known properties (DC gain, separability, frequency response)
- Learned kernels adapt to the task and the data, and the first layer learns general, reusable edge and colour detectors
- Separable kernels cut the cost from $k^2$ to $2k$ per output

**Limitations**

- Hand-designed banks cannot match learned features on recognition tasks
- Learned kernels beyond the first layer are hard to interpret from their weights alone
- Large kernels are expensive unless separable, depthwise or factorised

**Question:** A kernel's weights sum to 0. What is its response to a perfectly flat (constant) region of the image?

- A. 0
- B. The constant value itself
- C. The constant times the number of weights
- D. It depends on the kernel size

**Answer:** A. 0 — On a constant patch $x\equiv c$ the response is $\sum K_{m,n}c=c\sum K=0$. Derivative-type kernels (Sobel, Laplacian, mean-subtracted Gabor) are built this way so they ignore brightness and respond only to change.

**Question:** A conv layer has weight shape (128, 64, 3, 3). Which statement is correct?

- A. It has 128 filters, each a 64×3×3 stack that produces one output channel
- B. It has 64 filters, each producing 128 output channels
- C. It applies one 3×3 kernel to all 64 channels and copies the result 128 times
- D. It has 128·3·3 = 1,152 weights

**Answer:** A. It has 128 filters, each a 64×3×3 stack that produces one output channel — PyTorch stores Conv2d weights as $(C_{out},C_{in},k_h,k_w)$. Each of the 128 filters covers all 64 input channels with its own $3\times3$ slice per channel, for $128\cdot64\cdot9=73{,}728$ weights.

**Key points**

- A kernel's response is a dot product with the window: it fires where the window looks like the kernel.
- DC gain $\sum K$: 1 for smoothers (box, Gaussian, sharpen), 0 for detectors (Sobel, Laplacian, Gabor).
- A layer's filter bank has shape $(C_{out},C_{in},k,k)$; each filter spans all input channels.
- First layers learn Gabor-like oriented edges and colour blobs (AlexNet, Zeiler and Fergus); deeper filters are visualised by what activates them.
- Separable (rank-1) kernels $K=uv^\top$ cost $2k$ instead of $k^2$ per output.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
