Feature maps

A conv layer outputs a stack of 2-D maps, one per filter: channel c says where in the image feature c was found. Deeper layers combine earlier channels into edges, textures, parts and objects while the maps shrink and the channels multiply.

11 min read1323 wordscore0 viewsupdated 2026-10-11

Definition

Feature map is the output of one filter of a convolutional layer: a 2-D grid ac[i,j]a_c[i,j] that holds the response of filter cc at every spatial position. A layer with CC filters produces CC feature maps, stored together as an activation tensor of shape (N,C,H,W)(N, C, H, W): batch, channels, height, width.

Read the tensor two ways. A slice a[n,c,:,:]a[n, c, :, :] is a map: where in image nn feature cc fires. A fibre a[n,:,i,j]a[n, :, i, j] is a CC-dimensional vector: what is present at position (i,j)(i,j), described in the vocabulary of that layer's CC detectors. The next convolution reads all CC channels in a small window, so it builds new detectors out of combinations of the old ones.

The name comes from the earliest CNNs. Fukushima's neocognitron (1980) arranged S-cells in "cell-planes", one per feature type, and LeCun et al.'s LeNet papers (1989, 1998) called the planes feature maps: every unit in a map shares the same weights, so the map is one feature detector scanned over the image. The word "channel" came later, by analogy with the RGB channels of the input. To the next layer, the 64 maps of a conv layer are simply a 64-channel image.

How a feature map is computed

ac(l)[i,j]=φ(bc(l)+∑c′=1Cl−1∑u,vWc,c′(l)[u,v]  ac′(l−1)[ si+u,  sj+v ])a^{(l)}_{c}[i,j]=\varphi\Big(b^{(l)}_c+\sum_{c'=1}^{C_{l-1}}\sum_{u,v}W^{(l)}_{c,c'}[u,v]\;a^{(l-1)}_{c'}[\,s i+u,\;s j+v\,]\Big)
ac(l)a^{(l)}_c
feature map cc of layer ll; a(0)a^{(0)} is the image, with C0=3C_0=3 for RGB
Wc,c′(l)W^{(l)}_{c,c'}
the k×kk\times k slice of filter cc that reads input channel c′c'; filter cc has Cl−1k2C_{l-1}k^2 weights in total
∑c′\sum_{c'}
every output map reads every input map: this is where features get combined
ss
stride; a stride-2 layer halves H and W
φ\varphi
the nonlinearity (usually ReLU, after BatchNorm); without it stacked layers collapse into one linear filter

The gradient tells the same story backwards. With zz the pre-activation, ∂L/∂Wc,c′[u,v]=∑n,i,jδc[n,i,j] ac′[n,si+u,sj+v]\partial L/\partial W_{c,c'}[u,v]=\sum_{n,i,j}\delta_c[n,i,j]\,a_{c'}[n,si+u,sj+v]: filter cc is updated by correlating its error map δc=∂L/∂zc\delta_c=\partial L/\partial z_c with every input map. Because the same weights are used at every (i,j)(i,j), every position contributes to one shared gradient. That is weight sharing, and it is why a single filter learns to detect its feature anywhere in the image.

At a glance

Shape
(N,C,H,W)(N,C,H,W) in PyTorch (channels-first); TensorFlow defaults to (N,H,W,C)(N,H,W,C)
One map
the response of one filter at every position: translation equivariant, a shifted input gives a shifted map
One fibre
a[n,:,i,j]a[n,:,i,j], the CC-dim descriptor of one location, seen through the layer's receptive field there
Typical sizes
ResNet-50 at 2242224^2: 64×112264\times112^2 after the stem conv, then 256×562256\times56^2, 512×282512\times28^2, 1024×1421024\times14^2, 2048×722048\times7^2
Memory
N⋅C⋅H⋅W⋅N\cdot C\cdot H\cdot W\cdot bytes per value, for every layer whose output backprop needs to keep
Sparsity
after ReLU many entries are exactly zero; a map is often blank for an image without its feature
input (1,1,20,20) → conv3×3×4+ReLU (1,4,20,20) → maxpool2 (1,4,10,10) → conv3×3×3+ReLU (1,3,10,10)input▢ layer-1 RF 3×3▢ layer-2 RF 8×80: vert edge →1: vert edge ←2: horiz edge ↓3: horiz edge ↑0: corner ⌜1: corner ⌟2: vert bareach layer-2 unit reads all 4 pooledlayer-1 channels: 4·3·3 + 1 = 37 weightschannel vector at the probelayer 1 [:, 5, 5]c0 3.0c1 0.0c2 3.0c3 0.0layer 2 [:, 2, 2]c0 4.0c1 0.0c2 0.0after ReLU: max · share of nonzero unitsL1 c0: 4.0 · 6%L1 c1: 4.0 · 6%L1 c2: 4.0 · 6%L1 c3: 4.0 · 6%L2 c0 corner ⌜: 4.0 · 1% · peak at (2, 2)L2 c1 corner ⌟: 4.0 · 1% · peak at (7, 7)L2 c2 vert bar: 0.0 · 0%tensor sizes: 400 → 1600 → 400 → 300 numbers; ×4 bytes in fp32, ×N for a batchshowing activations after ReLU; toggle to see the negative half
click any map to probe
fig.A two-layer toy CNN with hand-set weights on a 20×20 image. Layer 1 has four Sobel filters, one per edge polarity, giving maps of shape (1, 4, 20, 20); after a 2×2 max-pool, layer 2 has three filters that each read all four pooled maps (4·3·3 + 1 = 37 weights). On the square, every edge map peaks at 4 (a Sobel step response, 1+2+1) and fires on 6% of positions. The corner ⌜ unit fires at exactly one pooled position, (2, 2), the top-left corner, with value 4, and the vertical-bar unit stays silent. Switch to bars and the bar unit reaches 12 on 18% of the map. Click any map to read the channel vector at that location, and toggle pre-ReLU to see the negative half that the ReLU discards.

Channels are detectors, depth builds a hierarchy

In the figure, a layer-1 map responds to one local pattern: a dark-to-bright step in one direction. Layer 2 cannot see pixels. It sees four edge maps, and its corner filter puts positive weight on "bright to the right" just below its centre and on "bright below" just to its right. Its bias of −10-10 acts as a threshold. A straight edge alone gives at most 4+4=8<104+4=8<10 and leaves the unit off. Only a place where both edges meet switches it on. That is the general mechanism: each layer is a set of soft AND/OR detectors over the previous layer's detectors.

Zeiler and Fergus (2014, "Visualizing and Understanding Convolutional Networks") made this visible in a trained ImageNet network. They projected strong activations back to pixel space with a deconvnet. Layer 1 holds oriented edges and colour blobs. Layer 2 responds to corners and edge-colour conjunctions. Layer 3 picks up textures and repeated patterns such as meshes and text. Layer 4 shows class-specific parts such as dog faces and bird legs, and layer 5 responds to whole objects across large pose changes. They also observed that the lower layers converge within a few epochs, while the upper layers only develop after 40 to 50.

Resolution down, channels up

Almost every CNN trades spatial resolution for channels as it goes deeper (see CNN architecture). Each stride-2 stage halves HH and WW, so the number of positions drops by 4. The channel count is usually doubled at the same time, so the total number of activation values per stage halves, while the weights per layer, CinCoutk2C_{in}C_{out}k^2, grow by 4. Early maps are large, cheap in weights and expensive in memory. Late maps are tiny, rich in channels, and hold most of the parameters.

TensorShape (C × H × W)Valuesfp32
conv1 (stem conv)64×112×11264\times112\times112802,8163.06 MiB
layer1256×56×56256\times56\times56802,8163.06 MiB
layer2512×28×28512\times28\times28401,4081.53 MiB
layer31024×14×141024\times14\times14200,7040.77 MiB
layer42048×7×72048\times7\times7100,3520.38 MiB
Stage outputs of torchvision ResNet-50 for one 224×224 image (values counted by the module's lib; fp32 = 4 bytes)

The final 2048×7×72048\times7\times7 tensor is the network's description of the image: 49 positions, each a 2048-dim fibre. Global average pooling turns it into one 2048-dim vector for the classifier. Detection and segmentation heads instead use the maps of several stages at once (a feature pyramid), because the early maps keep the spatial detail that the late ones have lost.

Worked example: one corner unit

Probe the square image at its top-left corner pixel (5, 5)

  1. 1Layer 1. The 3×33\times3 window at (5,5)(5,5) covers rows and columns 4 to 6. The square starts at row and column 5, so the Sobel-xx filter sees +1+1 and +2+2 in its right column and nothing on the left: 33. Sobel-yy gives 33 too. The fibre is z=(3,−3,3,−3)z=(3,-3,3,-3) before ReLU and a=(3,0,3,0)a=(3,0,3,0) after it: each opposite-polarity filter is the exact negative of its partner, so ReLU keeps one of each pair.
  2. 2Pool. The 2×22\times2 max-pool maps (5,5)(5,5) to pooled cell (2,2)(2,2). Around it, the "edge →" map has 33 at (2,2)(2,2) and 44 just below it at (3,2)(3,2), where the edge runs straight. The "edge ↓" map has 33 at (2,2)(2,2) and 44 to its right at (2,3)(2,3).
  3. 3Layer 2. The ⌜ filter has weight 1 on exactly those four taps: z=(3+4)+(3+4)−10=4z=(3+4)+(3+4)-10=4, so a=4a=4. Its ⌟ partner reads the opposite polarities, which are 0 here: z=−10z=-10, a=0a=0. The bar filter gets z=−17z=-17. One position out of 100 lights up: sparsity 1%.
  4. 4Shapes and memory. (1,1,20,20)→(1,4,20,20)→(1,4,10,10)→(1,3,10,10)(1,1,20,20)\to(1,4,20,20)\to(1,4,10,10)\to(1,3,10,10): 400→1600→400→300400\to1600\to400\to300 values. At this scale it is nothing. A real layer, e.g. nn.Conv2d(64, 128, 3, padding=1) on a (32,64,56,56)(32,64,56,56) batch, outputs 32⋅128⋅56⋅56=12,845,05632\cdot128\cdot56\cdot56=12{,}845{,}056 values (51.4 MB in fp32) from just 73,856 weights.

Adding pixel noise in the figure shows why the bias threshold matters. At noise 0.3 every layer-1 map is nonzero at about half of all positions, because ReLU passes any positive flicker. The layer-2 corner maps stay sparse (3% and 7% nonzero), and their peaks stay at the true corners, (2,2)(2,2) and (7,7)(7,7).

Memory: activations dominate training

Backprop needs the input of every layer to compute that layer's weight gradient, so training keeps most feature maps alive until the backward pass reaches them. Their size scales with batch size and image area, and weights do not. Counting only the conv outputs and residual sums of ResNet-50 gives about 16.8 million values per 2242224^2 image, 64 MiB in fp32. A batch of 256 then needs over 16 GiB for activations, against about 100 MiB for the 25.6M weights. The BatchNorm and ReLU outputs that autograd also saves come on top of that.

Looking inside: forward hooks

python
1import torch
2from torchvision.models import resnet50, ResNet50_Weights
3from torchvision.models.feature_extraction import create_feature_extractor
4
5model = resnet50(weights=ResNet50_Weights.DEFAULT).eval()
6
7# 1) Forward hooks: called as hook(module, inputs, output) after module.forward
8acts = {}
9def save(name):
10 def hook(module, inputs, output):
11 acts[name] = output.detach()
12 return hook
13
14names = ["conv1", "layer1", "layer2", "layer3", "layer4"]
15handles = [getattr(model, n).register_forward_hook(save(n)) for n in names]
16
17x = torch.randn(1, 3, 224, 224) # use a real, normalised image in practice
18with torch.inference_mode():
19 model(x)
20for n in names:
21 a = acts[n]
22 print(f"{n:7s} {tuple(a.shape)} {a.numel() * 4 / 2**20:.2f} MiB")
23# conv1 (1, 64, 112, 112) 3.06 MiB
24# layer1 (1, 256, 56, 56) 3.06 MiB
25# layer2 (1, 512, 28, 28) 1.53 MiB
26# layer3 (1, 1024, 14, 14) 0.77 MiB
27# layer4 (1, 2048, 7, 7) 0.38 MiB
28for h in handles:
29 h.remove() # hooks stay attached until removed
30
31# 2) Or build a model that returns the maps directly
32fx = create_feature_extractor(model, return_nodes={"layer2": "mid", "layer4": "top"})
33feats = fx(x) # {"mid": (1,512,28,28), "top": (1,2048,7,7)}
34
35# Which channels fire most at the centre of the last map? (a fibre)
36fibre = feats["top"][0, :, 3, 3]
37print(fibre.topk(5).indices)
38
39# Show the first 16 channels of layer1 as a grid of images
40import torchvision.utils as vutils
41maps = acts["layer1"][0, :16].unsqueeze(1) # (16, 1, 56, 56)
42grid = vutils.make_grid(maps, nrow=4, normalize=True, scale_each=True)
Capture and inspect intermediate feature maps of a pretrained ResNet-50

Use when

  • +Spatial layout is preserved, so maps can be visualised, localised (Grad-CAM, detection) and reused by dense prediction heads
  • +Weight sharing: one detector per channel scanned over every position, with translation equivariance for free
  • +Multi-scale features come out of one forward pass: early maps are detailed, late maps are semantic
  • +ReLU maps are sparse and compress well

Avoid when

  • −Activation memory scales with batch × resolution and dominates training memory
  • −Early high-resolution maps cost the most memory and bandwidth for the fewest parameters
  • −Individual channels are often polysemantic and hard to interpret
  • −Late maps are coarse (7×7 at stride 32), which loses small objects and fine boundaries

Check yourself

A conv layer maps a (16, 64, 56, 56) input to 128 channels with a 3×3 kernel, stride 2 and padding 1. What is the output shape?

Check yourself

Doubling the input resolution of a CNN from 224 to 448 (same model, same batch) changes training memory how?

Remember

  • ✓A conv layer outputs (N,C,H,W)(N,C,H,W): CC feature maps, one per filter. A map says where its feature is, a fibre a[n,:,i,j]a[n,:,i,j] says what is at one place.
  • ✓Every output channel reads every input channel, so deeper filters are combinations of earlier ones: edges → corners/textures → parts → objects (Zeiler & Fergus 2014).
  • ✓Going deeper, H and W halve per stage while channels double: fewer, richer positions, and more weights per layer.
  • ✓Activation memory scales with batch × resolution and usually dominates training memory; checkpointing, mixed precision and smaller inputs reduce it.
  • ✓Inspect maps with register_forward_hook or create_feature_extractor; remember to remove hooks and to run under inference_mode.
← back to Deep Learning roadmap