ZFNet

Zeiler & Fergus (2013) projected activations back to pixels with a deconvnet, saw that AlexNet's 11×11 stride-4 first layer wasted filters and aliased, and fixed it with 7×7 stride 2. Occluding image patches showed what the classifier actually relies on.

11 min read934 wordsadvanced0 viewsupdated 2026-10-11

Definition

ZFNet is the network of Zeiler and Fergus, "Visualizing and Understanding Convolutional Networks" (arXiv 2013, ECCV 2014). It is AlexNet with a smaller, denser front end (7×77\times7 stride-2 first layer, stride-2 second layer) found by looking at what AlexNet had learned. The paper's lasting contributions are its two diagnostic tools: the deconvnet, which maps a single hidden activation back to the input pixels that caused it, and occlusion sensitivity, which slides a grey patch over the image and watches the class score.

After AlexNet won ILSVRC-2012, nobody could say why it worked or how to improve it, except by trial and error. Zeiler and Fergus built a way to see inside. Attach to every conv layer a "deconvolutional network" (an idea from Zeiler's earlier work on unsupervised deconvolutional nets) that runs the layer's operations in reverse. Then pick one strong activation, zero all the others, and project it down to pixel space. What appears is the image structure that drove that unit, at its location and scale.

Applied to AlexNet, the projections showed two problems. The first-layer filters were a mix of very high- and very low-frequency patterns with little in between, and several were dead. The second-layer features showed aliasing artifacts caused by the large stride of 4. The fix was to shrink the first layer to 7×77\times7 with stride 2 and to move the downsampling into conv2. The single modified network reached 16.5% top-5 validation error, against 18.1% for their re-implementation of AlexNet (the AlexNet paper reports 18.2%). An ensemble of 6 such nets reached 14.8% test error using only the ILSVRC-2012 training data. The Clarifai entries, built on the same design, won the ILSVRC-2013 classification task with 11.7% top-5 error (11.2% with extra training data).

The architecture: AlexNet with a finer front end

layerZFNetAlexNetZFNet outputAlexNet output
conv196 × 7×7, stride 296 × 11×11, stride 496×110×11096×55×55
pool13×3 / 2 → normnorm → 3×3 / 296×55×5596×27×27
conv2256 × 5×5, stride 2256 × 5×5, stride 1256×26×26256×27×27
pool23×3 / 2 → normnorm → 3×3 / 2256×13×13256×13×13
conv3–5384, 384, 256 (3×3)same256×13×13256×13×13
fc6–84096, 4096, 1000same10001000
total62,357,608 params62,378,344 params1.17 G MACs1.14 G MACs
ZFNet (Fig. 3 of the paper, 224 input) next to single-GPU AlexNet (227 input). Computed by the figure's lib; pools use ceil rounding where the paper's sizes require it

From conv3 onwards the two networks are identical, and since ZFNet was trained on one GPU it has no grouped layers. The parameter counts are almost equal. ZFNet's conv1 is smaller (96⋅3⋅49+96=14,20896\cdot3\cdot49+96=14{,}208 against 34,94434{,}944) but does more work, because its output map is four times larger. The real change is sampling density. A stride of 2 instead of 4 means neighbouring conv1 units are 2 pixels apart, not 4, and a 7×77\times7 kernel cannot hold the very coarse, very fine mixture that the 11×1111\times11 ones did.

Worked example: the shapes in Fig. 3

  1. 1conv1: ⌊(224+2⋅1−7)/2⌋+1=⌊109.5⌋+1=110\lfloor(224+2\cdot1-7)/2\rfloor+1=\lfloor109.5\rfloor+1=110. Parameters 96⋅(3⋅7⋅7)+96=14,20896\cdot(3\cdot7\cdot7)+96=14{,}208; multiply-adds 1102⋅147⋅96=170.8110^2\cdot147\cdot96=170.8M.
  2. 2pool1, 3×33\times3 stride 2: (110−3)/2+1=54.5(110-3)/2+1=54.5. The paper's 55 needs ceil rounding (ceil_mode=True).
  3. 3conv2, 5×55\times5 stride 2, no padding: (55−5)/2+1=26(55-5)/2+1=26. pool2: (26−3)/2+1=12.5→13(26-3)/2+1=12.5\to13, again with ceil.
  4. 4Receptive fields: a conv1 unit sees 77 pixels (AlexNet: 1111), a conv2 unit 2727 (AlexNet: 5151), a conv5 unit 139139 (AlexNet: 163163). The front of ZFNet looks at the image in finer, denser pieces.

The deconvnet: projecting one activation to pixels

To visualise unit uu in layer ll, run the image forward and keep every max-pooling layer's switches, the positions of the maxima. Set all activations of layer ll to zero except the chosen one. Then apply, layer by layer down to the input:

a^(l−1)=ReLU(unpool⁡ s(l−1)(Fl⊤ a^(l))),a^(L)=one-hot(u)⋅au(L)\hat a^{(l-1)}=\mathrm{ReLU}\Big(\operatorname{unpool}_{\,s^{(l-1)}}\big(F_l^{\top}\,\hat a^{(l)}\big)\Big),\qquad \hat a^{(L)}=\text{one-hot}(u)\cdot a^{(L)}_u
Fl⊤F_l^{\top}
filtering with the transposed filters of layer ll, i.e. each kernel flipped and applied as a transposed convolution
unpool⁡s\operatorname{unpool}_{s}
put each value back at the position its max-pooling switch recorded; the other positions get 0
ReLU\mathrm{ReLU}
keep the reconstruction positive, as the forward features were

Without the switches the projection would be a blur. With them it is sharp and image-specific: it shows which pixels of this image drove the unit. The reverse pass is almost the backward pass of the gradient. The only difference is at the ReLUs. The deconvnet rectifies the signal coming down, while backpropagation masks it by where the forward input was positive. Springenberg et al.'s "guided backpropagation" (2015) combines the two masks.

ZFNet (Zeiler & Fergus, Fig. 3) · click a layer62,357,608 params · 1.17 GMACs224²·3inconv1110²96pool155²96norm155²96conv226²256pool213²256norm213²256conv313²384conv413²384conv513²256pool56²256fc614096fc714096fc811000layerconv 7×7, 96 maps, s2 p1in → out3×224×224 → 96×110×110params · MACs14,208 · 170.76M (14.6%)recept. field7×7 px, stride 2same layer in AlexNet96×55×55 · RF 117×7 stride 2: more mid-frequency filtersconv1 map: 110² (7×7 s2) vs 55² (11×11 s4)input · click: move occluderdeconvnet: top “cross” unitscore vs occluder position“cross” score 10.50 → 5.50 (−48%)top unit at pooled (9,1) = 5.50worst spot: 5.50occluder filled with the mean pixel 0.063 · deconv ■ + ■ − · layer-1 filters — | \ /
networklayer-2 unit
fig.Top: ZFNet layer by layer. Toggle to AlexNet to compare shapes (conv1: 110² maps from 7×7 stride 2 vs 55² from 11×11 stride 4), receptive fields and totals (62,357,608 vs 62,378,344 parameters). Bottom: both diagnostic tools on a hand-wired 2-layer convnet with four line detectors and three layer-2 units (cross, corner, diagonal). The middle panel is the deconvnet projection of the strongest layer-2 activation, unpooled through the recorded switches. The right panel is the unit's score as a 6×6 occluder filled with the mean pixel slides over the image: dark = big drop. With the occluder on the cross, the cross unit's score falls from 10.50 to 5.50 (−48%), and the deconvnet shows that its best evidence is now the corner of the square at pooled cell (9,1). The diagonal unit falls from 6.30 to 0.93 under a 6×6 occluder and to 0 under 8×8. The corner unit only drops from 9.30 to 8.30, because the centre of the cross also looks like a corner to it.

Occlusion sensitivity

The second tool needs no access to the network's insides. Cover part of the image with a grey square (the mean pixel), run the classifier, record the probability of the correct class, and repeat for every position of the square. The result is a map of where the evidence is. For a Pomeranian the probability collapses when the dog's face is covered, and for a car wheel when the wheel is. This answers a question the deconvnet cannot: is the network classifying the object, or its surroundings? The paper's maps showed it localises the object. The method is still a standard model-agnostic baseline for attribution.

S(i,j)=pθ(y⋆∣x⊙(1−Mij)+xˉ Mij),Δ(i,j)=pθ(y⋆∣x)−S(i,j)S(i,j)=p_\theta\big(y^\star\mid x\odot(1-M_{ij})+\bar x\,M_{ij}\big),\qquad \Delta(i,j)=p_\theta(y^\star\mid x)-S(i,j)
MijM_{ij}
a binary mask that is 1 on the square centred at (i,j)(i,j)
xˉ\bar x
the fill value: the dataset's mean pixel (0 after mean/std normalisation)
Δ(i,j)\Delta(i,j)
how much the class probability drops when that region is hidden; large = important

What the visualisations showed

Where it's used

A feature hierarchy

Layer 2 responds to corners, edge and colour conjunctions; layer 3 to textures and repeated patterns such as mesh; layer 4 to class-specific parts such as dog faces and bird legs; layer 5 to whole objects with large pose variation.

Training dynamics

Lower layers converge within a few epochs. Upper layers only develop after a considerable number of epochs (40 to 50), so stopping early leaves the top of the network immature.

Invariance

Small shifts and scalings change layer-1 features a lot but the top layers much less. The network is not rotation invariant except for objects with rotational symmetry.

Depth matters

Removing the two dense layers or two middle conv layers raised the error only slightly, but removing both at once hurt badly: the depth of the whole model, not any single layer, carried the performance.

Transfer

Keeping the ImageNet-trained layers fixed and retraining only a softmax on top gave 86.5% on Caltech-101 and 74.2% on Caltech-256, well above the previous best methods. This was early evidence that ImageNet features transfer.

At a glance

Paper
Zeiler & Fergus, arXiv 1311.2901 (2013), ECCV 2014
Change vs AlexNet
conv1 11×1111\times11/4 → 7×77\times7/2; conv2 stride 1 → 2; dense conv3–5 (one GPU)
Training
one GTX 580 for 12 days, 70 epochs; SGD batch 128, lr 0.01, momentum 0.9; dropout 0.5 in fc6/fc7
Filter renormalisation
any conv filter whose RMS exceeds 10−110^{-1} is rescaled back to 10−110^{-1}, to stop a few first-layer filters dominating
ImageNet
16.5% top-5 val (1 net); 14.8% test (6 nets); ILSVRC-2013 winner Clarifai: 11.7%
Tools
deconvnet projection with max-pool switches; occlusion sensitivity
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3lrn = lambda: nn.LocalResponseNorm(5, alpha=5e-4, beta=0.75, k=2.0) # constants not given in the paper
4zfnet = nn.Sequential(
5 nn.Conv2d(3, 96, 7, stride=2, padding=1), nn.ReLU(), nn.MaxPool2d(3, 2, ceil_mode=True), lrn(), # 110 -> 55
6 nn.Conv2d(96, 256, 5, stride=2), nn.ReLU(), nn.MaxPool2d(3, 2, ceil_mode=True), lrn(), # 26 -> 13
7 nn.Conv2d(256, 384, 3, padding=1), nn.ReLU(),
8 nn.Conv2d(384, 384, 3, padding=1), nn.ReLU(),
9 nn.Conv2d(384, 256, 3, padding=1), nn.ReLU(), nn.MaxPool2d(3, 2), # 13 -> 6
10 nn.Flatten(), nn.Dropout(0.5), nn.Linear(9216, 4096), nn.ReLU(),
11 nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Linear(4096, 1000),
12)
13print(sum(p.numel() for p in zfnet.parameters())) # 62357608
14
15# --- deconvnet: project the strongest pool1 activation of map m back to pixels ---
16conv1 = zfnet[0]
17x = torch.randn(1, 3, 224, 224)
18with torch.no_grad():
19 a = F.relu(conv1(x)) # 1x96x110x110
20 p, switches = F.max_pool2d(a, 3, 2, ceil_mode=True, return_indices=True) # 1x96x55x55
21 m = 5
22 keep = torch.zeros_like(p)
23 i = p[0, m].flatten().argmax()
24 keep[0, m].view(-1)[i] = p[0, m].flatten()[i] # one-hot activation
25 u = F.max_unpool2d(keep, switches, 3, 2, output_size=a.shape[-2:]) # back to 110x110
26 img = F.conv_transpose2d(F.relu(u), conv1.weight, stride=2, padding=1, output_padding=1)
27print(img.shape) # torch.Size([1, 3, 224, 224])
28
29# --- occlusion sensitivity for any classifier ---
30@torch.no_grad()
31def occlusion_map(model, x, target, size=32, stride=16, fill=0.0): # fill 0 = mean pixel after normalisation
32 H, W = x.shape[-2:]
33 rows, cols = range(0, H - size + 1, stride), range(0, W - size + 1, stride)
34 out = torch.empty(len(rows), len(cols))
35 for a_, r0 in enumerate(rows):
36 for b_, c0 in enumerate(cols):
37 xo = x.clone()
38 xo[..., r0:r0 + size, c0:c0 + size] = fill
39 out[a_, b_] = model(xo).softmax(-1)[0, target]
40 return out # low values = regions the prediction depends on
ZFNet, a one-layer deconvnet projection, and an occlusion map, in PyTorch

Use when

  • +Deconvnet projections are sharp and image-specific thanks to the pooling switches
  • +Occlusion sensitivity is model-agnostic and easy to explain
  • +The architecture fix (smaller first-layer kernel and stride) improved accuracy at almost no change in size
  • +Showed early that ImageNet features transfer to other datasets

Avoid when

  • −A deconvnet visualises one unit on one image at a time; it is not a full explanation
  • −Occlusion needs one forward pass per patch position, and the result depends on patch size and fill value
  • −ZFNet keeps AlexNet's 58.6M-parameter dense head and LRN
  • −Larger conv1/conv2 maps make the whole network about 3% more expensive (1.17 vs 1.14 G multiply-adds)

Check yourself

What does the deconvnet need from the forward pass that a plain transposed-convolution stack would not have?

Check yourself

Zeiler and Fergus changed AlexNet's first layer from 11×11 stride 4 to 7×7 stride 2. Which observation motivated the stride change?

Remember

  • ✓ZFNet = AlexNet with conv1 7×7 stride 2 (was 11×11 stride 4) and conv2 stride 2; about 62.4M parameters.
  • ✓Single net: 16.5% top-5 val vs 18.1% for their AlexNet; Clarifai's entries on this design won ILSVRC-2013 (11.7% top-5).
  • ✓Deconvnet: one-hot an activation, then repeat transposed filtering, unpooling with the recorded max switches, and ReLU down to the pixels.
  • ✓Occlusion sensitivity: slide a mean-grey patch and record the class probability; the drops map where the evidence is.
  • ✓Findings: a part hierarchy across layers, upper layers converging late, and features that transfer to Caltech-101/256.
← back to Deep Learning roadmap