Definition
ZFNet is the network of Zeiler and Fergus, "Visualizing and Understanding Convolutional Networks" (arXiv 2013, ECCV 2014). It is AlexNet with a smaller, denser front end ( stride-2 first layer, stride-2 second layer) found by looking at what AlexNet had learned. The paper's lasting contributions are its two diagnostic tools: the deconvnet, which maps a single hidden activation back to the input pixels that caused it, and occlusion sensitivity, which slides a grey patch over the image and watches the class score.
After AlexNet won ILSVRC-2012, nobody could say why it worked or how to improve it, except by trial and error. Zeiler and Fergus built a way to see inside. Attach to every conv layer a "deconvolutional network" (an idea from Zeiler's earlier work on unsupervised deconvolutional nets) that runs the layer's operations in reverse. Then pick one strong activation, zero all the others, and project it down to pixel space. What appears is the image structure that drove that unit, at its location and scale.
Applied to AlexNet, the projections showed two problems. The first-layer filters were a mix of very high- and very low-frequency patterns with little in between, and several were dead. The second-layer features showed aliasing artifacts caused by the large stride of 4. The fix was to shrink the first layer to with stride 2 and to move the downsampling into conv2. The single modified network reached 16.5% top-5 validation error, against 18.1% for their re-implementation of AlexNet (the AlexNet paper reports 18.2%). An ensemble of 6 such nets reached 14.8% test error using only the ILSVRC-2012 training data. The Clarifai entries, built on the same design, won the ILSVRC-2013 classification task with 11.7% top-5 error (11.2% with extra training data).
The architecture: AlexNet with a finer front end
| layer | ZFNet | AlexNet | ZFNet output | AlexNet output |
|---|---|---|---|---|
| conv1 | 96 × 7×7, stride 2 | 96 × 11×11, stride 4 | 96×110×110 | 96×55×55 |
| pool1 | 3×3 / 2 → norm | norm → 3×3 / 2 | 96×55×55 | 96×27×27 |
| conv2 | 256 × 5×5, stride 2 | 256 × 5×5, stride 1 | 256×26×26 | 256×27×27 |
| pool2 | 3×3 / 2 → norm | norm → 3×3 / 2 | 256×13×13 | 256×13×13 |
| conv3–5 | 384, 384, 256 (3×3) | same | 256×13×13 | 256×13×13 |
| fc6–8 | 4096, 4096, 1000 | same | 1000 | 1000 |
| total | 62,357,608 params | 62,378,344 params | 1.17 G MACs | 1.14 G MACs |
From conv3 onwards the two networks are identical, and since ZFNet was trained on one GPU it has no grouped layers. The parameter counts are almost equal. ZFNet's conv1 is smaller ( against ) but does more work, because its output map is four times larger. The real change is sampling density. A stride of 2 instead of 4 means neighbouring conv1 units are 2 pixels apart, not 4, and a kernel cannot hold the very coarse, very fine mixture that the ones did.
Worked example: the shapes in Fig. 3
- 1conv1: . Parameters ; multiply-adds M.
- 2pool1, stride 2: . The paper's 55 needs ceil rounding (
ceil_mode=True). - 3conv2, stride 2, no padding: . pool2: , again with ceil.
- 4Receptive fields: a conv1 unit sees pixels (AlexNet: ), a conv2 unit (AlexNet: ), a conv5 unit (AlexNet: ). The front of ZFNet looks at the image in finer, denser pieces.
The deconvnet: projecting one activation to pixels
To visualise unit in layer , run the image forward and keep every max-pooling layer's switches, the positions of the maxima. Set all activations of layer to zero except the chosen one. Then apply, layer by layer down to the input:
- filtering with the transposed filters of layer , i.e. each kernel flipped and applied as a transposed convolution
- put each value back at the position its max-pooling switch recorded; the other positions get 0
- keep the reconstruction positive, as the forward features were
Without the switches the projection would be a blur. With them it is sharp and image-specific: it shows which pixels of this image drove the unit. The reverse pass is almost the backward pass of the gradient. The only difference is at the ReLUs. The deconvnet rectifies the signal coming down, while backpropagation masks it by where the forward input was positive. Springenberg et al.'s "guided backpropagation" (2015) combines the two masks.
Occlusion sensitivity
The second tool needs no access to the network's insides. Cover part of the image with a grey square (the mean pixel), run the classifier, record the probability of the correct class, and repeat for every position of the square. The result is a map of where the evidence is. For a Pomeranian the probability collapses when the dog's face is covered, and for a car wheel when the wheel is. This answers a question the deconvnet cannot: is the network classifying the object, or its surroundings? The paper's maps showed it localises the object. The method is still a standard model-agnostic baseline for attribution.
- a binary mask that is 1 on the square centred at
- the fill value: the dataset's mean pixel (0 after mean/std normalisation)
- how much the class probability drops when that region is hidden; large = important
What the visualisations showed
Where it's used
A feature hierarchy
Layer 2 responds to corners, edge and colour conjunctions; layer 3 to textures and repeated patterns such as mesh; layer 4 to class-specific parts such as dog faces and bird legs; layer 5 to whole objects with large pose variation.
Training dynamics
Lower layers converge within a few epochs. Upper layers only develop after a considerable number of epochs (40 to 50), so stopping early leaves the top of the network immature.
Invariance
Small shifts and scalings change layer-1 features a lot but the top layers much less. The network is not rotation invariant except for objects with rotational symmetry.
Depth matters
Removing the two dense layers or two middle conv layers raised the error only slightly, but removing both at once hurt badly: the depth of the whole model, not any single layer, carried the performance.
Transfer
Keeping the ImageNet-trained layers fixed and retraining only a softmax on top gave 86.5% on Caltech-101 and 74.2% on Caltech-256, well above the previous best methods. This was early evidence that ImageNet features transfer.
At a glance
- Paper
- Zeiler & Fergus, arXiv 1311.2901 (2013), ECCV 2014
- Change vs AlexNet
- conv1 /4 → /2; conv2 stride 1 → 2; dense conv3–5 (one GPU)
- Training
- one GTX 580 for 12 days, 70 epochs; SGD batch 128, lr 0.01, momentum 0.9; dropout 0.5 in fc6/fc7
- Filter renormalisation
- any conv filter whose RMS exceeds is rescaled back to , to stop a few first-layer filters dominating
- ImageNet
- 16.5% top-5 val (1 net); 14.8% test (6 nets); ILSVRC-2013 winner Clarifai: 11.7%
- Tools
- deconvnet projection with max-pool switches; occlusion sensitivity
1import torch, torch.nn as nn, torch.nn.functional as F23lrn = lambda: nn.LocalResponseNorm(5, alpha=5e-4, beta=0.75, k=2.0) # constants not given in the paper4zfnet = nn.Sequential(5 nn.Conv2d(3, 96, 7, stride=2, padding=1), nn.ReLU(), nn.MaxPool2d(3, 2, ceil_mode=True), lrn(), # 110 -> 556 nn.Conv2d(96, 256, 5, stride=2), nn.ReLU(), nn.MaxPool2d(3, 2, ceil_mode=True), lrn(), # 26 -> 137 nn.Conv2d(256, 384, 3, padding=1), nn.ReLU(),8 nn.Conv2d(384, 384, 3, padding=1), nn.ReLU(),9 nn.Conv2d(384, 256, 3, padding=1), nn.ReLU(), nn.MaxPool2d(3, 2), # 13 -> 610 nn.Flatten(), nn.Dropout(0.5), nn.Linear(9216, 4096), nn.ReLU(),11 nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Linear(4096, 1000),12)13print(sum(p.numel() for p in zfnet.parameters())) # 623576081415# --- deconvnet: project the strongest pool1 activation of map m back to pixels ---16conv1 = zfnet[0]17x = torch.randn(1, 3, 224, 224)18with torch.no_grad():19 a = F.relu(conv1(x)) # 1x96x110x11020 p, switches = F.max_pool2d(a, 3, 2, ceil_mode=True, return_indices=True) # 1x96x55x5521 m = 522 keep = torch.zeros_like(p)23 i = p[0, m].flatten().argmax()24 keep[0, m].view(-1)[i] = p[0, m].flatten()[i] # one-hot activation25 u = F.max_unpool2d(keep, switches, 3, 2, output_size=a.shape[-2:]) # back to 110x11026 img = F.conv_transpose2d(F.relu(u), conv1.weight, stride=2, padding=1, output_padding=1)27print(img.shape) # torch.Size([1, 3, 224, 224])2829# --- occlusion sensitivity for any classifier ---30@torch.no_grad()31def occlusion_map(model, x, target, size=32, stride=16, fill=0.0): # fill 0 = mean pixel after normalisation32 H, W = x.shape[-2:]33 rows, cols = range(0, H - size + 1, stride), range(0, W - size + 1, stride)34 out = torch.empty(len(rows), len(cols))35 for a_, r0 in enumerate(rows):36 for b_, c0 in enumerate(cols):37 xo = x.clone()38 xo[..., r0:r0 + size, c0:c0 + size] = fill39 out[a_, b_] = model(xo).softmax(-1)[0, target]40 return out # low values = regions the prediction depends onUse when
- +Deconvnet projections are sharp and image-specific thanks to the pooling switches
- +Occlusion sensitivity is model-agnostic and easy to explain
- +The architecture fix (smaller first-layer kernel and stride) improved accuracy at almost no change in size
- +Showed early that ImageNet features transfer to other datasets
Avoid when
- −A deconvnet visualises one unit on one image at a time; it is not a full explanation
- −Occlusion needs one forward pass per patch position, and the result depends on patch size and fill value
- −ZFNet keeps AlexNet's 58.6M-parameter dense head and LRN
- −Larger conv1/conv2 maps make the whole network about 3% more expensive (1.17 vs 1.14 G multiply-adds)
Check yourself
What does the deconvnet need from the forward pass that a plain transposed-convolution stack would not have?
Check yourself
Zeiler and Fergus changed AlexNet's first layer from 11×11 stride 4 to 7×7 stride 2. Which observation motivated the stride change?
Remember
- ✓ZFNet = AlexNet with conv1 7×7 stride 2 (was 11×11 stride 4) and conv2 stride 2; about 62.4M parameters.
- ✓Single net: 16.5% top-5 val vs 18.1% for their AlexNet; Clarifai's entries on this design won ILSVRC-2013 (11.7% top-5).
- ✓Deconvnet: one-hot an activation, then repeat transposed filtering, unpooling with the recorded max switches, and ReLU down to the pixels.
- ✓Occlusion sensitivity: slide a mean-grey patch and record the class probability; the drops map where the evidence is.
- ✓Findings: a part hierarchy across layers, upper layers converging late, and features that transfer to Caltech-101/256.