# ZFNet

> Zeiler & Fergus (2013) projected activations back to pixels with a deconvnet, saw that AlexNet's 11×11 stride-4 first layer wasted filters and aliased, and fixed it with 7×7 stride 2. Occluding image patches showed what the classifier actually relies on.

Source: https://www.mlfoundry.in/learn/zfnet  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 11 min read · updated 2026-10-11

**ZFNet:** is the network of Zeiler and Fergus, "Visualizing and Understanding Convolutional Networks" (arXiv 2013, ECCV 2014). It is AlexNet with a smaller, denser front end ($7\times7$ stride-2 first layer, stride-2 second layer) found by **looking** at what AlexNet had learned. The paper's lasting contributions are its two diagnostic tools: the **deconvnet**, which maps a single hidden activation back to the input pixels that caused it, and **occlusion sensitivity**, which slides a grey patch over the image and watches the class score.

After **AlexNet** won ILSVRC-2012, nobody could say why it worked or how to improve it, except by trial and error. Zeiler and Fergus built a way to see inside. Attach to every conv layer a "deconvolutional network" (an idea from Zeiler's earlier work on unsupervised deconvolutional nets) that runs the layer's operations in reverse. Then pick one strong activation, zero all the others, and project it down to pixel space. What appears is the image structure that drove that unit, at its location and scale.

Applied to AlexNet, the projections showed two problems. The first-layer filters were a mix of very high- and very low-frequency patterns with little in between, and several were dead. The second-layer features showed aliasing artifacts caused by the large stride of 4. The fix was to shrink the first layer to $7\times7$ with stride 2 and to move the downsampling into conv2. The single modified network reached 16.5% top-5 validation error, against 18.1% for their re-implementation of AlexNet (the AlexNet paper reports 18.2%). An ensemble of 6 such nets reached 14.8% test error using only the ILSVRC-2012 training data. The Clarifai entries, built on the same design, won the ILSVRC-2013 classification task with 11.7% top-5 error (11.2% with extra training data).

## The architecture: AlexNet with a finer front end

ZFNet (Fig. 3 of the paper, 224 input) next to single-GPU AlexNet (227 input). Computed by the figure's lib; pools use ceil rounding where the paper's sizes require it

| layer | ZFNet | AlexNet | ZFNet output | AlexNet output |
| --- | --- | --- | --- | --- |
| conv1 | 96 × 7×7, stride 2 | 96 × 11×11, stride 4 | 96×110×110 | 96×55×55 |
| pool1 | 3×3 / 2 → norm | norm → 3×3 / 2 | 96×55×55 | 96×27×27 |
| conv2 | 256 × 5×5, **stride 2** | 256 × 5×5, stride 1 | 256×26×26 | 256×27×27 |
| pool2 | 3×3 / 2 → norm | norm → 3×3 / 2 | 256×13×13 | 256×13×13 |
| conv3–5 | 384, 384, 256 (3×3) | same | 256×13×13 | 256×13×13 |
| fc6–8 | 4096, 4096, 1000 | same | 1000 | 1000 |
| **total** | 62,357,608 params | 62,378,344 params | 1.17 G MACs | 1.14 G MACs |

From conv3 onwards the two networks are identical, and since ZFNet was trained on one GPU it has no grouped layers. The parameter counts are almost equal. ZFNet's conv1 is smaller ($96\cdot3\cdot49+96=14{,}208$ against $34{,}944$) but does more work, because its output map is four times larger. The real change is **sampling density**. A stride of 2 instead of 4 means neighbouring conv1 units are 2 pixels apart, not 4, and a $7\times7$ kernel cannot hold the very coarse, very fine mixture that the $11\times11$ ones did.

**Worked example: the shapes in Fig. 3**

1. conv1: $\lfloor(224+2\cdot1-7)/2\rfloor+1=\lfloor109.5\rfloor+1=110$. Parameters $96\cdot(3\cdot7\cdot7)+96=14{,}208$; multiply-adds $110^2\cdot147\cdot96=170.8$M.
2. pool1, $3\times3$ stride 2: $(110-3)/2+1=54.5$. The paper's 55 needs ceil rounding (`ceil_mode=True`).
3. conv2, $5\times5$ stride 2, no padding: $(55-5)/2+1=26$. pool2: $(26-3)/2+1=12.5\to13$, again with ceil.
4. Receptive fields: a conv1 unit sees $7$ pixels (AlexNet: $11$), a conv2 unit $27$ (AlexNet: $51$), a conv5 unit $139$ (AlexNet: $163$). The front of ZFNet looks at the image in finer, denser pieces.

## The deconvnet: projecting one activation to pixels

To visualise unit $u$ in layer $l$, run the image forward and keep every max-pooling layer's **switches**, the positions of the maxima. Set all activations of layer $l$ to zero except the chosen one. Then apply, layer by layer down to the input:

$$
\hat a^{(l-1)}=\mathrm{ReLU}\Big(\operatorname{unpool}_{\,s^{(l-1)}}\big(F_l^{\top}\,\hat a^{(l)}\big)\Big),\qquad \hat a^{(L)}=\text{one-hot}(u)\cdot a^{(L)}_u
$$

- $F_l^{\top}$: filtering with the transposed filters of layer $l$, i.e. each kernel flipped and applied as a transposed convolution
- $\operatorname{unpool}_{s}$: put each value back at the position its max-pooling switch recorded; the other positions get 0
- $\mathrm{ReLU}$: keep the reconstruction positive, as the forward features were

Without the switches the projection would be a blur. With them it is sharp and image-specific: it shows **which** pixels of **this** image drove the unit. The reverse pass is almost the backward pass of the gradient. The only difference is at the ReLUs. The deconvnet rectifies the signal coming down, while backpropagation masks it by where the forward input was positive. Springenberg et al.'s "guided backpropagation" (2015) combines the two masks.

> Figure: Top: ZFNet layer by layer. Toggle to AlexNet to compare shapes (conv1: 110² maps from 7×7 stride 2 vs 55² from 11×11 stride 4), receptive fields and totals (62,357,608 vs 62,378,344 parameters). Bottom: both diagnostic tools on a hand-wired 2-layer convnet with four line detectors and three layer-2 units (cross, corner, diagonal). The middle panel is the deconvnet projection of the strongest layer-2 activation, unpooled through the recorded switches. The right panel is the unit's score as a 6×6 occluder filled with the mean pixel slides over the image: dark = big drop. With the occluder on the cross, the cross unit's score falls from 10.50 to 5.50 (−48%), and the deconvnet shows that its best evidence is now the corner of the square at pooled cell (9,1). The diagonal unit falls from 6.30 to 0.93 under a 6×6 occluder and to 0 under 8×8. The corner unit only drops from 9.30 to 8.30, because the centre of the cross also looks like a corner to it.

## Occlusion sensitivity

The second tool needs no access to the network's insides. Cover part of the image with a grey square (the mean pixel), run the classifier, record the probability of the correct class, and repeat for every position of the square. The result is a map of where the evidence is. For a Pomeranian the probability collapses when the dog's face is covered, and for a car wheel when the wheel is. This answers a question the deconvnet cannot: is the network classifying the object, or its surroundings? The paper's maps showed it localises the object. The method is still a standard model-agnostic baseline for attribution.

$$
S(i,j)=p_\theta\big(y^\star\mid x\odot(1-M_{ij})+\bar x\,M_{ij}\big),\qquad \Delta(i,j)=p_\theta(y^\star\mid x)-S(i,j)
$$

- $M_{ij}$: a binary mask that is 1 on the square centred at $(i,j)$
- $\bar x$: the fill value: the dataset's mean pixel (0 after mean/std normalisation)
- $\Delta(i,j)$: how much the class probability drops when that region is hidden; large = important

## What the visualisations showed

- **A feature hierarchy:** Layer 2 responds to corners, edge and colour conjunctions; layer 3 to textures and repeated patterns such as mesh; layer 4 to class-specific parts such as dog faces and bird legs; layer 5 to whole objects with large pose variation.
- **Training dynamics:** Lower layers converge within a few epochs. Upper layers only develop after a considerable number of epochs (40 to 50), so stopping early leaves the top of the network immature.
- **Invariance:** Small shifts and scalings change layer-1 features a lot but the top layers much less. The network is not rotation invariant except for objects with rotational symmetry.
- **Depth matters:** Removing the two dense layers or two middle conv layers raised the error only slightly, but removing both at once hurt badly: the depth of the whole model, not any single layer, carried the performance.
- **Transfer:** Keeping the ImageNet-trained layers fixed and retraining only a softmax on top gave 86.5% on Caltech-101 and 74.2% on Caltech-256, well above the previous best methods. This was early evidence that ImageNet features transfer.

**At a glance**

- **Paper:** Zeiler & Fergus, arXiv 1311.2901 (2013), ECCV 2014
- **Change vs AlexNet:** conv1 $11\times11$/4 → $7\times7$/2; conv2 stride 1 → 2; dense conv3–5 (one GPU)
- **Training:** one GTX 580 for 12 days, 70 epochs; SGD batch 128, lr 0.01, momentum 0.9; dropout 0.5 in fc6/fc7
- **Filter renormalisation:** any conv filter whose RMS exceeds $10^{-1}$ is rescaled back to $10^{-1}$, to stop a few first-layer filters dominating
- **ImageNet:** 16.5% top-5 val (1 net); 14.8% test (6 nets); ILSVRC-2013 winner Clarifai: 11.7%
- **Tools:** deconvnet projection with max-pool switches; occlusion sensitivity

```python
import torch, torch.nn as nn, torch.nn.functional as F

lrn = lambda: nn.LocalResponseNorm(5, alpha=5e-4, beta=0.75, k=2.0)  # constants not given in the paper
zfnet = nn.Sequential(
    nn.Conv2d(3, 96, 7, stride=2, padding=1), nn.ReLU(), nn.MaxPool2d(3, 2, ceil_mode=True), lrn(),  # 110 -> 55
    nn.Conv2d(96, 256, 5, stride=2), nn.ReLU(), nn.MaxPool2d(3, 2, ceil_mode=True), lrn(),         # 26 -> 13
    nn.Conv2d(256, 384, 3, padding=1), nn.ReLU(),
    nn.Conv2d(384, 384, 3, padding=1), nn.ReLU(),
    nn.Conv2d(384, 256, 3, padding=1), nn.ReLU(), nn.MaxPool2d(3, 2),                              # 13 -> 6
    nn.Flatten(), nn.Dropout(0.5), nn.Linear(9216, 4096), nn.ReLU(),
    nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(), nn.Linear(4096, 1000),
)
print(sum(p.numel() for p in zfnet.parameters()))   # 62357608

# --- deconvnet: project the strongest pool1 activation of map m back to pixels ---
conv1 = zfnet[0]
x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    a = F.relu(conv1(x))                                                     # 1x96x110x110
    p, switches = F.max_pool2d(a, 3, 2, ceil_mode=True, return_indices=True)  # 1x96x55x55
    m = 5
    keep = torch.zeros_like(p)
    i = p[0, m].flatten().argmax()
    keep[0, m].view(-1)[i] = p[0, m].flatten()[i]                            # one-hot activation
    u = F.max_unpool2d(keep, switches, 3, 2, output_size=a.shape[-2:])       # back to 110x110
    img = F.conv_transpose2d(F.relu(u), conv1.weight, stride=2, padding=1, output_padding=1)
print(img.shape)                                                             # torch.Size([1, 3, 224, 224])

# --- occlusion sensitivity for any classifier ---
@torch.no_grad()
def occlusion_map(model, x, target, size=32, stride=16, fill=0.0):  # fill 0 = mean pixel after normalisation
    H, W = x.shape[-2:]
    rows, cols = range(0, H - size + 1, stride), range(0, W - size + 1, stride)
    out = torch.empty(len(rows), len(cols))
    for a_, r0 in enumerate(rows):
        for b_, c0 in enumerate(cols):
            xo = x.clone()
            xo[..., r0:r0 + size, c0:c0 + size] = fill
            out[a_, b_] = model(xo).softmax(-1)[0, target]
    return out   # low values = regions the prediction depends on
```

ZFNet, a one-layer deconvnet projection, and an occlusion map, in PyTorch

> **A deconvnet is not a decoder:** Despite the name, nothing in a deconvnet is trained. It reuses the forward filters (transposed) and the forward pass's switches, so it can only explain the network on the image it was run on. It is also not the "deconvolution" of signal processing, and it is not the learned **Transposed convolution** layer used for upsampling, even though both apply $F^\top$.

> **Visualisation as a design tool:** ZFNet's architecture change came from looking at pictures, not from a hyperparameter sweep: dead and aliased filters were visible, so the stride and kernel size were the obvious knobs. Every later classic network kept the lesson. **VGGNet** went further, with only $3\times3$ kernels and stride 1 everywhere except the pools.

> **Occlusion in the toy figure:** In the figure, hiding the cross only halves the cross unit's score, from 10.50 to 5.50, because the corner of the square is a weak cross too. The deconvnet panel shows the evidence jumping there. Real networks behave the same way: an occlusion map measures what the **maximum** over the remaining evidence is, so redundant cues make a region look less important than it is.

**Strengths**

- Deconvnet projections are sharp and image-specific thanks to the pooling switches
- Occlusion sensitivity is model-agnostic and easy to explain
- The architecture fix (smaller first-layer kernel and stride) improved accuracy at almost no change in size
- Showed early that ImageNet features transfer to other datasets

**Limitations**

- A deconvnet visualises one unit on one image at a time; it is not a full explanation
- Occlusion needs one forward pass per patch position, and the result depends on patch size and fill value
- ZFNet keeps AlexNet's 58.6M-parameter dense head and LRN
- Larger conv1/conv2 maps make the whole network about 3% more expensive (1.17 vs 1.14 G multiply-adds)

**Question:** What does the deconvnet need from the forward pass that a plain transposed-convolution stack would not have?

- A. The class label
- B. The max-pooling switches (where each maximum came from)
- C. The gradients of the loss
- D. The batch-norm statistics

**Answer:** B. The max-pooling switches (where each maximum came from) — Unpooling has to put each value back where the maximum was. Those positions, the switches, depend on the input image and are recorded during the forward pass. Without them the reconstruction smears each value over the whole pooling window.

**Question:** Zeiler and Fergus changed AlexNet's first layer from 11×11 stride 4 to 7×7 stride 2. Which observation motivated the stride change?

- A. The first layer had too many parameters
- B. The network overfit on ImageNet
- C. The GPUs ran out of memory
- D. Aliasing artifacts in the layer-2 features

**Answer:** D. Aliasing artifacts in the layer-2 features — Projections of layer 2 showed aliasing, a symptom of sampling the first-layer responses too coarsely with stride 4. The kernel-size change addressed the other symptom: first-layer filters that were all very high or very low frequency, some of them dead.

**Key points**

- ZFNet = AlexNet with conv1 7×7 stride 2 (was 11×11 stride 4) and conv2 stride 2; about 62.4M parameters.
- Single net: 16.5% top-5 val vs 18.1% for their AlexNet; Clarifai's entries on this design won ILSVRC-2013 (11.7% top-5).
- Deconvnet: one-hot an activation, then repeat transposed filtering, unpooling with the recorded max switches, and ReLU down to the pixels.
- Occlusion sensitivity: slide a mean-grey patch and record the class probability; the drops map where the evidence is.
- Findings: a part hierarchy across layers, upper layers converging late, and features that transfer to Caltech-101/256.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
