# SqueezeNet

> AlexNet-level ImageNet accuracy with 50× fewer parameters (1.25M). Fire modules squeeze channels with 1×1 convs before a mix of 1×1 and 3×3 expand filters. With Deep Compression the whole model fits in under 0.5 MB.

Source: https://www.mlfoundry.in/learn/squeezenet  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 10 min read · updated 2026-10-11

**SqueezeNet:** is a small CNN by Iandola, Han, Moskewicz, Ashraf, Dally and Keutzer ("SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size", 2016). It is built from **Fire modules**: a **squeeze** layer of $1\times1$ convolutions that cuts the number of channels, followed by an **expand** layer that applies $1\times1$ and $3\times3$ convolutions in parallel to the squeezed tensor and concatenates their outputs.

Most efficiency work targets compute. SqueezeNet targets **parameter count**, i.e. model size. The authors gave three reasons why that matters: distributed training sends less over the network, over-the-air updates to cars and phones get smaller, and a small model can fit in on-chip memory of an FPGA or embedded chip. AlexNet's 240 MB of fp32 weights fail all three. SqueezeNet matched AlexNet's ImageNet accuracy (57.5% top-1, 80.3% top-5, against AlexNet's 57.2% and 80.3%) with a 4.8 MB model.

**At a glance**

- **Paper:** Iandola et al., arXiv 2016 (Berkeley and Stanford)
- **Building block:** Fire module: squeeze $s_{1\times1}$ → expand $e_{1\times1}\,\|\,e_{3\times3}$, concatenated, with $s_{1\times1}<e_{1\times1}+e_{3\times3}$
- **Size:** 1,248,424 parameters (4.76 MiB fp32); 0.47 MB with Deep Compression at 6 bits
- **Compute:** 819M multiply-adds at 224×224 (v1.0); v1.1 needs 349M with the same accuracy
- **Accuracy:** 57.5% top-1 / 80.3% top-5 on ImageNet (paper); torchvision weights: 58.09% / 80.42%
- **No FC layers:** a $1\times1$ conv to 1000 channels and global average pooling produce the logits
- **In PyTorch:** `torchvision.models.squeezenet1_0`, `squeezenet1_1`

## Three design strategies

**From the paper, Section 3.1**

1. **Replace $3\times3$ filters with $1\times1$ filters.** A $1\times1$ filter has 9× fewer weights. Use them for as many of the filters as possible.
2. **Decrease the number of input channels to the $3\times3$ filters.** A $3\times3$ layer costs $9\cdot C_{in}\cdot C_{out}$ weights, so cutting $C_{in}$ cuts the cost directly. The squeeze layer does this.
3. **Downsample late.** Keep large activation maps for most of the network, because the paper's authors (and He & Sun, 2015) found that late downsampling raises accuracy. Strategies 1 and 2 save parameters; strategy 3 spends some of that saving on accuracy.

## The Fire module

$$
z=\mathrm{ReLU}\big(W_s * x\big)\in\mathbb R^{s\times H\times W},\qquad y=\Big[\;\mathrm{ReLU}(W_{1}*z)\;\Big\|\;\mathrm{ReLU}(W_{3}*z)\;\Big]\in\mathbb R^{(e_1+e_3)\times H\times W}
$$

- $W_s$: squeeze: $s$ filters of size $1\times1\times C_{in}$
- $W_1,\;W_3$: expand: $e_1$ filters of $1\times1\times s$ and $e_3$ filters of $3\times3\times s$ (padding 1 keeps the size)
- $\|$: concatenation along the channel axis

$$
\text{params}=\underbrace{C_{in}s+s}_{\text{squeeze}}+\underbrace{s\,e_1+e_1}_{\text{expand }1\times1}+\underbrace{9\,s\,e_3+e_3}_{\text{expand }3\times3}\qquad\text{vs. a plain }3\times3:\;9\,C_{in}(e_1+e_3)+(e_1+e_3)
$$

- $s$: squeeze width; SqueezeNet uses $s=(e_1+e_3)/8$, the squeeze ratio $\mathrm{SR}=0.125$
- $e_3/(e_1+e_3)$: share of $3\times3$ filters, $\mathrm{pct}_{3\times3}=0.5$ in SqueezeNet

In the backward pass the concatenation simply splits the incoming gradient into its $e_1$ and $e_3$ channel slices. The squeeze output receives the **sum** of what both expand branches send back: $\partial L/\partial z=\big(W_1^\top\delta_1+W_3^{\top}\star\delta_3\big)\odot\mathbb 1[z>0]$, where $\star$ is the transposed $3\times3$ convolution. The narrow squeeze layer is therefore a bottleneck in both directions.

The macro-architecture is a $7\times7$/2 conv with 96 filters, eight Fire modules (fire2–fire9) whose width grows from 128 to 512, max-pools after conv1, fire4 and fire8, then dropout, a $1\times1$ conv to 1000 classes and global average pooling. The paper generates the eight modules from a few metaparameters: total expand width $e_i=128+128\lfloor i/2\rfloor$, and the squeeze ratio and $3\times3$ share above.

> Figure: SqueezeNet v1.0 layer by layer (torchvision shapes at 224×224). Bars are parameters per layer. Click a Fire module to see its squeeze, expand-1×1 and expand-3×3 parts. At the default fire3 (128 → 128 channels on 54×54) the module has 12,432 parameters, against 147,584 for a plain 3×3 conv with the same input and output widths (11.9×). The total is 1,248,424 parameters (4.76 MiB in fp32), 49× fewer than torchvision's AlexNet (61.1M). Notice that conv10 alone holds 513,000 of them (41%). Raise the squeeze ratio to 0.75: 4.84M parameters, 18.5 MiB. That is the setting where the paper's sweep plateaued at 86.0% top-5. Switch to v1.1 (3×3 stem with 64 filters, earlier pooling): 1,235,496 parameters and 349M instead of 819M multiply-adds.

Fire modules of SqueezeNet v1.0 (SR = 0.125, pct3x3 = 0.5). Counts include biases, from the figure's library

| module | C_in → s → e₁ + e₃ | params | plain 3×3 params | saving |
| --- | --- | --- | --- | --- |
| fire2 | 96 → 16 → 64 + 64 | 11,920 | 110,720 | 9.3× |
| fire3 | 128 → 16 → 64 + 64 | 12,432 | 147,584 | 11.9× |
| fire4 | 128 → 32 → 128 + 128 | 45,344 | 295,168 | 6.5× |
| fire5 | 256 → 32 → 128 + 128 | 49,440 | 590,080 | 11.9× |
| fire8 | 384 → 64 → 256 + 256 | 188,992 | 1,769,984 | 9.4× |
| fire9 | 512 → 64 → 256 + 256 | 197,184 | 2,359,808 | 12.0× |

**Worked example: fire2**

1. Squeeze: $96\cdot16+16=1{,}552$ parameters, giving 16 channels.
2. Expand $1\times1$: $16\cdot64+64=1{,}088$. Expand $3\times3$: $9\cdot16\cdot64+64=9{,}280$. Total $11{,}920$, and the output has $64+64=128$ channels.
3. Without the squeeze (strategy 2 off), the same expand layer on all 96 input channels would need $96\cdot64+64+9\cdot96\cdot64+64=61{,}568$ parameters: 5.2× more.
4. A plain $3\times3$ conv $96\to128$ needs $9\cdot96\cdot128+128=110{,}720$: 9.3× more than the Fire module.

```python
import torch, torch.nn as nn
from torchvision.models import squeezenet1_0, squeezenet1_1, SqueezeNet1_0_Weights

class Fire(nn.Module):
    def __init__(self, cin, s, e1, e3):
        super().__init__()
        self.squeeze = nn.Conv2d(cin, s, 1)
        self.expand1 = nn.Conv2d(s, e1, 1)
        self.expand3 = nn.Conv2d(s, e3, 3, padding=1)
        self.act = nn.ReLU(inplace=True)

    def forward(self, x):
        z = self.act(self.squeeze(x))
        return torch.cat([self.act(self.expand1(z)), self.act(self.expand3(z))], dim=1)

count = lambda m: sum(p.numel() for p in m.parameters())
print(count(Fire(96, 16, 64, 64)))       # 11920
print(Fire(96, 16, 64, 64)(torch.randn(1, 96, 54, 54)).shape)   # (1, 128, 54, 54)

m = squeezenet1_0(weights=SqueezeNet1_0_Weights.IMAGENET1K_V1).eval()
print(count(m), count(squeezenet1_1()))  # 1248424 1235496
print(m.classifier)                      # Dropout -> Conv2d(512, 1000, 1) -> ReLU -> AdaptiveAvgPool2d(1)
```

A Fire module and the torchvision models

> **Few parameters is not the same as fast:** SqueezeNet v1.0 has 1.25M parameters but 819M multiply-adds, more than MobileNetV1 (569M) and over 2.7× MobileNetV2 (300M). Strategy 3 keeps the feature maps large, and that costs compute. If latency matters more than download size, look at **MobileNet** or **ShuffleNet**. Version 1.1 (a $3\times3$ stem and earlier pooling) cuts compute 2.35× at the same accuracy and is almost always the better choice.

> **Architecture and compression multiply:** Deep Compression (Han, Mao and Dally, 2016: pruning, trained quantization and Huffman coding) shrank AlexNet 35×. Applied to an already small SqueezeNet it still worked: 6-bit weights gave a 0.47 MB model with unchanged 57.5% / 80.3% accuracy, 510× smaller than the 240 MB AlexNet. A compact architecture does not leave compression with nothing to remove.

> **Design-space exploration:** The paper also sweeps the metaparameters. Raising the squeeze ratio from 0.125 to 0.75 raised top-5 from 80.3% to 86.0% (model 4.8 → 19 MB), and accuracy plateaued there. Raising the $3\times3$ share beyond 50% gave no gain (plateau at 85.6% top-5). Adding ResNet-style bypass connections around some Fire modules gave 60.4% top-1 and 82.5% top-5 at the same size: an early sign that shortcuts help even small networks (see **Residual neural network (ResNet)**).

**Strengths**

- Tiny models: 1.25M parameters, under 5 MB fp32 and under 0.5 MB compressed
- Simple, regular module with two easy knobs (squeeze ratio, 3×3 share)
- No fully connected layers: works at any input size
- Good for on-chip memory, FPGAs and frequent over-the-air updates

**Limitations**

- High compute for its size: 819M multiply-adds (v1.0)
- Accuracy is only AlexNet-level; modern small nets are far more accurate
- Many small layers and concatenations limit hardware utilization
- No batch normalization in the original design, so it is touchier to train

**Question:** A Fire module takes 256 input channels and has s = 32, e₁ = 128, e₃ = 128. Which part has the most parameters?

- A. The squeeze layer
- B. The 1×1 expand filters
- C. The 3×3 expand filters
- D. They are equal

**Answer:** C. The 3×3 expand filters — Squeeze: $256\cdot32+32=8{,}224$. Expand $1\times1$: $32\cdot128+128=4{,}224$. Expand $3\times3$: $9\cdot32\cdot128+128=36{,}992$. Even on a 32-channel input, the $3\times3$ filters dominate. That is why strategies 1 and 2 go after them.

**Question:** SqueezeNet v1.0 and v1.1 have almost the same number of parameters. Why does v1.1 need 2.35× fewer multiply-adds?

- A. It uses depthwise convolutions
- B. It has a smaller 3×3 stem and pools earlier, so most Fire modules run on smaller feature maps
- C. It removes the 3×3 expand filters
- D. It uses 8-bit weights

**Answer:** B. It has a smaller 3×3 stem and pools earlier, so most Fire modules run on smaller feature maps — Multiply-adds scale with the feature-map area, parameters do not. v1.1 replaces the $7\times7$/96 stem with $3\times3$/64 and moves the max-pools to after fire3 and fire5, so fire4–fire8 run at $27^2$ or $13^2$ instead of $54^2$ or $27^2$.

**Key points**

- Fire module: squeeze with $s$ $1\times1$ filters, then expand with $e_1$ $1\times1$ and $e_3$ $3\times3$ filters, concatenated.
- Three strategies: prefer 1×1 filters, feed fewer channels to the 3×3 filters, downsample late.
- 1,248,424 parameters (≈50× fewer than AlexNet) at AlexNet accuracy; 0.47 MB with Deep Compression.
- Small in parameters, not in compute: 819M multiply-adds for v1.0, 349M for v1.1.
- No FC layers: a 1×1 conv to the classes plus global average pooling.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
