SqueezeNet

AlexNet-level ImageNet accuracy with 50× fewer parameters (1.25M). Fire modules squeeze channels with 1×1 convs before a mix of 1×1 and 3×3 expand filters. With Deep Compression the whole model fits in under 0.5 MB.

10 min read666 wordsadvanced0 viewsupdated 2026-10-11

Definition

SqueezeNet is a small CNN by Iandola, Han, Moskewicz, Ashraf, Dally and Keutzer ("SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size", 2016). It is built from Fire modules: a squeeze layer of 1×11\times1 convolutions that cuts the number of channels, followed by an expand layer that applies 1×11\times1 and 3×33\times3 convolutions in parallel to the squeezed tensor and concatenates their outputs.

Most efficiency work targets compute. SqueezeNet targets parameter count, i.e. model size. The authors gave three reasons why that matters: distributed training sends less over the network, over-the-air updates to cars and phones get smaller, and a small model can fit in on-chip memory of an FPGA or embedded chip. AlexNet's 240 MB of fp32 weights fail all three. SqueezeNet matched AlexNet's ImageNet accuracy (57.5% top-1, 80.3% top-5, against AlexNet's 57.2% and 80.3%) with a 4.8 MB model.

At a glance

Paper
Iandola et al., arXiv 2016 (Berkeley and Stanford)
Building block
Fire module: squeeze s1×1s_{1\times1} → expand e1×1 ∥ e3×3e_{1\times1}\,\|\,e_{3\times3}, concatenated, with s1×1<e1×1+e3×3s_{1\times1}<e_{1\times1}+e_{3\times3}
Size
1,248,424 parameters (4.76 MiB fp32); 0.47 MB with Deep Compression at 6 bits
Compute
819M multiply-adds at 224×224 (v1.0); v1.1 needs 349M with the same accuracy
Accuracy
57.5% top-1 / 80.3% top-5 on ImageNet (paper); torchvision weights: 58.09% / 80.42%
No FC layers
a 1×11\times1 conv to 1000 channels and global average pooling produce the logits
In PyTorch
torchvision.models.squeezenet1_0, squeezenet1_1

Three design strategies

From the paper, Section 3.1

  1. 1Replace 3×33\times3 filters with 1×11\times1 filters. A 1×11\times1 filter has 9× fewer weights. Use them for as many of the filters as possible.
  2. 2Decrease the number of input channels to the 3×33\times3 filters. A 3×33\times3 layer costs 9⋅Cin⋅Cout9\cdot C_{in}\cdot C_{out} weights, so cutting CinC_{in} cuts the cost directly. The squeeze layer does this.
  3. 3Downsample late. Keep large activation maps for most of the network, because the paper's authors (and He & Sun, 2015) found that late downsampling raises accuracy. Strategies 1 and 2 save parameters; strategy 3 spends some of that saving on accuracy.

The Fire module

z=ReLU(Ws∗x)∈Rs×H×W,y=[  ReLU(W1∗z)  ∥  ReLU(W3∗z)  ]∈R(e1+e3)×H×Wz=\mathrm{ReLU}\big(W_s * x\big)\in\mathbb R^{s\times H\times W},\qquad y=\Big[\;\mathrm{ReLU}(W_{1}*z)\;\Big\|\;\mathrm{ReLU}(W_{3}*z)\;\Big]\in\mathbb R^{(e_1+e_3)\times H\times W}
WsW_s
squeeze: ss filters of size 1×1×Cin1\times1\times C_{in}
W1,  W3W_1,\;W_3
expand: e1e_1 filters of 1×1×s1\times1\times s and e3e_3 filters of 3×3×s3\times3\times s (padding 1 keeps the size)
∥\|
concatenation along the channel axis
params=Cins+s⏟squeeze+s e1+e1⏟expand 1×1+9 s e3+e3⏟expand 3×3vs. a plain 3×3:  9 Cin(e1+e3)+(e1+e3)\text{params}=\underbrace{C_{in}s+s}_{\text{squeeze}}+\underbrace{s\,e_1+e_1}_{\text{expand }1\times1}+\underbrace{9\,s\,e_3+e_3}_{\text{expand }3\times3}\qquad\text{vs. a plain }3\times3:\;9\,C_{in}(e_1+e_3)+(e_1+e_3)
ss
squeeze width; SqueezeNet uses s=(e1+e3)/8s=(e_1+e_3)/8, the squeeze ratio SR=0.125\mathrm{SR}=0.125
e3/(e1+e3)e_3/(e_1+e_3)
share of 3×33\times3 filters, pct3×3=0.5\mathrm{pct}_{3\times3}=0.5 in SqueezeNet

In the backward pass the concatenation simply splits the incoming gradient into its e1e_1 and e3e_3 channel slices. The squeeze output receives the sum of what both expand branches send back: ∂L/∂z=(W1⊤δ1+W3⊤⋆δ3)⊙1[z>0]\partial L/\partial z=\big(W_1^\top\delta_1+W_3^{\top}\star\delta_3\big)\odot\mathbb 1[z>0], where ⋆\star is the transposed 3×33\times3 convolution. The narrow squeeze layer is therefore a bottleneck in both directions.

The macro-architecture is a 7×77\times7/2 conv with 96 filters, eight Fire modules (fire2–fire9) whose width grows from 128 to 512, max-pools after conv1, fire4 and fire8, then dropout, a 1×11\times1 conv to 1000 classes and global average pooling. The paper generates the eight modules from a few metaparameters: total expand width ei=128+128⌊i/2⌋e_i=128+128\lfloor i/2\rfloor, and the squeeze ratio and 3×33\times3 share above.

SqueezeNet v1.0 · SR = 0.125 · pct3x3 = 0.5parameters per layerconv1 7×7/214.2k · 96×109²maxpool 3×3/2→ 54×54fire211.9k · 128×54²fire312.4k · 128×54²fire445.3k · 256×54²maxpool 3×3/2→ 27×27fire549.4k · 256×27²fire6104.9k · 384×27²fire7111.0k · 384×27²fire8189.0k · 512×27²maxpool 3×3/2→ 13×13fire9197.2k · 512×13²conv10 1×1513.0k · 1000×13²fire3: 128×54² → 128×54²squeeze 1×1 128→16: 2.1kexpand 1×1 16→64: 1.1kexpand 3×3 16→64: 9.3kfire total 12,432 paramsplain 3×3 128→128: 147,584 (11.9×)35.83M MACs at 54×54input128squeeze161×1 / 3×364 / 64concat128total 1,248,424 params · 819M MACs4.76 MiB fp32 · AlexNet 61.1M = 48.9×published SqueezeNet: 57.5% top-1, 80.3% top-5+ Deep Compression (6-bit): 0.47 MB, same accuracy
squeeze ratio
3×3 share
fig.SqueezeNet v1.0 layer by layer (torchvision shapes at 224×224). Bars are parameters per layer. Click a Fire module to see its squeeze, expand-1×1 and expand-3×3 parts. At the default fire3 (128 → 128 channels on 54×54) the module has 12,432 parameters, against 147,584 for a plain 3×3 conv with the same input and output widths (11.9×). The total is 1,248,424 parameters (4.76 MiB in fp32), 49× fewer than torchvision's AlexNet (61.1M). Notice that conv10 alone holds 513,000 of them (41%). Raise the squeeze ratio to 0.75: 4.84M parameters, 18.5 MiB. That is the setting where the paper's sweep plateaued at 86.0% top-5. Switch to v1.1 (3×3 stem with 64 filters, earlier pooling): 1,235,496 parameters and 349M instead of 819M multiply-adds.
moduleC_in → s → e₁ + e₃paramsplain 3×3 paramssaving
fire296 → 16 → 64 + 6411,920110,7209.3×
fire3128 → 16 → 64 + 6412,432147,58411.9×
fire4128 → 32 → 128 + 12845,344295,1686.5×
fire5256 → 32 → 128 + 12849,440590,08011.9×
fire8384 → 64 → 256 + 256188,9921,769,9849.4×
fire9512 → 64 → 256 + 256197,1842,359,80812.0×
Fire modules of SqueezeNet v1.0 (SR = 0.125, pct3x3 = 0.5). Counts include biases, from the figure's library

Worked example: fire2

  1. 1Squeeze: 96⋅16+16=1,55296\cdot16+16=1{,}552 parameters, giving 16 channels.
  2. 2Expand 1×11\times1: 16⋅64+64=1,08816\cdot64+64=1{,}088. Expand 3×33\times3: 9⋅16⋅64+64=9,2809\cdot16\cdot64+64=9{,}280. Total 11,92011{,}920, and the output has 64+64=12864+64=128 channels.
  3. 3Without the squeeze (strategy 2 off), the same expand layer on all 96 input channels would need 96⋅64+64+9⋅96⋅64+64=61,56896\cdot64+64+9\cdot96\cdot64+64=61{,}568 parameters: 5.2× more.
  4. 4A plain 3×33\times3 conv 96→12896\to128 needs 9⋅96⋅128+128=110,7209\cdot96\cdot128+128=110{,}720: 9.3× more than the Fire module.
python
1import torch, torch.nn as nn
2from torchvision.models import squeezenet1_0, squeezenet1_1, SqueezeNet1_0_Weights
3
4class Fire(nn.Module):
5 def __init__(self, cin, s, e1, e3):
6 super().__init__()
7 self.squeeze = nn.Conv2d(cin, s, 1)
8 self.expand1 = nn.Conv2d(s, e1, 1)
9 self.expand3 = nn.Conv2d(s, e3, 3, padding=1)
10 self.act = nn.ReLU(inplace=True)
11
12 def forward(self, x):
13 z = self.act(self.squeeze(x))
14 return torch.cat([self.act(self.expand1(z)), self.act(self.expand3(z))], dim=1)
15
16count = lambda m: sum(p.numel() for p in m.parameters())
17print(count(Fire(96, 16, 64, 64))) # 11920
18print(Fire(96, 16, 64, 64)(torch.randn(1, 96, 54, 54)).shape) # (1, 128, 54, 54)
19
20m = squeezenet1_0(weights=SqueezeNet1_0_Weights.IMAGENET1K_V1).eval()
21print(count(m), count(squeezenet1_1())) # 1248424 1235496
22print(m.classifier) # Dropout -> Conv2d(512, 1000, 1) -> ReLU -> AdaptiveAvgPool2d(1)
A Fire module and the torchvision models

Use when

  • +Tiny models: 1.25M parameters, under 5 MB fp32 and under 0.5 MB compressed
  • +Simple, regular module with two easy knobs (squeeze ratio, 3×3 share)
  • +No fully connected layers: works at any input size
  • +Good for on-chip memory, FPGAs and frequent over-the-air updates

Avoid when

  • −High compute for its size: 819M multiply-adds (v1.0)
  • −Accuracy is only AlexNet-level; modern small nets are far more accurate
  • −Many small layers and concatenations limit hardware utilization
  • −No batch normalization in the original design, so it is touchier to train

Check yourself

A Fire module takes 256 input channels and has s = 32, e₁ = 128, e₃ = 128. Which part has the most parameters?

Check yourself

SqueezeNet v1.0 and v1.1 have almost the same number of parameters. Why does v1.1 need 2.35× fewer multiply-adds?

Remember

  • ✓Fire module: squeeze with ss 1×11\times1 filters, then expand with e1e_1 1×11\times1 and e3e_3 3×33\times3 filters, concatenated.
  • ✓Three strategies: prefer 1×1 filters, feed fewer channels to the 3×3 filters, downsample late.
  • ✓1,248,424 parameters (≈50× fewer than AlexNet) at AlexNet accuracy; 0.47 MB with Deep Compression.
  • ✓Small in parameters, not in compute: 819M multiply-adds for v1.0, 349M for v1.1.
  • ✓No FC layers: a 1×1 conv to the classes plus global average pooling.
← back to Deep Learning roadmap