Definition
SqueezeNet is a small CNN by Iandola, Han, Moskewicz, Ashraf, Dally and Keutzer ("SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size", 2016). It is built from Fire modules: a squeeze layer of convolutions that cuts the number of channels, followed by an expand layer that applies and convolutions in parallel to the squeezed tensor and concatenates their outputs.
Most efficiency work targets compute. SqueezeNet targets parameter count, i.e. model size. The authors gave three reasons why that matters: distributed training sends less over the network, over-the-air updates to cars and phones get smaller, and a small model can fit in on-chip memory of an FPGA or embedded chip. AlexNet's 240 MB of fp32 weights fail all three. SqueezeNet matched AlexNet's ImageNet accuracy (57.5% top-1, 80.3% top-5, against AlexNet's 57.2% and 80.3%) with a 4.8 MB model.
At a glance
- Paper
- Iandola et al., arXiv 2016 (Berkeley and Stanford)
- Building block
- Fire module: squeeze → expand , concatenated, with
- Size
- 1,248,424 parameters (4.76 MiB fp32); 0.47 MB with Deep Compression at 6 bits
- Compute
- 819M multiply-adds at 224×224 (v1.0); v1.1 needs 349M with the same accuracy
- Accuracy
- 57.5% top-1 / 80.3% top-5 on ImageNet (paper); torchvision weights: 58.09% / 80.42%
- No FC layers
- a conv to 1000 channels and global average pooling produce the logits
- In PyTorch
torchvision.models.squeezenet1_0,squeezenet1_1
Three design strategies
From the paper, Section 3.1
- 1Replace filters with filters. A filter has 9× fewer weights. Use them for as many of the filters as possible.
- 2Decrease the number of input channels to the filters. A layer costs weights, so cutting cuts the cost directly. The squeeze layer does this.
- 3Downsample late. Keep large activation maps for most of the network, because the paper's authors (and He & Sun, 2015) found that late downsampling raises accuracy. Strategies 1 and 2 save parameters; strategy 3 spends some of that saving on accuracy.
The Fire module
- squeeze: filters of size
- expand: filters of and filters of (padding 1 keeps the size)
- concatenation along the channel axis
- squeeze width; SqueezeNet uses , the squeeze ratio
- share of filters, in SqueezeNet
In the backward pass the concatenation simply splits the incoming gradient into its and channel slices. The squeeze output receives the sum of what both expand branches send back: , where is the transposed convolution. The narrow squeeze layer is therefore a bottleneck in both directions.
The macro-architecture is a /2 conv with 96 filters, eight Fire modules (fire2–fire9) whose width grows from 128 to 512, max-pools after conv1, fire4 and fire8, then dropout, a conv to 1000 classes and global average pooling. The paper generates the eight modules from a few metaparameters: total expand width , and the squeeze ratio and share above.
| module | C_in → s → e₁ + e₃ | params | plain 3×3 params | saving |
|---|---|---|---|---|
| fire2 | 96 → 16 → 64 + 64 | 11,920 | 110,720 | 9.3× |
| fire3 | 128 → 16 → 64 + 64 | 12,432 | 147,584 | 11.9× |
| fire4 | 128 → 32 → 128 + 128 | 45,344 | 295,168 | 6.5× |
| fire5 | 256 → 32 → 128 + 128 | 49,440 | 590,080 | 11.9× |
| fire8 | 384 → 64 → 256 + 256 | 188,992 | 1,769,984 | 9.4× |
| fire9 | 512 → 64 → 256 + 256 | 197,184 | 2,359,808 | 12.0× |
Worked example: fire2
- 1Squeeze: parameters, giving 16 channels.
- 2Expand : . Expand : . Total , and the output has channels.
- 3Without the squeeze (strategy 2 off), the same expand layer on all 96 input channels would need parameters: 5.2× more.
- 4A plain conv needs : 9.3× more than the Fire module.
1import torch, torch.nn as nn2from torchvision.models import squeezenet1_0, squeezenet1_1, SqueezeNet1_0_Weights34class Fire(nn.Module):5 def __init__(self, cin, s, e1, e3):6 super().__init__()7 self.squeeze = nn.Conv2d(cin, s, 1)8 self.expand1 = nn.Conv2d(s, e1, 1)9 self.expand3 = nn.Conv2d(s, e3, 3, padding=1)10 self.act = nn.ReLU(inplace=True)1112 def forward(self, x):13 z = self.act(self.squeeze(x))14 return torch.cat([self.act(self.expand1(z)), self.act(self.expand3(z))], dim=1)1516count = lambda m: sum(p.numel() for p in m.parameters())17print(count(Fire(96, 16, 64, 64))) # 1192018print(Fire(96, 16, 64, 64)(torch.randn(1, 96, 54, 54)).shape) # (1, 128, 54, 54)1920m = squeezenet1_0(weights=SqueezeNet1_0_Weights.IMAGENET1K_V1).eval()21print(count(m), count(squeezenet1_1())) # 1248424 123549622print(m.classifier) # Dropout -> Conv2d(512, 1000, 1) -> ReLU -> AdaptiveAvgPool2d(1)Use when
- +Tiny models: 1.25M parameters, under 5 MB fp32 and under 0.5 MB compressed
- +Simple, regular module with two easy knobs (squeeze ratio, 3×3 share)
- +No fully connected layers: works at any input size
- +Good for on-chip memory, FPGAs and frequent over-the-air updates
Avoid when
- −High compute for its size: 819M multiply-adds (v1.0)
- −Accuracy is only AlexNet-level; modern small nets are far more accurate
- −Many small layers and concatenations limit hardware utilization
- −No batch normalization in the original design, so it is touchier to train
Check yourself
A Fire module takes 256 input channels and has s = 32, e₁ = 128, e₃ = 128. Which part has the most parameters?
Check yourself
SqueezeNet v1.0 and v1.1 have almost the same number of parameters. Why does v1.1 need 2.35× fewer multiply-adds?
Remember
- ✓Fire module: squeeze with filters, then expand with and filters, concatenated.
- ✓Three strategies: prefer 1×1 filters, feed fewer channels to the 3×3 filters, downsample late.
- ✓1,248,424 parameters (≈50× fewer than AlexNet) at AlexNet accuracy; 0.47 MB with Deep Compression.
- ✓Small in parameters, not in compute: 819M multiply-adds for v1.0, 349M for v1.1.
- ✓No FC layers: a 1×1 conv to the classes plus global average pooling.