# MobileNet

> A family of mobile CNNs built from depthwise separable convolutions. V1 adds width and resolution knobs (4.2M params, 569M mult-adds), V2 inverted residuals with linear bottlenecks, and V3 a searched architecture with h-swish and squeeze-and-excitation.

Source: https://www.mlfoundry.in/learn/mobilenet  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 13 min read · updated 2026-10-11

**MobileNet:** is a family of convolutional networks from Google designed for phones and embedded devices. Nearly all of their spatial filtering is done by cheap **depthwise** $3\times3$ (or $5\times5$) convolutions, and nearly all of their computation is in $1\times1$ convolutions. Three versions matter: **V1** (Howard et al., 2017), **V2** (Sandler et al., 2018) and **V3** (Howard et al., 2019).

By 2016 the accurate ImageNet models (VGG, Inception, ResNet) cost billions of multiply-adds per image, far too much for a phone CPU. Howard et al.'s "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications" (2017) took a deliberately plain approach. Stack **Depthwise separable convolution** blocks in a straight line, like VGG. Then give the user two global knobs to trade accuracy for cost. The result reached 70.6% ImageNet top-1 with 569M multiply-adds, close to GoogLeNet's accuracy at about a third of its compute. It became the default backbone for on-device detection (SSD-MobileNet) and segmentation.

**Three versions**

- **V1 (2017):** $3\times3$ stem + 13 depthwise separable blocks, BN + ReLU everywhere; width multiplier $\alpha$, resolution multiplier $\rho$. 4.2M params, 569M mult-adds, 70.6% top-1
- **V2 (2018):** inverted residual blocks: $1\times1$ expand ($t=6$) → depthwise $3\times3$ → **linear** $1\times1$ project, ReLU6, residual between bottlenecks. 3.4M params, 300M mult-adds, 72.0% top-1
- **V3 (2019):** block structure found by platform-aware NAS + NetAdapt; h-swish, squeeze-and-excitation, $5\times5$ depthwise, cheaper head. Large: 5.4M, 219M, 75.2%; Small: 2.5M, 56M, 67.4%
- **Accuracy source:** ImageNet top-1, 224×224 single crop, as reported in each paper
- **In PyTorch:** `torchvision.models.mobilenet_v2`, `mobilenet_v3_large`, `mobilenet_v3_small` (V1 is in `timm`, not torchvision)

## V1: separable blocks and two multipliers

Each V1 block is a depthwise $3\times3$ conv followed by a pointwise $1\times1$ conv, each with BatchNorm and ReLU. Downsampling happens with stride 2 in the depthwise conv. Channels go $32\to64\to128\to\dots\to1024$ while the map shrinks $112\to7$, then global average pooling and one fully connected layer. Counting depthwise and pointwise separately, the network is 28 layers deep.

The **width multiplier** $\alpha\in(0,1]$ thins every layer to $\alpha C$ channels. The **resolution multiplier** $\rho$ shrinks the input image (224 → 192, 160, 128), which shrinks every feature map. A block's cost becomes

$$
\underbrace{k^2\,\alpha C_{in}\,(\rho D)^2}_{\text{depthwise}}\;+\;\underbrace{\alpha C_{in}\,\alpha C_{out}\,(\rho D)^2}_{\text{pointwise}}\;\approx\;\alpha^2\rho^2\cdot\text{cost}(\alpha{=}1,\rho{=}1)
$$

- $\alpha$: width multiplier: cost and parameters scale roughly as $\alpha^2$ (the pointwise term dominates)
- $\rho$: resolution multiplier: cost scales as $\rho^2$; parameters do not change
- $D$: the side of the block's output feature map at full resolution

The paper also reports where the work goes: 95% of the multiply-adds and 75% of the parameters are in $1\times1$ convolutions. That is good news for speed, because a $1\times1$ conv is a plain matrix multiply (GEMM) with no im2col reshuffle. The depthwise convs that do all the spatial filtering account for only about 3% of the compute.

> Figure: MobileNetV1 and V2 built layer by layer from the papers' configurations. Bars are multiply-adds per block, split into depthwise 3×3 (blue), 1×1 (orange) and other (stem, FC). Click a row for its shapes, parameters and cost. At the default V1, α = 1, 224×224, the library counts 4,210,088 parameters and 568.7M multiply-adds, matching the paper's 4.2M and 569M. The 1×1 convs take 94.9% of the multiply-adds, the depthwise convs 3.1%. The selected block (512 → 512 on 14×14) costs 52.3M multiply-adds, against 462.4M for a full 3×3 conv. Try α = 0.5 (149.5M, paper 149M) or 128×128 (186.4M, paper 186M). Switch to V2: torchvision's counting gives 3,504,872 parameters and 300.8M multiply-adds (paper 300M). The expansion slider shows the cost of t: 63.7M at t = 1 and 387.0M at t = 8.

MobileNetV1 trade-off points. Params and multiply-adds from the figure's library; top-1 and paper columns from Howard et al. 2017, Tables 6 and 7

| setting | params (lib) | mult-adds (lib) | paper | top-1 |
| --- | --- | --- | --- | --- |
| $\alpha=1$, 224 | 4,210,088 | 568.7M | 4.2M / 569M | 70.6% |
| $\alpha=0.75$, 224 | 2,569,144 | 325.4M | 2.6M / 325M | 68.4% |
| $\alpha=0.5$, 224 | 1,320,648 | 149.5M | 1.3M / 149M | 63.7% |
| $\alpha=0.25$, 224 | 464,600 | 41.0M | 0.5M / 41M | 50.6% |
| $\alpha=1$, 192 | 4,210,088 | 418.1M | 4.2M / 418M | 69.1% |
| $\alpha=1$, 160 | 4,210,088 | 290.7M | 4.2M / 290M | 67.2% |
| $\alpha=1$, 128 | 4,210,088 | 186.4M | 4.2M / 186M | 64.4% |

Two things to read from the table. Accuracy falls smoothly down to $\alpha=0.5$ and then collapses at $0.25$. And parameters do **not** shrink as $\alpha^2$ at small widths: 1,025,000 of the 4.2M parameters are the final FC layer ($1024\times1000$ plus biases), which only shrinks linearly with $\alpha$.

## V2: inverted residuals and linear bottlenecks

Sandler et al. ("MobileNetV2: Inverted Residuals and Linear Bottlenecks", CVPR 2018) changed the block. A ResNet bottleneck is wide → narrow → wide, with the shortcut between the wide ends. MobileNetV2 inverts it: the block input and output are **narrow** bottlenecks, and the inside is **expanded** by a factor $t$ (6 in the paper) so that the depthwise conv works in a high-dimensional space.

$$
x\in\mathbb R^{c}\;\xrightarrow{\;1\times1,\ \text{ReLU6}\;}\;\mathbb R^{tc}\;\xrightarrow{\;\text{dw }3\times3,\ \text{ReLU6}\;}\;\mathbb R^{tc}\;\xrightarrow{\;1\times1,\ \textbf{linear}\;}\;\mathbb R^{c'}\qquad y=x+F(x)\ \text{ if stride 1 and } c=c'
$$

- $t$: expansion factor; the first block uses $t=1$ and all others $t=6$
- $\text{ReLU6}$: $\min(\max(0,x),6)$, kept for robustness in low-precision arithmetic
- $\text{linear}$: no activation after the projection: the bottleneck stays linear

**Why linear?** The paper's argument: if the "manifold of interest" lies in a low-dimensional subspace, a ReLU applied in a narrow layer zeroes out whole directions and loses information that later layers cannot recover. Expanding first and applying the ReLU in the wide space loses much less. Applying no ReLU after projecting back to the narrow space loses none. Their ablation found that adding a ReLU after the projection lowers accuracy.

**Why connect the narrow ends?** The residual shortcut joins the bottlenecks, which carry the information, and the stored tensor between blocks is small. MobileNetV2 has 17 such blocks with widths 16, 24, 32, 64, 96, 160, 320, then a $1\times1$ conv to 1280 and the classifier.

**Worked example: the 24 → 24 bottleneck at 56×56 (block 3), t = 6**

1. Expand $1\times1$: $24\to144$ channels, $24\cdot144=3{,}456$ weights, $3{,}456\cdot56^2=10{,}838{,}016$ multiply-adds.
2. Depthwise $3\times3$ on 144 channels: $9\cdot144=1{,}296$ weights, $1{,}296\cdot3{,}136=4{,}064{,}256$ multiply-adds.
3. Project $1\times1$: $144\to24$, another 3,456 weights and 10,838,016 multiply-adds. No ReLU.
4. Total 25,740,288 multiply-adds and 8,832 parameters including BatchNorm ($8{,}208$ weights $+\,2\cdot(144+144+24)$). Stride 1 and $24=24$, so the input is added to the output. The $1\times1$ convs are 84.2% of the block's cost.

## V3: searched, with h-swish and squeeze-and-excitation

Howard et al. ("Searching for MobileNetV3", ICCV 2019) kept the inverted residual block and let search tune it. Platform-aware neural architecture search (MnasNet-style, with measured phone latency in the reward) chose the block-level structure, and NetAdapt then trimmed individual layer widths. Several hand-made changes came on top:

$$
\text{h-swish}(x)=x\cdot\frac{\mathrm{ReLU6}(x+3)}{6},\qquad \text{SE}(u)=u\odot\text{h-sigmoid}\big(W_2\,\mathrm{ReLU}(W_1\,\mathrm{pool}(u))\big)
$$

- $\text{h-swish}$: a piecewise-polynomial stand-in for swish $x\sigma(x)$, cheap and quantization-friendly; used mainly in the deeper layers
- $\text{SE}$: squeeze-and-excitation (Hu et al., 2018): reweights the expanded channels by a gate computed from their global average
- $W_1$: reduces to 1/4 of the expanded width, then $W_2$ expands back

They also halved the stem to 16 filters and redesigned the expensive last stage: the final $1\times1$ expansion moves after global pooling, so it runs on a $1\times1$ map instead of $7\times7$. MobileNetV3-Large reports 75.2% top-1 at 219M multiply-adds (5.4M parameters), and V3-Small 67.4% at 56M (2.5M). torchvision's versions have 5,483,032 and 2,542,856 parameters.

```python
import torch, torch.nn as nn
from torchvision.models import (mobilenet_v2, MobileNet_V2_Weights,
                                mobilenet_v3_large, MobileNet_V3_Large_Weights)

class InvertedResidual(nn.Module):
    def __init__(self, cin, cout, stride=1, t=6):
        super().__init__()
        hid = cin * t
        self.use_res = stride == 1 and cin == cout
        layers = []
        if t != 1:                                            # 1x1 expand
            layers += [nn.Conv2d(cin, hid, 1, bias=False), nn.BatchNorm2d(hid), nn.ReLU6(inplace=True)]
        layers += [nn.Conv2d(hid, hid, 3, stride, 1, groups=hid, bias=False),   # depthwise 3x3
                   nn.BatchNorm2d(hid), nn.ReLU6(inplace=True),
                   nn.Conv2d(hid, cout, 1, bias=False), nn.BatchNorm2d(cout)]  # linear bottleneck
        self.conv = nn.Sequential(*layers)

    def forward(self, x):
        return x + self.conv(x) if self.use_res else self.conv(x)

count = lambda m: sum(p.numel() for p in m.parameters())
print(count(InvertedResidual(24, 24)))          # 8832

m = mobilenet_v2(weights=MobileNet_V2_Weights.IMAGENET1K_V1).eval()
print(count(m))                                 # 3504872
print(count(mobilenet_v2(width_mult=0.5)))      # 1968680: the width multiplier
print(count(mobilenet_v3_large()))              # 5483032 (uses nn.Hardswish and SE blocks)

with torch.no_grad():
    logits = m(torch.randn(1, 3, 224, 224))     # any input side works: global pooling at the end
```

An inverted residual block, and the torchvision models

> **Multiply-adds are not milliseconds:** Models with similar FLOP counts can differ a lot in speed. Depthwise convs are memory-bound, the expanded tensors of V2 cost memory traffic, and SE and h-swish add element-wise passes. That is why MobileNetV3 searched with **measured** latency on a Pixel phone, not FLOPs. **ShuffleNet** v2 makes the same point and measures it.

> **Two knobs instead of a new design:** The width and resolution multipliers make one architecture into a whole family of sizes. You pick the point on the accuracy/latency curve that fits the device. In V1, shrinking the input from 224 to 128 costs 6.2 points of top-1, and shrinking the width to $\alpha=0.5$ costs 6.9, at roughly the same compute. EfficientNet later formalized this as compound scaling of width, depth and resolution together.

> **Where you meet it:** MobileNets are the standard backbones for on-device vision: SSDLite and DeepLabv3 with MobileNetV2/V3 (torchvision ships `ssdlite320_mobilenet_v3_large` and `deeplabv3_mobilenet_v3_large`). They train with **Batch normalization** and usually with **Label smoothing**, and their depthwise layers are known to be harder to quantize to int8 than dense convs.

**Strengths**

- Very low multiply-adds and parameters for their accuracy
- Width and resolution multipliers give a family of models from one design
- Most compute is in 1×1 convs, which are plain matrix multiplies
- Mature tooling: torchvision/timm weights, TFLite and mobile runtimes optimize them

**Limitations**

- Depthwise layers are memory-bound: GPU speed-ups are much smaller than the FLOP savings
- Lower accuracy than large CNNs or ViTs; $\alpha\le0.25$ degrades sharply
- Depthwise weights and ReLU6/h-swish make int8 quantization more delicate
- V3's searched design is tuned for one phone CPU and may not be optimal elsewhere

**Question:** MobileNetV1 at α = 1, 224×224 costs 569M multiply-adds. Roughly what does α = 0.5 at 224×224 cost?

- A. About 285M (half)
- B. About 150M (a quarter)
- C. About 71M (an eighth)
- D. About 569M (unchanged)

**Answer:** B. About 150M (a quarter) — The pointwise convs dominate and their cost is $\alpha C_{in}\cdot\alpha C_{out}$, so the total scales as about $\alpha^2=0.25$. The figure's library gives 149.5M and the paper 149M.

**Question:** Why does MobileNetV2 omit the activation after the 1×1 projection of each block?

- A. To save one ReLU6 per block
- B. A ReLU in the narrow bottleneck zeroes directions and destroys information; keeping it linear preserves it
- C. Because BatchNorm already acts as a nonlinearity
- D. So the residual addition does not overflow ReLU6's range

**Answer:** B. A ReLU in the narrow bottleneck zeroes directions and destroys information; keeping it linear preserves it — The bottleneck is low-dimensional, and a ReLU there throws away whatever falls on its negative side. Sandler et al. keep the nonlinearity in the expanded space, where it loses little, and found empirically that a nonlinear projection lowers ImageNet accuracy.

**Key points**

- V1: straight stack of depthwise separable blocks; 4.2M params, 569M mult-adds, 70.6% top-1; 95% of compute is $1\times1$ convs.
- Width multiplier $\alpha$ scales cost by about $\alpha^2$; resolution $\rho$ scales cost by $\rho^2$ and leaves parameters unchanged.
- V2: inverted residuals (narrow → expand $t=6$ → depthwise → narrow) with a linear projection and shortcuts between bottlenecks; 3.4M, 300M, 72.0%.
- V3: NAS + NetAdapt, h-swish, squeeze-and-excitation, a cheaper head; Large 75.2% at 219M mult-adds.
- Low FLOPs do not guarantee low latency: measure on the target device.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
