# EfficientNet

> Scale depth, width and input resolution together, by α^φ, β^φ, γ^φ with α·β²·γ² ≈ 2, so each step of φ doubles FLOPs. Applied to a small NAS-found MBConv network (B0: 5.3M params, 0.39 GFLOPs), it gives B1–B7, up to 84.3% ImageNet top-1.

Source: https://www.mlfoundry.in/learn/efficientnet  
Section: Deep Learning › Convolutional Neural Networks  
Level: core · 14 min read · updated 2026-10-11

**EfficientNet:** is a family of convolutional networks, B0 to B7, obtained by **compound scaling** of one baseline. The baseline, EfficientNet-B0, was found by neural architecture search and is built from mobile inverted bottleneck (**MBConv**) blocks with squeeze-and-excitation and the Swish (SiLU) activation. Larger models multiply its depth by $\alpha^\phi$, its width by $\beta^\phi$ and its input resolution by $\gamma^\phi$ for a single user-chosen coefficient $\phi$.

Mingxing Tan and Quoc V. Le introduced EfficientNet in "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" (ICML 2019). Before it, scaling a ConvNet up usually meant turning one knob: more layers (ResNet-18 to ResNet-200), more channels (Wide ResNet), or larger input images (GPipe used 480×480). The paper measured each knob on its own and found that accuracy improves and then quickly **saturates near 80%** top-1 for all three. The dimensions are not independent. A larger image has larger objects, so it needs more layers to grow the receptive field and more channels to capture finer patterns. Scaling all three in a fixed ratio worked better than any single one.

**At a glance**

- **Paper:** Tan & Le, ICML 2019 (Google Brain)
- **Scaling rule:** depth $\alpha^\phi$, width $\beta^\phi$, resolution $\gamma^\phi$; $\alpha=1.2,\ \beta=1.1,\ \gamma=1.15$
- **Constraint:** $\alpha\beta^2\gamma^2\approx2$, so FLOPs grow by about $2^\phi$
- **Baseline B0:** NAS (MnasNet search space) with a 400M FLOPs target; MBConv + SE + SiLU; 5.3M params, 0.39G FLOPs
- **Top model:** B7: 84.3% top-1, 66M params, 37B FLOPs (paper); 8.4× smaller than GPipe at the same accuracy
- **Successor:** EfficientNetV2 (Tan & Le, 2021): Fused-MBConv, training-aware NAS, progressive learning
- **In PyTorch:** `torchvision.models.efficientnet_b0` … `efficientnet_b7`, `efficientnet_v2_s/m/l`; timm

## Compound scaling

The FLOPs of a convolution are proportional to depth (number of layers), to the square of width (input times output channels) and to the square of resolution (height times width of the map). Doubling depth doubles the cost; doubling width or resolution quadruples it. So if depth, width and resolution are multiplied by $d$, $w$, $r$, total FLOPs scale by roughly $d\,w^2r^2$. EfficientNet ties all three to one coefficient:

$$
d=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi},\qquad \text{s.t.}\ \ \alpha\cdot\beta^{2}\cdot\gamma^{2}\approx2,\ \ \alpha,\beta,\gamma\ge1\qquad\Rightarrow\qquad \text{FLOPs}\propto(\alpha\beta^2\gamma^2)^{\phi}\approx2^{\phi}
$$

- $\phi$: compound coefficient: how much more compute is available (each +1 roughly doubles FLOPs)
- $d,\ w,\ r$: multipliers for the number of layers, the channels and the input side length
- $\alpha,\beta,\gamma$: per-dimension growth constants, found once by a small grid search at $\phi=1$
- $\beta^2,\ \gamma^2$: width and resolution enter squared because FLOPs are quadratic in both

The procedure has two steps. **Step 1:** fix $\phi=1$, assume twice the resources, and grid-search $\alpha,\beta,\gamma$ under the constraint on B0; the result is $\alpha=1.2$, $\beta=1.1$, $\gamma=1.15$. **Step 2:** fix those constants and scale B0 with increasing $\phi$ to obtain B1 to B7. Searching the constants directly on a big model might do better, but it would cost far more; the paper does the search once, on the small model. With these values $\alpha\beta^2\gamma^2=1.2\cdot1.21\cdot1.3225=1.920$, so the predicted growth is $1.92^\phi$.

Taking logarithms shows what the rule really does: $\log d:\log w:\log r=\log1.2:\log1.1:\log1.15\approx1.91:1:1.47$. In log terms, each step of $\phi$ takes 28% of its extra FLOPs from depth ($\log1.2/\log1.92=0.28$), 29% from width ($2\log1.1/\log1.92=0.29$) and 43% from resolution ($2\log1.15/\log1.92=0.43$).

> Figure: Drag φ to scale EfficientNet-B0 and compare strategies that all aim at about 2^φ FLOPs. The square on the left is the input, the coloured slabs are the MBConv layers of stages 1–7 (width ∝ √channels), and the panel recounts parameters and multiply-adds from the real torchvision rules (channels rounded to multiples of 8, layer counts rounded up). The B0–B7 buttons load the official multipliers, and every one reproduces torchvision's parameter count: 5,288,548 for B0 at 0.386G multiply-adds, up to 66,347,960 and 37.75G for B7. Rounding matters. At φ = 1 compound scaling gives 23 blocks instead of 16 and 2.57× the FLOPs of B0, against the 1.92× the formula predicts. At φ = 3, depth-only scaling reaches 8.19× FLOPs with 39.8M parameters, while resolution-only reaches 8.10× with B0's 5.29M.

## The baseline: EfficientNet-B0

Compound scaling only changes the size of the layers, not their type, so the baseline matters. B0 comes from a multi-objective architecture search in the MnasNet search space that maximizes $\mathrm{ACC}(m)\cdot[\mathrm{FLOPS}(m)/T]^{w}$ with target $T=400$M FLOPs and $w=-0.07$. Unlike MnasNet, it optimizes FLOPs rather than measured latency on a specific phone, so it is not tied to one device.

EfficientNet-B0 (Tan & Le 2019, Table 1). MBConv6 = expansion ratio 6. Params and multiply-adds per stage counted by the figure's code (torchvision layout, 224×224); the total is 5,288,548 parameters and 0.386G multiply-adds.

| Stage | Operator | Input res. | Channels | Layers | Params | MACs |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | Conv 3×3, stride 2 | 224² | 32 | 1 | 928 | 10.8M |
| 2 | MBConv1, k3×3 | 112² | 16 | 1 | 1,448 | 10.0M |
| 3 | MBConv6, k3×3 | 112² | 24 | 2 | 16,714 | 54.9M |
| 4 | MBConv6, k5×5 | 56² | 40 | 2 | 46,640 | 37.9M |
| 5 | MBConv6, k3×3 | 28² | 80 | 3 | 242,930 | 43.6M |
| 6 | MBConv6, k5×5 | 14² | 112 | 3 | 543,148 | 86.1M |
| 7 | MBConv6, k5×5 | 14² | 192 | 4 | 2,026,348 | 91.5M |
| 8 | MBConv6, k3×3 | 7² | 320 | 1 | 717,232 | 29.5M |
| 9 | Conv 1×1, pool, FC | 7² | 1280 | 1 | 1,693,160 | 21.4M |

Each MBConv block (from MobileNetV2, with SE as in MnasNet) expands the input with a $1\times1$ conv by factor 6, applies a depthwise $k\times k$ conv (stride 2 in the first block of a downsampling stage), recalibrates channels with squeeze-and-excitation (squeeze to a quarter of the block's input channels), and projects back with a linear $1\times1$ conv. A residual connection, with **Stochastic depth** during training, is used when input and output shapes match:

$$
y=x+P\Big(\mathrm{SE}\big(\mathrm{SiLU}(\mathrm{DW}_k(\mathrm{SiLU}(E\,x)))\big)\Big),\qquad \mathrm{SiLU}(z)=z\,\sigma(z),\qquad \mathrm{SiLU}'(z)=\sigma(z)\big(1+z\,(1-\sigma(z))\big)
$$

- $E$: $1\times1$ expansion conv $c\to6c$ + BN (absent in MBConv1)
- $\mathrm{DW}_k$: depthwise $k\times k$ conv + BN: one filter per channel, $k^2\cdot6c$ weights
- $\mathrm{SE}$: squeeze-and-excitation gate, $6c\to\lfloor c/4\rfloor\to6c$, sigmoid output
- $P$: $1\times1$ projection $6c\to c'$ + BN, no activation (linear bottleneck)
- $\mathrm{SiLU}'$: derivative used in backprop: 0.5 at $z=0$, tends to 1 for large $z$, slightly negative for $z<-1.28$

Almost all of B0's compute sits in $1\times1$ convs; the depthwise convs are cheap in FLOPs but, as the RegNet paper later stressed, produce large activations and run with low arithmetic intensity on GPUs. 32% of B0's parameters (1.69M) are in the final $1\times1$ conv to 1280 channels and the classifier.

## Worked example: φ = 1

**Compound-scaling B0 by one step (as in the figure)**

1. Multipliers: $d=1.2$, $w=1.1$, $r=1.15$, so resolution $224\cdot1.15=257.6\to258$.
2. Width: channels are $c\cdot1.1$ rounded to a multiple of 8. $32\to35.2\to32$, $40\to44\to48$, $112\to123.2\to120$, $192\to211.2\to208$, $320\to352$, head $4\cdot352=1408$.
3. Depth: layers are $\lceil n\cdot1.2\rceil$: $(1,2,2,3,3,4,1)\to(2,3,3,4,4,5,2)$, 23 blocks instead of 16. The ceiling turns every 1-layer stage into 2 layers.
4. Result: 9,109,994 parameters and 0.993G multiply-adds, **2.57×** B0's FLOPs. The formula predicted $1.92\times$; the gap is almost all rounding of layer counts. This $(w,d)=(1.1,1.2)$ is exactly B2's width and depth; the official models were hand-adjusted (B1 is $(1.0,1.1)$ at 240 px, B2 is $(1.1,1.2)$ at 260 px in the paper).
5. Same target, one knob only, at $\phi=3$ (8× FLOPs): depth $\times8$ gives 8.19× FLOPs and 39.8M parameters, width $\times2.83$ gives 7.35× and 34.6M, resolution $\times2.83$ (634 px) gives 8.10× with B0's 5.29M parameters, and compound gives 8.43× with 16.3M.

The figure shows the cost side. The accuracy side is the paper's result: at equal FLOPs, compound scaling beat single-dimension scaling. For MobileNetV2 (72.0% baseline at 0.3B FLOPs) the paper reports 76.8% for depth ×4, 76.4% for width ×2, 74.8% for resolution ×2 and 77.4% for compound scaling, all at 1.1 to 1.3B FLOPs.

```python
import math, torch
from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights, efficientnet_v2_s

alpha, beta, gamma = 1.2, 1.1, 1.15
print(alpha * beta**2 * gamma**2)            # 1.920...: FLOPs grow ~1.92^phi
for phi in range(8):
    d, w, r = alpha**phi, beta**phi, gamma**phi
    print(phi, f"depth x{d:.2f}  width x{w:.2f}  res {round(224 * r)}")

weights = EfficientNet_B0_Weights.IMAGENET1K_V1
model = efficientnet_b0(weights=weights).eval()
print(sum(p.numel() for p in model.parameters()))   # 5288548
print(model.features[2][0])                         # MBConv: expand 16->96, dw 3x3 s2, SE 96->4->96, project ->24
preprocess = weights.transforms()                   # resize 256, center-crop 224
x = preprocess(torch.rand(3, 300, 300)).unsqueeze(0)
print(model(x).softmax(-1).shape)                   # torch.Size([1, 1000])

v2 = efficientnet_v2_s()                            # Fused-MBConv early stages, 21.5M params
```

Compound-scaling arithmetic and torchvision's EfficientNets

ImageNet. Paper: Tan & Le 2019, Table 2 (top-1, params, FLOPs at each model's own resolution). torchvision: documented acc@1 of the IMAGENET1K_V1 weights (B1 also has V2 weights at 79.84%), params and GFLOPs, which the figure's code reproduces.

| Model | res. | paper top-1 | paper params / FLOPs | torchvision acc@1 | tv params / GFLOPs |
| --- | --- | --- | --- | --- | --- |
| B0 | 224 | 77.1% | 5.3M / 0.39B | 77.69% | 5.3M / 0.39 |
| B1 | 240 | 79.1% | 7.8M / 0.70B | 78.64% | 7.8M / 0.69 |
| B2 | 260 (tv 288) | 80.1% | 9.2M / 1.0B | 80.61% | 9.1M / 1.09 |
| B3 | 300 | 81.6% | 12M / 1.8B | 82.01% | 12.2M / 1.83 |
| B4 | 380 | 82.9% | 19M / 4.2B | 83.38% | 19.3M / 4.39 |
| B5 | 456 | 83.6% | 30M / 9.9B | 83.44% | 30.4M / 10.27 |
| B6 | 528 | 84.0% | 43M / 19B | 84.01% | 43.0M / 19.07 |
| B7 | 600 | 84.3% | 66M / 37B | 84.12% | 66.3M / 37.75 |
| ResNet-50 (paper) | 224 | 76.0% | 26M / 4.1B | 76.13% | 25.6M / 4.09 |

B0 matches ResNet-50's accuracy with about 5× fewer parameters and 11× fewer FLOPs, and B7 matches GPipe's 84.3% with 8.4× fewer parameters and 6.1× faster inference (paper). Training used RMSProp (decay 0.9, momentum 0.9), weight decay $10^{-5}$, learning rate 0.256 decayed by 0.97 every 2.4 epochs, SiLU, AutoAugment, stochastic depth with survival probability 0.8, and dropout before the classifier rising from 0.2 (B0) to 0.5 (B7). See **Swish**, **Dropout** and **Stochastic depth**.

## EfficientNetV2

Tan and Le's "EfficientNetV2: Smaller Models and Faster Training" (ICML 2021) targets the weaknesses of V1: very large training images are slow and memory-hungry, and depthwise convs in early layers are slow on accelerators. V2 replaces early MBConv blocks with **Fused-MBConv**, where a regular $3\times3$ conv does the work of the expansion $1\times1$ plus the depthwise conv, searches with training speed in the objective, scales less aggressively in resolution, and uses **progressive learning**: image size grows during training, with regularization (dropout, RandAugment, **Mixup**) increased along with it. In torchvision, `efficientnet_v2_s` has 21.5M parameters, 8.37 GFLOPs and 84.23% acc@1.

> **Use each model's own resolution:** Resolution is one of the three scaled dimensions, so B4 evaluated at 224 is not B4. torchvision's `weights.transforms()` applies the right resize and crop (for example 380 for B4, 600 for B7, and 288 for its B2 weights rather than the paper's 260). Feeding 224 crops to a large EfficientNet silently loses accuracy, and fine-tuning at a different resolution needs a short adaptation.

> **FLOPs are not speed:** EfficientNets are FLOP- and parameter-efficient, but depthwise convs and high resolutions create many large activations and memory traffic. On GPUs, RegNet's authors reported RegNets up to 5× faster at comparable FLOPs and training, and EfficientNetV2's Fused-MBConv exists for the same reason. Pick by measured throughput on your hardware, not by FLOPs alone.

> **Compound scaling beyond EfficientNet:** The rule is a general recipe. The paper applied it to MobileNets and ResNet-50 as well, and later families (EfficientDet for detection, EfficientNetV2, scaled ResNets) reuse the idea that depth, width and resolution should grow together, though the best ratios depend on the architecture and the training recipe.

**Strengths**

- One coefficient $\phi$ scales a whole family; the constants are searched once on a small model
- Excellent accuracy per parameter and per FLOP (B0: 77.1% with 5.3M parameters and 0.39B FLOPs)
- Strong transfer: the paper reports state-of-the-art accuracy on 5 of 8 transfer datasets
- Widely available pretrained weights in torchvision and timm

**Limitations**

- Depthwise convs and large inputs make it slower on GPUs than its FLOPs suggest
- High resolution (up to 600 px) makes training memory-hungry
- Training recipe is heavy (RMSProp, AutoAugment, stochastic depth, long schedules) and sensitive
- Rounding of layers and channels makes the realised FLOPs drift from 2^φ

**Question:** With α = 1.2, β = 1.1, γ = 1.15, by roughly what factor do FLOPs grow when φ goes from 0 to 3?

- A. 3×
- B. About 7 to 8×
- C. About 27×
- D. 2×

**Answer:** B. About 7 to 8× — FLOPs scale by $(\alpha\beta^2\gamma^2)^\phi=1.92^3\approx7.1$, close to the design target $2^3=8$. Width and resolution count twice because FLOPs are quadratic in both. Rounding of layer counts makes real networks somewhat costlier, as the figure shows.

**Question:** Why does EfficientNet scale width and resolution with depth instead of scaling just one of them?

- A. Because depth cannot be scaled
- B. Because a single dimension saturates (near 80% top-1 in the paper); larger images need more layers for receptive field and more channels for finer patterns
- C. Because it keeps the parameter count constant
- D. Because depthwise convolutions require it

**Answer:** B. Because a single dimension saturates (near 80% top-1 in the paper); larger images need more layers for receptive field and more channels for finer patterns — The paper found that scaling any single dimension improves accuracy and then plateaus, and that balancing the three gives better accuracy at the same FLOPs. Resolution-only scaling, for example, leaves the receptive field and channel capacity unchanged while multiplying the cost.

**Key points**

- Compound scaling: $d=\alpha^\phi$, $w=\beta^\phi$, $r=\gamma^\phi$ with $\alpha\beta^2\gamma^2\approx2$, so FLOPs $\approx2^\phi$.
- Found constants: $\alpha=1.2$, $\beta=1.1$, $\gamma=1.15$ by a grid search at $\phi=1$ on B0.
- B0: NAS-found MBConv network with SE and SiLU, 5.3M params, 0.39 GFLOPs, 77.1% top-1 (paper).
- B1–B7 scale B0 up to 66M params and 84.3% top-1; evaluate each at its own resolution.
- Efficient in FLOPs, not necessarily in GPU time; EfficientNetV2 adds Fused-MBConv and progressive learning.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
