EfficientNet

Scale depth, width and input resolution together, by α^φ, β^φ, γ^φ with α·β²·γ² ≈ 2, so each step of φ doubles FLOPs. Applied to a small NAS-found MBConv network (B0: 5.3M params, 0.39 GFLOPs), it gives B1–B7, up to 84.3% ImageNet top-1.

14 min read1130 wordscore0 viewsupdated 2026-10-11

Definition

EfficientNet is a family of convolutional networks, B0 to B7, obtained by compound scaling of one baseline. The baseline, EfficientNet-B0, was found by neural architecture search and is built from mobile inverted bottleneck (MBConv) blocks with squeeze-and-excitation and the Swish (SiLU) activation. Larger models multiply its depth by αϕ\alpha^\phi, its width by βϕ\beta^\phi and its input resolution by γϕ\gamma^\phi for a single user-chosen coefficient ϕ\phi.

Mingxing Tan and Quoc V. Le introduced EfficientNet in "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" (ICML 2019). Before it, scaling a ConvNet up usually meant turning one knob: more layers (ResNet-18 to ResNet-200), more channels (Wide ResNet), or larger input images (GPipe used 480×480). The paper measured each knob on its own and found that accuracy improves and then quickly saturates near 80% top-1 for all three. The dimensions are not independent. A larger image has larger objects, so it needs more layers to grow the receptive field and more channels to capture finer patterns. Scaling all three in a fixed ratio worked better than any single one.

At a glance

Paper
Tan & Le, ICML 2019 (Google Brain)
Scaling rule
depth αϕ\alpha^\phi, width βϕ\beta^\phi, resolution γϕ\gamma^\phi; α=1.2, β=1.1, γ=1.15\alpha=1.2,\ \beta=1.1,\ \gamma=1.15
Constraint
αβ2γ2≈2\alpha\beta^2\gamma^2\approx2, so FLOPs grow by about 2ϕ2^\phi
Baseline B0
NAS (MnasNet search space) with a 400M FLOPs target; MBConv + SE + SiLU; 5.3M params, 0.39G FLOPs
Top model
B7: 84.3% top-1, 66M params, 37B FLOPs (paper); 8.4× smaller than GPipe at the same accuracy
Successor
EfficientNetV2 (Tan & Le, 2021): Fused-MBConv, training-aware NAS, progressive learning
In PyTorch
torchvision.models.efficientnet_b0 … efficientnet_b7, efficientnet_v2_s/m/l; timm

Compound scaling

The FLOPs of a convolution are proportional to depth (number of layers), to the square of width (input times output channels) and to the square of resolution (height times width of the map). Doubling depth doubles the cost; doubling width or resolution quadruples it. So if depth, width and resolution are multiplied by dd, ww, rr, total FLOPs scale by roughly d w2r2d\,w^2r^2. EfficientNet ties all three to one coefficient:

d=αϕ,w=βϕ,r=γϕ,s.t.  α⋅β2⋅γ2≈2,  α,β,γ≥1⇒FLOPs∝(αβ2γ2)ϕ≈2ϕd=\alpha^{\phi},\quad w=\beta^{\phi},\quad r=\gamma^{\phi},\qquad \text{s.t.}\ \ \alpha\cdot\beta^{2}\cdot\gamma^{2}\approx2,\ \ \alpha,\beta,\gamma\ge1\qquad\Rightarrow\qquad \text{FLOPs}\propto(\alpha\beta^2\gamma^2)^{\phi}\approx2^{\phi}
ϕ\phi
compound coefficient: how much more compute is available (each +1 roughly doubles FLOPs)
d, w, rd,\ w,\ r
multipliers for the number of layers, the channels and the input side length
α,β,γ\alpha,\beta,\gamma
per-dimension growth constants, found once by a small grid search at ϕ=1\phi=1
β2, γ2\beta^2,\ \gamma^2
width and resolution enter squared because FLOPs are quadratic in both

The procedure has two steps. Step 1: fix ϕ=1\phi=1, assume twice the resources, and grid-search α,β,γ\alpha,\beta,\gamma under the constraint on B0; the result is α=1.2\alpha=1.2, β=1.1\beta=1.1, γ=1.15\gamma=1.15. Step 2: fix those constants and scale B0 with increasing ϕ\phi to obtain B1 to B7. Searching the constants directly on a big model might do better, but it would cost far more; the paper does the search once, on the small model. With these values αβ2γ2=1.2⋅1.21⋅1.3225=1.920\alpha\beta^2\gamma^2=1.2\cdot1.21\cdot1.3225=1.920, so the predicted growth is 1.92ϕ1.92^\phi.

Taking logarithms shows what the rule really does: log⁡d:log⁡w:log⁡r=log⁡1.2:log⁡1.1:log⁡1.15≈1.91:1:1.47\log d:\log w:\log r=\log1.2:\log1.1:\log1.15\approx1.91:1:1.47. In log terms, each step of ϕ\phi takes 28% of its extra FLOPs from depth (log⁡1.2/log⁡1.92=0.28\log1.2/\log1.92=0.28), 29% from width (2log⁡1.1/log⁡1.92=0.292\log1.1/\log1.92=0.29) and 43% from resolution (2log⁡1.15/log⁡1.92=0.432\log1.15/\log1.92=0.43).

φ = 1.0: d = 1.2^φ, w = 1.1^φ, r = 1.15^φ258²input216129²32465²34833²48817²412017²52089²23529²stages 1–7: slabs = MBConv layers, width ∝ √channelsstem 3×3/2 → 32 ch · head 1×1 → 1408 ch · pool · FC 1000multipliers vs B0depth d1.20×width w1.10×res r1.15×FLOPs ∝ d·w²·r²parameters9,109,994multiply-adds0.993GFLOPs vs B0 (actual)2.57×target 2^φ2.00×(αβ²γ²)^φ = 1.92^φ1.92×MBConv blocks23stage 5: k5 MBConv64 × 120 @17² params / MACs0.86M / 201M
click a stage
fig.Drag φ to scale EfficientNet-B0 and compare strategies that all aim at about 2^φ FLOPs. The square on the left is the input, the coloured slabs are the MBConv layers of stages 1–7 (width ∝ √channels), and the panel recounts parameters and multiply-adds from the real torchvision rules (channels rounded to multiples of 8, layer counts rounded up). The B0–B7 buttons load the official multipliers, and every one reproduces torchvision's parameter count: 5,288,548 for B0 at 0.386G multiply-adds, up to 66,347,960 and 37.75G for B7. Rounding matters. At φ = 1 compound scaling gives 23 blocks instead of 16 and 2.57× the FLOPs of B0, against the 1.92× the formula predicts. At φ = 3, depth-only scaling reaches 8.19× FLOPs with 39.8M parameters, while resolution-only reaches 8.10× with B0's 5.29M.

The baseline: EfficientNet-B0

Compound scaling only changes the size of the layers, not their type, so the baseline matters. B0 comes from a multi-objective architecture search in the MnasNet search space that maximizes ACC(m)⋅[FLOPS(m)/T]w\mathrm{ACC}(m)\cdot[\mathrm{FLOPS}(m)/T]^{w} with target T=400T=400M FLOPs and w=−0.07w=-0.07. Unlike MnasNet, it optimizes FLOPs rather than measured latency on a specific phone, so it is not tied to one device.

StageOperatorInput res.ChannelsLayersParamsMACs
1Conv 3×3, stride 2224²32192810.8M
2MBConv1, k3×3112²1611,44810.0M
3MBConv6, k3×3112²24216,71454.9M
4MBConv6, k5×556²40246,64037.9M
5MBConv6, k3×328²803242,93043.6M
6MBConv6, k5×514²1123543,14886.1M
7MBConv6, k5×514²19242,026,34891.5M
8MBConv6, k3×37²3201717,23229.5M
9Conv 1×1, pool, FC7²128011,693,16021.4M
EfficientNet-B0 (Tan & Le 2019, Table 1). MBConv6 = expansion ratio 6. Params and multiply-adds per stage counted by the figure's code (torchvision layout, 224×224); the total is 5,288,548 parameters and 0.386G multiply-adds.

Each MBConv block (from MobileNetV2, with SE as in MnasNet) expands the input with a 1×11\times1 conv by factor 6, applies a depthwise k×kk\times k conv (stride 2 in the first block of a downsampling stage), recalibrates channels with squeeze-and-excitation (squeeze to a quarter of the block's input channels), and projects back with a linear 1×11\times1 conv. A residual connection, with Stochastic depth during training, is used when input and output shapes match:

y=x+P(SE(SiLU(DWk(SiLU(E x))))),SiLU(z)=z σ(z),SiLU′(z)=σ(z)(1+z (1−σ(z)))y=x+P\Big(\mathrm{SE}\big(\mathrm{SiLU}(\mathrm{DW}_k(\mathrm{SiLU}(E\,x)))\big)\Big),\qquad \mathrm{SiLU}(z)=z\,\sigma(z),\qquad \mathrm{SiLU}'(z)=\sigma(z)\big(1+z\,(1-\sigma(z))\big)
EE
1×11\times1 expansion conv c→6cc\to6c + BN (absent in MBConv1)
DWk\mathrm{DW}_k
depthwise k×kk\times k conv + BN: one filter per channel, k2⋅6ck^2\cdot6c weights
SE\mathrm{SE}
squeeze-and-excitation gate, 6c→⌊c/4⌋→6c6c\to\lfloor c/4\rfloor\to6c, sigmoid output
PP
1×11\times1 projection 6c→c′6c\to c' + BN, no activation (linear bottleneck)
SiLU′\mathrm{SiLU}'
derivative used in backprop: 0.5 at z=0z=0, tends to 1 for large zz, slightly negative for z<−1.28z<-1.28

Almost all of B0's compute sits in 1×11\times1 convs; the depthwise convs are cheap in FLOPs but, as the RegNet paper later stressed, produce large activations and run with low arithmetic intensity on GPUs. 32% of B0's parameters (1.69M) are in the final 1×11\times1 conv to 1280 channels and the classifier.

Worked example: φ = 1

Compound-scaling B0 by one step (as in the figure)

  1. 1Multipliers: d=1.2d=1.2, w=1.1w=1.1, r=1.15r=1.15, so resolution 224⋅1.15=257.6→258224\cdot1.15=257.6\to258.
  2. 2Width: channels are c⋅1.1c\cdot1.1 rounded to a multiple of 8. 32→35.2→3232\to35.2\to32, 40→44→4840\to44\to48, 112→123.2→120112\to123.2\to120, 192→211.2→208192\to211.2\to208, 320→352320\to352, head 4⋅352=14084\cdot352=1408.
  3. 3Depth: layers are ⌈n⋅1.2⌉\lceil n\cdot1.2\rceil: (1,2,2,3,3,4,1)→(2,3,3,4,4,5,2)(1,2,2,3,3,4,1)\to(2,3,3,4,4,5,2), 23 blocks instead of 16. The ceiling turns every 1-layer stage into 2 layers.
  4. 4Result: 9,109,994 parameters and 0.993G multiply-adds, 2.57× B0's FLOPs. The formula predicted 1.92×1.92\times; the gap is almost all rounding of layer counts. This (w,d)=(1.1,1.2)(w,d)=(1.1,1.2) is exactly B2's width and depth; the official models were hand-adjusted (B1 is (1.0,1.1)(1.0,1.1) at 240 px, B2 is (1.1,1.2)(1.1,1.2) at 260 px in the paper).
  5. 5Same target, one knob only, at ϕ=3\phi=3 (8× FLOPs): depth ×8\times8 gives 8.19× FLOPs and 39.8M parameters, width ×2.83\times2.83 gives 7.35× and 34.6M, resolution ×2.83\times2.83 (634 px) gives 8.10× with B0's 5.29M parameters, and compound gives 8.43× with 16.3M.

The figure shows the cost side. The accuracy side is the paper's result: at equal FLOPs, compound scaling beat single-dimension scaling. For MobileNetV2 (72.0% baseline at 0.3B FLOPs) the paper reports 76.8% for depth ×4, 76.4% for width ×2, 74.8% for resolution ×2 and 77.4% for compound scaling, all at 1.1 to 1.3B FLOPs.

python
1import math, torch
2from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights, efficientnet_v2_s
3
4alpha, beta, gamma = 1.2, 1.1, 1.15
5print(alpha * beta**2 * gamma**2) # 1.920...: FLOPs grow ~1.92^phi
6for phi in range(8):
7 d, w, r = alpha**phi, beta**phi, gamma**phi
8 print(phi, f"depth x{d:.2f} width x{w:.2f} res {round(224 * r)}")
9
10weights = EfficientNet_B0_Weights.IMAGENET1K_V1
11model = efficientnet_b0(weights=weights).eval()
12print(sum(p.numel() for p in model.parameters())) # 5288548
13print(model.features[2][0]) # MBConv: expand 16->96, dw 3x3 s2, SE 96->4->96, project ->24
14preprocess = weights.transforms() # resize 256, center-crop 224
15x = preprocess(torch.rand(3, 300, 300)).unsqueeze(0)
16print(model(x).softmax(-1).shape) # torch.Size([1, 1000])
17
18v2 = efficientnet_v2_s() # Fused-MBConv early stages, 21.5M params
Compound-scaling arithmetic and torchvision's EfficientNets
Modelres.paper top-1paper params / FLOPstorchvision acc@1tv params / GFLOPs
B022477.1%5.3M / 0.39B77.69%5.3M / 0.39
B124079.1%7.8M / 0.70B78.64%7.8M / 0.69
B2260 (tv 288)80.1%9.2M / 1.0B80.61%9.1M / 1.09
B330081.6%12M / 1.8B82.01%12.2M / 1.83
B438082.9%19M / 4.2B83.38%19.3M / 4.39
B545683.6%30M / 9.9B83.44%30.4M / 10.27
B652884.0%43M / 19B84.01%43.0M / 19.07
B760084.3%66M / 37B84.12%66.3M / 37.75
ResNet-50 (paper)22476.0%26M / 4.1B76.13%25.6M / 4.09
ImageNet. Paper: Tan & Le 2019, Table 2 (top-1, params, FLOPs at each model's own resolution). torchvision: documented acc@1 of the IMAGENET1K_V1 weights (B1 also has V2 weights at 79.84%), params and GFLOPs, which the figure's code reproduces.

B0 matches ResNet-50's accuracy with about 5× fewer parameters and 11× fewer FLOPs, and B7 matches GPipe's 84.3% with 8.4× fewer parameters and 6.1× faster inference (paper). Training used RMSProp (decay 0.9, momentum 0.9), weight decay 10−510^{-5}, learning rate 0.256 decayed by 0.97 every 2.4 epochs, SiLU, AutoAugment, stochastic depth with survival probability 0.8, and dropout before the classifier rising from 0.2 (B0) to 0.5 (B7). See Swish, Dropout and Stochastic depth.

EfficientNetV2

Tan and Le's "EfficientNetV2: Smaller Models and Faster Training" (ICML 2021) targets the weaknesses of V1: very large training images are slow and memory-hungry, and depthwise convs in early layers are slow on accelerators. V2 replaces early MBConv blocks with Fused-MBConv, where a regular 3×33\times3 conv does the work of the expansion 1×11\times1 plus the depthwise conv, searches with training speed in the objective, scales less aggressively in resolution, and uses progressive learning: image size grows during training, with regularization (dropout, RandAugment, Mixup) increased along with it. In torchvision, efficientnet_v2_s has 21.5M parameters, 8.37 GFLOPs and 84.23% acc@1.

Use when

  • +One coefficient ϕ\phi scales a whole family; the constants are searched once on a small model
  • +Excellent accuracy per parameter and per FLOP (B0: 77.1% with 5.3M parameters and 0.39B FLOPs)
  • +Strong transfer: the paper reports state-of-the-art accuracy on 5 of 8 transfer datasets
  • +Widely available pretrained weights in torchvision and timm

Avoid when

  • −Depthwise convs and large inputs make it slower on GPUs than its FLOPs suggest
  • −High resolution (up to 600 px) makes training memory-hungry
  • −Training recipe is heavy (RMSProp, AutoAugment, stochastic depth, long schedules) and sensitive
  • −Rounding of layers and channels makes the realised FLOPs drift from 2^φ

Check yourself

With α = 1.2, β = 1.1, γ = 1.15, by roughly what factor do FLOPs grow when φ goes from 0 to 3?

Check yourself

Why does EfficientNet scale width and resolution with depth instead of scaling just one of them?

Remember

  • ✓Compound scaling: d=αϕd=\alpha^\phi, w=βϕw=\beta^\phi, r=γϕr=\gamma^\phi with αβ2γ2≈2\alpha\beta^2\gamma^2\approx2, so FLOPs ≈2ϕ\approx2^\phi.
  • ✓Found constants: α=1.2\alpha=1.2, β=1.1\beta=1.1, γ=1.15\gamma=1.15 by a grid search at ϕ=1\phi=1 on B0.
  • ✓B0: NAS-found MBConv network with SE and SiLU, 5.3M params, 0.39 GFLOPs, 77.1% top-1 (paper).
  • ✓B1–B7 scale B0 up to 66M params and 84.3% top-1; evaluate each at its own resolution.
  • ✓Efficient in FLOPs, not necessarily in GPU time; EfficientNetV2 adds Fused-MBConv and progressive learning.
← back to Deep Learning roadmap