Definition
EfficientNet is a family of convolutional networks, B0 to B7, obtained by compound scaling of one baseline. The baseline, EfficientNet-B0, was found by neural architecture search and is built from mobile inverted bottleneck (MBConv) blocks with squeeze-and-excitation and the Swish (SiLU) activation. Larger models multiply its depth by , its width by and its input resolution by for a single user-chosen coefficient .
Mingxing Tan and Quoc V. Le introduced EfficientNet in "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" (ICML 2019). Before it, scaling a ConvNet up usually meant turning one knob: more layers (ResNet-18 to ResNet-200), more channels (Wide ResNet), or larger input images (GPipe used 480×480). The paper measured each knob on its own and found that accuracy improves and then quickly saturates near 80% top-1 for all three. The dimensions are not independent. A larger image has larger objects, so it needs more layers to grow the receptive field and more channels to capture finer patterns. Scaling all three in a fixed ratio worked better than any single one.
At a glance
- Paper
- Tan & Le, ICML 2019 (Google Brain)
- Scaling rule
- depth , width , resolution ;
- Constraint
- , so FLOPs grow by about
- Baseline B0
- NAS (MnasNet search space) with a 400M FLOPs target; MBConv + SE + SiLU; 5.3M params, 0.39G FLOPs
- Top model
- B7: 84.3% top-1, 66M params, 37B FLOPs (paper); 8.4× smaller than GPipe at the same accuracy
- Successor
- EfficientNetV2 (Tan & Le, 2021): Fused-MBConv, training-aware NAS, progressive learning
- In PyTorch
torchvision.models.efficientnet_b0…efficientnet_b7,efficientnet_v2_s/m/l; timm
Compound scaling
The FLOPs of a convolution are proportional to depth (number of layers), to the square of width (input times output channels) and to the square of resolution (height times width of the map). Doubling depth doubles the cost; doubling width or resolution quadruples it. So if depth, width and resolution are multiplied by , , , total FLOPs scale by roughly . EfficientNet ties all three to one coefficient:
- compound coefficient: how much more compute is available (each +1 roughly doubles FLOPs)
- multipliers for the number of layers, the channels and the input side length
- per-dimension growth constants, found once by a small grid search at
- width and resolution enter squared because FLOPs are quadratic in both
The procedure has two steps. Step 1: fix , assume twice the resources, and grid-search under the constraint on B0; the result is , , . Step 2: fix those constants and scale B0 with increasing to obtain B1 to B7. Searching the constants directly on a big model might do better, but it would cost far more; the paper does the search once, on the small model. With these values , so the predicted growth is .
Taking logarithms shows what the rule really does: . In log terms, each step of takes 28% of its extra FLOPs from depth (), 29% from width () and 43% from resolution ().
The baseline: EfficientNet-B0
Compound scaling only changes the size of the layers, not their type, so the baseline matters. B0 comes from a multi-objective architecture search in the MnasNet search space that maximizes with target M FLOPs and . Unlike MnasNet, it optimizes FLOPs rather than measured latency on a specific phone, so it is not tied to one device.
| Stage | Operator | Input res. | Channels | Layers | Params | MACs |
|---|---|---|---|---|---|---|
| 1 | Conv 3×3, stride 2 | 224² | 32 | 1 | 928 | 10.8M |
| 2 | MBConv1, k3×3 | 112² | 16 | 1 | 1,448 | 10.0M |
| 3 | MBConv6, k3×3 | 112² | 24 | 2 | 16,714 | 54.9M |
| 4 | MBConv6, k5×5 | 56² | 40 | 2 | 46,640 | 37.9M |
| 5 | MBConv6, k3×3 | 28² | 80 | 3 | 242,930 | 43.6M |
| 6 | MBConv6, k5×5 | 14² | 112 | 3 | 543,148 | 86.1M |
| 7 | MBConv6, k5×5 | 14² | 192 | 4 | 2,026,348 | 91.5M |
| 8 | MBConv6, k3×3 | 7² | 320 | 1 | 717,232 | 29.5M |
| 9 | Conv 1×1, pool, FC | 7² | 1280 | 1 | 1,693,160 | 21.4M |
Each MBConv block (from MobileNetV2, with SE as in MnasNet) expands the input with a conv by factor 6, applies a depthwise conv (stride 2 in the first block of a downsampling stage), recalibrates channels with squeeze-and-excitation (squeeze to a quarter of the block's input channels), and projects back with a linear conv. A residual connection, with Stochastic depth during training, is used when input and output shapes match:
- expansion conv + BN (absent in MBConv1)
- depthwise conv + BN: one filter per channel, weights
- squeeze-and-excitation gate, , sigmoid output
- projection + BN, no activation (linear bottleneck)
- derivative used in backprop: 0.5 at , tends to 1 for large , slightly negative for
Almost all of B0's compute sits in convs; the depthwise convs are cheap in FLOPs but, as the RegNet paper later stressed, produce large activations and run with low arithmetic intensity on GPUs. 32% of B0's parameters (1.69M) are in the final conv to 1280 channels and the classifier.
Worked example: φ = 1
Compound-scaling B0 by one step (as in the figure)
- 1Multipliers: , , , so resolution .
- 2Width: channels are rounded to a multiple of 8. , , , , , head .
- 3Depth: layers are : , 23 blocks instead of 16. The ceiling turns every 1-layer stage into 2 layers.
- 4Result: 9,109,994 parameters and 0.993G multiply-adds, 2.57× B0's FLOPs. The formula predicted ; the gap is almost all rounding of layer counts. This is exactly B2's width and depth; the official models were hand-adjusted (B1 is at 240 px, B2 is at 260 px in the paper).
- 5Same target, one knob only, at (8× FLOPs): depth gives 8.19× FLOPs and 39.8M parameters, width gives 7.35× and 34.6M, resolution (634 px) gives 8.10× with B0's 5.29M parameters, and compound gives 8.43× with 16.3M.
The figure shows the cost side. The accuracy side is the paper's result: at equal FLOPs, compound scaling beat single-dimension scaling. For MobileNetV2 (72.0% baseline at 0.3B FLOPs) the paper reports 76.8% for depth ×4, 76.4% for width ×2, 74.8% for resolution ×2 and 77.4% for compound scaling, all at 1.1 to 1.3B FLOPs.
1import math, torch2from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights, efficientnet_v2_s34alpha, beta, gamma = 1.2, 1.1, 1.155print(alpha * beta**2 * gamma**2) # 1.920...: FLOPs grow ~1.92^phi6for phi in range(8):7 d, w, r = alpha**phi, beta**phi, gamma**phi8 print(phi, f"depth x{d:.2f} width x{w:.2f} res {round(224 * r)}")910weights = EfficientNet_B0_Weights.IMAGENET1K_V111model = efficientnet_b0(weights=weights).eval()12print(sum(p.numel() for p in model.parameters())) # 528854813print(model.features[2][0]) # MBConv: expand 16->96, dw 3x3 s2, SE 96->4->96, project ->2414preprocess = weights.transforms() # resize 256, center-crop 22415x = preprocess(torch.rand(3, 300, 300)).unsqueeze(0)16print(model(x).softmax(-1).shape) # torch.Size([1, 1000])1718v2 = efficientnet_v2_s() # Fused-MBConv early stages, 21.5M params| Model | res. | paper top-1 | paper params / FLOPs | torchvision acc@1 | tv params / GFLOPs |
|---|---|---|---|---|---|
| B0 | 224 | 77.1% | 5.3M / 0.39B | 77.69% | 5.3M / 0.39 |
| B1 | 240 | 79.1% | 7.8M / 0.70B | 78.64% | 7.8M / 0.69 |
| B2 | 260 (tv 288) | 80.1% | 9.2M / 1.0B | 80.61% | 9.1M / 1.09 |
| B3 | 300 | 81.6% | 12M / 1.8B | 82.01% | 12.2M / 1.83 |
| B4 | 380 | 82.9% | 19M / 4.2B | 83.38% | 19.3M / 4.39 |
| B5 | 456 | 83.6% | 30M / 9.9B | 83.44% | 30.4M / 10.27 |
| B6 | 528 | 84.0% | 43M / 19B | 84.01% | 43.0M / 19.07 |
| B7 | 600 | 84.3% | 66M / 37B | 84.12% | 66.3M / 37.75 |
| ResNet-50 (paper) | 224 | 76.0% | 26M / 4.1B | 76.13% | 25.6M / 4.09 |
B0 matches ResNet-50's accuracy with about 5× fewer parameters and 11× fewer FLOPs, and B7 matches GPipe's 84.3% with 8.4× fewer parameters and 6.1× faster inference (paper). Training used RMSProp (decay 0.9, momentum 0.9), weight decay , learning rate 0.256 decayed by 0.97 every 2.4 epochs, SiLU, AutoAugment, stochastic depth with survival probability 0.8, and dropout before the classifier rising from 0.2 (B0) to 0.5 (B7). See Swish, Dropout and Stochastic depth.
EfficientNetV2
Tan and Le's "EfficientNetV2: Smaller Models and Faster Training" (ICML 2021) targets the weaknesses of V1: very large training images are slow and memory-hungry, and depthwise convs in early layers are slow on accelerators. V2 replaces early MBConv blocks with Fused-MBConv, where a regular conv does the work of the expansion plus the depthwise conv, searches with training speed in the objective, scales less aggressively in resolution, and uses progressive learning: image size grows during training, with regularization (dropout, RandAugment, Mixup) increased along with it. In torchvision, efficientnet_v2_s has 21.5M parameters, 8.37 GFLOPs and 84.23% acc@1.
Use when
- +One coefficient scales a whole family; the constants are searched once on a small model
- +Excellent accuracy per parameter and per FLOP (B0: 77.1% with 5.3M parameters and 0.39B FLOPs)
- +Strong transfer: the paper reports state-of-the-art accuracy on 5 of 8 transfer datasets
- +Widely available pretrained weights in torchvision and timm
Avoid when
- −Depthwise convs and large inputs make it slower on GPUs than its FLOPs suggest
- −High resolution (up to 600 px) makes training memory-hungry
- −Training recipe is heavy (RMSProp, AutoAugment, stochastic depth, long schedules) and sensitive
- −Rounding of layers and channels makes the realised FLOPs drift from 2^φ
Check yourself
With α = 1.2, β = 1.1, γ = 1.15, by roughly what factor do FLOPs grow when φ goes from 0 to 3?
Check yourself
Why does EfficientNet scale width and resolution with depth instead of scaling just one of them?
Remember
- ✓Compound scaling: , , with , so FLOPs .
- ✓Found constants: , , by a grid search at on B0.
- ✓B0: NAS-found MBConv network with SE and SiLU, 5.3M params, 0.39 GFLOPs, 77.1% top-1 (paper).
- ✓B1–B7 scale B0 up to 66M params and 84.3% top-1; evaluate each at its own resolution.
- ✓Efficient in FLOPs, not necessarily in GPU time; EfficientNetV2 adds Fused-MBConv and progressive learning.