Definition
RegNet is a family of convolutional networks defined by six numbers: depth , initial width , width slope , width multiplier , bottleneck ratio and group width . The block widths follow a quantized linear function of the block index, which splits the network into stages of residual blocks with grouped convolutions. RegNetX uses plain ResNeXt-style blocks; RegNetY adds squeeze-and-excitation.
Radosavovic, Kosaraju, Girshick, He and Dollár introduced RegNet in "Designing Network Design Spaces" (CVPR 2020). Its main contribution is a method rather than a model. Neural architecture search looks for a single best network in a fixed space. Radosavovic et al. instead treat the design space, a population of networks, as the object being designed. They sample models from it, measure the distribution of their errors, and keep changes that make the whole population better. The rules that survive are meant to be general principles, and the RegNet models are what those rules produce at each compute budget.
The tool for comparing design spaces comes from the same group's earlier "On Network Design Spaces for Visual Recognition" (Radosavovic et al., ICCV 2019): the error empirical distribution function (EDF). Sample models, train each one, and plot the fraction whose error is below :
- number of models sampled from the design space (500 in the paper)
- ImageNet top-1 error of model i after a short schedule (10 epochs, 360MF to 400MF models)
- fraction of the design space below error e; a curve further left is a better space
According to the paper, training 100 models at 400MF for 10 epochs costs about as many FLOPs as training one ResNet-50 for 100 epochs, so a 500-model comparison is affordable.
At a glance
- Paper
- Radosavovic, Kosaraju, Girshick, He, Dollár, CVPR 2020
- Block
- ResNeXt-style residual bottleneck: → grouped (group width ) → ; RegNetY adds SE
- Width rule
- , quantized to , a multiple of 8
- Parameters
- : 6 numbers, against 16 for AnyNetX
- Findings
- (no bottleneck), about 20 blocks at every budget, , widths grow with depth
- RegNetY-400MF
- 4.34M params, 0.40G multiply-adds, 74.05% top-1 (torchvision V1 weights)
- In PyTorch
torchvision.models.regnet_y_400mf…regnet_y_128gf,regnet_x_*
From AnyNet to RegNet
The starting point, AnyNetX, fixes the overall shape: a stem (a stride-2 conv to 32 channels), a body of 4 stages at decreasing resolution, and a head (pooling and a linear classifier). Each stage is a stack of identical "X blocks" (1×1, grouped 3×3, 1×1 with a residual connection, the first block of a stage having stride 2) with width , bottleneck ratio and group width . That is 16 free parameters. With , in multiples of 8, and there are networks.
Refining the design space (each step keeps or improves the error EDF)
- 1AnyNetX → : a single bottleneck ratio for all stages. The EDF is unchanged. Size .
- 2 → : a single group width for all stages. Again no loss; 6 fewer degrees of freedom than , size .
- 3 → : widths must not decrease, . The EDF improves clearly. Size .
- 4 → : depths must not decrease either, . Improves again; size .
- 5 → RegNet: the per-block widths of the best models lie close to a line, . Replacing the free widths and depths by a quantized linear rule gives a space of about networks, nearly 10 orders of magnitude smaller than AnyNetX, and with a better EDF.
The quantized linear width rule
- block index over the whole network, to
- continuous target width of block , linear in
- initial width and slope (both )
- how many factors of block is above
- rounded to an integer; blocks with the same value form a stage
- quantized width, finally rounded to a multiple of 8
Rounding turns the line into a staircase: stage has width and depth . Consecutive stage widths therefore differ by the factor , before the rounding to multiples of 8. Two last adjustments in torchvision's implementation: the stage width is rounded to a multiple of the group width (and is capped at the stage width), and each stage starts with a stride-2 block. The figure implements exactly this procedure.
What the population analysis found
With the RegNetX space in hand, the paper analyses which parameter values the best models share across compute budgets. Some findings contradict common practice:
Empirical findings (Radosavovic et al., 2020)
- 1Depth is stable. The best models have about 20 blocks (about 60 layers) at every budget. They do not get deeper as FLOPs grow, unlike the usual practice of scaling depth.
- 2No bottleneck. The best bottleneck ratio is : the middle of the block is as wide as its output. Inverted bottlenecks (, as in MobileNetV2) make the EDF worse, and depthwise convolution () worse still.
- 3Width multiplier about 2.5. Good models multiply the width by per stage, close to but not exactly the classic doubling. The group width , and all increase with the budget.
- 4Activations predict runtime. Defining activations as the total size of all conv output tensors, GPU inference time correlates more strongly with activations than with FLOPs. For the best models, activations grow like the square root of FLOPs and parameters grow linearly.
- 5Resolution 224 is best. Unlike EfficientNet's compound scaling, keeping input was best for RegNetX even at higher FLOPs.
- 6SE helps. Adding squeeze-and-excitation gives RegNetY, with clear gains over RegNetX.
Worked example: building RegNetY-400MF
d = 16, w₀ = 48, w_a = 27.89, w_m = 2.09, g = 8, b = 1, SE ratio 0.25
- 1Block : , , rounded to 1. Width , nearest multiple of 8: 104.
- 2Block : , , width 208. Blocks 4 to 9 ( up to 2.48) all round to 2.
- 3Block : , , width 440, the last stage up to .
- 4Stages: depths , widths , all divisible by , so groups of 8 channels at resolutions .
- 5Parameters per stage: 10,040 + 102,296 + 737,438 + 3,052,442, plus 928 for the stem and 441,000 for the 440→1000 classifier: 4,344,144, exactly torchvision's count. Multiply-adds: 0.402G.
- 6The SE modules (squeeze width from the block input width) account for 668,024 of those parameters, 15%, but only 0.66M multiply-adds, under 0.2% of the compute.
Squeeze-and-excitation in RegNetY
The SE operation (Hu et al., 2018) rescales each channel of the output by a learned gate computed from its global average:
- output of the grouped (after BN, ReLU), channels
- squeeze: one global-average number per channel
- convs (with bias)
- excitation: sigmoid gate for channel
- small: each position contributes through the pooled
The gradient through a gated channel is mostly the gate itself, . SE learns to turn whole channels down for a given image, and every channel's gate depends on global context, which a conv cannot see. Its cost is in parameters, not in FLOPs, because it works on pooled vectors.
1import math, torch2from torchvision.models import regnet_y_400mf, RegNet_Y_400MF_Weights3from torchvision.models.regnet import BlockParams45def regnet_widths(d, w0, wa, wm, q=8):6 u = [w0 + wa * j for j in range(d)] # linear widths7 s = [round(math.log(uj / w0) / math.log(wm)) for uj in u] # stage index per block8 return [int(round(w0 * wm ** sj / q) * q) for sj in s] # quantized widths910print(regnet_widths(16, 48, 27.89, 2.09))11# [48, 104, 104, 104, 208, 208, 208, 208, 208, 208, 440, 440, 440, 440, 440, 440]1213bp = BlockParams.from_init_params(depth=16, w_0=48, w_a=27.89, w_m=2.09,14 group_width=8, se_ratio=0.25)15print(bp.depths, bp.widths) # [1, 3, 6, 6] [48, 104, 208, 440]1617model = regnet_y_400mf(weights=RegNet_Y_400MF_Weights.IMAGENET1K_V2).eval()18print(sum(p.numel() for p in model.parameters())) # 434414419print(model.trunk_output.block2[0].f.b[0]) # Conv2d(104, 104, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=13, bias=False)| Model | stages (depth × width) | Params | GFLOPs | acc@1 V1 | acc@1 V2 |
|---|---|---|---|---|---|
| regnet_x_400mf | 1×32, 2×64, 7×160, 12×400 | 5.50M | 0.41 | 72.83% | 74.86% |
| regnet_y_400mf | 1×48, 3×104, 6×208, 6×440 | 4.34M | 0.40 | 74.05% | 75.80% |
| regnet_y_800mf | 1×64, 3×144, 8×320, 2×784 | 6.43M | 0.83 | 76.42% | 78.83% |
| regnet_y_1_6gf | 2×48, 6×120, 17×336, 2×888 | 11.20M | 1.61 | 77.95% | 80.88% |
| regnet_x_3_2gf | 2×96, 6×192, 15×432, 2×1008 | 15.30M | 3.18 | 78.36% | 81.20% |
| regnet_y_3_2gf | 2×72, 5×216, 13×576, 1×1512 | 19.44M | 3.18 | 78.95% | 81.98% |
The stage layout of RegNetX-3.2GF, depths and widths with , is the one shown in the paper's Figure 15. In the paper's mobile-regime comparison (Table 2), RegNetX-600MF reaches 25.9% and RegNetY-600MF 24.5% top-1 error, trained with a basic 100-epoch schedule and only weight decay as regularization, against mobile networks trained with much heavier recipes.
Use when
- +Six interpretable parameters generate a whole family (400MF to 128GF in torchvision)
- +Fast on GPUs: no depthwise convs, standard resolution, low activation count
- +The design-space method (EDFs over populations) transfers to other architecture questions
- +Clean torchvision implementations with strong V2 weights; a common detection backbone
Avoid when
- −Less FLOP-efficient than EfficientNet on mobile-style FLOP counts
- −The findings come from short 10-epoch training runs; their transfer was checked but not guaranteed
- −Width, depth and group width interact through the quantization; small changes can regroup stages
- −Still a plain ConvNet design; later work (ConvNeXt) gained more from modernizing training and blocks
Check yourself
With w₀ = 48, w_a = 27.89, w_m = 2.09, block j = 4 has u₄ = 159.56. What is its quantized width?
Check yourself
Which of these did the RegNet population analysis find for the best models?
Remember
- ✓RegNet designs the design space: sample populations, compare error EDFs, keep restrictions that improve the whole population.
- ✓AnyNetX (16 parameters, nets) is refined to RegNet (6 parameters: ).
- ✓Widths: , quantized to and multiples of 8; equal widths form stages.
- ✓Findings: about 20 blocks, , , resolution 224; activations predict GPU runtime better than FLOPs.
- ✓RegNetY = RegNetX + squeeze-and-excitation; RegNetY-400MF has 4.34M params and 0.40 GFLOPs.