RegNet

Instead of searching for one best network, RegNet narrows a whole design space of ResNeXt-style models until it is described by a quantized linear rule for widths, w_j = w₀ + w_a·j, plus depth and group width. Simple, regular nets fall out that rival EfficientNet and run up to 5× faster on GPUs.

14 min read1238 wordsadvanced0 viewsupdated 2026-10-11

Definition

RegNet is a family of convolutional networks defined by six numbers: depth dd, initial width w0w_0, width slope waw_a, width multiplier wmw_m, bottleneck ratio bb and group width gg. The block widths follow a quantized linear function of the block index, which splits the network into stages of residual blocks with grouped 3×33\times3 convolutions. RegNetX uses plain ResNeXt-style blocks; RegNetY adds squeeze-and-excitation.

Radosavovic, Kosaraju, Girshick, He and Dollár introduced RegNet in "Designing Network Design Spaces" (CVPR 2020). Its main contribution is a method rather than a model. Neural architecture search looks for a single best network in a fixed space. Radosavovic et al. instead treat the design space, a population of networks, as the object being designed. They sample models from it, measure the distribution of their errors, and keep changes that make the whole population better. The rules that survive are meant to be general principles, and the RegNet models are what those rules produce at each compute budget.

The tool for comparing design spaces comes from the same group's earlier "On Network Design Spaces for Visual Recognition" (Radosavovic et al., ICCV 2019): the error empirical distribution function (EDF). Sample nn models, train each one, and plot the fraction whose error is below ee:

F(e)=1n∑i=1n1[ei<e]F(e)=\frac1n\sum_{i=1}^{n}\mathbf 1\left[e_i<e\right]
nn
number of models sampled from the design space (500 in the paper)
eie_i
ImageNet top-1 error of model i after a short schedule (10 epochs, 360MF to 400MF models)
F(e)F(e)
fraction of the design space below error e; a curve further left is a better space

According to the paper, training 100 models at 400MF for 10 epochs costs about as many FLOPs as training one ResNet-50 for 100 epochs, so a 500-model comparison is affordable.

At a glance

Paper
Radosavovic, Kosaraju, Girshick, He, Dollár, CVPR 2020
Block
ResNeXt-style residual bottleneck: 1×11\times1 → grouped 3×33\times3 (group width gg) → 1×11\times1; RegNetY adds SE
Width rule
uj=w0+waju_j=w_0+w_a j, quantized to wj=w0wm⌊sj⌉w_j=w_0w_m^{\lfloor s_j\rceil}, a multiple of 8
Parameters
d,w0,wa,wm,b,gd,w_0,w_a,w_m,b,g: 6 numbers, against 16 for AnyNetX
Findings
b=1b=1 (no bottleneck), about 20 blocks at every budget, wm≈2.5w_m\approx2.5, widths grow with depth
RegNetY-400MF
4.34M params, 0.40G multiply-adds, 74.05% top-1 (torchvision V1 weights)
In PyTorch
torchvision.models.regnet_y_400mf … regnet_y_128gf, regnet_x_*

From AnyNet to RegNet

The starting point, AnyNetX, fixes the overall shape: a stem (a stride-2 3×33\times3 conv to 32 channels), a body of 4 stages at decreasing resolution, and a head (pooling and a linear classifier). Each stage ii is a stack of did_i identical "X blocks" (1×1, grouped 3×3, 1×1 with a residual connection, the first block of a stage having stride 2) with width wiw_i, bottleneck ratio bib_i and group width gig_i. That is 16 free parameters. With di≤16d_i\le16, wi≤1024w_i\le1024 in multiples of 8, bi∈{1,2,4}b_i\in\{1,2,4\} and gi∈{1,2,…,32}g_i\in\{1,2,\dots,32\} there are (16⋅128⋅3⋅6)4≈1.8⋅1018(16\cdot128\cdot3\cdot6)^4\approx1.8\cdot10^{18} networks.

Refining the design space (each step keeps or improves the error EDF)

  1. 1AnyNetXA_A → B_B: a single bottleneck ratio bb for all stages. The EDF is unchanged. Size ≈6.8⋅1016\approx6.8\cdot10^{16}.
  2. 2B_B → C_C: a single group width gg for all stages. Again no loss; 6 fewer degrees of freedom than AA, size ≈3.2⋅1014\approx3.2\cdot10^{14}.
  3. 3C_C → D_D: widths must not decrease, wi+1≥wiw_{i+1}\ge w_i. The EDF improves clearly. Size ≈1.3⋅1013\approx1.3\cdot10^{13}.
  4. 4D_D → E_E: depths must not decrease either, di+1≥did_{i+1}\ge d_i. Improves again; size ≈5.5⋅1011\approx5.5\cdot10^{11}.
  5. 5E_E → RegNet: the per-block widths of the best E_E models lie close to a line, wj≈48⋅(j+1)w_j\approx48\cdot(j+1). Replacing the free widths and depths by a quantized linear rule gives a space of about 3.0⋅1083.0\cdot10^{8} networks, nearly 10 orders of magnitude smaller than AnyNetX, and with a better EDF.

The quantized linear width rule

uj=w0+wa j    (0≤j<d),uj=w0 wm sj,wj=w0 wm⌊sj⌉u_j=w_0+w_a\,j\;\;(0\le j<d),\qquad u_j=w_0\,w_m^{\,s_j},\qquad w_j=w_0\,w_m^{\lfloor s_j\rceil}
jj
block index over the whole network, 00 to d−1d-1
uju_j
continuous target width of block jj, linear in jj
w0,  waw_0,\;w_a
initial width and slope (both >0>0)
sj=log⁡(uj/w0)/log⁡wms_j=\log(u_j/w_0)/\log w_m
how many factors of wmw_m block jj is above w0w_0
⌊sj⌉\lfloor s_j\rceil
sjs_j rounded to an integer; blocks with the same value form a stage
wjw_j
quantized width, finally rounded to a multiple of 8

Rounding sjs_j turns the line into a staircase: stage ii has width w0wmiw_0w_m^{i} and depth di=∑j1[⌊sj⌉=i]d_i=\sum_j\mathbf 1[\lfloor s_j\rceil=i]. Consecutive stage widths therefore differ by the factor wmw_m, before the rounding to multiples of 8. Two last adjustments in torchvision's implementation: the stage width is rounded to a multiple of the group width (and gg is capped at the stage width), and each stage starts with a stride-2 block. The figure implements exactly this procedure.

uⱼ = 48 + 27.89·j → wⱼ = 48·2.09^⌊sⱼ⌉, multiple of 80120248368496s1s2s3s4block j (0 … 15)dashed: linear uⱼ · dots: quantized wⱼstage 3: 6 blocks, width 208, 26 groups of 8 ch, 14×14 + SEstage depth × width groupsparams1 · 1 × 48 @56² 6×80.01M2 · 3 × 104 @28² 13×80.10M3 · 6 × 208 @14² 26×80.74M4 · 6 × 440 @7² 55×83.05Mstage 3 multiply-adds132.2Mstage 3 activations0.90MRegNetY totalparameters4,344,144multiply-adds0.402Gactivations3.89Mblocks d16torchvision regnet_y_400mfmatch ✓
click a stage
fig.Dashed line: the linear widths u_j = w₀ + w_a·j. Dots: the quantized widths w_j, coloured by stage; each flat run is one stage. The default is RegNetY-400MF (d = 16, w₀ = 48, w_a = 27.89, w_m = 2.09, g = 8), which gives stages of 1, 3, 6, 6 blocks at widths 48, 104, 208, 440. The panel counts parameters, multiply-adds and activations from the generated configuration: 4,344,144 parameters, 0.402G multiply-adds and 3.89M activations, exactly torchvision's regnet_y_400mf. All six presets reproduce torchvision's parameter counts. Move a slider to see the stages regroup. Toggle X/Y to add or remove squeeze-and-excitation. 'sample a RegNet' draws a random 4-stage member of the design space.

What the population analysis found

With the RegNetX space in hand, the paper analyses which parameter values the best models share across compute budgets. Some findings contradict common practice:

Empirical findings (Radosavovic et al., 2020)

  1. 1Depth is stable. The best models have about 20 blocks (about 60 layers) at every budget. They do not get deeper as FLOPs grow, unlike the usual practice of scaling depth.
  2. 2No bottleneck. The best bottleneck ratio is b=1b=1: the middle of the block is as wide as its output. Inverted bottlenecks (b<1b<1, as in MobileNetV2) make the EDF worse, and depthwise convolution (g=1g=1) worse still.
  3. 3Width multiplier about 2.5. Good models multiply the width by wm≈2.5w_m\approx2.5 per stage, close to but not exactly the classic doubling. The group width gg, waw_a and w0w_0 all increase with the budget.
  4. 4Activations predict runtime. Defining activations as the total size of all conv output tensors, GPU inference time correlates more strongly with activations than with FLOPs. For the best models, activations grow like the square root of FLOPs and parameters grow linearly.
  5. 5Resolution 224 is best. Unlike EfficientNet's compound scaling, keeping 224×224224\times224 input was best for RegNetX even at higher FLOPs.
  6. 6SE helps. Adding squeeze-and-excitation gives RegNetY, with clear gains over RegNetX.

Worked example: building RegNetY-400MF

d = 16, w₀ = 48, w_a = 27.89, w_m = 2.09, g = 8, b = 1, SE ratio 0.25

  1. 1Block j=1j=1: u1=48+27.89=75.89u_1=48+27.89=75.89, s1=log⁡(75.89/48)/log⁡2.09=0.62s_1=\log(75.89/48)/\log2.09=0.62, rounded to 1. Width 48⋅2.09=100.348\cdot2.09=100.3, nearest multiple of 8: 104.
  2. 2Block j=4j=4: u4=159.56u_4=159.56, s4=1.63→2s_4=1.63\to2, width 48⋅2.092=209.7→48\cdot2.09^2=209.7\to 208. Blocks 4 to 9 (sjs_j up to 2.48) all round to 2.
  3. 3Block j=10j=10: u10=326.9u_{10}=326.9, s10=2.60→3s_{10}=2.60\to3, width 48⋅2.093=438.2→48\cdot2.09^3=438.2\to 440, the last stage up to j=15j=15.
  4. 4Stages: depths (1,3,6,6)(1,3,6,6), widths (48,104,208,440)(48,104,208,440), all divisible by g=8g=8, so (6,13,26,55)(6,13,26,55) groups of 8 channels at resolutions 56,28,14,756,28,14,7.
  5. 5Parameters per stage: 10,040 + 102,296 + 737,438 + 3,052,442, plus 928 for the stem and 441,000 for the 440→1000 classifier: 4,344,144, exactly torchvision's count. Multiply-adds: 0.402G.
  6. 6The SE modules (squeeze width ⌊0.25 win⌉\lfloor0.25\,w_{\text{in}}\rceil from the block input width) account for 668,024 of those parameters, 15%, but only 0.66M multiply-adds, under 0.2% of the compute.

Squeeze-and-excitation in RegNetY

The SE operation (Hu et al., 2018) rescales each channel of the 3×33\times3 output hh by a learned gate computed from its global average:

z=1HW∑php,a=σ(W2 ReLU(W1z)),yc,p=ac hc,p,∂yc,p∂hc,p=ac+hc,p ∂ac∂hc,pz=\tfrac1{HW}\sum_{p}h_{p},\qquad a=\sigma\big(W_2\,\mathrm{ReLU}(W_1z)\big),\qquad y_{c,p}=a_c\,h_{c,p},\qquad \frac{\partial y_{c,p}}{\partial h_{c,p}}=a_c+h_{c,p}\,\frac{\partial a_c}{\partial h_{c,p}}
hh
output of the grouped 3×33\times3 (after BN, ReLU), wbw_b channels
zz
squeeze: one global-average number per channel
W1,  W2W_1,\;W_2
1×11\times1 convs (with bias) wb→⌊0.25win⌉→wbw_b\to\lfloor0.25w_{\text{in}}\rceil\to w_b
ac∈(0,1)a_c\in(0,1)
excitation: sigmoid gate for channel cc
∂ac/∂hc,p\partial a_c/\partial h_{c,p}
small: each position contributes 1/HW1/HW through the pooled zz

The gradient through a gated channel is mostly the gate itself, aca_c. SE learns to turn whole channels down for a given image, and every channel's gate depends on global context, which a 3×33\times3 conv cannot see. Its cost is in parameters, not in FLOPs, because it works on pooled vectors.

python
1import math, torch
2from torchvision.models import regnet_y_400mf, RegNet_Y_400MF_Weights
3from torchvision.models.regnet import BlockParams
4
5def regnet_widths(d, w0, wa, wm, q=8):
6 u = [w0 + wa * j for j in range(d)] # linear widths
7 s = [round(math.log(uj / w0) / math.log(wm)) for uj in u] # stage index per block
8 return [int(round(w0 * wm ** sj / q) * q) for sj in s] # quantized widths
9
10print(regnet_widths(16, 48, 27.89, 2.09))
11# [48, 104, 104, 104, 208, 208, 208, 208, 208, 208, 440, 440, 440, 440, 440, 440]
12
13bp = BlockParams.from_init_params(depth=16, w_0=48, w_a=27.89, w_m=2.09,
14 group_width=8, se_ratio=0.25)
15print(bp.depths, bp.widths) # [1, 3, 6, 6] [48, 104, 208, 440]
16
17model = regnet_y_400mf(weights=RegNet_Y_400MF_Weights.IMAGENET1K_V2).eval()
18print(sum(p.numel() for p in model.parameters())) # 4344144
19print(model.trunk_output.block2[0].f.b[0]) # Conv2d(104, 104, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=13, bias=False)
The width rule in a few lines, and torchvision's RegNets
Modelstages (depth × width)ParamsGFLOPsacc@1 V1acc@1 V2
regnet_x_400mf1×32, 2×64, 7×160, 12×4005.50M0.4172.83%74.86%
regnet_y_400mf1×48, 3×104, 6×208, 6×4404.34M0.4074.05%75.80%
regnet_y_800mf1×64, 3×144, 8×320, 2×7846.43M0.8376.42%78.83%
regnet_y_1_6gf2×48, 6×120, 17×336, 2×88811.20M1.6177.95%80.88%
regnet_x_3_2gf2×96, 6×192, 15×432, 2×100815.30M3.1878.36%81.20%
regnet_y_3_2gf2×72, 5×216, 13×576, 1×151219.44M3.1878.95%81.98%
torchvision ImageNet weights (documented). Params and GFLOPs are reproduced exactly by the figure's code. V1 = paper-style recipe, V2 = torchvision's improved 2021 recipe. Stages = blocks × width, as generated.

The stage layout of RegNetX-3.2GF, depths (2,6,15,2)(2,6,15,2) and widths (96,192,432,1008)(96,192,432,1008) with g=48g=48, is the one shown in the paper's Figure 15. In the paper's mobile-regime comparison (Table 2), RegNetX-600MF reaches 25.9% and RegNetY-600MF 24.5% top-1 error, trained with a basic 100-epoch schedule and only weight decay as regularization, against mobile networks trained with much heavier recipes.

Use when

  • +Six interpretable parameters generate a whole family (400MF to 128GF in torchvision)
  • +Fast on GPUs: no depthwise convs, standard resolution, low activation count
  • +The design-space method (EDFs over populations) transfers to other architecture questions
  • +Clean torchvision implementations with strong V2 weights; a common detection backbone

Avoid when

  • −Less FLOP-efficient than EfficientNet on mobile-style FLOP counts
  • −The findings come from short 10-epoch training runs; their transfer was checked but not guaranteed
  • −Width, depth and group width interact through the quantization; small changes can regroup stages
  • −Still a plain ConvNet design; later work (ConvNeXt) gained more from modernizing training and blocks

Check yourself

With w₀ = 48, w_a = 27.89, w_m = 2.09, block j = 4 has u₄ = 159.56. What is its quantized width?

Check yourself

Which of these did the RegNet population analysis find for the best models?

Remember

  • ✓RegNet designs the design space: sample populations, compare error EDFs, keep restrictions that improve the whole population.
  • ✓AnyNetX (16 parameters, ∼1018\sim10^{18} nets) is refined to RegNet (6 parameters: d,w0,wa,wm,b,gd,w_0,w_a,w_m,b,g).
  • ✓Widths: uj=w0+waju_j=w_0+w_aj, quantized to w0wm⌊sj⌉w_0w_m^{\lfloor s_j\rceil} and multiples of 8; equal widths form stages.
  • ✓Findings: about 20 blocks, b=1b=1, wm≈2.5w_m\approx2.5, resolution 224; activations predict GPU runtime better than FLOPs.
  • ✓RegNetY = RegNetX + squeeze-and-excitation; RegNetY-400MF has 4.34M params and 0.40 GFLOPs.
← back to Deep Learning roadmap