# RegNet

> Instead of searching for one best network, RegNet narrows a whole design space of ResNeXt-style models until it is described by a quantized linear rule for widths, w_j = w₀ + w_a·j, plus depth and group width. Simple, regular nets fall out that rival EfficientNet and run up to 5× faster on GPUs.

Source: https://www.mlfoundry.in/learn/regnet  
Section: Deep Learning › Convolutional Neural Networks  
Level: advanced · 14 min read · updated 2026-10-11

**RegNet:** is a family of convolutional networks defined by six numbers: depth $d$, initial width $w_0$, width slope $w_a$, width multiplier $w_m$, bottleneck ratio $b$ and group width $g$. The block widths follow a **quantized linear** function of the block index, which splits the network into stages of residual blocks with grouped $3\times3$ convolutions. **RegNetX** uses plain ResNeXt-style blocks; **RegNetY** adds squeeze-and-excitation.

Radosavovic, Kosaraju, Girshick, He and Dollár introduced RegNet in "Designing Network Design Spaces" (CVPR 2020). Its main contribution is a method rather than a model. Neural architecture search looks for a single best network in a fixed space. Radosavovic et al. instead treat the **design space**, a population of networks, as the object being designed. They sample models from it, measure the distribution of their errors, and keep changes that make the whole population better. The rules that survive are meant to be general principles, and the RegNet models are what those rules produce at each compute budget.

The tool for comparing design spaces comes from the same group's earlier "On Network Design Spaces for Visual Recognition" (Radosavovic et al., ICCV 2019): the **error empirical distribution function** (EDF). Sample $n$ models, train each one, and plot the fraction whose error is below $e$:

$$
F(e)=\frac1n\sum_{i=1}^{n}\mathbf 1\left[e_i<e\right]
$$

- $n$: number of models sampled from the design space (500 in the paper)
- $e_i$: ImageNet top-1 error of model i after a short schedule (10 epochs, 360MF to 400MF models)
- $F(e)$: fraction of the design space below error e; a curve further left is a better space

According to the paper, training 100 models at 400MF for 10 epochs costs about as many FLOPs as training one ResNet-50 for 100 epochs, so a 500-model comparison is affordable.

**At a glance**

- **Paper:** Radosavovic, Kosaraju, Girshick, He, Dollár, CVPR 2020
- **Block:** ResNeXt-style residual bottleneck: $1\times1$ → grouped $3\times3$ (group width $g$) → $1\times1$; RegNetY adds SE
- **Width rule:** $u_j=w_0+w_a j$, quantized to $w_j=w_0w_m^{\lfloor s_j\rceil}$, a multiple of 8
- **Parameters:** $d,w_0,w_a,w_m,b,g$: 6 numbers, against 16 for AnyNetX
- **Findings:** $b=1$ (no bottleneck), about 20 blocks at every budget, $w_m\approx2.5$, widths grow with depth
- **RegNetY-400MF:** 4.34M params, 0.40G multiply-adds, 74.05% top-1 (torchvision V1 weights)
- **In PyTorch:** `torchvision.models.regnet_y_400mf` … `regnet_y_128gf`, `regnet_x_*`

## From AnyNet to RegNet

The starting point, **AnyNetX**, fixes the overall shape: a stem (a stride-2 $3\times3$ conv to 32 channels), a body of 4 stages at decreasing resolution, and a head (pooling and a linear classifier). Each stage $i$ is a stack of $d_i$ identical "X blocks" (1×1, grouped 3×3, 1×1 with a residual connection, the first block of a stage having stride 2) with width $w_i$, bottleneck ratio $b_i$ and group width $g_i$. That is 16 free parameters. With $d_i\le16$, $w_i\le1024$ in multiples of 8, $b_i\in\{1,2,4\}$ and $g_i\in\{1,2,\dots,32\}$ there are $(16\cdot128\cdot3\cdot6)^4\approx1.8\cdot10^{18}$ networks.

**Refining the design space (each step keeps or improves the error EDF)**

1. **AnyNetX$_A$ → $_B$:** a single bottleneck ratio $b$ for all stages. The EDF is unchanged. Size $\approx6.8\cdot10^{16}$.
2. **$_B$ → $_C$:** a single group width $g$ for all stages. Again no loss; 6 fewer degrees of freedom than $A$, size $\approx3.2\cdot10^{14}$.
3. **$_C$ → $_D$:** widths must not decrease, $w_{i+1}\ge w_i$. The EDF improves clearly. Size $\approx1.3\cdot10^{13}$.
4. **$_D$ → $_E$:** depths must not decrease either, $d_{i+1}\ge d_i$. Improves again; size $\approx5.5\cdot10^{11}$.
5. **$_E$ → RegNet:** the per-block widths of the best $_E$ models lie close to a line, $w_j\approx48\cdot(j+1)$. Replacing the free widths and depths by a quantized linear rule gives a space of about $3.0\cdot10^{8}$ networks, nearly 10 orders of magnitude smaller than AnyNetX, and with a better EDF.

## The quantized linear width rule

$$
u_j=w_0+w_a\,j\;\;(0\le j<d),\qquad u_j=w_0\,w_m^{\,s_j},\qquad w_j=w_0\,w_m^{\lfloor s_j\rceil}
$$

- $j$: block index over the whole network, $0$ to $d-1$
- $u_j$: continuous target width of block $j$, linear in $j$
- $w_0,\;w_a$: initial width and slope (both $>0$)
- $s_j=\log(u_j/w_0)/\log w_m$: how many factors of $w_m$ block $j$ is above $w_0$
- $\lfloor s_j\rceil$: $s_j$ rounded to an integer; blocks with the same value form a stage
- $w_j$: quantized width, finally rounded to a multiple of 8

Rounding $s_j$ turns the line into a staircase: stage $i$ has width $w_0w_m^{i}$ and depth $d_i=\sum_j\mathbf 1[\lfloor s_j\rceil=i]$. Consecutive stage widths therefore differ by the factor $w_m$, before the rounding to multiples of 8. Two last adjustments in torchvision's implementation: the stage width is rounded to a multiple of the group width (and $g$ is capped at the stage width), and each stage starts with a stride-2 block. The figure implements exactly this procedure.

> Figure: Dashed line: the linear widths u_j = w₀ + w_a·j. Dots: the quantized widths w_j, coloured by stage; each flat run is one stage. The default is RegNetY-400MF (d = 16, w₀ = 48, w_a = 27.89, w_m = 2.09, g = 8), which gives stages of 1, 3, 6, 6 blocks at widths 48, 104, 208, 440. The panel counts parameters, multiply-adds and activations from the generated configuration: 4,344,144 parameters, 0.402G multiply-adds and 3.89M activations, exactly torchvision's regnet_y_400mf. All six presets reproduce torchvision's parameter counts. Move a slider to see the stages regroup. Toggle X/Y to add or remove squeeze-and-excitation. 'sample a RegNet' draws a random 4-stage member of the design space.

## What the population analysis found

With the RegNetX space in hand, the paper analyses which parameter values the best models share across compute budgets. Some findings contradict common practice:

**Empirical findings (Radosavovic et al., 2020)**

1. **Depth is stable.** The best models have about 20 blocks (about 60 layers) at every budget. They do not get deeper as FLOPs grow, unlike the usual practice of scaling depth.
2. **No bottleneck.** The best bottleneck ratio is $b=1$: the middle of the block is as wide as its output. Inverted bottlenecks ($b<1$, as in MobileNetV2) make the EDF worse, and depthwise convolution ($g=1$) worse still.
3. **Width multiplier about 2.5.** Good models multiply the width by $w_m\approx2.5$ per stage, close to but not exactly the classic doubling. The group width $g$, $w_a$ and $w_0$ all increase with the budget.
4. **Activations predict runtime.** Defining activations as the total size of all conv output tensors, GPU inference time correlates more strongly with activations than with FLOPs. For the best models, activations grow like the square root of FLOPs and parameters grow linearly.
5. **Resolution 224 is best.** Unlike EfficientNet's compound scaling, keeping $224\times224$ input was best for RegNetX even at higher FLOPs.
6. **SE helps.** Adding squeeze-and-excitation gives RegNetY, with clear gains over RegNetX.

> **Random search becomes easy:** Because the RegNet space is so concentrated around good models, the paper finds that sampling about 32 random models is likely to yield a good one. Its released models were picked as the best of 25 random settings of $(d,g,w_m,w_a,w_0)$ per flop regime, then retrained. There is no expensive search controller; the design space does the work.

## Worked example: building RegNetY-400MF

**d = 16, w₀ = 48, w_a = 27.89, w_m = 2.09, g = 8, b = 1, SE ratio 0.25**

1. Block $j=1$: $u_1=48+27.89=75.89$, $s_1=\log(75.89/48)/\log2.09=0.62$, rounded to 1. Width $48\cdot2.09=100.3$, nearest multiple of 8: **104**.
2. Block $j=4$: $u_4=159.56$, $s_4=1.63\to2$, width $48\cdot2.09^2=209.7\to$ **208**. Blocks 4 to 9 ($s_j$ up to 2.48) all round to 2.
3. Block $j=10$: $u_{10}=326.9$, $s_{10}=2.60\to3$, width $48\cdot2.09^3=438.2\to$ **440**, the last stage up to $j=15$.
4. Stages: depths $(1,3,6,6)$, widths $(48,104,208,440)$, all divisible by $g=8$, so $(6,13,26,55)$ groups of 8 channels at resolutions $56,28,14,7$.
5. Parameters per stage: 10,040 + 102,296 + 737,438 + 3,052,442, plus 928 for the stem and 441,000 for the 440→1000 classifier: **4,344,144**, exactly torchvision's count. Multiply-adds: 0.402G.
6. The SE modules (squeeze width $\lfloor0.25\,w_{\text{in}}\rceil$ from the block **input** width) account for 668,024 of those parameters, 15%, but only 0.66M multiply-adds, under 0.2% of the compute.

## Squeeze-and-excitation in RegNetY

The SE operation (Hu et al., 2018) rescales each channel of the $3\times3$ output $h$ by a learned gate computed from its global average:

$$
z=\tfrac1{HW}\sum_{p}h_{p},\qquad a=\sigma\big(W_2\,\mathrm{ReLU}(W_1z)\big),\qquad y_{c,p}=a_c\,h_{c,p},\qquad \frac{\partial y_{c,p}}{\partial h_{c,p}}=a_c+h_{c,p}\,\frac{\partial a_c}{\partial h_{c,p}}
$$

- $h$: output of the grouped $3\times3$ (after BN, ReLU), $w_b$ channels
- $z$: squeeze: one global-average number per channel
- $W_1,\;W_2$: $1\times1$ convs (with bias) $w_b\to\lfloor0.25w_{\text{in}}\rceil\to w_b$
- $a_c\in(0,1)$: excitation: sigmoid gate for channel $c$
- $\partial a_c/\partial h_{c,p}$: small: each position contributes $1/HW$ through the pooled $z$

The gradient through a gated channel is mostly the gate itself, $a_c$. SE learns to turn whole channels down for a given image, and every channel's gate depends on global context, which a $3\times3$ conv cannot see. Its cost is in parameters, not in FLOPs, because it works on pooled vectors.

```python
import math, torch
from torchvision.models import regnet_y_400mf, RegNet_Y_400MF_Weights
from torchvision.models.regnet import BlockParams

def regnet_widths(d, w0, wa, wm, q=8):
    u = [w0 + wa * j for j in range(d)]                       # linear widths
    s = [round(math.log(uj / w0) / math.log(wm)) for uj in u] # stage index per block
    return [int(round(w0 * wm ** sj / q) * q) for sj in s]    # quantized widths

print(regnet_widths(16, 48, 27.89, 2.09))
# [48, 104, 104, 104, 208, 208, 208, 208, 208, 208, 440, 440, 440, 440, 440, 440]

bp = BlockParams.from_init_params(depth=16, w_0=48, w_a=27.89, w_m=2.09,
                                  group_width=8, se_ratio=0.25)
print(bp.depths, bp.widths)            # [1, 3, 6, 6] [48, 104, 208, 440]

model = regnet_y_400mf(weights=RegNet_Y_400MF_Weights.IMAGENET1K_V2).eval()
print(sum(p.numel() for p in model.parameters()))   # 4344144
print(model.trunk_output.block2[0].f.b[0])  # Conv2d(104, 104, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=13, bias=False)
```

The width rule in a few lines, and torchvision's RegNets

torchvision ImageNet weights (documented). Params and GFLOPs are reproduced exactly by the figure's code. V1 = paper-style recipe, V2 = torchvision's improved 2021 recipe. Stages = blocks × width, as generated.

| Model | stages (depth × width) | Params | GFLOPs | acc@1 V1 | acc@1 V2 |
| --- | --- | --- | --- | --- | --- |
| regnet_x_400mf | 1×32, 2×64, 7×160, 12×400 | 5.50M | 0.41 | 72.83% | 74.86% |
| regnet_y_400mf | 1×48, 3×104, 6×208, 6×440 | 4.34M | 0.40 | 74.05% | 75.80% |
| regnet_y_800mf | 1×64, 3×144, 8×320, 2×784 | 6.43M | 0.83 | 76.42% | 78.83% |
| regnet_y_1_6gf | 2×48, 6×120, 17×336, 2×888 | 11.20M | 1.61 | 77.95% | 80.88% |
| regnet_x_3_2gf | 2×96, 6×192, 15×432, 2×1008 | 15.30M | 3.18 | 78.36% | 81.20% |
| regnet_y_3_2gf | 2×72, 5×216, 13×576, 1×1512 | 19.44M | 3.18 | 78.95% | 81.98% |

The stage layout of RegNetX-3.2GF, depths $(2,6,15,2)$ and widths $(96,192,432,1008)$ with $g=48$, is the one shown in the paper's Figure 15. In the paper's mobile-regime comparison (Table 2), RegNetX-600MF reaches 25.9% and RegNetY-600MF 24.5% top-1 error, trained with a basic 100-epoch schedule and only weight decay as regularization, against mobile networks trained with much heavier recipes.

> **Group width, not number of groups:** RegNet's $g$ is the number of **channels per group**, so the number of groups is $w/g$ and changes from stage to stage (6, 13, 26, 55 in RegNetY-400MF). PyTorch's `groups=` argument is the number of groups. Also, $g$ and $w$ must be compatible: in the figure, raising $g$ from 8 to 16 at the 400MF settings rounds the widths to $(48,112,208,448)$ and the cost to 4.78M parameters and 0.474G multiply-adds, while $g=48$ gives widths $(48,96,192,432)$, 5.57M parameters and 0.631G. Change one parameter and the others shift.

> **FLOPs, activations and EfficientNet:** EfficientNet optimizes FLOPs and uses depthwise convs, inverted bottlenecks and high input resolution, all of which increase activations. RegNet's analysis says GPU runtime follows activations more than FLOPs, and the paper reports RegNets that outperform EfficientNets at comparable training settings and FLOPs while being up to 5× faster on GPUs. Which is better depends on the hardware and the training recipe; see **EfficientNet** for the other side.

**Strengths**

- Six interpretable parameters generate a whole family (400MF to 128GF in torchvision)
- Fast on GPUs: no depthwise convs, standard resolution, low activation count
- The design-space method (EDFs over populations) transfers to other architecture questions
- Clean torchvision implementations with strong V2 weights; a common detection backbone

**Limitations**

- Less FLOP-efficient than EfficientNet on mobile-style FLOP counts
- The findings come from short 10-epoch training runs; their transfer was checked but not guaranteed
- Width, depth and group width interact through the quantization; small changes can regroup stages
- Still a plain ConvNet design; later work (ConvNeXt) gained more from modernizing training and blocks

**Question:** With w₀ = 48, w_a = 27.89, w_m = 2.09, block j = 4 has u₄ = 159.56. What is its quantized width?

- A. 160
- B. 208
- C. 104
- D. 440

**Answer:** B. 208 — $s_4=\log(159.56/48)/\log2.09=1.63$, which rounds to 2. The width is $48\cdot2.09^2=209.7$, and the nearest multiple of 8 is 208. The quantization snaps every block to a power of $w_m$ times $w_0$, so blocks with nearby $u_j$ share a stage.

**Question:** Which of these did the RegNet population analysis find for the best models?

- A. Deeper networks at higher FLOPs, with inverted bottlenecks
- B. About 20 blocks at every budget, bottleneck ratio b = 1, widths growing by about 2.5× per stage
- C. Depthwise convolutions (g = 1) at every budget
- D. Higher input resolution as FLOPs increase

**Answer:** B. About 20 blocks at every budget, bottleneck ratio b = 1, widths growing by about 2.5× per stage — Depth stayed near 20 blocks across regimes, $b=1$ beat both bottlenecks and inverted bottlenecks, $w_m\approx2.5$, and a fixed $224\times224$ resolution was best. Depthwise convs and inverted bottlenecks made the error distribution worse.

**Key points**

- RegNet designs the design space: sample populations, compare error EDFs, keep restrictions that improve the whole population.
- AnyNetX (16 parameters, $\sim10^{18}$ nets) is refined to RegNet (6 parameters: $d,w_0,w_a,w_m,b,g$).
- Widths: $u_j=w_0+w_aj$, quantized to $w_0w_m^{\lfloor s_j\rceil}$ and multiples of 8; equal widths form stages.
- Findings: about 20 blocks, $b=1$, $w_m\approx2.5$, resolution 224; activations predict GPU runtime better than FLOPs.
- RegNetY = RegNetX + squeeze-and-excitation; RegNetY-400MF has 4.34M params and 0.40 GFLOPs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
