Sparse MoE routing

A sparse MoE router sends each token to its top-k experts (or lets each expert pick its top tokens). Fixed expert capacity C = cf·kT/E drops overflow tokens, so routers are kept balanced by an auxiliary loss N·Σfᵢ·Pᵢ, kept stable by a z-loss, or balanced by bias terms.

15 min read1348 wordsadvanced0 viewsupdated 2026-10-12

Definition

Sparse MoE routing is the rule that decides which of the EE experts of a Mixture of experts (MoE) layer process each token, and with what weight. In token choice routing each token takes the kk experts with the largest router logits. In expert choice routing each expert takes the CC tokens with the largest scores. Because each expert processes a fixed-size buffer of CC tokens (its capacity), routing comes with extra machinery. Capacity limits and token dropping handle overflow. Auxiliary losses or bias terms keep the load balanced. A router z-loss keeps the logits small and training stable.

A learned router left alone is unstable in a simple way. Experts that receive more tokens early get more gradient updates, become better, and attract even more tokens. Shazeer et al. (ICLR 2017) saw this collapse. Their fix was noisy top-k gating plus two auxiliary losses: an importance loss on the coefficient of variation of the summed gate values, and a smooth load estimate. GShard (Lepikhin et al., ICLR 2021) introduced expert capacity, a differentiable auxiliary balancing loss and random routing of the second expert for top-2. The Switch Transformer (Fedus, Zoph and Shazeer, JMLR 2022, arXiv 2021) showed that top-1 routing works, with one simple auxiliary loss. Later work addressed the remaining problems. ST-MoE (Zoph et al. 2022) added the router z-loss for stability. Expert choice (Zhou et al., NeurIPS 2022) reverses the choice so that balance holds by construction. DeepSeek (2024) introduced shared experts and balancing without an auxiliary loss.

Token choice: noisy top-k and Switch top-1

H(x)i=(xWg)i+εi⋅softplus((xWnoise)i),  εi∼N(0,1),G(x)=softmax(KeepTopK(H(x),k))H(x)_i=(xW_g)_i+\varepsilon_i\cdot\mathrm{softplus}\big((xW_{\text{noise}})_i\big),\ \ \varepsilon_i\sim\mathcal N(0,1),\qquad G(x)=\mathrm{softmax}\big(\mathrm{KeepTopK}(H(x),k)\big)
WgW_g
router weights, d×Ed\times E
WnoiseW_{\text{noise}}
learned per-expert noise scale; the noise lets near-ties switch experts during training and feeds the smooth load estimate of Shazeer et al.
KeepTopK\mathrm{KeepTopK}
keeps the kk largest entries and sets the rest to −∞-\infty, so their gate is exactly 0

Switch routing drops the noise and sets k=1k=1. The router computes p(x)=softmax(xWr)p(x)=\mathrm{softmax}(xW_r), sends the token to i∗=arg⁡max⁡ipi(x)i^*=\arg\max_i p_i(x), and outputs y=pi∗(x) Ei∗(x)y=p_{i^*}(x)\,E_{i^*}(x). Keeping pi∗p_{i^*} in the output gives the router its gradient. Top-1 halves the expert compute and the dispatch traffic of top-2, and the paper found it no worse in quality per FLOP. The authors also kept the router in float32 inside bfloat16 training ("selective precision") and scaled down the weight initialisation by 10×, both for stability.

Capacity and dropped tokens

C=⌊cf⋅k TE⌋C=\Big\lfloor \mathrm{cf}\cdot\frac{k\,T}{E}\Big\rfloor
TT
tokens in the routing group (a batch, or a slice of it on one device)
kT/EkT/E
the load of each expert if routing were perfectly uniform
cf\mathrm{cf}
capacity factor, e.g. 1.0–1.25 in Switch pre-training; values above 1 leave slack for imbalance

Each expert's buffer has CC slots. Tokens are placed in order (by position, or by router score in the batch-prioritised routing of V-MoE, Riquelme et al. 2021), and the ones that arrive after the buffer is full are dropped: the MoE contributes nothing for them in that layer, and they continue unchanged through the residual connection. For top-2, GShard fills all first choices before any second choice, so a token loses its second expert before its first. A larger cf drops fewer tokens but pads every buffer to CC, wasting compute and memory on empty slots. Switch found that low capacity factors (1.0–1.25) gave the best quality per unit of compute, provided the balancing loss keeps the drop rate small.

The load-balancing loss

Laux=α N∑i=1Nfi Pi,fi=1T∑x∈B1{arg⁡max⁡jpj(x)=i},Pi=1T∑x∈Bpi(x)\mathcal L_{\text{aux}}=\alpha\,N\sum_{i=1}^{N}f_i\,P_i,\qquad f_i=\frac1T\sum_{x\in\mathcal B}\mathbb 1\{\arg\max_j p_j(x)=i\},\qquad P_i=\frac1T\sum_{x\in\mathcal B}p_i(x)
NN
number of experts (EE elsewhere in this article)
fif_i
fraction of the TT tokens dispatched to expert ii; piecewise constant, so no gradient flows through it
PiP_i
fraction of router probability given to expert ii; differentiable, so the loss pushes probability away from overloaded experts
α\alpha
10−210^{-2} in Switch: large enough to balance, small enough not to override the task loss

Under uniform routing fi=Pi=1/Nf_i=P_i=1/N and N∑ifiPi=1N\sum_i f_iP_i=1, whatever NN is, so the loss reads the same for 8 or 2048 experts. The gradient ∂Laux/∂Pi=αNfi\partial\mathcal L_{\text{aux}}/\partial P_i=\alpha Nf_i is largest for the busiest experts. It lowers their probability on every token, which in turn lowers their future fif_i. The product fiPif_iP_i is a differentiable stand-in for fi2f_i^2: minimising ∑fi2\sum f_i^2 at fixed ∑fi=1\sum f_i=1 means equal loads. GShard's loss has the same form with the mean gate and the first-choice fraction.

Router z-loss

Lz=1T∑x∈B(log⁡∑j=1Nehj(x))2\mathcal L_z=\frac1T\sum_{x\in\mathcal B}\Big(\log\sum_{j=1}^{N}e^{h_j(x)}\Big)^{2}
hj(x)h_j(x)
router logits before the softmax
log⁡∑eh\log\sum e^{h}
the log-partition; it grows with the scale of the logits
czc_z
weight 10−310^{-3} in ST-MoE

ST-MoE (Zoph, Bello, Kumar, Du, Huang, Dean, Shazeer and Fedus, 2022) traced many training divergences to the router. Large logits make the exponentials in the softmax overflow bfloat16 rounding, and a small error in a large logit changes the routing. The z-loss penalises large logits directly. It does not change which expert wins, because adding a constant to all logits leaves the softmax unchanged. The constant is pushed toward log⁡∑eh=0\log\sum e^h=0. ST-MoE found that it removed the instabilities without the quality cost of other fixes. They used it to train ST-MoE-32B, a 269B-parameter sparse model with the compute of a 32B dense model.

p(e|x) = softmax(x·W_r) · cell shade = p · each token picks its best expert0481216202428e0e1e2e3e4e5e6e7□ kept · ✕ dropped (over C) · ● token gets no expertC 55+8e05+3e15+1e22e32e41e50e60e7routertop-1capacity C5max load5dropped12 of 32no expert12 tokens≥3 experts0 tokensaux loss1.782z-loss7.71C = ⌊cf·k·T/E⌋ = ⌊1.25·1·32/8⌋ = 5 slots per expert; late tokens past C skip the layeraux = N·Σᵢ fᵢPᵢ = 1.782 (1 if uniform); fᵢ = share of assignments, Pᵢ = mean p(i|x)z-loss = mean over tokens of (log Σₑ exp hₑ)² = 7.71; grows with the logit scaleno balancing: requested loads 13 8 6 2 2 1 0 0 against C = 5
router

bias steps: DeepSeek-V3-style auxiliary-loss-free balancing on this batch (γ = 0.1). Skew adds a fixed preference for low-numbered experts.

fig.32 tokens, 8 experts. The heat map shows router probabilities p(e|x), with a fixed preference for low-numbered experts set by the skew slider. With top-1, cf = 1.25 and skew 1, the capacity is ⌊1.25·32/8⌋ = 5. The tokens ask for loads of 13, 8, 6, 2, 2, 1, 0, 0, so 12 of 32 are dropped (red), the aux loss N·Σf·P is 1.782 and the z-loss 7.71. cf = 1.0 (C = 4) drops 15; cf = 2.0 (C = 8) still drops 5. At skew 0 the requests are 6, 5, 7, 2, 2, 3, 6, 1: only 4 are dropped, and the aux loss is 1.054, close to its uniform value of 1. Raising the skew to 2 pushes the aux loss to 3.257 and the z-loss to 11.90. Ten bias updates (DeepSeek-style, γ = 0.1, selection only) bring the skew-1 requests to 5, 5, 4, 5, 4, 5, 3, 1 with no drops. After 20 the sign updates overshoot (one drop). Top-2 at cf 1.25 (C = 10) drops 22 of 64 copies, and 2 tokens get no expert at all. Expert choice at the same capacity drops nothing: every expert takes exactly 10 tokens, 15 tokens get three or more experts, and at cf = 0.5 eight tokens get none.

Expert choice routing

S=softmax(XWg)∈RT×E,I:,e=TopK⁡(S:,e, C),C=T cES=\mathrm{softmax}(XW_g)\in\mathbb R^{T\times E},\qquad I_{:,e}=\operatorname{TopK}\big(S_{:,e},\,C\big),\qquad C=\frac{T\,c}{E}
St,eS_{t,e}
token-to-expert affinity, a softmax over experts for each token
I:,eI_{:,e}
the CC tokens expert ee selects, taking the top CC of its column
cc
average experts per token; c=2c=2 matches top-2's compute

Zhou, Lei, Liu, Du, Huang, Zhao, Dai, Chen, Le and Laudon ("Mixture-of-Experts with Expert Choice Routing", NeurIPS 2022) let the experts choose. Every expert processes exactly CC tokens, so the load is perfectly balanced, nothing overflows, and no auxiliary loss is needed. The number of experts per token now varies: hard tokens can get several, easy ones zero. They reported more than 2× faster training convergence than Switch top-1 and GShard top-2 at equal compute. The catch is that the choice for one token depends on the other tokens in the batch. In autoregressive decoding the future tokens of the batch are not available, so expert choice breaks causality and is mostly used in encoders or during training only.

Shared experts and balancing without an auxiliary loss

DeepSeekMoE (Dai et al., 2024) changed the expert layout in two ways. Fine-grained experts: each FFN is split into mm smaller experts and mkmk of them are activated, so the same compute allows many more combinations. Shared experts: a few experts are always active for every token and capture common knowledge, so the routed experts need not each relearn it. DeepSeek-V3 (2024) has 1 shared and 256 routed experts per MoE layer and activates 8 routed ones: 671B total and 37B active parameters.

T(x)=TopK⁡(si(x)+bi, k),gi(x)=si(x)∑j∈T(x)sj(x),bi←bi−γ sign(loadi−load‾)\mathcal T(x)=\operatorname{TopK}\big(s_i(x)+b_i,\ k\big),\qquad g_i(x)=\frac{s_i(x)}{\sum_{j\in\mathcal T(x)}s_j(x)},\qquad b_i\leftarrow b_i-\gamma\,\mathrm{sign}\big(\mathrm{load}_i-\overline{\mathrm{load}}\big)
si(x)s_i(x)
token–expert affinity (a sigmoid in DeepSeek-V3)
bib_i
a per-expert bias used only to select experts, never in the gate value, so the output is not distorted
γ\gamma
bias update speed; overloaded experts become less likely to be selected, underloaded ones more likely

The bias update is not a gradient step and adds no term to the loss, so balancing no longer trades off against the language-modelling objective (Wang et al., "Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts", 2024). DeepSeek-V3 still adds a very small sequence-level balance loss against extreme imbalance within one sequence, and it drops no tokens in training or inference.

modelroutingexperts per MoE layerbalancingnotes
Sparsely-gated MoE (Shazeer et al. 2017)noisy top-kup to 131,072 (hierarchical)importance + load lossesbetween LSTM layers
GShard (2020/21)top-2, random 2nd2048aux loss, capacity600B, every other FFN
Switch Transformer (2021/22)top-1up to 2048 (Switch-C)aux loss α=10−2\alpha=10^{-2}, capacity1.6T parameters
ST-MoE (2022)top-2aux loss + z-loss 10−310^{-3}269B parameters
Expert choice (2022)experts pick tokens64 in the main 8B runsby constructionnot causal
Mixtral 8x7B (2024)top-2, softmax over the 28not described in the paper46.7B / 12.9B active
DeepSeek-V3 (2024)top-8 + 1 shared256 routedbias, tiny seq. loss671B / 37B active
Routers in published models (as described in each paper)

Worked example: Switch routing of 8 tokens to 4 experts

tokene0e1e2e3argmax
t00.600.200.100.10e0
t10.500.300.150.05e0
t20.700.100.120.08e0
t30.200.500.200.10e1
t40.400.300.250.05e0
t50.100.600.180.12e1
t60.300.200.450.05e2
t70.450.100.140.31e0
Router probabilities p(i|x); bold = argmax (checked in Node)

Capacity, drops, the balancing loss and expert choice

  1. 1Dispatch fractions: f=(5,2,1,0)/8=(0.625, 0.25, 0.125, 0)f=(5,2,1,0)/8=(0.625,\,0.25,\,0.125,\,0). Mean probabilities (column means): P=(0.40625, 0.2875, 0.19875, 0.1075)P=(0.40625,\,0.2875,\,0.19875,\,0.1075).
  2. 2N∑ifiPi=4 (0.625⋅0.40625+0.25⋅0.2875+0.125⋅0.19875+0)=N\sum_if_iP_i=4\,(0.625\cdot0.40625+0.25\cdot0.2875+0.125\cdot0.19875+0)= 1.4025, against 1 for uniform routing. The gradient is largest on P0P_0 (coefficient α⋅4⋅0.625\alpha\cdot4\cdot0.625).
  3. 3Capacity with cf = 1: C=⌊1⋅8/4⌋=2C=\lfloor1\cdot8/4\rfloor=2. Expert 0 keeps t0 and t1 and drops t2, t4, t7 (37.5% of the batch), even though t2 is its most confident token. With cf = 1.5, C=3C=3 and only t4, t7 are dropped. With cf = 2.5, C=5C=5 and none are, but expert 3's 5 slots are all padding.
  4. 4Expert choice with C=2C=2: e0 takes t2, t0; e1 takes t5, t3; e2 takes t6, t4; e3 takes t7, t5. Every expert is full and none overflows, but t1 gets no expert and t5 gets two.
  5. 5z-loss of one token with logits (2, 0.5, −1, 0)(2,\,0.5,\,-1,\,0): log⁡∑eh=2.3423\log\sum e^h=2.3423, so its term is 5.48665.4866. Multiplying the logits by 10 leaves the argmax unchanged, but the term becomes 400.00 and the softmax is almost one-hot. The z-loss pushes against exactly that growth.
python
1import torch, torch.nn.functional as F
2
3def switch_route(logits, capacity_factor=1.0):
4 """logits: (T, E) router logits. Top-1 routing with a capacity per expert."""
5 T, E = logits.shape
6 probs = logits.softmax(dim=-1) # (T, E)
7 gate, expert = probs.max(dim=-1) # (T,), (T,)
8 onehot = F.one_hot(expert, E).float() # (T, E)
9 pos = (onehot.cumsum(dim=0) - 1) * onehot # slot index inside the chosen expert, token order
10 C = int(capacity_factor * T / E)
11 kept = ((pos < C).float() * onehot).sum(dim=-1) # (T,): 1 = processed, 0 = dropped (residual only)
12 f = onehot.mean(dim=0) # share of tokens dispatched to each expert
13 P = probs.mean(dim=0) # mean router probability per expert
14 aux = E * (f * P).sum() # multiply by alpha = 1e-2 in the total loss
15 z = torch.logsumexp(logits, dim=-1).pow(2).mean()
16 return expert, gate * kept, aux, z, C
17
18p = torch.tensor([[.60, .20, .10, .10], [.50, .30, .15, .05], [.70, .10, .12, .08],
19 [.20, .50, .20, .10], [.40, .30, .25, .05], [.10, .60, .18, .12],
20 [.30, .20, .45, .05], [.45, .10, .14, .31]])
21expert, gate, aux, z, C = switch_route(p.log()) # log-probabilities as logits
22print(C, expert.tolist()) # 2 [0, 0, 0, 1, 0, 1, 2, 0]
23print((gate == 0).nonzero().flatten().tolist()) # [2, 4, 7] dropped tokens
24print(round(aux.item(), 4)) # 1.4025
25print(round(z.item(), 4)) # 0.0: log-probabilities already have logsumexp 0
Switch top-1 routing with capacity, load-balancing loss and z-loss (the worked example's probabilities as input)

Use when

  • +Top-1 (Switch): least compute and communication per token; one simple aux loss
  • +Top-2: more robust, each token gets a second opinion and survives losing one slot
  • +Expert choice: perfect load balance, no drops, variable compute per token
  • +Bias-based (aux-loss-free) balancing: no gradient interference with the task loss

Avoid when

  • −Fixed capacity drops tokens under imbalance; large cf wastes padded compute
  • −Auxiliary loss weight α\alpha must be tuned; too large hurts quality, too small collapses
  • −Expert choice is not causal, so it is awkward for autoregressive decoding
  • −Router logits can blow up in low precision; needs float32 routing and/or z-loss

Check yourself

A Switch layer routes T = 4096 tokens to E = 64 experts with capacity factor 1.25. What is the expert capacity, and what happens to the 81st token sent to one expert?

Check yourself

Why does the Switch balancing loss use the product fᵢ·Pᵢ rather than penalising fᵢ alone?

Remember

  • ✓Switch balancing loss αN∑ifiPi\alpha N\sum_i f_iP_i (α=10−2\alpha=10^{-2}): fif_i = share of tokens dispatched to expert ii, PiP_i = mean router probability; equals α\alpha under uniform routing.
  • ✓Capacity C=⌊cf⋅kT/E⌋C=\lfloor\mathrm{cf}\cdot kT/E\rfloor slots per expert; overflow tokens are dropped and pass through the residual connection.
  • ✓Router z-loss 1T∑x(log⁡∑jehj(x))2\frac1T\sum_x(\log\sum_j e^{h_j(x)})^2, weight 10−310^{-3} (ST-MoE), keeps logits small for stable low-precision training.
  • ✓Expert choice: each expert takes its top-C tokens; perfect balance and no drops, but variable experts per token and not causal.
  • ✓DeepSeek: shared experts always on, fine-grained routed experts, and per-expert selection biases updated by sign(load − mean) in place of an aux loss.
← back to Deep Learning roadmap