# Sparse MoE routing

> A sparse MoE router sends each token to its top-k experts (or lets each expert pick its top tokens). Fixed expert capacity C = cf·kT/E drops overflow tokens, so routers are kept balanced by an auxiliary loss N·Σfᵢ·Pᵢ, kept stable by a z-loss, or balanced by bias terms.

Source: https://www.mlfoundry.in/learn/sparse-moe-routing  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 15 min read · updated 2026-10-12

**Sparse MoE routing:** is the rule that decides which of the $E$ experts of a **Mixture of experts (MoE)** layer process each token, and with what weight. In **token choice** routing each token takes the $k$ experts with the largest router logits. In **expert choice** routing each expert takes the $C$ tokens with the largest scores. Because each expert processes a fixed-size buffer of $C$ tokens (its **capacity**), routing comes with extra machinery. Capacity limits and token dropping handle overflow. Auxiliary losses or bias terms keep the load balanced. A router z-loss keeps the logits small and training stable.

A learned router left alone is unstable in a simple way. Experts that receive more tokens early get more gradient updates, become better, and attract even more tokens. Shazeer et al. (ICLR 2017) saw this collapse. Their fix was **noisy top-k gating** plus two auxiliary losses: an importance loss on the coefficient of variation of the summed gate values, and a smooth load estimate. GShard (Lepikhin et al., ICLR 2021) introduced **expert capacity**, a differentiable auxiliary balancing loss and random routing of the second expert for top-2. The Switch Transformer (Fedus, Zoph and Shazeer, *JMLR* 2022, arXiv 2021) showed that **top-1** routing works, with one simple auxiliary loss. Later work addressed the remaining problems. ST-MoE (Zoph et al. 2022) added the router z-loss for stability. Expert choice (Zhou et al., NeurIPS 2022) reverses the choice so that balance holds by construction. DeepSeek (2024) introduced shared experts and balancing without an auxiliary loss.

## Token choice: noisy top-k and Switch top-1

$$
H(x)_i=(xW_g)_i+\varepsilon_i\cdot\mathrm{softplus}\big((xW_{\text{noise}})_i\big),\ \ \varepsilon_i\sim\mathcal N(0,1),\qquad G(x)=\mathrm{softmax}\big(\mathrm{KeepTopK}(H(x),k)\big)
$$

- $W_g$: router weights, $d\times E$
- $W_{\text{noise}}$: learned per-expert noise scale; the noise lets near-ties switch experts during training and feeds the smooth load estimate of Shazeer et al.
- $\mathrm{KeepTopK}$: keeps the $k$ largest entries and sets the rest to $-\infty$, so their gate is exactly 0

Switch routing drops the noise and sets $k=1$. The router computes $p(x)=\mathrm{softmax}(xW_r)$, sends the token to $i^*=\arg\max_i p_i(x)$, and outputs $y=p_{i^*}(x)\,E_{i^*}(x)$. Keeping $p_{i^*}$ in the output gives the router its gradient. Top-1 halves the expert compute and the dispatch traffic of top-2, and the paper found it no worse in quality per FLOP. The authors also kept the router in float32 inside bfloat16 training ("selective precision") and scaled down the weight initialisation by 10×, both for stability.

## Capacity and dropped tokens

$$
C=\Big\lfloor \mathrm{cf}\cdot\frac{k\,T}{E}\Big\rfloor
$$

- $T$: tokens in the routing group (a batch, or a slice of it on one device)
- $kT/E$: the load of each expert if routing were perfectly uniform
- $\mathrm{cf}$: capacity factor, e.g. 1.0–1.25 in Switch pre-training; values above 1 leave slack for imbalance

Each expert's buffer has $C$ slots. Tokens are placed in order (by position, or by router score in the batch-prioritised routing of V-MoE, Riquelme et al. 2021), and the ones that arrive after the buffer is full are **dropped**: the MoE contributes nothing for them in that layer, and they continue unchanged through the residual connection. For top-2, GShard fills all first choices before any second choice, so a token loses its second expert before its first. A larger cf drops fewer tokens but pads every buffer to $C$, wasting compute and memory on empty slots. Switch found that low capacity factors (1.0–1.25) gave the best quality per unit of compute, provided the balancing loss keeps the drop rate small.

## The load-balancing loss

$$
\mathcal L_{\text{aux}}=\alpha\,N\sum_{i=1}^{N}f_i\,P_i,\qquad f_i=\frac1T\sum_{x\in\mathcal B}\mathbb 1\{\arg\max_j p_j(x)=i\},\qquad P_i=\frac1T\sum_{x\in\mathcal B}p_i(x)
$$

- $N$: number of experts ($E$ elsewhere in this article)
- $f_i$: fraction of the $T$ tokens dispatched to expert $i$; piecewise constant, so no gradient flows through it
- $P_i$: fraction of router probability given to expert $i$; differentiable, so the loss pushes probability away from overloaded experts
- $\alpha$: $10^{-2}$ in Switch: large enough to balance, small enough not to override the task loss

Under uniform routing $f_i=P_i=1/N$ and $N\sum_i f_iP_i=1$, whatever $N$ is, so the loss reads the same for 8 or 2048 experts. The gradient $\partial\mathcal L_{\text{aux}}/\partial P_i=\alpha Nf_i$ is largest for the busiest experts. It lowers their probability on every token, which in turn lowers their future $f_i$. The product $f_iP_i$ is a differentiable stand-in for $f_i^2$: minimising $\sum f_i^2$ at fixed $\sum f_i=1$ means equal loads. GShard's loss has the same form with the mean gate and the first-choice fraction.

## Router z-loss

$$
\mathcal L_z=\frac1T\sum_{x\in\mathcal B}\Big(\log\sum_{j=1}^{N}e^{h_j(x)}\Big)^{2}
$$

- $h_j(x)$: router logits before the softmax
- $\log\sum e^{h}$: the log-partition; it grows with the scale of the logits
- $c_z$: weight $10^{-3}$ in ST-MoE

ST-MoE (Zoph, Bello, Kumar, Du, Huang, Dean, Shazeer and Fedus, 2022) traced many training divergences to the router. Large logits make the exponentials in the softmax overflow bfloat16 rounding, and a small error in a large logit changes the routing. The z-loss penalises large logits directly. It does not change which expert wins, because adding a constant to all logits leaves the softmax unchanged. The constant is pushed toward $\log\sum e^h=0$. ST-MoE found that it removed the instabilities without the quality cost of other fixes. They used it to train ST-MoE-32B, a 269B-parameter sparse model with the compute of a 32B dense model.

> Figure: 32 tokens, 8 experts. The heat map shows router probabilities p(e|x), with a fixed preference for low-numbered experts set by the skew slider. With top-1, cf = 1.25 and skew 1, the capacity is ⌊1.25·32/8⌋ = 5. The tokens ask for loads of 13, 8, 6, 2, 2, 1, 0, 0, so 12 of 32 are dropped (red), the aux loss N·Σf·P is 1.782 and the z-loss 7.71. cf = 1.0 (C = 4) drops 15; cf = 2.0 (C = 8) still drops 5. At skew 0 the requests are 6, 5, 7, 2, 2, 3, 6, 1: only 4 are dropped, and the aux loss is 1.054, close to its uniform value of 1. Raising the skew to 2 pushes the aux loss to 3.257 and the z-loss to 11.90. Ten bias updates (DeepSeek-style, γ = 0.1, selection only) bring the skew-1 requests to 5, 5, 4, 5, 4, 5, 3, 1 with no drops. After 20 the sign updates overshoot (one drop). Top-2 at cf 1.25 (C = 10) drops 22 of 64 copies, and 2 tokens get no expert at all. Expert choice at the same capacity drops nothing: every expert takes exactly 10 tokens, 15 tokens get three or more experts, and at cf = 0.5 eight tokens get none.

## Expert choice routing

$$
S=\mathrm{softmax}(XW_g)\in\mathbb R^{T\times E},\qquad I_{:,e}=\operatorname{TopK}\big(S_{:,e},\,C\big),\qquad C=\frac{T\,c}{E}
$$

- $S_{t,e}$: token-to-expert affinity, a softmax over experts for each token
- $I_{:,e}$: the $C$ tokens expert $e$ selects, taking the top $C$ of its **column**
- $c$: average experts per token; $c=2$ matches top-2's compute

Zhou, Lei, Liu, Du, Huang, Zhao, Dai, Chen, Le and Laudon ("Mixture-of-Experts with Expert Choice Routing", NeurIPS 2022) let the experts choose. Every expert processes exactly $C$ tokens, so the load is perfectly balanced, nothing overflows, and no auxiliary loss is needed. The number of experts per token now varies: hard tokens can get several, easy ones zero. They reported more than 2× faster training convergence than Switch top-1 and GShard top-2 at equal compute. The catch is that the choice for one token depends on the other tokens in the batch. In autoregressive decoding the future tokens of the batch are not available, so expert choice breaks causality and is mostly used in encoders or during training only.

## Shared experts and balancing without an auxiliary loss

DeepSeekMoE (Dai et al., 2024) changed the expert layout in two ways. **Fine-grained experts**: each FFN is split into $m$ smaller experts and $mk$ of them are activated, so the same compute allows many more combinations. **Shared experts**: a few experts are always active for every token and capture common knowledge, so the routed experts need not each relearn it. DeepSeek-V3 (2024) has 1 shared and 256 routed experts per MoE layer and activates 8 routed ones: 671B total and 37B active parameters.

$$
\mathcal T(x)=\operatorname{TopK}\big(s_i(x)+b_i,\ k\big),\qquad g_i(x)=\frac{s_i(x)}{\sum_{j\in\mathcal T(x)}s_j(x)},\qquad b_i\leftarrow b_i-\gamma\,\mathrm{sign}\big(\mathrm{load}_i-\overline{\mathrm{load}}\big)
$$

- $s_i(x)$: token–expert affinity (a sigmoid in DeepSeek-V3)
- $b_i$: a per-expert bias used **only to select** experts, never in the gate value, so the output is not distorted
- $\gamma$: bias update speed; overloaded experts become less likely to be selected, underloaded ones more likely

The bias update is not a gradient step and adds no term to the loss, so balancing no longer trades off against the language-modelling objective (Wang et al., "Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts", 2024). DeepSeek-V3 still adds a very small sequence-level balance loss against extreme imbalance within one sequence, and it drops no tokens in training or inference.

Routers in published models (as described in each paper)

| model | routing | experts per MoE layer | balancing | notes |
| --- | --- | --- | --- | --- |
| Sparsely-gated MoE (Shazeer et al. 2017) | noisy top-k | up to 131,072 (hierarchical) | importance + load losses | between LSTM layers |
| GShard (2020/21) | top-2, random 2nd | 2048 | aux loss, capacity | 600B, every other FFN |
| Switch Transformer (2021/22) | top-1 | up to 2048 (Switch-C) | aux loss $\alpha=10^{-2}$, capacity | 1.6T parameters |
| ST-MoE (2022) | top-2 |  | aux loss + z-loss $10^{-3}$ | 269B parameters |
| Expert choice (2022) | experts pick tokens | 64 in the main 8B runs | by construction | not causal |
| Mixtral 8x7B (2024) | top-2, softmax over the 2 | 8 | not described in the paper | 46.7B / 12.9B active |
| DeepSeek-V3 (2024) | top-8 + 1 shared | 256 routed | bias, tiny seq. loss | 671B / 37B active |

## Worked example: Switch routing of 8 tokens to 4 experts

Router probabilities p(i|x); bold = argmax (checked in Node)

| token | e0 | e1 | e2 | e3 | argmax |
| --- | --- | --- | --- | --- | --- |
| t0 | **0.60** | 0.20 | 0.10 | 0.10 | e0 |
| t1 | **0.50** | 0.30 | 0.15 | 0.05 | e0 |
| t2 | **0.70** | 0.10 | 0.12 | 0.08 | e0 |
| t3 | 0.20 | **0.50** | 0.20 | 0.10 | e1 |
| t4 | **0.40** | 0.30 | 0.25 | 0.05 | e0 |
| t5 | 0.10 | **0.60** | 0.18 | 0.12 | e1 |
| t6 | 0.30 | 0.20 | **0.45** | 0.05 | e2 |
| t7 | **0.45** | 0.10 | 0.14 | 0.31 | e0 |

**Capacity, drops, the balancing loss and expert choice**

1. Dispatch fractions: $f=(5,2,1,0)/8=(0.625,\,0.25,\,0.125,\,0)$. Mean probabilities (column means): $P=(0.40625,\,0.2875,\,0.19875,\,0.1075)$.
2. $N\sum_if_iP_i=4\,(0.625\cdot0.40625+0.25\cdot0.2875+0.125\cdot0.19875+0)=$ **1.4025**, against 1 for uniform routing. The gradient is largest on $P_0$ (coefficient $\alpha\cdot4\cdot0.625$).
3. Capacity with cf = 1: $C=\lfloor1\cdot8/4\rfloor=2$. Expert 0 keeps t0 and t1 and **drops t2, t4, t7** (37.5% of the batch), even though t2 is its most confident token. With cf = 1.5, $C=3$ and only t4, t7 are dropped. With cf = 2.5, $C=5$ and none are, but expert 3's 5 slots are all padding.
4. Expert choice with $C=2$: e0 takes t2, t0; e1 takes t5, t3; e2 takes t6, t4; e3 takes t7, t5. Every expert is full and none overflows, but **t1 gets no expert** and t5 gets two.
5. z-loss of one token with logits $(2,\,0.5,\,-1,\,0)$: $\log\sum e^h=2.3423$, so its term is $5.4866$. Multiplying the logits by 10 leaves the argmax unchanged, but the term becomes 400.00 and the softmax is almost one-hot. The z-loss pushes against exactly that growth.

```python
import torch, torch.nn.functional as F

def switch_route(logits, capacity_factor=1.0):
    """logits: (T, E) router logits. Top-1 routing with a capacity per expert."""
    T, E = logits.shape
    probs = logits.softmax(dim=-1)                   # (T, E)
    gate, expert = probs.max(dim=-1)                 # (T,), (T,)
    onehot = F.one_hot(expert, E).float()            # (T, E)
    pos = (onehot.cumsum(dim=0) - 1) * onehot        # slot index inside the chosen expert, token order
    C = int(capacity_factor * T / E)
    kept = ((pos < C).float() * onehot).sum(dim=-1)  # (T,): 1 = processed, 0 = dropped (residual only)
    f = onehot.mean(dim=0)                           # share of tokens dispatched to each expert
    P = probs.mean(dim=0)                            # mean router probability per expert
    aux = E * (f * P).sum()                          # multiply by alpha = 1e-2 in the total loss
    z = torch.logsumexp(logits, dim=-1).pow(2).mean()
    return expert, gate * kept, aux, z, C

p = torch.tensor([[.60, .20, .10, .10], [.50, .30, .15, .05], [.70, .10, .12, .08],
                  [.20, .50, .20, .10], [.40, .30, .25, .05], [.10, .60, .18, .12],
                  [.30, .20, .45, .05], [.45, .10, .14, .31]])
expert, gate, aux, z, C = switch_route(p.log())      # log-probabilities as logits
print(C, expert.tolist())          # 2 [0, 0, 0, 1, 0, 1, 2, 0]
print((gate == 0).nonzero().flatten().tolist())      # [2, 4, 7]  dropped tokens
print(round(aux.item(), 4))        # 1.4025
print(round(z.item(), 4))          # 0.0: log-probabilities already have logsumexp 0
```

Switch top-1 routing with capacity, load-balancing loss and z-loss (the worked example's probabilities as input)

> **Dropped tokens depend on the batch:** With capacity limits a token's output depends on which other tokens share its routing group and where it sits in the order. The same sentence can be processed differently in two batches, and evaluation can change with the batch size. At inference, use a larger capacity factor, or a dropless implementation (MegaBlocks-style grouped matmuls, or DeepSeek-V3's no-drop routing), and do not compare drop rates across different group sizes.

> **Balance is about hardware as much as quality:** Every expert buffer is padded to $C$, and the step waits for the busiest device. Imbalance costs twice: dropped tokens lose quality, and idle experts waste the compute you paid for. Expert choice and the bias method fix balance without adding a loss term. The auxiliary loss fixes it at the price of a small gradient that is not about the task. All three trade some routing freedom for utilisation.

> **Simpler alternatives that work:** Routing need not be learned per token. Hash layers (Roller et al., 2021) route by a fixed hash of the token id. BASE layers (Lewis et al., 2021) solve a balanced linear assignment problem, so every expert gets exactly $T/E$ tokens. Both are competitive baselines, which suggests that much of the benefit of MoE comes from extra parameters with sparse use, not from clever routing.

**Strengths**

- Top-1 (Switch): least compute and communication per token; one simple aux loss
- Top-2: more robust, each token gets a second opinion and survives losing one slot
- Expert choice: perfect load balance, no drops, variable compute per token
- Bias-based (aux-loss-free) balancing: no gradient interference with the task loss

**Limitations**

- Fixed capacity drops tokens under imbalance; large cf wastes padded compute
- Auxiliary loss weight $\alpha$ must be tuned; too large hurts quality, too small collapses
- Expert choice is not causal, so it is awkward for autoregressive decoding
- Router logits can blow up in low precision; needs float32 routing and/or z-loss

**Question:** A Switch layer routes T = 4096 tokens to E = 64 experts with capacity factor 1.25. What is the expert capacity, and what happens to the 81st token sent to one expert?

- A. 64; it is sent to its second-best expert
- B. 80; it is dropped from this layer and passes through the residual connection
- C. 80; it waits for the next batch
- D. 5120; nothing, capacity is per batch

**Answer:** B. 80; it is dropped from this layer and passes through the residual connection — $C=\lfloor1.25\cdot4096/64\rfloor=\lfloor80\rfloor=80$. Switch has no second choice. Overflow tokens skip the expert computation, so the MoE output for that token is zero and its hidden state continues unchanged through the residual connection.

**Question:** Why does the Switch balancing loss use the product fᵢ·Pᵢ rather than penalising fᵢ alone?

- A. fᵢ is a count of argmax decisions and has no gradient; Pᵢ is differentiable, and weighting it by fᵢ pushes probability away from overloaded experts
- B. Pᵢ is always equal to fᵢ
- C. The product is needed to make the loss non-negative
- D. fᵢ alone would make the loss depend on the number of experts

**Answer:** A. fᵢ is a count of argmax decisions and has no gradient; Pᵢ is differentiable, and weighting it by fᵢ pushes probability away from overloaded experts — $f_i$ comes from a discrete argmax, so its gradient is zero almost everywhere. $\partial(N\sum f_iP_i)/\partial P_i=Nf_i$: the router is told to lower the probability of the experts that currently receive the most tokens, which lowers their future $f_i$. The loss equals 1 under uniform routing for any $N$.

**Key points**

- Switch balancing loss $\alpha N\sum_i f_iP_i$ ($\alpha=10^{-2}$): $f_i$ = share of tokens dispatched to expert $i$, $P_i$ = mean router probability; equals $\alpha$ under uniform routing.
- Capacity $C=\lfloor\mathrm{cf}\cdot kT/E\rfloor$ slots per expert; overflow tokens are dropped and pass through the residual connection.
- Router z-loss $\frac1T\sum_x(\log\sum_j e^{h_j(x)})^2$, weight $10^{-3}$ (ST-MoE), keeps logits small for stable low-precision training.
- Expert choice: each expert takes its top-C tokens; perfect balance and no drops, but variable experts per token and not causal.
- DeepSeek: shared experts always on, fine-grained routed experts, and per-expert selection biases updated by sign(load − mean) in place of an aux loss.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
