Definition
Sparse MoE routing is the rule that decides which of the experts of a Mixture of experts (MoE) layer process each token, and with what weight. In token choice routing each token takes the experts with the largest router logits. In expert choice routing each expert takes the tokens with the largest scores. Because each expert processes a fixed-size buffer of tokens (its capacity), routing comes with extra machinery. Capacity limits and token dropping handle overflow. Auxiliary losses or bias terms keep the load balanced. A router z-loss keeps the logits small and training stable.
A learned router left alone is unstable in a simple way. Experts that receive more tokens early get more gradient updates, become better, and attract even more tokens. Shazeer et al. (ICLR 2017) saw this collapse. Their fix was noisy top-k gating plus two auxiliary losses: an importance loss on the coefficient of variation of the summed gate values, and a smooth load estimate. GShard (Lepikhin et al., ICLR 2021) introduced expert capacity, a differentiable auxiliary balancing loss and random routing of the second expert for top-2. The Switch Transformer (Fedus, Zoph and Shazeer, JMLR 2022, arXiv 2021) showed that top-1 routing works, with one simple auxiliary loss. Later work addressed the remaining problems. ST-MoE (Zoph et al. 2022) added the router z-loss for stability. Expert choice (Zhou et al., NeurIPS 2022) reverses the choice so that balance holds by construction. DeepSeek (2024) introduced shared experts and balancing without an auxiliary loss.
Token choice: noisy top-k and Switch top-1
- router weights,
- learned per-expert noise scale; the noise lets near-ties switch experts during training and feeds the smooth load estimate of Shazeer et al.
- keeps the largest entries and sets the rest to , so their gate is exactly 0
Switch routing drops the noise and sets . The router computes , sends the token to , and outputs . Keeping in the output gives the router its gradient. Top-1 halves the expert compute and the dispatch traffic of top-2, and the paper found it no worse in quality per FLOP. The authors also kept the router in float32 inside bfloat16 training ("selective precision") and scaled down the weight initialisation by 10×, both for stability.
Capacity and dropped tokens
- tokens in the routing group (a batch, or a slice of it on one device)
- the load of each expert if routing were perfectly uniform
- capacity factor, e.g. 1.0–1.25 in Switch pre-training; values above 1 leave slack for imbalance
Each expert's buffer has slots. Tokens are placed in order (by position, or by router score in the batch-prioritised routing of V-MoE, Riquelme et al. 2021), and the ones that arrive after the buffer is full are dropped: the MoE contributes nothing for them in that layer, and they continue unchanged through the residual connection. For top-2, GShard fills all first choices before any second choice, so a token loses its second expert before its first. A larger cf drops fewer tokens but pads every buffer to , wasting compute and memory on empty slots. Switch found that low capacity factors (1.0–1.25) gave the best quality per unit of compute, provided the balancing loss keeps the drop rate small.
The load-balancing loss
- number of experts ( elsewhere in this article)
- fraction of the tokens dispatched to expert ; piecewise constant, so no gradient flows through it
- fraction of router probability given to expert ; differentiable, so the loss pushes probability away from overloaded experts
- in Switch: large enough to balance, small enough not to override the task loss
Under uniform routing and , whatever is, so the loss reads the same for 8 or 2048 experts. The gradient is largest for the busiest experts. It lowers their probability on every token, which in turn lowers their future . The product is a differentiable stand-in for : minimising at fixed means equal loads. GShard's loss has the same form with the mean gate and the first-choice fraction.
Router z-loss
- router logits before the softmax
- the log-partition; it grows with the scale of the logits
- weight in ST-MoE
ST-MoE (Zoph, Bello, Kumar, Du, Huang, Dean, Shazeer and Fedus, 2022) traced many training divergences to the router. Large logits make the exponentials in the softmax overflow bfloat16 rounding, and a small error in a large logit changes the routing. The z-loss penalises large logits directly. It does not change which expert wins, because adding a constant to all logits leaves the softmax unchanged. The constant is pushed toward . ST-MoE found that it removed the instabilities without the quality cost of other fixes. They used it to train ST-MoE-32B, a 269B-parameter sparse model with the compute of a 32B dense model.
bias steps: DeepSeek-V3-style auxiliary-loss-free balancing on this batch (γ = 0.1). Skew adds a fixed preference for low-numbered experts.
Expert choice routing
- token-to-expert affinity, a softmax over experts for each token
- the tokens expert selects, taking the top of its column
- average experts per token; matches top-2's compute
Zhou, Lei, Liu, Du, Huang, Zhao, Dai, Chen, Le and Laudon ("Mixture-of-Experts with Expert Choice Routing", NeurIPS 2022) let the experts choose. Every expert processes exactly tokens, so the load is perfectly balanced, nothing overflows, and no auxiliary loss is needed. The number of experts per token now varies: hard tokens can get several, easy ones zero. They reported more than 2× faster training convergence than Switch top-1 and GShard top-2 at equal compute. The catch is that the choice for one token depends on the other tokens in the batch. In autoregressive decoding the future tokens of the batch are not available, so expert choice breaks causality and is mostly used in encoders or during training only.
Shared experts and balancing without an auxiliary loss
DeepSeekMoE (Dai et al., 2024) changed the expert layout in two ways. Fine-grained experts: each FFN is split into smaller experts and of them are activated, so the same compute allows many more combinations. Shared experts: a few experts are always active for every token and capture common knowledge, so the routed experts need not each relearn it. DeepSeek-V3 (2024) has 1 shared and 256 routed experts per MoE layer and activates 8 routed ones: 671B total and 37B active parameters.
- token–expert affinity (a sigmoid in DeepSeek-V3)
- a per-expert bias used only to select experts, never in the gate value, so the output is not distorted
- bias update speed; overloaded experts become less likely to be selected, underloaded ones more likely
The bias update is not a gradient step and adds no term to the loss, so balancing no longer trades off against the language-modelling objective (Wang et al., "Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts", 2024). DeepSeek-V3 still adds a very small sequence-level balance loss against extreme imbalance within one sequence, and it drops no tokens in training or inference.
| model | routing | experts per MoE layer | balancing | notes |
|---|---|---|---|---|
| Sparsely-gated MoE (Shazeer et al. 2017) | noisy top-k | up to 131,072 (hierarchical) | importance + load losses | between LSTM layers |
| GShard (2020/21) | top-2, random 2nd | 2048 | aux loss, capacity | 600B, every other FFN |
| Switch Transformer (2021/22) | top-1 | up to 2048 (Switch-C) | aux loss , capacity | 1.6T parameters |
| ST-MoE (2022) | top-2 | aux loss + z-loss | 269B parameters | |
| Expert choice (2022) | experts pick tokens | 64 in the main 8B runs | by construction | not causal |
| Mixtral 8x7B (2024) | top-2, softmax over the 2 | 8 | not described in the paper | 46.7B / 12.9B active |
| DeepSeek-V3 (2024) | top-8 + 1 shared | 256 routed | bias, tiny seq. loss | 671B / 37B active |
Worked example: Switch routing of 8 tokens to 4 experts
| token | e0 | e1 | e2 | e3 | argmax |
|---|---|---|---|---|---|
| t0 | 0.60 | 0.20 | 0.10 | 0.10 | e0 |
| t1 | 0.50 | 0.30 | 0.15 | 0.05 | e0 |
| t2 | 0.70 | 0.10 | 0.12 | 0.08 | e0 |
| t3 | 0.20 | 0.50 | 0.20 | 0.10 | e1 |
| t4 | 0.40 | 0.30 | 0.25 | 0.05 | e0 |
| t5 | 0.10 | 0.60 | 0.18 | 0.12 | e1 |
| t6 | 0.30 | 0.20 | 0.45 | 0.05 | e2 |
| t7 | 0.45 | 0.10 | 0.14 | 0.31 | e0 |
Capacity, drops, the balancing loss and expert choice
- 1Dispatch fractions: . Mean probabilities (column means): .
- 2 1.4025, against 1 for uniform routing. The gradient is largest on (coefficient ).
- 3Capacity with cf = 1: . Expert 0 keeps t0 and t1 and drops t2, t4, t7 (37.5% of the batch), even though t2 is its most confident token. With cf = 1.5, and only t4, t7 are dropped. With cf = 2.5, and none are, but expert 3's 5 slots are all padding.
- 4Expert choice with : e0 takes t2, t0; e1 takes t5, t3; e2 takes t6, t4; e3 takes t7, t5. Every expert is full and none overflows, but t1 gets no expert and t5 gets two.
- 5z-loss of one token with logits : , so its term is . Multiplying the logits by 10 leaves the argmax unchanged, but the term becomes 400.00 and the softmax is almost one-hot. The z-loss pushes against exactly that growth.
1import torch, torch.nn.functional as F23def switch_route(logits, capacity_factor=1.0):4 """logits: (T, E) router logits. Top-1 routing with a capacity per expert."""5 T, E = logits.shape6 probs = logits.softmax(dim=-1) # (T, E)7 gate, expert = probs.max(dim=-1) # (T,), (T,)8 onehot = F.one_hot(expert, E).float() # (T, E)9 pos = (onehot.cumsum(dim=0) - 1) * onehot # slot index inside the chosen expert, token order10 C = int(capacity_factor * T / E)11 kept = ((pos < C).float() * onehot).sum(dim=-1) # (T,): 1 = processed, 0 = dropped (residual only)12 f = onehot.mean(dim=0) # share of tokens dispatched to each expert13 P = probs.mean(dim=0) # mean router probability per expert14 aux = E * (f * P).sum() # multiply by alpha = 1e-2 in the total loss15 z = torch.logsumexp(logits, dim=-1).pow(2).mean()16 return expert, gate * kept, aux, z, C1718p = torch.tensor([[.60, .20, .10, .10], [.50, .30, .15, .05], [.70, .10, .12, .08],19 [.20, .50, .20, .10], [.40, .30, .25, .05], [.10, .60, .18, .12],20 [.30, .20, .45, .05], [.45, .10, .14, .31]])21expert, gate, aux, z, C = switch_route(p.log()) # log-probabilities as logits22print(C, expert.tolist()) # 2 [0, 0, 0, 1, 0, 1, 2, 0]23print((gate == 0).nonzero().flatten().tolist()) # [2, 4, 7] dropped tokens24print(round(aux.item(), 4)) # 1.402525print(round(z.item(), 4)) # 0.0: log-probabilities already have logsumexp 0Use when
- +Top-1 (Switch): least compute and communication per token; one simple aux loss
- +Top-2: more robust, each token gets a second opinion and survives losing one slot
- +Expert choice: perfect load balance, no drops, variable compute per token
- +Bias-based (aux-loss-free) balancing: no gradient interference with the task loss
Avoid when
- −Fixed capacity drops tokens under imbalance; large cf wastes padded compute
- −Auxiliary loss weight must be tuned; too large hurts quality, too small collapses
- −Expert choice is not causal, so it is awkward for autoregressive decoding
- −Router logits can blow up in low precision; needs float32 routing and/or z-loss
Check yourself
A Switch layer routes T = 4096 tokens to E = 64 experts with capacity factor 1.25. What is the expert capacity, and what happens to the 81st token sent to one expert?
Check yourself
Why does the Switch balancing loss use the product fᵢ·Pᵢ rather than penalising fᵢ alone?
Remember
- ✓Switch balancing loss (): = share of tokens dispatched to expert , = mean router probability; equals under uniform routing.
- ✓Capacity slots per expert; overflow tokens are dropped and pass through the residual connection.
- ✓Router z-loss , weight (ST-MoE), keeps logits small for stable low-precision training.
- ✓Expert choice: each expert takes its top-C tokens; perfect balance and no drops, but variable experts per token and not causal.
- ✓DeepSeek: shared experts always on, fine-grained routed experts, and per-expert selection biases updated by sign(load − mean) in place of an aux loss.