Mixture of experts (MoE)

An MoE layer holds E expert networks and a learned router that sends each token to only k of them, y = Σᵢ∈TopK gᵢ(x)·Eᵢ(x). Parameters grow with E while per-token compute grows only with k: Mixtral 8x7B has 46.7B parameters but uses 12.9B per token.

13 min read1277 wordscore0 viewsupdated 2026-10-12

Definition

Mixture of experts (MoE) is a layer made of EE sub-networks, the experts E1,…,EEE_1,\dots,E_E, and a gating network (router) that computes input-dependent weights g(x)g(x). The output is y=∑igi(x) Ei(x)y=\sum_i g_i(x)\,E_i(x). In a sparse MoE the gate is non-zero for only k≪Ek\ll E experts per input, so only those experts are evaluated. In Transformers the experts are usually the position-wise feed-forward networks (FFNs): each token is sent to its top-kk FFNs, and the model has many more parameters than any one token uses.

The idea is older than deep learning. Jacobs, Jordan, Nowlan and Hinton, "Adaptive Mixtures of Local Experts" (Neural Computation, 1991), trained several small networks and a softmax gating network jointly. Their error −log⁡∑igi e−12∥d−oi∥2-\log\sum_i g_i\,e^{-\frac12\lVert d-o_i\rVert^2} makes the experts compete for each training case instead of cooperating, so each one specialises on a region of the input space. Jordan and Jacobs (1994) stacked gates into a tree (hierarchical mixtures of experts) and fitted it with EM. Eigen, Ranzato and Sutskever (2013) stacked MoE layers inside a deep network. In all of these every expert is evaluated for every input, so the mixture costs as much as running all experts.

The modern form comes from Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton and Dean, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (ICLR 2017). Their gate keeps only the top kk logits, so the other experts are never computed. With thousands of feed-forward experts applied between stacked LSTM layers, they trained language-modelling and translation models with up to 137 billion parameters at modest compute. GShard (Lepikhin et al., ICLR 2021) moved the layer into the Transformer: every other FFN became a top-2 MoE, giving a 600B-parameter translation model with 2048 experts per MoE layer. The Switch Transformer (Fedus, Zoph and Shazeer, JMLR 2022) used top-1 and scaled to 1.6 trillion parameters. Mixtral 8x7B (Jiang et al., 2024) made top-2 MoE a standard open-weights Decoder-only transformer. How the router chooses, and how it is kept balanced, is the subject of Sparse MoE routing.

The layer

h(x)=x Wg∈RE,T(x)=TopK⁡(h(x),k),gi(x)={ehi(x)∑j∈T(x)ehj(x)i∈T(x)0otherwise,y=∑i∈T(x)gi(x) Ei(x)h(x)=x\,W_g\in\mathbb R^{E},\qquad \mathcal T(x)=\operatorname{TopK}\big(h(x),k\big),\qquad g_i(x)=\begin{cases}\dfrac{e^{h_i(x)}}{\sum_{j\in\mathcal T(x)}e^{h_j(x)}} & i\in\mathcal T(x)\\[4pt] 0 & \text{otherwise}\end{cases},\qquad y=\sum_{i\in\mathcal T(x)}g_i(x)\,E_i(x)
x∈Rdx\in\mathbb R^{d}
one token's hidden state at the input of the FFN sub-layer (after the attention sub-layer and its norm)
Wg∈Rd×EW_g\in\mathbb R^{d\times E}
the router: one linear map, so it adds only dEdE parameters per layer
T(x)\mathcal T(x)
indices of the kk largest router logits; k=2k=2 in GShard and Mixtral, k=1k=1 in Switch
gi(x)g_i(x)
gate weight: a softmax over the selected logits only, which equals Shazeer's softmax(KeepTopK(h,k))\mathrm{softmax}(\mathrm{KeepTopK}(h,k)) with the others set to −∞-\infty
EiE_i
expert ii: in Transformers an FFN, d→dff→dd\to d_{ff}\to d; in Mixtral a SwiGLU FFN W2(SiLU(W1x)⊙W3x)W_2(\mathrm{SiLU}(W_1x)\odot W_3x)

Only the selected experts run, so the FFN cost per token is kk times that of one expert, whatever EE is. The attention sub-layers, the embeddings and the norms are shared by all tokens, as in a dense model. A Transformer MoE block is therefore x←x+Attn(LN(x))x\leftarrow x+\mathrm{Attn}(\mathrm{LN}(x)), then x←x+MoE(LN(x))x\leftarrow x+\mathrm{MoE}(\mathrm{LN}(x)). Some models put an MoE in every block (Mixtral). Others alternate MoE and dense blocks (GShard replaced every other FFN).

The router is trained by backpropagation through the gate values. ∂y/∂hi\partial y/\partial h_i is non-zero only for selected experts, and it pushes hih_i up when Ei(x)E_i(x) helps the loss more than the other chosen experts do. The discrete top-kk choice itself has no gradient. With k=1k=1 a renormalised gate would always equal 1, and the router would receive no gradient at all. That is why Switch multiplies the expert output by the un-normalised probability pi(x)p_i(x), not by 1. Without extra terms, the router tends to collapse onto a few experts that then train faster, which is why every practical MoE adds a balancing mechanism (see Sparse MoE routing).

Total versus active parameters

Ntotal=Nshared+L E Nexpert,Nactive=Nshared+L k Nexpert,FLOPs/token≈2NactiveN_{\text{total}}=N_{\text{shared}}+L\,E\,N_{\text{expert}},\qquad N_{\text{active}}=N_{\text{shared}}+L\,k\,N_{\text{expert}},\qquad \text{FLOPs/token}\approx2N_{\text{active}}
NsharedN_{\text{shared}}
embeddings, attention, norms and routers: used by every token
NexpertN_{\text{expert}}
parameters of one expert FFN (3 d dff3\,d\,d_{ff} for SwiGLU)
LL
number of MoE layers
2Nactive2N_{\text{active}}
forward multiply-adds counted as 2 FLOPs per weight, ignoring the attention-score term

The two numbers separate what dense models tie together. Capacity (what the model can store) follows NtotalN_{\text{total}}. Compute per token follows NactiveN_{\text{active}}. Memory has to hold NtotalN_{\text{total}}, because any token may need any expert. An MoE is cheap to train and run per token but not cheap to host. At small batch sizes, inference is limited by reading weights from memory, and a batch of tokens touches most experts. The speed advantage over a dense model of the same total size is real. The memory requirement is the same as that dense model's.

Mixtral 8x7B (Jiang et al. 2024; released configuration)

Shape
32 layers, d=4096d=4096, SwiGLU experts with dff=14336d_{ff}=14336, 32 query heads and 8 KV heads of 128 (Grouped-query attention (GQA)), vocabulary 32,000, context 32k
Experts
8 per layer, top-2 routing in every layer; gate = softmax over the two selected logits
Parameters
46.7B total, 12.9B active per token (the paper rounds to 47B and 13B)
Dense counterpart
the same shape with one FFN is Mistral 7B (7.24B parameters)
Results reported
matches or outperforms Llama 2 70B and GPT-3.5 on most of the paper's benchmarks, with about 5× fewer active parameters than Llama 2 70B
Routing analysis
experts do not specialise by topic or domain in any obvious way; consecutive tokens are often routed to the same experts, and the choice tracks syntax (e.g. Python indentation, self)
y = Σi∈TopK(h) gᵢ·Eᵢ(x), h = x·W_g, g = softmax over the k chosen logitsThecatsatonthemat;deff(x)tokensexperts · loadFFN02FFN14FFN23FFN32FFN42FFN51FFN64FFN76“cat”: □ softmax(h) over all 8 · ■ gate g (top-2, renormalised)01234567Mixtral shape: d 4096, 32 layersSwiGLU experts 4096→14336→4096experts E · top-k 8 · 2total parameters 46.70Bactive per token 12.88B (27.6%)dense (E = 1) model 7.24Bexpert weights 96.6% of totalFFN FLOPs / token 2× one expert12 tokens on 2 devices:token→expert copies 24max / mean load 6 / 3.00sent off-device 13 of 24“cat” → 2:0.75 1:0.25colour = device · dashed edge =token leaves its device(all-to-all dispatch and return)■ busiest expert · click a token
devices
fig.Twelve tokens are routed by a random linear router to their top-k of E experts. Edge width is the gate weight, and the bars show how many token copies each expert received. With E = 8 and k = 2, 'cat' goes to FFN2 with weight 0.751 and FFN1 with 0.249. The loads are 2, 4, 3, 2, 2, 1, 4, 6: no balancing loss is applied, so FFN7 gets twice the mean of 3. The right panel counts a Mixtral-shaped model exactly: 46.70B total, 12.88B active (27.6%), against 7.24B for the dense E = 1 model. Setting k = 1 keeps the 46.70B but drops the active count to 7.24B. E = 16, k = 4 gives 91.80B total and 24.16B active. Expert weights are 96.6% of all parameters at E = 8. With the experts split over 2 devices, 13 of the 24 token copies must cross devices (dashed edges). With 4 devices it is 20 of 24. That is the all-to-all traffic expert parallelism pays for.

Expert parallelism

A model whose experts do not fit on one accelerator is sharded by expert, as GShard did. Each of DD devices holds E/DE/D experts and also processes its own slice of the batch, with the shared layers replicated or sharded as usual. An MoE layer then runs in four steps: route locally, all-to-all dispatch (each token's hidden state is sent to the devices that hold its chosen experts), run the experts on the tokens received, and all-to-all combine (send the outputs back and sum them with the gates). With experts spread evenly, a fraction (D−1)/D(D-1)/D of the kk copies per token leaves its device. The communication is proportional to k⋅dk\cdot d per token, and it sits on the critical path twice per MoE layer.

Static tensor shapes make this efficient on accelerators, so each expert receives a fixed-size buffer of CC tokens per batch, the expert capacity. Overflow tokens are dropped from that layer and continue through the residual connection. Load imbalance is not only a quality problem: the busiest expert sets the step time, and a device whose experts are popular stalls the others. Later systems added refinements: DeepSpeed-MoE and Tutel optimise the all-to-all, MegaBlocks avoids dropping with block-sparse kernels, and DeepSeek-V3 limits each token to experts on at most 4 nodes.

Worked example: one token through a top-2 layer

Four experts with router logits h=(1.2, −0.3, 2.0, 0.4)h=(1.2,\,-0.3,\,2.0,\,0.4) for one token. Each expert maps the token to a 2-vector; experts 0 and 2 return E0(x)=(1,−1)E_0(x)=(1,-1) and E2(x)=(0.5, 2)E_2(x)=(0.5,\,2).

expert ilogit hᵢsoftmax over all 4top-2?gate gᵢ (softmax over the 2)
01.20.2565yes0.3100
1−0.30.0572no0
22.00.5709yes0.6900
30.40.1153no0
Gate weights (computed in Node)

From logits to the layer output

  1. 1Top-2 of hh: experts 2 and 0. Gates g2=e2.0/(e2.0+e1.2)=1/(1+e−0.8)=0.6900g_2=e^{2.0}/(e^{2.0}+e^{1.2})=1/(1+e^{-0.8})=0.6900 and g0=0.3100g_0=0.3100. Renormalising the full-softmax probabilities, 0.5709/(0.5709+0.2565)0.5709/(0.5709+0.2565), gives the same 0.6900.
  2. 2Only E0E_0 and E2E_2 are evaluated: y=0.31 (1,−1)+0.69 (0.5,2)=(0.6550, 1.0699)y=0.31\,(1,-1)+0.69\,(0.5,2)=(0.6550,\ 1.0699). Experts 1 and 3 cost nothing for this token.
  3. 3A Switch-style top-1 layer would pick expert 2 alone and output p2E2(x)=0.5709 (0.5,2)=(0.2855, 1.1419)p_2E_2(x)=0.5709\,(0.5,2)=(0.2855,\ 1.1419), keeping p2p_2 so the router gets a gradient.
  4. 4Counting Mixtral exactly: attention per layer 2⋅4096⋅4096+2⋅4096⋅1024=41,943,0402\cdot4096\cdot4096+2\cdot4096\cdot1024=41{,}943{,}040; one SwiGLU expert 3⋅4096⋅14336=176,160,7683\cdot4096\cdot14336=176{,}160{,}768; router 4096⋅84096\cdot8; two RMSNorms 2⋅40962\cdot4096. Per layer with 8 experts: 1,451,270,144.
  5. 5Times 32 layers, plus embeddings and output head 2⋅32000⋅40962\cdot32000\cdot4096 and the final norm: 46,702,792,704 total. Replacing 88 experts by the k=2k=2 used per token: 12,879,925,248 active. Mixtral's "46.7B / 12.9B" are these two numbers.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class SwiGLU(nn.Module):
4 def __init__(self, d, h):
5 super().__init__()
6 self.w1 = nn.Linear(d, h, bias=False)
7 self.w3 = nn.Linear(d, h, bias=False)
8 self.w2 = nn.Linear(h, d, bias=False)
9
10 def forward(self, x):
11 return self.w2(F.silu(self.w1(x)) * self.w3(x))
12
13class MoE(nn.Module):
14 def __init__(self, d=64, h=128, n_experts=8, k=2):
15 super().__init__()
16 self.k = k
17 self.gate = nn.Linear(d, n_experts, bias=False) # router W_g
18 self.experts = nn.ModuleList(SwiGLU(d, h) for _ in range(n_experts))
19
20 def forward(self, x): # x: (B, T, d)
21 B, T, d = x.shape
22 x = x.reshape(-1, d) # (N, d) with N = B*T tokens
23 logits = self.gate(x) # (N, E)
24 top, idx = logits.topk(self.k, dim=-1) # (N, k) values and expert ids
25 w = top.softmax(dim=-1) # gates: softmax over the k chosen logits
26 y = torch.zeros_like(x)
27 for e, expert in enumerate(self.experts):
28 tok, slot = (idx == e).nonzero(as_tuple=True) # tokens that chose expert e
29 if tok.numel():
30 y.index_add_(0, tok, w[tok, slot, None] * expert(x[tok]))
31 return y.reshape(B, T, d), logits
32
33moe = MoE()
34y, logits = moe(torch.randn(2, 10, 64))
35print(y.shape, logits.shape) # torch.Size([2, 10, 64]) torch.Size([20, 8])
36total = sum(p.numel() for p in moe.parameters())
37active = moe.gate.weight.numel() + moe.k * sum(p.numel() for p in moe.experts[0].parameters())
38print(total, active) # 197120 49664 (8 vs 2 experts of 3*64*128 = 24576, + 512 router)
A top-k MoE layer with SwiGLU experts (loop over experts; real systems batch this with grouped matmuls)

Use when

  • +Parameter count, and so capacity, grows with EE at constant FLOPs per token
  • +Better quality per training FLOP than dense models in language modelling and translation
  • +Drop-in replacement for the FFN sub-layer; attention, embeddings and the training loop are unchanged
  • +Expert parallelism adds a new axis for sharding very large models

Avoid when

  • −Memory holds all NtotalN_{\text{total}} parameters; serving needs as much memory as a dense model of the same total size
  • −Two all-to-all exchanges per MoE layer; communication can dominate on slow interconnects
  • −Needs load balancing; unbalanced routers collapse, drop tokens and leave hardware idle
  • −Training instabilities and overfitting on fine-tuning are more common than in dense models

Check yourself

A model has 32 layers with 8 experts each and top-2 routing. Shared parameters are 1.6B and each expert has 176M parameters. Roughly how many parameters does one token use?

Check yourself

Why does the Switch Transformer scale the chosen expert's output by its router probability p_i(x) instead of renormalising the single selected gate to 1?

Remember

  • ✓MoE layer: y=∑i∈TopK(xWg,k)gi(x)Ei(x)y=\sum_{i\in\mathrm{TopK}(xW_g,k)}g_i(x)E_i(x), gates = softmax over the kk selected router logits; only kk of EE experts run per token.
  • ✓Ntotal=Nshared+LENexpertN_{\text{total}}=N_{\text{shared}}+LEN_{\text{expert}} vs Nactive=Nshared+LkNexpertN_{\text{active}}=N_{\text{shared}}+LkN_{\text{expert}}: Mixtral 8x7B has 46.7B total, 12.9B active (top-2 of 8).
  • ✓Origins: Jacobs et al. 1991 (competing experts with a softmax gate); Shazeer et al. 2017 made it sparse with top-k gating; GShard and Switch put it in place of the Transformer FFN.
  • ✓Expert parallelism shards experts over devices; each MoE layer needs an all-to-all dispatch and an all-to-all combine.
  • ✓Compute per token follows active parameters, memory follows total parameters; balancing the router is necessary (see Sparse MoE routing).
← back to Deep Learning roadmap