Definition
Mixture of experts (MoE) is a layer made of sub-networks, the experts , and a gating network (router) that computes input-dependent weights . The output is . In a sparse MoE the gate is non-zero for only experts per input, so only those experts are evaluated. In Transformers the experts are usually the position-wise feed-forward networks (FFNs): each token is sent to its top- FFNs, and the model has many more parameters than any one token uses.
The idea is older than deep learning. Jacobs, Jordan, Nowlan and Hinton, "Adaptive Mixtures of Local Experts" (Neural Computation, 1991), trained several small networks and a softmax gating network jointly. Their error makes the experts compete for each training case instead of cooperating, so each one specialises on a region of the input space. Jordan and Jacobs (1994) stacked gates into a tree (hierarchical mixtures of experts) and fitted it with EM. Eigen, Ranzato and Sutskever (2013) stacked MoE layers inside a deep network. In all of these every expert is evaluated for every input, so the mixture costs as much as running all experts.
The modern form comes from Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton and Dean, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (ICLR 2017). Their gate keeps only the top logits, so the other experts are never computed. With thousands of feed-forward experts applied between stacked LSTM layers, they trained language-modelling and translation models with up to 137 billion parameters at modest compute. GShard (Lepikhin et al., ICLR 2021) moved the layer into the Transformer: every other FFN became a top-2 MoE, giving a 600B-parameter translation model with 2048 experts per MoE layer. The Switch Transformer (Fedus, Zoph and Shazeer, JMLR 2022) used top-1 and scaled to 1.6 trillion parameters. Mixtral 8x7B (Jiang et al., 2024) made top-2 MoE a standard open-weights Decoder-only transformer. How the router chooses, and how it is kept balanced, is the subject of Sparse MoE routing.
The layer
- one token's hidden state at the input of the FFN sub-layer (after the attention sub-layer and its norm)
- the router: one linear map, so it adds only parameters per layer
- indices of the largest router logits; in GShard and Mixtral, in Switch
- gate weight: a softmax over the selected logits only, which equals Shazeer's with the others set to
- expert : in Transformers an FFN, ; in Mixtral a SwiGLU FFN
Only the selected experts run, so the FFN cost per token is times that of one expert, whatever is. The attention sub-layers, the embeddings and the norms are shared by all tokens, as in a dense model. A Transformer MoE block is therefore , then . Some models put an MoE in every block (Mixtral). Others alternate MoE and dense blocks (GShard replaced every other FFN).
The router is trained by backpropagation through the gate values. is non-zero only for selected experts, and it pushes up when helps the loss more than the other chosen experts do. The discrete top- choice itself has no gradient. With a renormalised gate would always equal 1, and the router would receive no gradient at all. That is why Switch multiplies the expert output by the un-normalised probability , not by 1. Without extra terms, the router tends to collapse onto a few experts that then train faster, which is why every practical MoE adds a balancing mechanism (see Sparse MoE routing).
Total versus active parameters
- embeddings, attention, norms and routers: used by every token
- parameters of one expert FFN ( for SwiGLU)
- number of MoE layers
- forward multiply-adds counted as 2 FLOPs per weight, ignoring the attention-score term
The two numbers separate what dense models tie together. Capacity (what the model can store) follows . Compute per token follows . Memory has to hold , because any token may need any expert. An MoE is cheap to train and run per token but not cheap to host. At small batch sizes, inference is limited by reading weights from memory, and a batch of tokens touches most experts. The speed advantage over a dense model of the same total size is real. The memory requirement is the same as that dense model's.
Mixtral 8x7B (Jiang et al. 2024; released configuration)
- Shape
- 32 layers, , SwiGLU experts with , 32 query heads and 8 KV heads of 128 (Grouped-query attention (GQA)), vocabulary 32,000, context 32k
- Experts
- 8 per layer, top-2 routing in every layer; gate = softmax over the two selected logits
- Parameters
- 46.7B total, 12.9B active per token (the paper rounds to 47B and 13B)
- Dense counterpart
- the same shape with one FFN is Mistral 7B (7.24B parameters)
- Results reported
- matches or outperforms Llama 2 70B and GPT-3.5 on most of the paper's benchmarks, with about 5× fewer active parameters than Llama 2 70B
- Routing analysis
- experts do not specialise by topic or domain in any obvious way; consecutive tokens are often routed to the same experts, and the choice tracks syntax (e.g. Python indentation,
self)
Expert parallelism
A model whose experts do not fit on one accelerator is sharded by expert, as GShard did. Each of devices holds experts and also processes its own slice of the batch, with the shared layers replicated or sharded as usual. An MoE layer then runs in four steps: route locally, all-to-all dispatch (each token's hidden state is sent to the devices that hold its chosen experts), run the experts on the tokens received, and all-to-all combine (send the outputs back and sum them with the gates). With experts spread evenly, a fraction of the copies per token leaves its device. The communication is proportional to per token, and it sits on the critical path twice per MoE layer.
Static tensor shapes make this efficient on accelerators, so each expert receives a fixed-size buffer of tokens per batch, the expert capacity. Overflow tokens are dropped from that layer and continue through the residual connection. Load imbalance is not only a quality problem: the busiest expert sets the step time, and a device whose experts are popular stalls the others. Later systems added refinements: DeepSpeed-MoE and Tutel optimise the all-to-all, MegaBlocks avoids dropping with block-sparse kernels, and DeepSeek-V3 limits each token to experts on at most 4 nodes.
Worked example: one token through a top-2 layer
Four experts with router logits for one token. Each expert maps the token to a 2-vector; experts 0 and 2 return and .
| expert i | logit hᵢ | softmax over all 4 | top-2? | gate gᵢ (softmax over the 2) |
|---|---|---|---|---|
| 0 | 1.2 | 0.2565 | yes | 0.3100 |
| 1 | −0.3 | 0.0572 | no | 0 |
| 2 | 2.0 | 0.5709 | yes | 0.6900 |
| 3 | 0.4 | 0.1153 | no | 0 |
From logits to the layer output
- 1Top-2 of : experts 2 and 0. Gates and . Renormalising the full-softmax probabilities, , gives the same 0.6900.
- 2Only and are evaluated: . Experts 1 and 3 cost nothing for this token.
- 3A Switch-style top-1 layer would pick expert 2 alone and output , keeping so the router gets a gradient.
- 4Counting Mixtral exactly: attention per layer ; one SwiGLU expert ; router ; two RMSNorms . Per layer with 8 experts: 1,451,270,144.
- 5Times 32 layers, plus embeddings and output head and the final norm: 46,702,792,704 total. Replacing experts by the used per token: 12,879,925,248 active. Mixtral's "46.7B / 12.9B" are these two numbers.
1import torch, torch.nn as nn, torch.nn.functional as F23class SwiGLU(nn.Module):4 def __init__(self, d, h):5 super().__init__()6 self.w1 = nn.Linear(d, h, bias=False)7 self.w3 = nn.Linear(d, h, bias=False)8 self.w2 = nn.Linear(h, d, bias=False)910 def forward(self, x):11 return self.w2(F.silu(self.w1(x)) * self.w3(x))1213class MoE(nn.Module):14 def __init__(self, d=64, h=128, n_experts=8, k=2):15 super().__init__()16 self.k = k17 self.gate = nn.Linear(d, n_experts, bias=False) # router W_g18 self.experts = nn.ModuleList(SwiGLU(d, h) for _ in range(n_experts))1920 def forward(self, x): # x: (B, T, d)21 B, T, d = x.shape22 x = x.reshape(-1, d) # (N, d) with N = B*T tokens23 logits = self.gate(x) # (N, E)24 top, idx = logits.topk(self.k, dim=-1) # (N, k) values and expert ids25 w = top.softmax(dim=-1) # gates: softmax over the k chosen logits26 y = torch.zeros_like(x)27 for e, expert in enumerate(self.experts):28 tok, slot = (idx == e).nonzero(as_tuple=True) # tokens that chose expert e29 if tok.numel():30 y.index_add_(0, tok, w[tok, slot, None] * expert(x[tok]))31 return y.reshape(B, T, d), logits3233moe = MoE()34y, logits = moe(torch.randn(2, 10, 64))35print(y.shape, logits.shape) # torch.Size([2, 10, 64]) torch.Size([20, 8])36total = sum(p.numel() for p in moe.parameters())37active = moe.gate.weight.numel() + moe.k * sum(p.numel() for p in moe.experts[0].parameters())38print(total, active) # 197120 49664 (8 vs 2 experts of 3*64*128 = 24576, + 512 router)Use when
- +Parameter count, and so capacity, grows with at constant FLOPs per token
- +Better quality per training FLOP than dense models in language modelling and translation
- +Drop-in replacement for the FFN sub-layer; attention, embeddings and the training loop are unchanged
- +Expert parallelism adds a new axis for sharding very large models
Avoid when
- −Memory holds all parameters; serving needs as much memory as a dense model of the same total size
- −Two all-to-all exchanges per MoE layer; communication can dominate on slow interconnects
- −Needs load balancing; unbalanced routers collapse, drop tokens and leave hardware idle
- −Training instabilities and overfitting on fine-tuning are more common than in dense models
Check yourself
A model has 32 layers with 8 experts each and top-2 routing. Shared parameters are 1.6B and each expert has 176M parameters. Roughly how many parameters does one token use?
Check yourself
Why does the Switch Transformer scale the chosen expert's output by its router probability p_i(x) instead of renormalising the single selected gate to 1?
Remember
- ✓MoE layer: , gates = softmax over the selected router logits; only of experts run per token.
- ✓ vs : Mixtral 8x7B has 46.7B total, 12.9B active (top-2 of 8).
- ✓Origins: Jacobs et al. 1991 (competing experts with a softmax gate); Shazeer et al. 2017 made it sparse with top-k gating; GShard and Switch put it in place of the Transformer FFN.
- ✓Expert parallelism shards experts over devices; each MoE layer needs an all-to-all dispatch and an all-to-all combine.
- ✓Compute per token follows active parameters, memory follows total parameters; balancing the router is necessary (see Sparse MoE routing).