# Mixture of experts (MoE)

> An MoE layer holds E expert networks and a learned router that sends each token to only k of them, y = Σᵢ∈TopK gᵢ(x)·Eᵢ(x). Parameters grow with E while per-token compute grows only with k: Mixtral 8x7B has 46.7B parameters but uses 12.9B per token.

Source: https://www.mlfoundry.in/learn/mixture-of-experts-moe  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 13 min read · updated 2026-10-12

**Mixture of experts (MoE):** is a layer made of $E$ sub-networks, the **experts** $E_1,\dots,E_E$, and a **gating network** (router) that computes input-dependent weights $g(x)$. The output is $y=\sum_i g_i(x)\,E_i(x)$. In a **sparse** MoE the gate is non-zero for only $k\ll E$ experts per input, so only those experts are evaluated. In Transformers the experts are usually the position-wise feed-forward networks (FFNs): each token is sent to its top-$k$ FFNs, and the model has many more parameters than any one token uses.

The idea is older than deep learning. Jacobs, Jordan, Nowlan and Hinton, "Adaptive Mixtures of Local Experts" (*Neural Computation*, 1991), trained several small networks and a softmax gating network jointly. Their error $-\log\sum_i g_i\,e^{-\frac12\lVert d-o_i\rVert^2}$ makes the experts **compete** for each training case instead of cooperating, so each one specialises on a region of the input space. Jordan and Jacobs (1994) stacked gates into a tree (hierarchical mixtures of experts) and fitted it with EM. Eigen, Ranzato and Sutskever (2013) stacked MoE layers inside a deep network. In all of these every expert is evaluated for every input, so the mixture costs as much as running all experts.

The modern form comes from Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton and Dean, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (ICLR 2017). Their gate keeps only the top $k$ logits, so the other experts are never computed. With thousands of feed-forward experts applied between stacked LSTM layers, they trained language-modelling and translation models with up to 137 billion parameters at modest compute. GShard (Lepikhin et al., ICLR 2021) moved the layer into the Transformer: every other FFN became a top-2 MoE, giving a 600B-parameter translation model with 2048 experts per MoE layer. The Switch Transformer (Fedus, Zoph and Shazeer, *JMLR* 2022) used top-1 and scaled to 1.6 trillion parameters. Mixtral 8x7B (Jiang et al., 2024) made top-2 MoE a standard open-weights **Decoder-only transformer**. How the router chooses, and how it is kept balanced, is the subject of **Sparse MoE routing**.

## The layer

$$
h(x)=x\,W_g\in\mathbb R^{E},\qquad \mathcal T(x)=\operatorname{TopK}\big(h(x),k\big),\qquad g_i(x)=\begin{cases}\dfrac{e^{h_i(x)}}{\sum_{j\in\mathcal T(x)}e^{h_j(x)}} & i\in\mathcal T(x)\\[4pt] 0 & \text{otherwise}\end{cases},\qquad y=\sum_{i\in\mathcal T(x)}g_i(x)\,E_i(x)
$$

- $x\in\mathbb R^{d}$: one token's hidden state at the input of the FFN sub-layer (after the attention sub-layer and its norm)
- $W_g\in\mathbb R^{d\times E}$: the router: one linear map, so it adds only $dE$ parameters per layer
- $\mathcal T(x)$: indices of the $k$ largest router logits; $k=2$ in GShard and Mixtral, $k=1$ in Switch
- $g_i(x)$: gate weight: a softmax over the selected logits only, which equals Shazeer's $\mathrm{softmax}(\mathrm{KeepTopK}(h,k))$ with the others set to $-\infty$
- $E_i$: expert $i$: in Transformers an FFN, $d\to d_{ff}\to d$; in Mixtral a **SwiGLU** FFN $W_2(\mathrm{SiLU}(W_1x)\odot W_3x)$

Only the selected experts run, so the FFN cost per token is $k$ times that of one expert, whatever $E$ is. The attention sub-layers, the embeddings and the norms are shared by all tokens, as in a dense model. A Transformer MoE block is therefore $x\leftarrow x+\mathrm{Attn}(\mathrm{LN}(x))$, then $x\leftarrow x+\mathrm{MoE}(\mathrm{LN}(x))$. Some models put an MoE in every block (Mixtral). Others alternate MoE and dense blocks (GShard replaced every other FFN).

The router is trained by backpropagation through the gate values. $\partial y/\partial h_i$ is non-zero only for selected experts, and it pushes $h_i$ up when $E_i(x)$ helps the loss more than the other chosen experts do. The discrete top-$k$ choice itself has no gradient. With $k=1$ a renormalised gate would always equal 1, and the router would receive no gradient at all. That is why Switch multiplies the expert output by the un-normalised probability $p_i(x)$, not by 1. Without extra terms, the router tends to collapse onto a few experts that then train faster, which is why every practical MoE adds a balancing mechanism (see **Sparse MoE routing**).

## Total versus active parameters

$$
N_{\text{total}}=N_{\text{shared}}+L\,E\,N_{\text{expert}},\qquad N_{\text{active}}=N_{\text{shared}}+L\,k\,N_{\text{expert}},\qquad \text{FLOPs/token}\approx2N_{\text{active}}
$$

- $N_{\text{shared}}$: embeddings, attention, norms and routers: used by every token
- $N_{\text{expert}}$: parameters of one expert FFN ($3\,d\,d_{ff}$ for SwiGLU)
- $L$: number of MoE layers
- $2N_{\text{active}}$: forward multiply-adds counted as 2 FLOPs per weight, ignoring the attention-score term

The two numbers separate what dense models tie together. **Capacity** (what the model can store) follows $N_{\text{total}}$. **Compute** per token follows $N_{\text{active}}$. **Memory** has to hold $N_{\text{total}}$, because any token may need any expert. An MoE is cheap to train and run per token but not cheap to host. At small batch sizes, inference is limited by reading weights from memory, and a batch of tokens touches most experts. The speed advantage over a dense model of the same total size is real. The memory requirement is the same as that dense model's.

**Mixtral 8x7B (Jiang et al. 2024; released configuration)**

- **Shape:** 32 layers, $d=4096$, SwiGLU experts with $d_{ff}=14336$, 32 query heads and 8 KV heads of 128 (**Grouped-query attention (GQA)**), vocabulary 32,000, context 32k
- **Experts:** 8 per layer, top-2 routing in every layer; gate = softmax over the two selected logits
- **Parameters:** 46.7B total, 12.9B active per token (the paper rounds to 47B and 13B)
- **Dense counterpart:** the same shape with one FFN is Mistral 7B (7.24B parameters)
- **Results reported:** matches or outperforms Llama 2 70B and GPT-3.5 on most of the paper's benchmarks, with about 5× fewer active parameters than Llama 2 70B
- **Routing analysis:** experts do not specialise by topic or domain in any obvious way; consecutive tokens are often routed to the same experts, and the choice tracks syntax (e.g. Python indentation, `self`)

> Figure: Twelve tokens are routed by a random linear router to their top-k of E experts. Edge width is the gate weight, and the bars show how many token copies each expert received. With E = 8 and k = 2, 'cat' goes to FFN2 with weight 0.751 and FFN1 with 0.249. The loads are 2, 4, 3, 2, 2, 1, 4, 6: no balancing loss is applied, so FFN7 gets twice the mean of 3. The right panel counts a Mixtral-shaped model exactly: 46.70B total, 12.88B active (27.6%), against 7.24B for the dense E = 1 model. Setting k = 1 keeps the 46.70B but drops the active count to 7.24B. E = 16, k = 4 gives 91.80B total and 24.16B active. Expert weights are 96.6% of all parameters at E = 8. With the experts split over 2 devices, 13 of the 24 token copies must cross devices (dashed edges). With 4 devices it is 20 of 24. That is the all-to-all traffic expert parallelism pays for.

## Expert parallelism

A model whose experts do not fit on one accelerator is sharded **by expert**, as GShard did. Each of $D$ devices holds $E/D$ experts and also processes its own slice of the batch, with the shared layers replicated or sharded as usual. An MoE layer then runs in four steps: route locally, **all-to-all dispatch** (each token's hidden state is sent to the devices that hold its chosen experts), run the experts on the tokens received, and **all-to-all combine** (send the outputs back and sum them with the gates). With experts spread evenly, a fraction $(D-1)/D$ of the $k$ copies per token leaves its device. The communication is proportional to $k\cdot d$ per token, and it sits on the critical path twice per MoE layer.

Static tensor shapes make this efficient on accelerators, so each expert receives a fixed-size buffer of $C$ tokens per batch, the **expert capacity**. Overflow tokens are dropped from that layer and continue through the residual connection. Load imbalance is not only a quality problem: the busiest expert sets the step time, and a device whose experts are popular stalls the others. Later systems added refinements: DeepSpeed-MoE and Tutel optimise the all-to-all, MegaBlocks avoids dropping with block-sparse kernels, and DeepSeek-V3 limits each token to experts on at most 4 nodes.

## Worked example: one token through a top-2 layer

Four experts with router logits $h=(1.2,\,-0.3,\,2.0,\,0.4)$ for one token. Each expert maps the token to a 2-vector; experts 0 and 2 return $E_0(x)=(1,-1)$ and $E_2(x)=(0.5,\,2)$.

Gate weights (computed in Node)

| expert i | logit hᵢ | softmax over all 4 | top-2? | gate gᵢ (softmax over the 2) |
| --- | --- | --- | --- | --- |
| 0 | 1.2 | 0.2565 | yes | 0.3100 |
| 1 | −0.3 | 0.0572 | no | 0 |
| 2 | 2.0 | 0.5709 | yes | 0.6900 |
| 3 | 0.4 | 0.1153 | no | 0 |

**From logits to the layer output**

1. Top-2 of $h$: experts 2 and 0. Gates $g_2=e^{2.0}/(e^{2.0}+e^{1.2})=1/(1+e^{-0.8})=0.6900$ and $g_0=0.3100$. Renormalising the full-softmax probabilities, $0.5709/(0.5709+0.2565)$, gives the same 0.6900.
2. Only $E_0$ and $E_2$ are evaluated: $y=0.31\,(1,-1)+0.69\,(0.5,2)=(0.6550,\ 1.0699)$. Experts 1 and 3 cost nothing for this token.
3. A Switch-style top-1 layer would pick expert 2 alone and output $p_2E_2(x)=0.5709\,(0.5,2)=(0.2855,\ 1.1419)$, keeping $p_2$ so the router gets a gradient.
4. Counting Mixtral exactly: attention per layer $2\cdot4096\cdot4096+2\cdot4096\cdot1024=41{,}943{,}040$; one SwiGLU expert $3\cdot4096\cdot14336=176{,}160{,}768$; router $4096\cdot8$; two RMSNorms $2\cdot4096$. Per layer with 8 experts: 1,451,270,144.
5. Times 32 layers, plus embeddings and output head $2\cdot32000\cdot4096$ and the final norm: **46,702,792,704** total. Replacing $8$ experts by the $k=2$ used per token: **12,879,925,248** active. Mixtral's "46.7B / 12.9B" are these two numbers.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class SwiGLU(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.w1 = nn.Linear(d, h, bias=False)
        self.w3 = nn.Linear(d, h, bias=False)
        self.w2 = nn.Linear(h, d, bias=False)

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

class MoE(nn.Module):
    def __init__(self, d=64, h=128, n_experts=8, k=2):
        super().__init__()
        self.k = k
        self.gate = nn.Linear(d, n_experts, bias=False)          # router W_g
        self.experts = nn.ModuleList(SwiGLU(d, h) for _ in range(n_experts))

    def forward(self, x):                        # x: (B, T, d)
        B, T, d = x.shape
        x = x.reshape(-1, d)                     # (N, d) with N = B*T tokens
        logits = self.gate(x)                    # (N, E)
        top, idx = logits.topk(self.k, dim=-1)   # (N, k) values and expert ids
        w = top.softmax(dim=-1)                  # gates: softmax over the k chosen logits
        y = torch.zeros_like(x)
        for e, expert in enumerate(self.experts):
            tok, slot = (idx == e).nonzero(as_tuple=True)    # tokens that chose expert e
            if tok.numel():
                y.index_add_(0, tok, w[tok, slot, None] * expert(x[tok]))
        return y.reshape(B, T, d), logits

moe = MoE()
y, logits = moe(torch.randn(2, 10, 64))
print(y.shape, logits.shape)       # torch.Size([2, 10, 64]) torch.Size([20, 8])
total = sum(p.numel() for p in moe.parameters())
active = moe.gate.weight.numel() + moe.k * sum(p.numel() for p in moe.experts[0].parameters())
print(total, active)               # 197120 49664  (8 vs 2 experts of 3*64*128 = 24576, + 512 router)
```

A top-k MoE layer with SwiGLU experts (loop over experts; real systems batch this with grouped matmuls)

> **Why sparsity pays:** For a fixed compute budget, quality in language modelling keeps improving as you add experts, because the extra parameters are almost free in FLOPs. Switch Transformer reported up to 7× faster pre-training than T5-Base at the same FLOPs per token. Mixtral matched a dense model with about 5× more active parameters. The gain shrinks as $E$ grows (scaling laws for routed models, Clark et al. 2022), and it is smallest when you are bound by memory, not compute.

> **Do not compare MoE and dense models on one number:** "An 8x7B model" is not 56B: the attention and embeddings are shared, so it is 46.7B. It is also not a 13B model in deployment, because all 46.7B must sit in memory. Quote both total and active parameters, and remember that fine-tuning an MoE has its own problems. Switch and ST-MoE found that sparse models overfit small downstream tasks more easily. Switch used a higher dropout inside the experts (0.4) during fine-tuning.

> **What the experts learn:** The name suggests topic specialists ("a biology expert"). Analyses of trained Transformer MoEs mostly find something else. Routing depends strongly on the token itself and on local syntax: punctuation, code indentation, particular sub-words. Consecutive tokens often go to the same experts, and the domain of the document matters much less. Mixtral's paper reports this. ST-MoE found clearer specialisation in the encoder than in the decoder. Specialisation is an emergent, token-level division of labour, not a designed one.

**Strengths**

- Parameter count, and so capacity, grows with $E$ at constant FLOPs per token
- Better quality per training FLOP than dense models in language modelling and translation
- Drop-in replacement for the FFN sub-layer; attention, embeddings and the training loop are unchanged
- Expert parallelism adds a new axis for sharding very large models

**Limitations**

- Memory holds all $N_{\text{total}}$ parameters; serving needs as much memory as a dense model of the same total size
- Two all-to-all exchanges per MoE layer; communication can dominate on slow interconnects
- Needs load balancing; unbalanced routers collapse, drop tokens and leave hardware idle
- Training instabilities and overfitting on fine-tuning are more common than in dense models

**Question:** A model has 32 layers with 8 experts each and top-2 routing. Shared parameters are 1.6B and each expert has 176M parameters. Roughly how many parameters does one token use?

- A. 1.6B + 32·8·0.176B ≈ 46.7B
- B. 1.6B + 32·2·0.176B ≈ 12.9B
- C. 46.7B / 8 ≈ 5.8B
- D. 2 × 7B = 14B

**Answer:** B. 1.6B + 32·2·0.176B ≈ 12.9B — Each token uses all shared parameters plus $k=2$ experts in each of the 32 layers: $1.6+64\cdot0.176\approx12.9$B. The 46.7B is the total that must be stored. Dividing by 8 or doubling a 7B model ignores that attention and embeddings are shared.

**Question:** Why does the Switch Transformer scale the chosen expert's output by its router probability p_i(x) instead of renormalising the single selected gate to 1?

- A. To keep the output norm below 1
- B. Because a renormalised top-1 gate is constant 1, so the router would receive no gradient
- C. To implement the load-balancing loss
- D. Because softmax cannot be computed over one element

**Answer:** B. Because a renormalised top-1 gate is constant 1, so the router would receive no gradient — With $k=1$, the softmax over the one selected logit is exactly 1 for every input, so $\partial y/\partial h=0$ and $W_g$ would never learn. Multiplying by $p_i(x)$ from the softmax over **all** experts makes the output depend smoothly on the router logits. With $k\ge2$, the renormalised gates already vary.

**Key points**

- MoE layer: $y=\sum_{i\in\mathrm{TopK}(xW_g,k)}g_i(x)E_i(x)$, gates = softmax over the $k$ selected router logits; only $k$ of $E$ experts run per token.
- $N_{\text{total}}=N_{\text{shared}}+LEN_{\text{expert}}$ vs $N_{\text{active}}=N_{\text{shared}}+LkN_{\text{expert}}$: Mixtral 8x7B has 46.7B total, 12.9B active (top-2 of 8).
- Origins: Jacobs et al. 1991 (competing experts with a softmax gate); Shazeer et al. 2017 made it sparse with top-k gating; GShard and Switch put it in place of the Transformer FFN.
- Expert parallelism shards experts over devices; each MoE layer needs an all-to-all dispatch and an all-to-all combine.
- Compute per token follows active parameters, memory follows total parameters; balancing the router is necessary (see Sparse MoE routing).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
