# Mistral

> Mistral 7B (Jiang et al., 2023) is a 32-layer decoder with grouped-query attention (32 query, 8 KV heads) and 4096-token sliding-window attention served from a rolling-buffer cache. Mixtral 8x7B swaps each FFN for a top-2-of-8 sparse MoE: 46.7B parameters, 12.9B active per token.

Source: https://www.mlfoundry.in/learn/mistral  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**Mistral 7B:** is the 7.24B-parameter open-weight decoder-only language model of Jiang, Sablayrolles, Mensch and colleagues at Mistral AI ("Mistral 7B", arXiv 2023), released under Apache 2.0. Its block is the Llama recipe (pre-norm RMSNorm, RoPE, SwiGLU, no biases) with two inference-driven changes: **grouped-query attention** with 8 key/value heads, and **sliding-window attention** (SWA) in which each token attends only to the previous $W=4096$ positions, so the KV cache can be a fixed-size **rolling buffer**. **Mixtral 8x7B** (Jiang et al., "Mixtral of Experts", 2024) keeps the same attention, drops the window, and replaces every feed-forward layer with a sparse mixture of 8 experts, of which a router picks 2 per token.

In 2023 the reference open models were Llama and Llama 2. Their 7B and 13B sizes used full multi-head attention, so the **KV cache** grew linearly with context and with all 32 heads. For a chat or code assistant, that cache, not the weights, often limits batch size and latency. Mistral AI's first model aimed at the best quality per inference FLOP and per byte of cache at 7B scale. They did not change the training objective or the block much. They changed what attention is allowed to look at.

The headline result was that Mistral 7B beat Llama 2 13B on every benchmark the paper reports. On MMLU it scored 60.1% against 55.6% for Llama 2 13B and 44.4% for Llama 2 7B. It also beat Llama 1 34B on reasoning, mathematics and code. The paper does not disclose the training data or the number of training tokens, so the gain cannot be attributed to architecture alone. Data almost certainly accounts for much of it.

## Sliding-window attention

$$
\mathrm{SWA}(Q,K,V)_i=\sum_{j=i-W+1}^{i}\frac{\exp\!\big(q_i^\top k_j/\sqrt{d_h}\big)}{\sum_{j'=i-W+1}^{i}\exp\!\big(q_i^\top k_{j'}/\sqrt{d_h}\big)}\,v_j,\qquad W=4096
$$

- $i-W<j\le i$: a band mask: causal **and** at most $W$ positions back (the paper writes the range as $i-W$ to $i$; implementations differ by one)
- $q_i,k_j,v_j$: per-head query, key and value after RoPE; with GQA, 4 query heads share each key/value head
- $d_h=128$: head dimension; 32 query heads × 128 = 4096
- $O(TW)$: attention cost for $T$ tokens, against $O(T^2)$ for full **Causal attention**

A single layer sees only $W$ tokens. Stacking layers widens the view. Position $i$ in layer $k$ reads layer $k-1$ at positions down to $i-W+1$, and each of those read layer $k-2$ another $W-1$ positions back. After $L$ layers, information can travel $L(W-1)$ positions. For Mistral 7B that is $32\times4095=131{,}040$, the "about 131K tokens" theoretical span of the paper, although the model was trained with an 8192-token context. This is the receptive-field argument of a deep stack of convolutions, applied to attention. **Longformer** and other **Sparse attention** models used local windows before. Mistral's contribution was to make the window the only attention pattern in a production LLM and design the cache around it.

The span is an upper bound on what *can* reach the top layer, not what does. Information from far back has to be relayed through intermediate positions, layer by layer, and each relay dilutes it. The interactive figure below computes that dilution exactly for uniform attention.

> Figure: The Mistral 7B block and its Mixtral 8x7B variant (click to switch). Both use 32 pre-norm layers at d = 4096 with grouped-query attention: 32 query heads and 8 key/value heads of 128. Mistral 7B restricts attention to a 4096-token window and has one SwiGLU FFN (4096 → 14336 → 4096, 176M parameters per layer). Mixtral has full attention over 32,768 tokens and 8 such FFNs per layer, of which a 4096 → 8 router uses 2 per token. Top right: the band mask for W = 3. Bottom right: the paper's pre-fill scheme with chunk size equal to W = 4. The third chunk ("the dog go to") attends to the cached second chunk inside the window and causally to itself. The first chunk is no longer in the cache.

## Rolling buffer cache

Because no query ever looks further back than $W$ positions, keys and values older than that can be discarded. Mistral stores them in a buffer of exactly $W$ slots per layer and writes position $i$ to slot $i \bmod W$. When $i\ge W$, the write overwrites position $i-W$, which has just fallen out of every future window. Cache memory is then $O(W)$ instead of $O(T)$. The paper reports an 8× reduction at a 32K-token sequence, "without impacting the model quality", which is simply $32768/4096$.

$$
\text{KV bytes}=2\cdot L\cdot n_{kv}\cdot d_h\cdot \min(T,W)\cdot b\;=\;2\cdot32\cdot8\cdot128\cdot\min(T,4096)\cdot2
$$

- $2$: one key and one value vector
- $n_{kv}=8$: key/value heads (see **Grouped-query attention (GQA)**); full MHA would be 32
- $\min(T,W)$: the rolling buffer caps the stored positions at W
- $b=2$: bytes per value in fp16/bf16

The two savings multiply. GQA divides the cache by $32/8=4$ at every length. The window divides it by $T/W$ once $T>W$. Slots are not in temporal order after wrap-around, but attention does not care about order: RoPE has already rotated each key by its absolute position before it was stored, so the dot products are correct wherever a key sits in the buffer.

## Pre-fill and chunking

Generation has two phases (see **KV cache**). **Pre-fill** runs the whole prompt in one forward pass to fill the cache. **Decoding** then adds one token at a time. With a very long prompt, pre-filling in one pass would need attention scores for all $T$ prompt tokens at once, and the cache would have to hold all of them. Mistral instead splits the prompt into chunks of $W$ tokens. Each chunk computes attention over the cache, which holds the previous chunk, plus itself, with the band mask applied to the combined keys. It then writes its own keys and values into the buffer. Each chunk costs a $W\times 2W$ score matrix, and peak memory stays fixed however long the prompt is.

These three pieces fit together. The band mask makes the outputs depend only on the last $W$ positions. The rolling buffer stores exactly those positions. Chunked pre-fill gives the same result as one big masked pass. The code below checks numerically that all three computations agree. The paper also reports that, using FlashAttention and xFormers kernels, SWA was about twice as fast as vanilla attention at a 16K sequence with $W=4096$.

**Configurations (from the two papers)**

- **Mistral 7B:** $L=32$, $d=4096$, 32 heads of $d_h=128$, 8 KV heads, FFN $d_{ff}=14336$ (SwiGLU), $W=4096$, context 8192, vocabulary 32,000, 7,241,732,096 parameters
- **Mixtral 8x7B:** same $L$, $d$, heads, $d_{ff}$ and vocabulary; 8 experts per layer, top-2 routing, no sliding window, context 32,768; 46.7B total and 12.9B active parameters
- **Block:** pre-norm RMSNorm, RoPE, SwiGLU, no biases, untied input embedding and LM head
- **Attention span:** theoretical $L(W-1)\approx131$K tokens for Mistral 7B; the rolling buffer reduces cache memory 8× at 32K tokens
- **Training data:** not disclosed for either model; Mixtral was pre-trained with a 32K context on data with more multilingual text than Mistral 7B
- **Instruct models:** Mistral 7B Instruct was fine-tuned on public instruction data and beat Llama 2 13B Chat on MT-Bench; Mixtral 8x7B Instruct used SFT plus **Direct preference optimization (DPO)** and scored 8.30 on MT-Bench
- **Licence:** Apache 2.0 weights for both

> Figure: A toy sliding-window stack over 48 positions. The top row is one query, and each row below shows how much every position in that layer contributes to it. Every head attends uniformly over its window, the least selective case, and shading is normalised per row. At the default W = 4, K = 4 layers and query 47, the rose line marks the reach K(W − 1) = 12 (13 reachable inputs). Influence is a 4-fold convolution of a box: its mean distance is 6.00 positions, 86.3% of it comes from beyond one window (green line), and both the oldest token and the query's own token weigh (1/4)⁴ = 0.0039. Turning on the residual stream (M = ½(I + A)) halves the mean distance to 3.00, cuts the share beyond one window to 36.6%, and gives the oldest token only 0.0002 while its own token keeps 0.1526. With residuals and K = 8 the reach doubles to 24, but the mean distance is still 6.00. Far tokens are reachable but heavily diluted. The rolling buffer below holds positions 44, 45, 46 and 47 in slots 0 to 3; position 48 will overwrite slot 0 and evict 44. The bars use the real Mistral 7B config. At T = 32,768 the fp16 cache would be 16.0 GiB with 32 KV heads and 4.00 GiB with 8, and the rolling buffer stores 0.50 GiB at any length. At T = 131,072 the first two grow to 64.0 and 16.0 GiB.

## Mixtral 8x7B: a sparse mixture of experts

$$
y=\sum_{i=1}^{8}G(x)_i\,E_i(x),\qquad G(x)=\mathrm{softmax}\big(\mathrm{TopK}(x\,W_g)\big),\qquad E_i(x)=\big(\mathrm{SiLU}(xW_1^{i})\odot xW_3^{i}\big)W_2^{i}
$$

- $W_g\in\mathbb R^{4096\times8}$: router (gating) weights, one logit per expert
- $\mathrm{TopK}$: keeps the $K=2$ largest logits and sets the rest to $-\infty$, so the softmax renormalises over the chosen two
- $E_i$: expert $i$: a full SwiGLU FFN of Mistral 7B's shape, $4096\to14336\to4096$
- $G(x)_i=0$: for the 6 unselected experts, whose FFNs are never computed for this token

The name 8x7B is misleading. Only the FFNs are replicated. Attention, embeddings and norms are shared, so the model has 46.7B parameters, not 56B. Each token passes through 2 of the 8 experts, so its forward pass touches 12.9B parameters, about 27.6% of the total. Compute per token is therefore close to a 13B dense model. Memory is not: all 46.7B weights must be loaded, because different tokens in a batch choose different experts. **Mixture of experts (MoE)** and **Sparse MoE routing** cover the general mechanism, including load-balancing losses. Here the point is what the routing turned out to do.

The Mixtral paper reports that it matches or beats Llama 2 70B and GPT-3.5 on most benchmarks they compare. On MMLU it scores 70.6% against 69.9% for Llama 2 70B. The margins are largest on mathematics, code generation and multilingual tasks. Its routing analysis found no clear topic specialisation: on subsets of The Pile, expert assignments were similar for ArXiv papers, biology and philosophy. Mathematics showed a modest difference. Routing tracked syntax more than topic. Consecutive tokens, and tokens such as `self` in Python or indentation, were often sent to the same experts.

## Worked example: parameters and cache

Exact parameter counts from the published configs (computed in Python)

| component | Mistral 7B | Mixtral 8x7B |
| --- | --- | --- |
| attention per layer: 4096·4096·2 + 4096·1024·2 | 41,943,040 | 41,943,040 |
| FFN per layer: 3·4096·14336 per expert | 176,160,768 | 8 × = 1,409,286,144 |
| router per layer: 4096·8 | — | 32,768 |
| two RMSNorms per layer | 8,192 | 8,192 |
| one layer | 218,112,000 | 1,451,270,144 |
| × 32 layers | 6,979,584,000 | 46,440,644,608 |
| embedding + LM head + final norm | 262,148,096 | 262,148,096 |
| total | 7,241,732,096 | 46,702,792,704 |
| active per token (2 experts) | 7,241,732,096 | 12,879,925,248 |

**Cache memory for one 32,768-token sequence in fp16**

1. Per token and layer, the cache stores a key and a value for each of 8 KV heads of 128 dimensions: $2\cdot8\cdot128=2048$ numbers, or 4096 bytes in fp16.
2. Across 32 layers that is $131{,}072$ bytes, 128 KiB per token. With full MHA (32 KV heads) it would be 512 KiB.
3. Without the window, 32,768 tokens need $32768\cdot128\text{ KiB}=4$ GiB, and with MHA 16 GiB.
4. With the rolling buffer only $\min(T,4096)=4096$ positions are stored: $4096\cdot128\text{ KiB}=0.5$ GiB. That is $32768/4096=8\times$ smaller, the paper's figure.
5. Position 32,767 goes to slot $32767 \bmod 4096=4095$. The next token, 32,768, goes to slot 0, overwriting position 28,672, which is exactly $W$ positions back and outside every future window.
6. Mixtral has no window, so at its full 32,768-token context it needs the full 4 GiB per sequence. The cache does not depend on the number of experts.

```python
import numpy as np

rng = np.random.default_rng(0)
T, d, W = 11, 8, 4                       # 11 tokens, head dim 8, window 4
q, k, v = (rng.standard_normal((T, d)) for _ in range(3))

def softmax(s):
    s = s - s.max(-1, keepdims=True)
    e = np.exp(s)
    return e / e.sum(-1, keepdims=True)

# 1) reference: full attention with the sliding-window mask  i - W < j <= i
i, j = np.arange(T)[:, None], np.arange(T)[None, :]
band = (j <= i) & (j > i - W)
s = q @ k.T / np.sqrt(d)
ref = softmax(np.where(band, s, -np.inf)) @ v

# 2) token-by-token decoding with a rolling buffer of W slots
K_buf, V_buf = np.zeros((W, d)), np.zeros((W, d))
pos = np.full(W, -1)                     # which position each slot holds
dec = np.zeros((T, d))
for t in range(T):
    slot = t % W                         # position t overwrites position t - W
    K_buf[slot], V_buf[slot], pos[slot] = k[t], v[t], t
    live = pos >= 0
    a = softmax(q[t] @ K_buf[live].T / np.sqrt(d))
    dec[t] = a @ V_buf[live]

# 3) chunked pre-fill: chunks of W queries against [cached previous chunk, current chunk]
pre = np.zeros((T, d))
for c0 in range(0, T, W):
    c1 = min(c0 + W, T)
    kv_lo = max(0, c0 - W)               # the cache holds at most the previous W positions
    keys = np.arange(kv_lo, c1)
    qi = np.arange(c0, c1)[:, None]
    m = (keys[None, :] <= qi) & (keys[None, :] > qi - W)
    sc = q[c0:c1] @ k[keys].T / np.sqrt(d)
    pre[c0:c1] = softmax(np.where(m, sc, -np.inf)) @ v[keys]

print(np.abs(ref - dec).max() < 1e-12, np.abs(ref - pre).max() < 1e-12)   # True True
print("slots after t = 10:", pos)                                          # [ 8  9 10  7]

# Mixtral-style sparse MoE layer: top-2 of 8 experts, softmax over the two kept logits
n_exp, d_model, d_ff = 8, 16, 32
Wg = rng.standard_normal((d_model, n_exp)) * 0.3
experts = [(rng.standard_normal((d_model, d_ff)) * 0.1,     # W1 (gate)
            rng.standard_normal((d_model, d_ff)) * 0.1,     # W3 (up)
            rng.standard_normal((d_ff, d_model)) * 0.1)     # W2 (down)
           for _ in range(n_exp)]
silu = lambda z: z / (1 + np.exp(-z))

def moe(x):                              # x: (d_model,)
    logits = x @ Wg
    top = np.argsort(logits)[-2:][::-1]  # the two highest-scoring experts
    g = softmax(logits[top])             # renormalise over the kept two only
    y = sum(gi * (silu(x @ W1) * (x @ W3)) @ W2
            for gi, (W1, W3, W2) in ((g[n], experts[e]) for n, e in enumerate(top)))
    return y, top, g

y, top, g = moe(rng.standard_normal(d_model))
print("experts", top, "weights", g.round(3), "output", y.shape)  # experts [3 7] weights [0.777 0.223] output (16,)
```

Sliding-window attention three ways (masked full pass, rolling-buffer decoding, chunked pre-fill), checked equal, plus a Mixtral-style top-2 router (numpy; run as shown)

> **Theoretical span is not usable context:** $L(W-1)\approx131$K says what *can* influence the last token, not what the model uses. Information from far back must hop through intermediate positions, layer by layer, and each hop mixes it with other tokens. In the figure, even with uniform weights the residual stream keeps most influence within a couple of windows, and the oldest reachable token carries a vanishing share. A sliding-window model cannot quote a passage it saw 100K tokens ago. If the task needs exact long-range retrieval, it needs full attention or interleaved global layers. Mistral's own later 7B release (v0.2) dropped the window in favour of a full 32K context.

> **Why slot order does not matter:** After wrap-around the buffer holds positions in rotated order, e.g. $[8,9,10,7]$. Attention is a weighted sum over a *set* of keys. Position enters only through RoPE, which was applied to each key before it was written. So the buffer needs no reordering and no stored positions, apart from tracking which slots are filled at the start. That is why the cache can be a fixed-size array rather than a growing list.

> **Active parameters set FLOPs, total parameters set memory:** Mixtral's 12.9B active parameters make each token cost roughly as much as a 13B dense model. All 46.7B parameters (about 93 GB in bf16) must still be in accelerator memory, and at small batch sizes decoding is limited by weight-read bandwidth, so the speed advantage over a 47B dense model is smaller than the FLOP ratio suggests. Throughput is highest when batches are large enough that every expert is busy.

**Strengths**

- SWA makes attention $O(TW)$ and the KV cache $O(W)$: 8× less cache memory at 32K tokens
- GQA cuts the cache another 4× with little quality loss; the two savings multiply
- Chunked pre-fill bounds peak memory for arbitrarily long prompts
- Mixtral gives roughly 70B-dense quality at about 13B-dense FLOPs per token
- Apache 2.0 weights, and a block close enough to Llama that most tooling supports it

**Limitations**

- Sliding windows lose exact access to anything more than W tokens back; the span is only theoretical
- Requires kernels and cache code that understand the band mask and the ring buffer
- Mixtral must hold all 46.7B parameters in memory, and its MoE layers complicate batching and parallelism
- Training data and token counts are undisclosed, so architectural gains cannot be separated from data gains

**Question:** Mistral 7B is decoding token 10,000 with W = 4096. Which keys does layer 1 attend to, and where is the key of position 10,000 written?

- A. Positions 0 to 10,000; it is appended to the end of the cache
- B. Positions 5905 to 10,000; it overwrites slot 10,000 mod 4096 = 1808, which held position 5904
- C. Positions 5905 to 10,000; it is written to slot 0 after shifting the buffer
- D. Only position 10,000, because the window applies per layer

**Answer:** B. Positions 5905 to 10,000; it overwrites slot 10,000 mod 4096 = 1808, which held position 5904 — The band is $i-W<j\le i$, i.e. $5905\dots10000$, 4096 keys. The rolling buffer writes position $i$ to slot $i \bmod W=10000-2\cdot4096=1808$. That slot held position $10000-4096=5904$, which has just left the window. Nothing is shifted.

**Question:** Why does Mixtral 8x7B have 46.7B parameters rather than 8 × 7.24B ≈ 58B?

- A. The experts are pruned to 75% of their size
- B. Only the feed-forward layers are replicated; attention, embeddings and norms are shared across experts
- C. Each expert uses a smaller hidden size of 7168
- D. The LM head is tied to the embedding

**Answer:** B. Only the feed-forward layers are replicated; attention, embeddings and norms are shared across experts — Each of the 32 layers has 8 SwiGLU experts of 176.2M parameters but one shared attention block of 41.9M. The embedding and head (262M) appear once. The total is 46,702,792,704. Two experts per token gives 12,879,925,248 active.

**Key points**

- Mistral 7B: 32 layers, $d=4096$, 32 query and 8 KV heads, SwiGLU 14336, 7.24B parameters, sliding window $W=4096$ with theoretical span $L(W-1)\approx131$K.
- Rolling buffer cache: position $i$ goes to slot $i \bmod W$, so the cache is fixed at $W$ entries. That is 8× smaller at 32K tokens, on top of GQA's 4×.
- Chunked pre-fill processes long prompts W tokens at a time against the cached previous chunk, giving identical outputs with bounded memory.
- Mixtral 8x7B: each FFN becomes 8 SwiGLU experts with $G(x)=\mathrm{softmax}(\mathrm{TopK}(xW_g))$, $K=2$; 46.7B total, 12.9B active, full 32K attention.
- Reachable is not used: influence from far tokens is diluted at every hop, and experts specialise by syntax more than by topic.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
