Mistral

Mistral 7B (Jiang et al., 2023) is a 32-layer decoder with grouped-query attention (32 query, 8 KV heads) and 4096-token sliding-window attention served from a rolling-buffer cache. Mixtral 8x7B swaps each FFN for a top-2-of-8 sparse MoE: 46.7B parameters, 12.9B active per token.

15 min read1462 wordsadvanced0 viewsupdated 2026-10-13

Definition

Mistral 7B is the 7.24B-parameter open-weight decoder-only language model of Jiang, Sablayrolles, Mensch and colleagues at Mistral AI ("Mistral 7B", arXiv 2023), released under Apache 2.0. Its block is the Llama recipe (pre-norm RMSNorm, RoPE, SwiGLU, no biases) with two inference-driven changes: grouped-query attention with 8 key/value heads, and sliding-window attention (SWA) in which each token attends only to the previous W=4096W=4096 positions, so the KV cache can be a fixed-size rolling buffer. Mixtral 8x7B (Jiang et al., "Mixtral of Experts", 2024) keeps the same attention, drops the window, and replaces every feed-forward layer with a sparse mixture of 8 experts, of which a router picks 2 per token.

In 2023 the reference open models were Llama and Llama 2. Their 7B and 13B sizes used full multi-head attention, so the KV cache grew linearly with context and with all 32 heads. For a chat or code assistant, that cache, not the weights, often limits batch size and latency. Mistral AI's first model aimed at the best quality per inference FLOP and per byte of cache at 7B scale. They did not change the training objective or the block much. They changed what attention is allowed to look at.

The headline result was that Mistral 7B beat Llama 2 13B on every benchmark the paper reports. On MMLU it scored 60.1% against 55.6% for Llama 2 13B and 44.4% for Llama 2 7B. It also beat Llama 1 34B on reasoning, mathematics and code. The paper does not disclose the training data or the number of training tokens, so the gain cannot be attributed to architecture alone. Data almost certainly accounts for much of it.

Sliding-window attention

SWA(Q,K,V)i=∑j=i−W+1iexp⁡ ⁣(qi⊤kj/dh)∑j′=i−W+1iexp⁡ ⁣(qi⊤kj′/dh) vj,W=4096\mathrm{SWA}(Q,K,V)_i=\sum_{j=i-W+1}^{i}\frac{\exp\!\big(q_i^\top k_j/\sqrt{d_h}\big)}{\sum_{j'=i-W+1}^{i}\exp\!\big(q_i^\top k_{j'}/\sqrt{d_h}\big)}\,v_j,\qquad W=4096
i−W<j≤ii-W<j\le i
a band mask: causal and at most WW positions back (the paper writes the range as i−Wi-W to ii; implementations differ by one)
qi,kj,vjq_i,k_j,v_j
per-head query, key and value after RoPE; with GQA, 4 query heads share each key/value head
dh=128d_h=128
head dimension; 32 query heads × 128 = 4096
O(TW)O(TW)
attention cost for TT tokens, against O(T2)O(T^2) for full Causal attention

A single layer sees only WW tokens. Stacking layers widens the view. Position ii in layer kk reads layer k−1k-1 at positions down to i−W+1i-W+1, and each of those read layer k−2k-2 another W−1W-1 positions back. After LL layers, information can travel L(W−1)L(W-1) positions. For Mistral 7B that is 32×4095=131,04032\times4095=131{,}040, the "about 131K tokens" theoretical span of the paper, although the model was trained with an 8192-token context. This is the receptive-field argument of a deep stack of convolutions, applied to attention. Longformer and other Sparse attention models used local windows before. Mistral's contribution was to make the window the only attention pattern in a production LLM and design the cache around it.

The span is an upper bound on what can reach the top layer, not what does. Information from far back has to be relayed through intermediate positions, layer by layer, and each relay dilutes it. The interactive figure below computes that dilution exactly for uniform attention.

Mistral 7BMixtral 8x7B(click)RMSNorm → LM head4096 × 32,000, untied× 32 layers, d = 4096SwiGLU feed-forward4096 → 14336 (×2 gates) → 4096176M parameters per layerRMSNormpre-normsliding-window GQA32 Q heads, 8 KV heads, d_h = 128RoPE, window W = 4096, cache WRMSNorm++token embedding32,000 × 40967.24B parameters, all activecontext 8192 · span 32 × 4095 ≈ 131K tokenssliding-window mask, W = 3Thecatsatonthematandsawrow i seesi − W < j ≤ islot = i mod Win a W-entryrolling cachepre-fill chunk 3 (chunk = W = 4)pastcachecurrent chunkthedoggotoThecatsatonthematandsawthedoggotolong prompts are fed W tokens at a time;each chunk sees the cached previous chunk(windowed) and itself (causal), thenoverwrites the cache with its own K, Vattention per chunk: W × 2W, never W × T
fig.The Mistral 7B block and its Mixtral 8x7B variant (click to switch). Both use 32 pre-norm layers at d = 4096 with grouped-query attention: 32 query heads and 8 key/value heads of 128. Mistral 7B restricts attention to a 4096-token window and has one SwiGLU FFN (4096 → 14336 → 4096, 176M parameters per layer). Mixtral has full attention over 32,768 tokens and 8 such FFNs per layer, of which a 4096 → 8 router uses 2 per token. Top right: the band mask for W = 3. Bottom right: the paper's pre-fill scheme with chunk size equal to W = 4. The third chunk ("the dog go to") attends to the cached second chunk inside the window and causally to itself. The first chunk is no longer in the cache.

Rolling buffer cache

Because no query ever looks further back than WW positions, keys and values older than that can be discarded. Mistral stores them in a buffer of exactly WW slots per layer and writes position ii to slot i mod Wi \bmod W. When i≥Wi\ge W, the write overwrites position i−Wi-W, which has just fallen out of every future window. Cache memory is then O(W)O(W) instead of O(T)O(T). The paper reports an 8× reduction at a 32K-token sequence, "without impacting the model quality", which is simply 32768/409632768/4096.

KV bytes=2⋅L⋅nkv⋅dh⋅min⁡(T,W)⋅b  =  2⋅32⋅8⋅128⋅min⁡(T,4096)⋅2\text{KV bytes}=2\cdot L\cdot n_{kv}\cdot d_h\cdot \min(T,W)\cdot b\;=\;2\cdot32\cdot8\cdot128\cdot\min(T,4096)\cdot2
22
one key and one value vector
nkv=8n_{kv}=8
key/value heads (see Grouped-query attention (GQA)); full MHA would be 32
min⁡(T,W)\min(T,W)
the rolling buffer caps the stored positions at W
b=2b=2
bytes per value in fp16/bf16

The two savings multiply. GQA divides the cache by 32/8=432/8=4 at every length. The window divides it by T/WT/W once T>WT>W. Slots are not in temporal order after wrap-around, but attention does not care about order: RoPE has already rotated each key by its absolute position before it was stored, so the dot products are correct wherever a key sits in the buffer.

Pre-fill and chunking

Generation has two phases (see KV cache). Pre-fill runs the whole prompt in one forward pass to fill the cache. Decoding then adds one token at a time. With a very long prompt, pre-filling in one pass would need attention scores for all TT prompt tokens at once, and the cache would have to hold all of them. Mistral instead splits the prompt into chunks of WW tokens. Each chunk computes attention over the cache, which holds the previous chunk, plus itself, with the band mask applied to the combined keys. It then writes its own keys and values into the buffer. Each chunk costs a W×2WW\times 2W score matrix, and peak memory stays fixed however long the prompt is.

These three pieces fit together. The band mask makes the outputs depend only on the last WW positions. The rolling buffer stores exactly those positions. Chunked pre-fill gives the same result as one big masked pass. The code below checks numerically that all three computations agree. The paper also reports that, using FlashAttention and xFormers kernels, SWA was about twice as fast as vanilla attention at a 16K sequence with W=4096W=4096.

Configurations (from the two papers)

Mistral 7B
L=32L=32, d=4096d=4096, 32 heads of dh=128d_h=128, 8 KV heads, FFN dff=14336d_{ff}=14336 (SwiGLU), W=4096W=4096, context 8192, vocabulary 32,000, 7,241,732,096 parameters
Mixtral 8x7B
same LL, dd, heads, dffd_{ff} and vocabulary; 8 experts per layer, top-2 routing, no sliding window, context 32,768; 46.7B total and 12.9B active parameters
Block
pre-norm RMSNorm, RoPE, SwiGLU, no biases, untied input embedding and LM head
Attention span
theoretical L(W−1)≈131L(W-1)\approx131K tokens for Mistral 7B; the rolling buffer reduces cache memory 8× at 32K tokens
Training data
not disclosed for either model; Mixtral was pre-trained with a 32K context on data with more multilingual text than Mistral 7B
Instruct models
Mistral 7B Instruct was fine-tuned on public instruction data and beat Llama 2 13B Chat on MT-Bench; Mixtral 8x7B Instruct used SFT plus Direct preference optimization (DPO) and scored 8.30 on MT-Bench
Licence
Apache 2.0 weights for both
layer ℓ, position i attends to i − W < j ≤ i · reach after K layers = K·(W − 1) · uniform AL4L3L2L1input081624324047reach K(W − 1) = 12 · reachable inputs 13 · mean distance 6.00share from beyond one window 86.3% · oldest token 0.0039 · own token 0.0039Mistral 7B: 32 layers × (4096 − 1) = 131,040 positions of span from a 4,096-entry cacherolling buffer, 4 slots, after writing position 47: position p lives in slot p mod 4s044s145s246s347just written slot 3 · position 48 goes to slot 0, evicting position 44Mistral 7B KV cache, fp16, one sequence of T = 32,768 tokensMHA, 32 KV heads16.0 GiBGQA, 8 KV heads4.00 GiBGQA + rolling buffer0.50 GiB
attention weights
fig.A toy sliding-window stack over 48 positions. The top row is one query, and each row below shows how much every position in that layer contributes to it. Every head attends uniformly over its window, the least selective case, and shading is normalised per row. At the default W = 4, K = 4 layers and query 47, the rose line marks the reach K(W − 1) = 12 (13 reachable inputs). Influence is a 4-fold convolution of a box: its mean distance is 6.00 positions, 86.3% of it comes from beyond one window (green line), and both the oldest token and the query's own token weigh (1/4)⁴ = 0.0039. Turning on the residual stream (M = ½(I + A)) halves the mean distance to 3.00, cuts the share beyond one window to 36.6%, and gives the oldest token only 0.0002 while its own token keeps 0.1526. With residuals and K = 8 the reach doubles to 24, but the mean distance is still 6.00. Far tokens are reachable but heavily diluted. The rolling buffer below holds positions 44, 45, 46 and 47 in slots 0 to 3; position 48 will overwrite slot 0 and evict 44. The bars use the real Mistral 7B config. At T = 32,768 the fp16 cache would be 16.0 GiB with 32 KV heads and 4.00 GiB with 8, and the rolling buffer stores 0.50 GiB at any length. At T = 131,072 the first two grow to 64.0 and 16.0 GiB.

Mixtral 8x7B: a sparse mixture of experts

y=∑i=18G(x)i Ei(x),G(x)=softmax(TopK(x Wg)),Ei(x)=(SiLU(xW1i)⊙xW3i)W2iy=\sum_{i=1}^{8}G(x)_i\,E_i(x),\qquad G(x)=\mathrm{softmax}\big(\mathrm{TopK}(x\,W_g)\big),\qquad E_i(x)=\big(\mathrm{SiLU}(xW_1^{i})\odot xW_3^{i}\big)W_2^{i}
Wg∈R4096×8W_g\in\mathbb R^{4096\times8}
router (gating) weights, one logit per expert
TopK\mathrm{TopK}
keeps the K=2K=2 largest logits and sets the rest to −∞-\infty, so the softmax renormalises over the chosen two
EiE_i
expert ii: a full SwiGLU FFN of Mistral 7B's shape, 4096→14336→40964096\to14336\to4096
G(x)i=0G(x)_i=0
for the 6 unselected experts, whose FFNs are never computed for this token

The name 8x7B is misleading. Only the FFNs are replicated. Attention, embeddings and norms are shared, so the model has 46.7B parameters, not 56B. Each token passes through 2 of the 8 experts, so its forward pass touches 12.9B parameters, about 27.6% of the total. Compute per token is therefore close to a 13B dense model. Memory is not: all 46.7B weights must be loaded, because different tokens in a batch choose different experts. Mixture of experts (MoE) and Sparse MoE routing cover the general mechanism, including load-balancing losses. Here the point is what the routing turned out to do.

The Mixtral paper reports that it matches or beats Llama 2 70B and GPT-3.5 on most benchmarks they compare. On MMLU it scores 70.6% against 69.9% for Llama 2 70B. The margins are largest on mathematics, code generation and multilingual tasks. Its routing analysis found no clear topic specialisation: on subsets of The Pile, expert assignments were similar for ArXiv papers, biology and philosophy. Mathematics showed a modest difference. Routing tracked syntax more than topic. Consecutive tokens, and tokens such as self in Python or indentation, were often sent to the same experts.

Worked example: parameters and cache

componentMistral 7BMixtral 8x7B
attention per layer: 4096·4096·2 + 4096·1024·241,943,04041,943,040
FFN per layer: 3·4096·14336 per expert176,160,7688 × = 1,409,286,144
router per layer: 4096·8—32,768
two RMSNorms per layer8,1928,192
one layer218,112,0001,451,270,144
× 32 layers6,979,584,00046,440,644,608
embedding + LM head + final norm262,148,096262,148,096
total7,241,732,09646,702,792,704
active per token (2 experts)7,241,732,09612,879,925,248
Exact parameter counts from the published configs (computed in Python)

Cache memory for one 32,768-token sequence in fp16

  1. 1Per token and layer, the cache stores a key and a value for each of 8 KV heads of 128 dimensions: 2⋅8⋅128=20482\cdot8\cdot128=2048 numbers, or 4096 bytes in fp16.
  2. 2Across 32 layers that is 131,072131{,}072 bytes, 128 KiB per token. With full MHA (32 KV heads) it would be 512 KiB.
  3. 3Without the window, 32,768 tokens need 32768⋅128 KiB=432768\cdot128\text{ KiB}=4 GiB, and with MHA 16 GiB.
  4. 4With the rolling buffer only min⁡(T,4096)=4096\min(T,4096)=4096 positions are stored: 4096⋅128 KiB=0.54096\cdot128\text{ KiB}=0.5 GiB. That is 32768/4096=8×32768/4096=8\times smaller, the paper's figure.
  5. 5Position 32,767 goes to slot 32767 mod 4096=409532767 \bmod 4096=4095. The next token, 32,768, goes to slot 0, overwriting position 28,672, which is exactly WW positions back and outside every future window.
  6. 6Mixtral has no window, so at its full 32,768-token context it needs the full 4 GiB per sequence. The cache does not depend on the number of experts.
python
1import numpy as np
2
3rng = np.random.default_rng(0)
4T, d, W = 11, 8, 4 # 11 tokens, head dim 8, window 4
5q, k, v = (rng.standard_normal((T, d)) for _ in range(3))
6
7def softmax(s):
8 s = s - s.max(-1, keepdims=True)
9 e = np.exp(s)
10 return e / e.sum(-1, keepdims=True)
11
12# 1) reference: full attention with the sliding-window mask i - W < j <= i
13i, j = np.arange(T)[:, None], np.arange(T)[None, :]
14band = (j <= i) & (j > i - W)
15s = q @ k.T / np.sqrt(d)
16ref = softmax(np.where(band, s, -np.inf)) @ v
17
18# 2) token-by-token decoding with a rolling buffer of W slots
19K_buf, V_buf = np.zeros((W, d)), np.zeros((W, d))
20pos = np.full(W, -1) # which position each slot holds
21dec = np.zeros((T, d))
22for t in range(T):
23 slot = t % W # position t overwrites position t - W
24 K_buf[slot], V_buf[slot], pos[slot] = k[t], v[t], t
25 live = pos >= 0
26 a = softmax(q[t] @ K_buf[live].T / np.sqrt(d))
27 dec[t] = a @ V_buf[live]
28
29# 3) chunked pre-fill: chunks of W queries against [cached previous chunk, current chunk]
30pre = np.zeros((T, d))
31for c0 in range(0, T, W):
32 c1 = min(c0 + W, T)
33 kv_lo = max(0, c0 - W) # the cache holds at most the previous W positions
34 keys = np.arange(kv_lo, c1)
35 qi = np.arange(c0, c1)[:, None]
36 m = (keys[None, :] <= qi) & (keys[None, :] > qi - W)
37 sc = q[c0:c1] @ k[keys].T / np.sqrt(d)
38 pre[c0:c1] = softmax(np.where(m, sc, -np.inf)) @ v[keys]
39
40print(np.abs(ref - dec).max() < 1e-12, np.abs(ref - pre).max() < 1e-12) # True True
41print("slots after t = 10:", pos) # [ 8 9 10 7]
42
43# Mixtral-style sparse MoE layer: top-2 of 8 experts, softmax over the two kept logits
44n_exp, d_model, d_ff = 8, 16, 32
45Wg = rng.standard_normal((d_model, n_exp)) * 0.3
46experts = [(rng.standard_normal((d_model, d_ff)) * 0.1, # W1 (gate)
47 rng.standard_normal((d_model, d_ff)) * 0.1, # W3 (up)
48 rng.standard_normal((d_ff, d_model)) * 0.1) # W2 (down)
49 for _ in range(n_exp)]
50silu = lambda z: z / (1 + np.exp(-z))
51
52def moe(x): # x: (d_model,)
53 logits = x @ Wg
54 top = np.argsort(logits)[-2:][::-1] # the two highest-scoring experts
55 g = softmax(logits[top]) # renormalise over the kept two only
56 y = sum(gi * (silu(x @ W1) * (x @ W3)) @ W2
57 for gi, (W1, W3, W2) in ((g[n], experts[e]) for n, e in enumerate(top)))
58 return y, top, g
59
60y, top, g = moe(rng.standard_normal(d_model))
61print("experts", top, "weights", g.round(3), "output", y.shape) # experts [3 7] weights [0.777 0.223] output (16,)
Sliding-window attention three ways (masked full pass, rolling-buffer decoding, chunked pre-fill), checked equal, plus a Mixtral-style top-2 router (numpy; run as shown)

Use when

  • +SWA makes attention O(TW)O(TW) and the KV cache O(W)O(W): 8× less cache memory at 32K tokens
  • +GQA cuts the cache another 4× with little quality loss; the two savings multiply
  • +Chunked pre-fill bounds peak memory for arbitrarily long prompts
  • +Mixtral gives roughly 70B-dense quality at about 13B-dense FLOPs per token
  • +Apache 2.0 weights, and a block close enough to Llama that most tooling supports it

Avoid when

  • −Sliding windows lose exact access to anything more than W tokens back; the span is only theoretical
  • −Requires kernels and cache code that understand the band mask and the ring buffer
  • −Mixtral must hold all 46.7B parameters in memory, and its MoE layers complicate batching and parallelism
  • −Training data and token counts are undisclosed, so architectural gains cannot be separated from data gains

Check yourself

Mistral 7B is decoding token 10,000 with W = 4096. Which keys does layer 1 attend to, and where is the key of position 10,000 written?

Check yourself

Why does Mixtral 8x7B have 46.7B parameters rather than 8 × 7.24B ≈ 58B?

Remember

  • ✓Mistral 7B: 32 layers, d=4096d=4096, 32 query and 8 KV heads, SwiGLU 14336, 7.24B parameters, sliding window W=4096W=4096 with theoretical span L(W−1)≈131L(W-1)\approx131K.
  • ✓Rolling buffer cache: position ii goes to slot i mod Wi \bmod W, so the cache is fixed at WW entries. That is 8× smaller at 32K tokens, on top of GQA's 4×.
  • ✓Chunked pre-fill processes long prompts W tokens at a time against the cached previous chunk, giving identical outputs with bounded memory.
  • ✓Mixtral 8x7B: each FFN becomes 8 SwiGLU experts with G(x)=softmax(TopK(xWg))G(x)=\mathrm{softmax}(\mathrm{TopK}(xW_g)), K=2K=2; 46.7B total, 12.9B active, full 32K attention.
  • ✓Reachable is not used: influence from far tokens is diluted at every hop, and experts specialise by syntax more than by topic.
← back to Deep Learning roadmap