Llama

Meta's Llama recipe is pre-RMSNorm + SwiGLU + RoPE + no biases, trained far past compute-optimal: LLaMA (2023, 7–65B, 1.0–1.4T tokens), Llama 2 (2T tokens, 4K context, GQA in 34B/70B), Llama 3/3.1 (128K tiktoken-based vocab, 15T+ tokens, 8K → 128K context, 405B).

15 min read1388 wordscore1 viewsupdated 2026-10-13

Definition

Llama is Meta's family of open-weight decoder-only language models. The series starts with Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (February 2023), continues with Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (July 2023), and reaches Llama 3 (April 2024) and Llama 3.1 (July 2024), described in the Llama Team's "The Llama 3 Herd of Models" (2024). Every generation uses the same block. It is a Decoder-only transformer with RMSNorm pre-normalisation, a SwiGLU MLP, Rotary positional embeddings (RoPE), no bias terms and an untied output head. Later generations add Grouped-query attention (GQA), a larger tokenizer, much more data and longer context.

By 2022 the strongest models (GPT-3, Gopher, PaLM, Chinchilla) were either closed or only reachable through an API. The LLaMA paper made two arguments. First, a competitive model can be trained only on publicly available data. Second, the scaling laws of Hoffmann et al. ("Chinchilla") optimise the training budget, but a deployed model's cost is set by inference. A smaller model trained for longer is cheaper to serve at the same quality. Hoffmann et al. recommend training a 10B model on 200B tokens. LLaMA trained its 7B model on 1T tokens and found that its performance was still improving.

The result was LLaMA-13B outperforming GPT-3 (175B) on most of the paper's benchmarks, and LLaMA-65B competitive with Chinchilla-70B and PaLM-540B. The weights were released to researchers under a non-commercial licence, and they leaked within a week. Llama 2 followed with a licence that allows commercial use with conditions, plus chat models aligned with RLHF. Llama 3.1 brought a 405B dense model with open weights.

The block: four changes to GPT-3

Llama 3.1 8B blockuntied head W_U4096 × 128,256 → logitsRMSNorm (final)× 32 layersSwiGLU MLPW₁ (gate), W₃ (up): 4096 → 14336W₂ (SiLU(W₁x) ⊙ W₃x): 14336 → 4096RMSNorm x / √(mean(x²) + ε) · gcausal attention, 4 q heads per kv headsoftmax(qkᵀ/√128) v, then W_o4096 → 4096, no biases anywhereRoPE on q, k · base θ = 500,000W_q 4096 → 32 heads × 128W_k, W_v 4096 → 8 kv heads × 128RMSNorm++token embedding 128,256 × 4096no position table8,030,261,248 parameters · KV cache 128 KiB/token (bf16)pre-norm residual: x ← x + f(RMSNorm(x))four releases (hover)LLaMA · Feb 20237B 13B 33B 65B · 1.0T / 1.4T tokenscontext 2K · SentencePiece BPE 32K · MHALlama 2 · Jul 20237B 13B 70B (34B unreleased) · 2T tokenscontext 4K · same 32K vocab · GQA in 34B/70BLlama 3 · Apr 20248B 70B · 15T+ tokens · context 8K128K vocab (tiktoken-based) · GQA all sizesLlama 3.1 · Jul 20248B 70B 405B · context 128K405B: 126 layers, d 16384, 128 q / 8 kvhover a release to see what it changedd_ff = round_up(m · ⌊8d/3⌋, k)7B: m = 1, k = 256 → 110083.1 8B: m = 1.3, k = 1024 → 14336405B: m = 1.2, k = 4096 → 53248
fig.One Llama 3.1 8B block with its real shapes: 32 query heads and 8 key/value heads of size 128, RoPE with base 500,000 applied to q and k, a SwiGLU MLP 4096 → 14336 → 4096, RMSNorm before each sub-layer and once at the end, a 128,256-token embedding and a separate output head. There are no biases and no position table. Hover a release on the right to see which parts it introduced: LLaMA brought RMSNorm, SwiGLU and RoPE; Llama 2 brought GQA (in 34B and 70B only); Llama 3 brought the new tokenizer, the larger RoPE base and GQA at every size; Llama 3.1 brought RoPE frequency scaling to 128K tokens. The d_ff rule at the bottom right reproduces every released MLP width.

The LLaMA paper lists its architecture as GPT-3 plus three borrowed ideas. Each came from an earlier model, and each is now standard. The fourth change, dropping biases, is in the released code rather than the paper's list.

RMSNorm(x)=g⊙x1d∑ixi2+ϵ,SwiGLU(x)=W2(SiLU(W1x)⊙W3x),SiLU(z)=z σ(z)\mathrm{RMSNorm}(x)=g\odot\frac{x}{\sqrt{\tfrac1d\sum_i x_i^2+\epsilon}},\qquad \mathrm{SwiGLU}(x)=W_2\big(\mathrm{SiLU}(W_1x)\odot W_3x\big),\qquad \mathrm{SiLU}(z)=z\,\sigma(z)
g∈Rdg\in\mathbb R^d
learned gain; there is no bias and no mean subtraction, so a norm has dd parameters instead of LayerNorm's 2d2d
ϵ\epsilon
10−610^{-6} in LLaMA, 10−510^{-5} in Llama 2 and Llama 3
W1,W3∈Rdff×dW_1,W_3\in\mathbb R^{d_{ff}\times d}
"gate" and "up" projections; W2∈Rd×dffW_2\in\mathbb R^{d\times d_{ff}} is "down"
dff≈23⋅4dd_{ff}\approx\tfrac23\cdot4d
three matrices of 83d2\tfrac83d^2 cost the same 8d28d^2 as a two-matrix GELU MLP of width 4d4d

Pre-normalisation with RMSNorm (Zhang and Sennrich, "Root Mean Square Layer Normalization", 2019). GPT-2 and GPT-3 had already moved the norm in front of each sub-layer (see GPT architecture). RMSNorm also drops LayerNorm's re-centring. The authors found that re-scaling alone gives most of the stability, at lower cost, because there is no mean to compute and subtract. The parameter saving is negligible, dd per norm. The point is a simpler, cheaper kernel.

SwiGLU (Shazeer, "GLU Variants Improve Transformer", 2020; used in PaLM). A gated linear unit multiplies one projection by a nonlinear function of another, so the MLP learns which features to pass, not just how to threshold them. Shazeer found SwiGLU and GeGLU gave the best perplexity among the variants he tested at matched parameters and compute. LLaMA follows PaLM in using 23⋅4d\tfrac23\cdot4d instead of 4d4d to keep the parameter count matched (see SwiGLU).

RoPE (Su et al., 2021; used before in GPT-J and GPT-NeoX). Queries and keys are rotated by a position-dependent angle in each layer, so qm⊤knq_m^\top k_n depends only on m−nm-n. There is no position table, so no hard context limit is stored in the weights, and the frequencies can be rescaled later to extend context. LLaMA used base θ=10,000\theta=10{,}000. Llama 3 raised it to 500,000500{,}000 (see Rotary positional embeddings (RoPE)).

No biases. None of the linear layers has a bias. That saves only a few million parameters, but it simplifies the kernels and the tensor-parallel sharding. Combined with RMSNorm, every parameter in a block is either a weight matrix or a norm gain.

How the MLP width is chosen

dff=k⋅⌈⌊m⋅⌊8d/3⌋⌋k⌉d_{ff}=k\cdot\Big\lceil \frac{\lfloor m\cdot\lfloor 8d/3\rfloor\rfloor}{k}\Big\rceil
⌊8d/3⌋\lfloor 8d/3\rfloor
23⋅4d\tfrac23\cdot4d, the parameter-matched SwiGLU width
mm
ffn_dim_multiplier in Meta's params.json: absent (1) for LLaMA and Llama 2 7B/13B, 1.3 for Llama 2 70B and Llama 3 8B/70B, 1.2 for 405B
kk
multiple_of: 256 for LLaMA and Llama 2 7B/13B, 1024 for Llama 3 8B, 4096 for the 70B models and 405B; it keeps matrix shapes friendly to GPU tiles and tensor-parallel splits

The multiplier matters more than it looks. With m=1.3m=1.3 the MLP is 3.5d3.5d wide instead of 2.67d2.67d, so Llama 2 70B and Llama 3 8B spend about 31% more parameters on the MLP than a parameter-matched SwiGLU would. One effect is to put back into the MLP the parameters that GQA takes out of WKW_K and WVW_V. In Llama 3 8B, attention loses 25.2M parameters per layer relative to MHA, and the MLP gains 41.9M relative to 8d28d^2.

Three generations

Llama 2 kept the architecture and changed the training. It used 40% more data (2T tokens), a 4096-token context, and GQA in the two largest models (34B, which was trained but not released, and 70B, with 8 key/value heads for 64 query heads). It came with Llama 2-Chat, trained with Supervised fine-tuning (SFT) and then RLHF. The RLHF stage used rejection sampling and PPO with separate helpfulness and safety reward models, and "Ghost Attention" kept a system instruction in force across many turns.

Llama 3 changed the tokenizer and the scale. The vocabulary grew from 32K SentencePiece BPE tokens to 128K: 100K from OpenAI's tiktoken plus 28K extra tokens for non-English languages. The paper reports that English compression improves from 3.17 to 3.94 characters per token, so the same text needs about 20% fewer tokens. Pre-training used about 15T multilingual tokens at an 8K context. GQA with 8 KV heads was used at every size, and a mask stopped attention from crossing document boundaries inside a packed sequence. Llama 3.1 added the 405B model and extended all sizes to 128K tokens. This was done by continued pre-training in six stages of increasing length (about 800B tokens in total) together with a RoPE frequency rescaling. Post-training used SFT, rejection sampling and Direct preference optimization (DPO) rather than PPO.

Published configurations

LLaMA
7B: L=32L=32, d=4096d=4096, 32 heads; 13B: 40, 5120, 40; 33B: 60, 6656, 52; 65B: 80, 8192, 64. Context 2048, 4M-token batches, peak lr 3⋅10−43\cdot10^{-4} (7B, 13B) and 1.5⋅10−41.5\cdot10^{-4} (33B, 65B). 1.0T tokens for 7B/13B, 1.4T for 33B/65B
LLaMA training
AdamW β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, weight decay 0.1, gradient clipping 1.0, 2000 warm-up steps, cosine decay to 10% of the peak. 65B processed about 380 tokens/s per GPU on 2048 A100-80GB, about 21 days for 1.4T tokens
LLaMA data
public sources only: CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, books (Gutenberg, Books3) 4.5%, arXiv 2.5%, StackExchange 2%. Tokenizer: SentencePiece BPE, 32K vocabulary, numbers split into digits, byte fallback
Llama 2
7B, 13B, 70B released (34B not); 2T tokens; context 4096; 70B: L=80L=80, d=8192d=8192, 64 query and 8 KV heads, dff=28672d_{ff}=28672, 68,976,648,192 parameters
Llama 3.1 8B / 70B
8B: L=32L=32, d=4096d=4096, 32/8 heads, dff=14336d_{ff}=14336, peak lr 3⋅10−43\cdot10^{-4}, 8,030,261,248 parameters. 70B: L=80L=80, d=8192d=8192, 64/8 heads, dff=28672d_{ff}=28672, lr 1.5⋅10−41.5\cdot10^{-4}
Llama 3.1 405B
L=126L=126, d=16384d=16384, 128 query and 8 KV heads, dff=53248d_{ff}=53248, peak lr 8⋅10−58\cdot10^{-5}; 15.6T tokens; 3.8⋅10253.8\cdot10^{25} FLOPs on up to 16K H100 GPUs
Llama 3 tokenizer and RoPE
128,000 tokens (checkpoints have 128,256 rows including reserved special tokens); RoPE base 500,000. 3.1 rescales the frequencies with factor 8 over an original 8192-token context
Llama 3.1 8B: L = 32 · d = 4096 · 32 heads of 128 · n_kv = 8 · V = 128,256 · 15T+ tokensuntied head W_U4096 × 128,256final RMSNorm (gain)× 32SwiGLU: W₂(SiLU(W₁x)⊙W₃x)4096→2×14336→4096RMSNorm (gain)attention 32 q / 8 kv headsRoPE on q, kθ-rotationRMSNorm (gain)++token embedding128,256 × 4096orange outline = changed from vanillad_ff = 1024·⌈1.3·⌊8d/3⌋/1024⌉ = 14336per block: 218.1M (vanilla 201.4M)vanilla 7,506,370,560 (7.51B)this block 8,030,261,248 (8.03B)Llama 8,030,261,248 (8.03B)■ embed/head ■ attention ■ MLP ■ norms + positionswhat each change does to Llama 3.1 8B (vs reverting it)RMSNorm: −266.2KSwiGLU: +1.34BRoPE: −536.9Mno biases: −1.4MGQA: −805.3Muntied head: +525.3Mbf16 weights14.96 GiBKV cache per token128 KiB (2·L·n_kv·d_h·2 B)KV cache at 8K tokens1.00 GiBMHA cache at same length4.00 GiBforward FLOPs per token15.0 GFLOP (2 × matmul weights)position table if learned536.9M params (T = 131,072)
model
changes
fig.Start from a vanilla GPT-3-style block at the same depth and width, then switch on the Llama changes one at a time. The counts are exact and match the released checkpoints with all six on. Llama 3.1 8B (the default) has 8,030,261,248 parameters, 218.1M per block. The vanilla version has 7,506,370,560 and 201.4M per block. The list shows what each change does to the finished model. RMSNorm saves only 266.2K and dropping biases 1.4M. RoPE saves 536.9M, because a learned table for 131,072 positions would need that many. GQA saves 805.3M. SwiGLU at d_ff = 14336 adds 1.34B and the untied head adds 525.3M. The bf16 weights take 14.96 GiB and the KV cache 128 KiB per token: 1.00 GiB at 8K and 16.00 GiB at 128K, where full MHA would need 64.00 GiB. A forward pass costs 15.0 GFLOP per token. Llama 2 70B has 68,976,648,192 parameters; switching GQA off gives 78.37B and raises the cache from 320 KiB to 2.5 MiB per token. Llama 3.1 405B needs 755.96 GiB of bf16 weights and 504 KiB per token, 63.00 GiB at 128K; with MHA the cache would be 1008.00 GiB, larger than the weights. For LLaMA 7B, GQA changes nothing because it has none: n_kv = H = 32.

Worked example: rounding, norms and the cache

Three small calculations that you will meet when you load a Llama checkpoint: the MLP width, what RMSNorm does to a vector, and how much the KV cache costs at long context.

modeld⌊8d/3⌋× mround up to kd_ff / d
LLaMA / Llama 2 7B40961092210922 (m = 1)11008 (k = 256)2.69
LLaMA 65B81922184521845 (m = 1)22016 (k = 256)2.69
Llama 2 70B, 3.1 70B81922184528398 (m = 1.3)28672 (k = 4096)3.50
Llama 3.1 8B40961092214198 (m = 1.3)14336 (k = 1024)3.50
Llama 3.1 405B163844369052428 (m = 1.2)53248 (k = 4096)3.25
MLP width from the rule above (computed in Python; matches every released config)

Step by step

  1. 1SwiGLU cost per layer for Llama 2 7B: 3⋅4096⋅11008=135,266,3043\cdot4096\cdot11008=135{,}266{,}304, against 8d2=134,217,7288d^2=134{,}217{,}728 for a GELU MLP of width 4d4d. The rounding adds 0.8%. For Llama 3.1 8B it is 176,160,768176{,}160{,}768, 31% above 8d28d^2, as the multiplier 1.3 implies.
  2. 2RMSNorm of x=(1,−2,3,0)x=(1,-2,3,0) with g=1g=1: 14(1+4+9+0)=3.5\tfrac14(1+4+9+0)=3.5, 3.5=1.8708\sqrt{3.5}=1.8708, so the output is (0.5345,−1.0690,1.6036,0)(0.5345,-1.0690,1.6036,0). LayerNorm would first subtract the mean 0.5 and divide by 3.25=1.8028\sqrt{3.25}=1.8028, giving (0.2774,−1.3868,1.3868,−0.2774)(0.2774,-1.3868,1.3868,-0.2774). RMSNorm keeps the mean direction, and LayerNorm removes it.
  3. 3Parameters of Llama 3.1 8B: each layer has 2d2+2⋅d⋅8⋅128=41,943,0402d^2+2\cdot d\cdot 8\cdot128=41{,}943{,}040 attention weights, 176,160,768176{,}160{,}768 MLP weights and 2d2d gains. 32 layers, two 128,256×4096128{,}256\times4096 tables and the final norm give 8,030,261,2488{,}030{,}261{,}248.
  4. 4KV cache of 405B in bf16: 2⋅126⋅8⋅128⋅2=516,0962\cdot126\cdot8\cdot128\cdot2=516{,}096 bytes = 504 KiB per token, so 63 GiB for one 128K-token sequence. With full multi-head attention (128 KV heads) it would be 16 times more, 1008 GiB, more than the 756 GiB of bf16 weights.
  5. 5Training compute check: 6ND=6⋅405.85×109⋅15.6×1012=3.80×10256ND=6\cdot405.85\times10^9\cdot15.6\times10^{12}=3.80\times10^{25} FLOPs, the paper's 3.8×10253.8\times10^{25}. That is about 38 tokens per parameter for 405B. Llama 2 7B saw 2×1012/6.74×109≈2972\times10^{12}/6.74\times10^9\approx297.
python
1import numpy as np
2
3def llama_dff(d, mult=None, multiple_of=256):
4 h = int(2 * 4 * d / 3) # 2/3 of a 4d MLP, so 3 matrices cost ~8d^2
5 if mult:
6 h = int(mult * h) # ffn_dim_multiplier (Llama 2 70B, Llama 3)
7 return multiple_of * ((h + multiple_of - 1) // multiple_of)
8
9def llama_params(L, d, H, kv, dff, V):
10 dh = d // H
11 attn = 2 * d * d + 2 * d * kv * dh # W_q, W_o full; W_k, W_v only kv heads (GQA)
12 mlp = 3 * d * dff # W1 (gate), W3 (up), W2 (down), no biases
13 return L * (attn + mlp + 2 * d) + 2 * V * d + d # 2 RMSNorm gains per layer, untied head, final norm
14
15def rms_norm(x, g, eps=1e-5):
16 return g * x / np.sqrt(np.mean(x ** 2, axis=-1, keepdims=True) + eps)
17
18def swiglu(x, W1, W3, W2):
19 a = x @ W1.T
20 return ((a / (1 + np.exp(-a))) * (x @ W3.T)) @ W2.T # W2(SiLU(W1 x) * W3 x)
21
22def llama31_inv_freq(dh=128, base=500_000.0, factor=8.0, low=1.0, high=4.0, orig=8192):
23 inv = base ** (-np.arange(0, dh, 2) / dh) # RoPE frequencies, one per pair of dims
24 wavelen = 2 * np.pi / inv
25 out = np.where(wavelen > orig / low, inv / factor, inv) # long wavelengths: interpolate
26 s = (orig / wavelen - low) / (high - low)
27 mid = (wavelen >= orig / high) & (wavelen <= orig / low) # in between: blend
28 return np.where(mid, (1 - s) * inv / factor + s * inv, out), inv
29
30print([llama_dff(4096), llama_dff(8192, 1.3, 4096), llama_dff(4096, 1.3, 1024), llama_dff(16384, 1.2, 4096)])
31print(f"{llama_params(32, 4096, 32, 32, llama_dff(4096), 32000):,}") # Llama 2 7B
32print(f"{llama_params(32, 4096, 32, 8, llama_dff(4096, 1.3, 1024), 128256):,}") # Llama 3.1 8B
33print(f"{llama_params(126, 16384, 128, 8, llama_dff(16384, 1.2, 4096), 128256):,}") # Llama 3.1 405B
34
35x = np.array([[1.0, -2.0, 3.0, 0.0]])
36print(rms_norm(x, np.ones(4)).round(4))
37
38rng = np.random.default_rng(0)
39d, f = 8, 24
40y = swiglu(rng.normal(size=(2, d)), *(rng.normal(size=s) * 0.1 for s in [(f, d), (f, d), (d, f)]))
41print(y.shape)
42
43new, old = llama31_inv_freq()
44ratio = old / new
45print((ratio == 1).sum(), ((ratio > 1) & (ratio < 8)).sum(), np.isclose(ratio, 8).sum()) # kept, blended, divided by 8
46
47# [11008, 28672, 14336, 53248]
48# 6,738,415,616
49# 8,030,261,248
50# 405,853,388,800
51# [[ 0.5345 -1.069 1.6036 0. ]]
52# (2, 8)
53# 29 6 29
The Llama recipe in numpy: MLP-width rule, exact parameter count, RMSNorm, SwiGLU and Llama 3.1's RoPE frequency scaling (run with numpy; outputs shown)

The last line shows how Llama 3.1 reaches 128K. Of the 64 frequency pairs per head, the 29 fastest (wavelength under 2048 tokens) are left alone, so local word order is resolved as before. The 29 slowest (wavelength over 8192 tokens, the original context) are divided by 8, the context ratio. The 6 in between are blended. This is the "by parts" idea that YaRN also uses (see Context window optimization).

Use when

  • +A simple, well-tested block (RMSNorm, SwiGLU, RoPE, no biases) that most open LLMs copy
  • +No position table: context can be extended by rescaling RoPE frequencies and training on longer data
  • +GQA cuts the KV cache 4× (8B) to 16× (405B) at little quality cost
  • +Small models trained far past compute-optimal are cheap to serve
  • +Open weights with exact configs make the family the reference for fine-tuning, quantisation and serving work

Avoid when

  • −Over-training costs extra training compute per unit of quality
  • −Untied 128K-vocabulary tables cost 1.05B parameters in the 8B model, 13% of it
  • −Dense: every token runs all parameters, unlike MoE models of similar quality
  • −Custom licences, not open-source licences; the training data is not released
  • −SwiGLU's three matrices mean three MLP matmuls and more activation memory per token

Check yourself

LLaMA replaced GPT-3's GELU MLP of width 4d with SwiGLU. Why is the hidden width about 2.67d rather than 4d?

Check yourself

Llama 3.1 405B has 126 layers, 8 KV heads of size 128 and 128 query heads. Roughly how large is the bf16 KV cache for one 128K-token sequence?

Remember

  • ✓The Llama block is a pre-norm decoder with RMSNorm g⊙x/rms(x)g\odot x/\mathrm{rms}(x), SwiGLU W2(SiLU(W1x)⊙W3x)W_2(\mathrm{SiLU}(W_1x)\odot W_3x), RoPE on q,kq,k, no biases and an untied head.
  • ✓dff=k⌈m⌊8d/3⌋/k⌉d_{ff}=k\lceil m\lfloor8d/3\rfloor/k\rceil gives every released MLP width: 11008, 14336, 28672, 53248.
  • ✓LLaMA: 7–65B, 1.0–1.4T public tokens, 2K context, 32K SentencePiece vocab; 13B beat GPT-3 on most benchmarks.
  • ✓Llama 2: 2T tokens, 4K context, GQA in 34B/70B, chat models via SFT + RLHF. Llama 3/3.1: 128K tiktoken-based vocab, 15T+ tokens, RoPE base 500,000, GQA everywhere, 8K → 128K context, 405B.
  • ✓The design goal is inference cost: train small models far past compute-optimal and shrink the KV cache with GQA.
← back to Deep Learning roadmap