Definition
Llama is Meta's family of open-weight decoder-only language models. The series starts with Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (February 2023), continues with Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (July 2023), and reaches Llama 3 (April 2024) and Llama 3.1 (July 2024), described in the Llama Team's "The Llama 3 Herd of Models" (2024). Every generation uses the same block. It is a Decoder-only transformer with RMSNorm pre-normalisation, a SwiGLU MLP, Rotary positional embeddings (RoPE), no bias terms and an untied output head. Later generations add Grouped-query attention (GQA), a larger tokenizer, much more data and longer context.
By 2022 the strongest models (GPT-3, Gopher, PaLM, Chinchilla) were either closed or only reachable through an API. The LLaMA paper made two arguments. First, a competitive model can be trained only on publicly available data. Second, the scaling laws of Hoffmann et al. ("Chinchilla") optimise the training budget, but a deployed model's cost is set by inference. A smaller model trained for longer is cheaper to serve at the same quality. Hoffmann et al. recommend training a 10B model on 200B tokens. LLaMA trained its 7B model on 1T tokens and found that its performance was still improving.
The result was LLaMA-13B outperforming GPT-3 (175B) on most of the paper's benchmarks, and LLaMA-65B competitive with Chinchilla-70B and PaLM-540B. The weights were released to researchers under a non-commercial licence, and they leaked within a week. Llama 2 followed with a licence that allows commercial use with conditions, plus chat models aligned with RLHF. Llama 3.1 brought a 405B dense model with open weights.
The block: four changes to GPT-3
The LLaMA paper lists its architecture as GPT-3 plus three borrowed ideas. Each came from an earlier model, and each is now standard. The fourth change, dropping biases, is in the released code rather than the paper's list.
- learned gain; there is no bias and no mean subtraction, so a norm has parameters instead of LayerNorm's
- in LLaMA, in Llama 2 and Llama 3
- "gate" and "up" projections; is "down"
- three matrices of cost the same as a two-matrix GELU MLP of width
Pre-normalisation with RMSNorm (Zhang and Sennrich, "Root Mean Square Layer Normalization", 2019). GPT-2 and GPT-3 had already moved the norm in front of each sub-layer (see GPT architecture). RMSNorm also drops LayerNorm's re-centring. The authors found that re-scaling alone gives most of the stability, at lower cost, because there is no mean to compute and subtract. The parameter saving is negligible, per norm. The point is a simpler, cheaper kernel.
SwiGLU (Shazeer, "GLU Variants Improve Transformer", 2020; used in PaLM). A gated linear unit multiplies one projection by a nonlinear function of another, so the MLP learns which features to pass, not just how to threshold them. Shazeer found SwiGLU and GeGLU gave the best perplexity among the variants he tested at matched parameters and compute. LLaMA follows PaLM in using instead of to keep the parameter count matched (see SwiGLU).
RoPE (Su et al., 2021; used before in GPT-J and GPT-NeoX). Queries and keys are rotated by a position-dependent angle in each layer, so depends only on . There is no position table, so no hard context limit is stored in the weights, and the frequencies can be rescaled later to extend context. LLaMA used base . Llama 3 raised it to (see Rotary positional embeddings (RoPE)).
No biases. None of the linear layers has a bias. That saves only a few million parameters, but it simplifies the kernels and the tensor-parallel sharding. Combined with RMSNorm, every parameter in a block is either a weight matrix or a norm gain.
How the MLP width is chosen
- , the parameter-matched SwiGLU width
ffn_dim_multiplierin Meta'sparams.json: absent (1) for LLaMA and Llama 2 7B/13B, 1.3 for Llama 2 70B and Llama 3 8B/70B, 1.2 for 405Bmultiple_of: 256 for LLaMA and Llama 2 7B/13B, 1024 for Llama 3 8B, 4096 for the 70B models and 405B; it keeps matrix shapes friendly to GPU tiles and tensor-parallel splits
The multiplier matters more than it looks. With the MLP is wide instead of , so Llama 2 70B and Llama 3 8B spend about 31% more parameters on the MLP than a parameter-matched SwiGLU would. One effect is to put back into the MLP the parameters that GQA takes out of and . In Llama 3 8B, attention loses 25.2M parameters per layer relative to MHA, and the MLP gains 41.9M relative to .
Three generations
Llama 2 kept the architecture and changed the training. It used 40% more data (2T tokens), a 4096-token context, and GQA in the two largest models (34B, which was trained but not released, and 70B, with 8 key/value heads for 64 query heads). It came with Llama 2-Chat, trained with Supervised fine-tuning (SFT) and then RLHF. The RLHF stage used rejection sampling and PPO with separate helpfulness and safety reward models, and "Ghost Attention" kept a system instruction in force across many turns.
Llama 3 changed the tokenizer and the scale. The vocabulary grew from 32K SentencePiece BPE tokens to 128K: 100K from OpenAI's tiktoken plus 28K extra tokens for non-English languages. The paper reports that English compression improves from 3.17 to 3.94 characters per token, so the same text needs about 20% fewer tokens. Pre-training used about 15T multilingual tokens at an 8K context. GQA with 8 KV heads was used at every size, and a mask stopped attention from crossing document boundaries inside a packed sequence. Llama 3.1 added the 405B model and extended all sizes to 128K tokens. This was done by continued pre-training in six stages of increasing length (about 800B tokens in total) together with a RoPE frequency rescaling. Post-training used SFT, rejection sampling and Direct preference optimization (DPO) rather than PPO.
Published configurations
- LLaMA
- 7B: , , 32 heads; 13B: 40, 5120, 40; 33B: 60, 6656, 52; 65B: 80, 8192, 64. Context 2048, 4M-token batches, peak lr (7B, 13B) and (33B, 65B). 1.0T tokens for 7B/13B, 1.4T for 33B/65B
- LLaMA training
- AdamW , , weight decay 0.1, gradient clipping 1.0, 2000 warm-up steps, cosine decay to 10% of the peak. 65B processed about 380 tokens/s per GPU on 2048 A100-80GB, about 21 days for 1.4T tokens
- LLaMA data
- public sources only: CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, books (Gutenberg, Books3) 4.5%, arXiv 2.5%, StackExchange 2%. Tokenizer: SentencePiece BPE, 32K vocabulary, numbers split into digits, byte fallback
- Llama 2
- 7B, 13B, 70B released (34B not); 2T tokens; context 4096; 70B: , , 64 query and 8 KV heads, , 68,976,648,192 parameters
- Llama 3.1 8B / 70B
- 8B: , , 32/8 heads, , peak lr , 8,030,261,248 parameters. 70B: , , 64/8 heads, , lr
- Llama 3.1 405B
- , , 128 query and 8 KV heads, , peak lr ; 15.6T tokens; FLOPs on up to 16K H100 GPUs
- Llama 3 tokenizer and RoPE
- 128,000 tokens (checkpoints have 128,256 rows including reserved special tokens); RoPE base 500,000. 3.1 rescales the frequencies with factor 8 over an original 8192-token context
Worked example: rounding, norms and the cache
Three small calculations that you will meet when you load a Llama checkpoint: the MLP width, what RMSNorm does to a vector, and how much the KV cache costs at long context.
| model | d | ⌊8d/3⌋ | × m | round up to k | d_ff / d |
|---|---|---|---|---|---|
| LLaMA / Llama 2 7B | 4096 | 10922 | 10922 (m = 1) | 11008 (k = 256) | 2.69 |
| LLaMA 65B | 8192 | 21845 | 21845 (m = 1) | 22016 (k = 256) | 2.69 |
| Llama 2 70B, 3.1 70B | 8192 | 21845 | 28398 (m = 1.3) | 28672 (k = 4096) | 3.50 |
| Llama 3.1 8B | 4096 | 10922 | 14198 (m = 1.3) | 14336 (k = 1024) | 3.50 |
| Llama 3.1 405B | 16384 | 43690 | 52428 (m = 1.2) | 53248 (k = 4096) | 3.25 |
Step by step
- 1SwiGLU cost per layer for Llama 2 7B: , against for a GELU MLP of width . The rounding adds 0.8%. For Llama 3.1 8B it is , 31% above , as the multiplier 1.3 implies.
- 2RMSNorm of with : , , so the output is . LayerNorm would first subtract the mean 0.5 and divide by , giving . RMSNorm keeps the mean direction, and LayerNorm removes it.
- 3Parameters of Llama 3.1 8B: each layer has attention weights, MLP weights and gains. 32 layers, two tables and the final norm give .
- 4KV cache of 405B in bf16: bytes = 504 KiB per token, so 63 GiB for one 128K-token sequence. With full multi-head attention (128 KV heads) it would be 16 times more, 1008 GiB, more than the 756 GiB of bf16 weights.
- 5Training compute check: FLOPs, the paper's . That is about 38 tokens per parameter for 405B. Llama 2 7B saw .
1import numpy as np23def llama_dff(d, mult=None, multiple_of=256):4 h = int(2 * 4 * d / 3) # 2/3 of a 4d MLP, so 3 matrices cost ~8d^25 if mult:6 h = int(mult * h) # ffn_dim_multiplier (Llama 2 70B, Llama 3)7 return multiple_of * ((h + multiple_of - 1) // multiple_of)89def llama_params(L, d, H, kv, dff, V):10 dh = d // H11 attn = 2 * d * d + 2 * d * kv * dh # W_q, W_o full; W_k, W_v only kv heads (GQA)12 mlp = 3 * d * dff # W1 (gate), W3 (up), W2 (down), no biases13 return L * (attn + mlp + 2 * d) + 2 * V * d + d # 2 RMSNorm gains per layer, untied head, final norm1415def rms_norm(x, g, eps=1e-5):16 return g * x / np.sqrt(np.mean(x ** 2, axis=-1, keepdims=True) + eps)1718def swiglu(x, W1, W3, W2):19 a = x @ W1.T20 return ((a / (1 + np.exp(-a))) * (x @ W3.T)) @ W2.T # W2(SiLU(W1 x) * W3 x)2122def llama31_inv_freq(dh=128, base=500_000.0, factor=8.0, low=1.0, high=4.0, orig=8192):23 inv = base ** (-np.arange(0, dh, 2) / dh) # RoPE frequencies, one per pair of dims24 wavelen = 2 * np.pi / inv25 out = np.where(wavelen > orig / low, inv / factor, inv) # long wavelengths: interpolate26 s = (orig / wavelen - low) / (high - low)27 mid = (wavelen >= orig / high) & (wavelen <= orig / low) # in between: blend28 return np.where(mid, (1 - s) * inv / factor + s * inv, out), inv2930print([llama_dff(4096), llama_dff(8192, 1.3, 4096), llama_dff(4096, 1.3, 1024), llama_dff(16384, 1.2, 4096)])31print(f"{llama_params(32, 4096, 32, 32, llama_dff(4096), 32000):,}") # Llama 2 7B32print(f"{llama_params(32, 4096, 32, 8, llama_dff(4096, 1.3, 1024), 128256):,}") # Llama 3.1 8B33print(f"{llama_params(126, 16384, 128, 8, llama_dff(16384, 1.2, 4096), 128256):,}") # Llama 3.1 405B3435x = np.array([[1.0, -2.0, 3.0, 0.0]])36print(rms_norm(x, np.ones(4)).round(4))3738rng = np.random.default_rng(0)39d, f = 8, 2440y = swiglu(rng.normal(size=(2, d)), *(rng.normal(size=s) * 0.1 for s in [(f, d), (f, d), (d, f)]))41print(y.shape)4243new, old = llama31_inv_freq()44ratio = old / new45print((ratio == 1).sum(), ((ratio > 1) & (ratio < 8)).sum(), np.isclose(ratio, 8).sum()) # kept, blended, divided by 84647# [11008, 28672, 14336, 53248]48# 6,738,415,61649# 8,030,261,24850# 405,853,388,80051# [[ 0.5345 -1.069 1.6036 0. ]]52# (2, 8)53# 29 6 29The last line shows how Llama 3.1 reaches 128K. Of the 64 frequency pairs per head, the 29 fastest (wavelength under 2048 tokens) are left alone, so local word order is resolved as before. The 29 slowest (wavelength over 8192 tokens, the original context) are divided by 8, the context ratio. The 6 in between are blended. This is the "by parts" idea that YaRN also uses (see Context window optimization).
Use when
- +A simple, well-tested block (RMSNorm, SwiGLU, RoPE, no biases) that most open LLMs copy
- +No position table: context can be extended by rescaling RoPE frequencies and training on longer data
- +GQA cuts the KV cache 4× (8B) to 16× (405B) at little quality cost
- +Small models trained far past compute-optimal are cheap to serve
- +Open weights with exact configs make the family the reference for fine-tuning, quantisation and serving work
Avoid when
- −Over-training costs extra training compute per unit of quality
- −Untied 128K-vocabulary tables cost 1.05B parameters in the 8B model, 13% of it
- −Dense: every token runs all parameters, unlike MoE models of similar quality
- −Custom licences, not open-source licences; the training data is not released
- −SwiGLU's three matrices mean three MLP matmuls and more activation memory per token
Check yourself
LLaMA replaced GPT-3's GELU MLP of width 4d with SwiGLU. Why is the hidden width about 2.67d rather than 4d?
Check yourself
Llama 3.1 405B has 126 layers, 8 KV heads of size 128 and 128 query heads. Roughly how large is the bf16 KV cache for one 128K-token sequence?
Remember
- ✓The Llama block is a pre-norm decoder with RMSNorm , SwiGLU , RoPE on , no biases and an untied head.
- ✓ gives every released MLP width: 11008, 14336, 28672, 53248.
- ✓LLaMA: 7–65B, 1.0–1.4T public tokens, 2K context, 32K SentencePiece vocab; 13B beat GPT-3 on most benchmarks.
- ✓Llama 2: 2T tokens, 4K context, GQA in 34B/70B, chat models via SFT + RLHF. Llama 3/3.1: 128K tiktoken-based vocab, 15T+ tokens, RoPE base 500,000, GQA everywhere, 8K → 128K context, 405B.
- ✓The design goal is inference cost: train small models far past compute-optimal and shrink the KV cache with GQA.