# Llama

> Meta's Llama recipe is pre-RMSNorm + SwiGLU + RoPE + no biases, trained far past compute-optimal: LLaMA (2023, 7–65B, 1.0–1.4T tokens), Llama 2 (2T tokens, 4K context, GQA in 34B/70B), Llama 3/3.1 (128K tiktoken-based vocab, 15T+ tokens, 8K → 128K context, 405B).

Source: https://www.mlfoundry.in/learn/llama  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Llama:** is Meta's family of open-weight decoder-only language models. The series starts with Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (February 2023), continues with Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (July 2023), and reaches Llama 3 (April 2024) and Llama 3.1 (July 2024), described in the Llama Team's "The Llama 3 Herd of Models" (2024). Every generation uses the same block. It is a **Decoder-only transformer** with **RMSNorm** pre-normalisation, a **SwiGLU** MLP, **Rotary positional embeddings (RoPE)**, no bias terms and an untied output head. Later generations add **Grouped-query attention (GQA)**, a larger tokenizer, much more data and longer context.

By 2022 the strongest models (GPT-3, Gopher, PaLM, Chinchilla) were either closed or only reachable through an API. The LLaMA paper made two arguments. First, a competitive model can be trained only on publicly available data. Second, the scaling laws of Hoffmann et al. ("Chinchilla") optimise the *training* budget, but a deployed model's cost is set by *inference*. A smaller model trained for longer is cheaper to serve at the same quality. Hoffmann et al. recommend training a 10B model on 200B tokens. LLaMA trained its 7B model on 1T tokens and found that its performance was still improving.

The result was LLaMA-13B outperforming GPT-3 (175B) on most of the paper's benchmarks, and LLaMA-65B competitive with Chinchilla-70B and PaLM-540B. The weights were released to researchers under a non-commercial licence, and they leaked within a week. Llama 2 followed with a licence that allows commercial use with conditions, plus chat models aligned with **RLHF**. Llama 3.1 brought a 405B dense model with open weights.

## The block: four changes to GPT-3

> Figure: One Llama 3.1 8B block with its real shapes: 32 query heads and 8 key/value heads of size 128, RoPE with base 500,000 applied to q and k, a SwiGLU MLP 4096 → 14336 → 4096, RMSNorm before each sub-layer and once at the end, a 128,256-token embedding and a separate output head. There are no biases and no position table. Hover a release on the right to see which parts it introduced: LLaMA brought RMSNorm, SwiGLU and RoPE; Llama 2 brought GQA (in 34B and 70B only); Llama 3 brought the new tokenizer, the larger RoPE base and GQA at every size; Llama 3.1 brought RoPE frequency scaling to 128K tokens. The d_ff rule at the bottom right reproduces every released MLP width.

The LLaMA paper lists its architecture as GPT-3 plus three borrowed ideas. Each came from an earlier model, and each is now standard. The fourth change, dropping biases, is in the released code rather than the paper's list.

$$
\mathrm{RMSNorm}(x)=g\odot\frac{x}{\sqrt{\tfrac1d\sum_i x_i^2+\epsilon}},\qquad \mathrm{SwiGLU}(x)=W_2\big(\mathrm{SiLU}(W_1x)\odot W_3x\big),\qquad \mathrm{SiLU}(z)=z\,\sigma(z)
$$

- $g\in\mathbb R^d$: learned gain; there is no bias and no mean subtraction, so a norm has $d$ parameters instead of LayerNorm's $2d$
- $\epsilon$: $10^{-6}$ in LLaMA, $10^{-5}$ in Llama 2 and Llama 3
- $W_1,W_3\in\mathbb R^{d_{ff}\times d}$: "gate" and "up" projections; $W_2\in\mathbb R^{d\times d_{ff}}$ is "down"
- $d_{ff}\approx\tfrac23\cdot4d$: three matrices of $\tfrac83d^2$ cost the same $8d^2$ as a two-matrix GELU MLP of width $4d$

**Pre-normalisation with RMSNorm** (Zhang and Sennrich, "Root Mean Square Layer Normalization", 2019). GPT-2 and GPT-3 had already moved the norm in front of each sub-layer (see **GPT architecture**). RMSNorm also drops LayerNorm's re-centring. The authors found that re-scaling alone gives most of the stability, at lower cost, because there is no mean to compute and subtract. The parameter saving is negligible, $d$ per norm. The point is a simpler, cheaper kernel.

**SwiGLU** (Shazeer, "GLU Variants Improve Transformer", 2020; used in PaLM). A gated linear unit multiplies one projection by a nonlinear function of another, so the MLP learns which features to pass, not just how to threshold them. Shazeer found SwiGLU and GeGLU gave the best perplexity among the variants he tested at matched parameters and compute. LLaMA follows PaLM in using $\tfrac23\cdot4d$ instead of $4d$ to keep the parameter count matched (see **SwiGLU**).

**RoPE** (Su et al., 2021; used before in GPT-J and GPT-NeoX). Queries and keys are rotated by a position-dependent angle in each layer, so $q_m^\top k_n$ depends only on $m-n$. There is no position table, so no hard context limit is stored in the weights, and the frequencies can be rescaled later to extend context. LLaMA used base $\theta=10{,}000$. Llama 3 raised it to $500{,}000$ (see **Rotary positional embeddings (RoPE)**).

**No biases.** None of the linear layers has a bias. That saves only a few million parameters, but it simplifies the kernels and the tensor-parallel sharding. Combined with RMSNorm, every parameter in a block is either a weight matrix or a norm gain.

## How the MLP width is chosen

$$
d_{ff}=k\cdot\Big\lceil \frac{\lfloor m\cdot\lfloor 8d/3\rfloor\rfloor}{k}\Big\rceil
$$

- $\lfloor 8d/3\rfloor$: $\tfrac23\cdot4d$, the parameter-matched SwiGLU width
- $m$: `ffn_dim_multiplier` in Meta's `params.json`: absent (1) for LLaMA and Llama 2 7B/13B, 1.3 for Llama 2 70B and Llama 3 8B/70B, 1.2 for 405B
- $k$: `multiple_of`: 256 for LLaMA and Llama 2 7B/13B, 1024 for Llama 3 8B, 4096 for the 70B models and 405B; it keeps matrix shapes friendly to GPU tiles and tensor-parallel splits

The multiplier matters more than it looks. With $m=1.3$ the MLP is $3.5d$ wide instead of $2.67d$, so Llama 2 70B and Llama 3 8B spend about 31% more parameters on the MLP than a parameter-matched SwiGLU would. One effect is to put back into the MLP the parameters that GQA takes out of $W_K$ and $W_V$. In Llama 3 8B, attention loses 25.2M parameters per layer relative to MHA, and the MLP gains 41.9M relative to $8d^2$.

## Three generations

**Llama 2** kept the architecture and changed the training. It used 40% more data (2T tokens), a 4096-token context, and GQA in the two largest models (34B, which was trained but not released, and 70B, with 8 key/value heads for 64 query heads). It came with **Llama 2-Chat**, trained with **Supervised fine-tuning (SFT)** and then **RLHF**. The RLHF stage used rejection sampling and PPO with separate helpfulness and safety reward models, and "Ghost Attention" kept a system instruction in force across many turns.

**Llama 3** changed the tokenizer and the scale. The vocabulary grew from 32K SentencePiece BPE tokens to 128K: 100K from OpenAI's tiktoken plus 28K extra tokens for non-English languages. The paper reports that English compression improves from 3.17 to 3.94 characters per token, so the same text needs about 20% fewer tokens. Pre-training used about 15T multilingual tokens at an 8K context. GQA with 8 KV heads was used at every size, and a mask stopped attention from crossing document boundaries inside a packed sequence. **Llama 3.1** added the 405B model and extended all sizes to 128K tokens. This was done by continued pre-training in six stages of increasing length (about 800B tokens in total) together with a RoPE frequency rescaling. Post-training used SFT, rejection sampling and **Direct preference optimization (DPO)** rather than PPO.

**Published configurations**

- **LLaMA:** 7B: $L=32$, $d=4096$, 32 heads; 13B: 40, 5120, 40; 33B: 60, 6656, 52; 65B: 80, 8192, 64. Context 2048, 4M-token batches, peak lr $3\cdot10^{-4}$ (7B, 13B) and $1.5\cdot10^{-4}$ (33B, 65B). 1.0T tokens for 7B/13B, 1.4T for 33B/65B
- **LLaMA training:** AdamW $\beta_1=0.9$, $\beta_2=0.95$, weight decay 0.1, gradient clipping 1.0, 2000 warm-up steps, cosine decay to 10% of the peak. 65B processed about 380 tokens/s per GPU on 2048 A100-80GB, about 21 days for 1.4T tokens
- **LLaMA data:** public sources only: CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, books (Gutenberg, Books3) 4.5%, arXiv 2.5%, StackExchange 2%. Tokenizer: SentencePiece BPE, 32K vocabulary, numbers split into digits, byte fallback
- **Llama 2:** 7B, 13B, 70B released (34B not); 2T tokens; context 4096; 70B: $L=80$, $d=8192$, 64 query and 8 KV heads, $d_{ff}=28672$, 68,976,648,192 parameters
- **Llama 3.1 8B / 70B:** 8B: $L=32$, $d=4096$, 32/8 heads, $d_{ff}=14336$, peak lr $3\cdot10^{-4}$, 8,030,261,248 parameters. 70B: $L=80$, $d=8192$, 64/8 heads, $d_{ff}=28672$, lr $1.5\cdot10^{-4}$
- **Llama 3.1 405B:** $L=126$, $d=16384$, 128 query and 8 KV heads, $d_{ff}=53248$, peak lr $8\cdot10^{-5}$; 15.6T tokens; $3.8\cdot10^{25}$ FLOPs on up to 16K H100 GPUs
- **Llama 3 tokenizer and RoPE:** 128,000 tokens (checkpoints have 128,256 rows including reserved special tokens); RoPE base 500,000. 3.1 rescales the frequencies with factor 8 over an original 8192-token context

> Figure: Start from a vanilla GPT-3-style block at the same depth and width, then switch on the Llama changes one at a time. The counts are exact and match the released checkpoints with all six on. Llama 3.1 8B (the default) has 8,030,261,248 parameters, 218.1M per block. The vanilla version has 7,506,370,560 and 201.4M per block. The list shows what each change does to the finished model. RMSNorm saves only 266.2K and dropping biases 1.4M. RoPE saves 536.9M, because a learned table for 131,072 positions would need that many. GQA saves 805.3M. SwiGLU at d_ff = 14336 adds 1.34B and the untied head adds 525.3M. The bf16 weights take 14.96 GiB and the KV cache 128 KiB per token: 1.00 GiB at 8K and 16.00 GiB at 128K, where full MHA would need 64.00 GiB. A forward pass costs 15.0 GFLOP per token. Llama 2 70B has 68,976,648,192 parameters; switching GQA off gives 78.37B and raises the cache from 320 KiB to 2.5 MiB per token. Llama 3.1 405B needs 755.96 GiB of bf16 weights and 504 KiB per token, 63.00 GiB at 128K; with MHA the cache would be 1008.00 GiB, larger than the weights. For LLaMA 7B, GQA changes nothing because it has none: n_kv = H = 32.

## Worked example: rounding, norms and the cache

Three small calculations that you will meet when you load a Llama checkpoint: the MLP width, what RMSNorm does to a vector, and how much the KV cache costs at long context.

MLP width from the rule above (computed in Python; matches every released config)

| model | d | ⌊8d/3⌋ | × m | round up to k | d_ff / d |
| --- | --- | --- | --- | --- | --- |
| LLaMA / Llama 2 7B | 4096 | 10922 | 10922 (m = 1) | 11008 (k = 256) | 2.69 |
| LLaMA 65B | 8192 | 21845 | 21845 (m = 1) | 22016 (k = 256) | 2.69 |
| Llama 2 70B, 3.1 70B | 8192 | 21845 | 28398 (m = 1.3) | 28672 (k = 4096) | 3.50 |
| Llama 3.1 8B | 4096 | 10922 | 14198 (m = 1.3) | 14336 (k = 1024) | 3.50 |
| Llama 3.1 405B | 16384 | 43690 | 52428 (m = 1.2) | 53248 (k = 4096) | 3.25 |

**Step by step**

1. SwiGLU cost per layer for Llama 2 7B: $3\cdot4096\cdot11008=135{,}266{,}304$, against $8d^2=134{,}217{,}728$ for a GELU MLP of width $4d$. The rounding adds 0.8%. For Llama 3.1 8B it is $176{,}160{,}768$, 31% above $8d^2$, as the multiplier 1.3 implies.
2. RMSNorm of $x=(1,-2,3,0)$ with $g=1$: $\tfrac14(1+4+9+0)=3.5$, $\sqrt{3.5}=1.8708$, so the output is $(0.5345,-1.0690,1.6036,0)$. LayerNorm would first subtract the mean 0.5 and divide by $\sqrt{3.25}=1.8028$, giving $(0.2774,-1.3868,1.3868,-0.2774)$. RMSNorm keeps the mean direction, and LayerNorm removes it.
3. Parameters of Llama 3.1 8B: each layer has $2d^2+2\cdot d\cdot 8\cdot128=41{,}943{,}040$ attention weights, $176{,}160{,}768$ MLP weights and $2d$ gains. 32 layers, two $128{,}256\times4096$ tables and the final norm give $8{,}030{,}261{,}248$.
4. KV cache of 405B in bf16: $2\cdot126\cdot8\cdot128\cdot2=516{,}096$ bytes = 504 KiB per token, so 63 GiB for one 128K-token sequence. With full multi-head attention (128 KV heads) it would be 16 times more, 1008 GiB, more than the 756 GiB of bf16 weights.
5. Training compute check: $6ND=6\cdot405.85\times10^9\cdot15.6\times10^{12}=3.80\times10^{25}$ FLOPs, the paper's $3.8\times10^{25}$. That is about 38 tokens per parameter for 405B. Llama 2 7B saw $2\times10^{12}/6.74\times10^9\approx297$.

```python
import numpy as np

def llama_dff(d, mult=None, multiple_of=256):
    h = int(2 * 4 * d / 3)                      # 2/3 of a 4d MLP, so 3 matrices cost ~8d^2
    if mult:
        h = int(mult * h)                       # ffn_dim_multiplier (Llama 2 70B, Llama 3)
    return multiple_of * ((h + multiple_of - 1) // multiple_of)

def llama_params(L, d, H, kv, dff, V):
    dh = d // H
    attn = 2 * d * d + 2 * d * kv * dh          # W_q, W_o full; W_k, W_v only kv heads (GQA)
    mlp = 3 * d * dff                           # W1 (gate), W3 (up), W2 (down), no biases
    return L * (attn + mlp + 2 * d) + 2 * V * d + d   # 2 RMSNorm gains per layer, untied head, final norm

def rms_norm(x, g, eps=1e-5):
    return g * x / np.sqrt(np.mean(x ** 2, axis=-1, keepdims=True) + eps)

def swiglu(x, W1, W3, W2):
    a = x @ W1.T
    return ((a / (1 + np.exp(-a))) * (x @ W3.T)) @ W2.T    # W2(SiLU(W1 x) * W3 x)

def llama31_inv_freq(dh=128, base=500_000.0, factor=8.0, low=1.0, high=4.0, orig=8192):
    inv = base ** (-np.arange(0, dh, 2) / dh)    # RoPE frequencies, one per pair of dims
    wavelen = 2 * np.pi / inv
    out = np.where(wavelen > orig / low, inv / factor, inv)       # long wavelengths: interpolate
    s = (orig / wavelen - low) / (high - low)
    mid = (wavelen >= orig / high) & (wavelen <= orig / low)      # in between: blend
    return np.where(mid, (1 - s) * inv / factor + s * inv, out), inv

print([llama_dff(4096), llama_dff(8192, 1.3, 4096), llama_dff(4096, 1.3, 1024), llama_dff(16384, 1.2, 4096)])
print(f"{llama_params(32, 4096, 32, 32, llama_dff(4096), 32000):,}")                 # Llama 2 7B
print(f"{llama_params(32, 4096, 32, 8, llama_dff(4096, 1.3, 1024), 128256):,}")      # Llama 3.1 8B
print(f"{llama_params(126, 16384, 128, 8, llama_dff(16384, 1.2, 4096), 128256):,}")  # Llama 3.1 405B

x = np.array([[1.0, -2.0, 3.0, 0.0]])
print(rms_norm(x, np.ones(4)).round(4))

rng = np.random.default_rng(0)
d, f = 8, 24
y = swiglu(rng.normal(size=(2, d)), *(rng.normal(size=s) * 0.1 for s in [(f, d), (f, d), (d, f)]))
print(y.shape)

new, old = llama31_inv_freq()
ratio = old / new
print((ratio == 1).sum(), ((ratio > 1) & (ratio < 8)).sum(), np.isclose(ratio, 8).sum())   # kept, blended, divided by 8

# [11008, 28672, 14336, 53248]
# 6,738,415,616
# 8,030,261,248
# 405,853,388,800
# [[ 0.5345 -1.069   1.6036  0.    ]]
# (2, 8)
# 29 6 29
```

The Llama recipe in numpy: MLP-width rule, exact parameter count, RMSNorm, SwiGLU and Llama 3.1's RoPE frequency scaling (run with numpy; outputs shown)

The last line shows how Llama 3.1 reaches 128K. Of the 64 frequency pairs per head, the 29 fastest (wavelength under 2048 tokens) are left alone, so local word order is resolved as before. The 29 slowest (wavelength over 8192 tokens, the original context) are divided by 8, the context ratio. The 6 in between are blended. This is the "by parts" idea that YaRN also uses (see **Context window optimization**).

> **Two RoPE layouts:** Meta's reference code rotates *adjacent* pairs $(x_0,x_1),(x_2,x_3),\dots$ using complex multiplication. Hugging Face's `LlamaForCausalLM` rotates *halves*, $(x_i,x_{i+d/2})$. The conversion script permutes the rows of $W_Q$ and $W_K$ so that both give the same outputs. Load Meta weights into a rotate-half implementation without that permutation and the model produces fluent-looking garbage, not an error.

> **Train past Chinchilla, serve cheaply:** Chinchilla's 20 tokens per parameter minimises loss for a fixed *training* budget. A model served to millions of users is paid for mostly at inference, and inference cost scales with $N$, not $D$. Llama 2 7B saw about 300 tokens per parameter, and Llama 3 8B about 1,900 (15T tokens). The extra training costs more once but makes every later token cheaper.

> **After 3.1:** Llama 3.2 (September 2024) added small 1B and 3B text models, which tie their input and output embeddings to save parameters, and 11B and 90B vision models. Llama 4 (April 2025) switched to **Mixture of experts (MoE)**. This article covers the dense Llama 1–3.1 line.

**Strengths**

- A simple, well-tested block (RMSNorm, SwiGLU, RoPE, no biases) that most open LLMs copy
- No position table: context can be extended by rescaling RoPE frequencies and training on longer data
- GQA cuts the KV cache 4× (8B) to 16× (405B) at little quality cost
- Small models trained far past compute-optimal are cheap to serve
- Open weights with exact configs make the family the reference for fine-tuning, quantisation and serving work

**Limitations**

- Over-training costs extra training compute per unit of quality
- Untied 128K-vocabulary tables cost 1.05B parameters in the 8B model, 13% of it
- Dense: every token runs all parameters, unlike MoE models of similar quality
- Custom licences, not open-source licences; the training data is not released
- SwiGLU's three matrices mean three MLP matmuls and more activation memory per token

**Question:** LLaMA replaced GPT-3's GELU MLP of width 4d with SwiGLU. Why is the hidden width about 2.67d rather than 4d?

- A. SiLU saturates beyond 2.67d
- B. SwiGLU has three weight matrices, so width 8d/3 keeps the parameter count at 8d², the same as a two-matrix MLP of width 4d
- C. RoPE needs the MLP width to be a multiple of the head size
- D. GQA removes a third of the MLP

**Answer:** B. SwiGLU has three weight matrices, so width 8d/3 keeps the parameter count at 8d², the same as a two-matrix MLP of width 4d — A GELU MLP has $W_1\in\mathbb R^{4d\times d}$ and $W_2\in\mathbb R^{d\times4d}$, $8d^2$ in total. SwiGLU adds the gate $W_3$, so three matrices of width $d_{ff}$ cost $3d\,d_{ff}$. Setting that equal to $8d^2$ gives $d_{ff}=\tfrac83d$. The released widths then round up to a multiple (11008 for $d=4096$).

**Question:** Llama 3.1 405B has 126 layers, 8 KV heads of size 128 and 128 query heads. Roughly how large is the bf16 KV cache for one 128K-token sequence?

- A. 4 GiB
- B. 63 GiB
- C. 504 GiB
- D. 1008 GiB

**Answer:** B. 63 GiB — $2\cdot126\cdot8\cdot128\cdot2$ bytes = 504 KiB per token, times $131{,}072$ tokens = 63 GiB. Full MHA with 128 KV heads would be 16× larger, 1008 GiB. This is why every Llama 3 size uses GQA.

**Key points**

- The Llama block is a pre-norm decoder with RMSNorm $g\odot x/\mathrm{rms}(x)$, SwiGLU $W_2(\mathrm{SiLU}(W_1x)\odot W_3x)$, RoPE on $q,k$, no biases and an untied head.
- $d_{ff}=k\lceil m\lfloor8d/3\rfloor/k\rceil$ gives every released MLP width: 11008, 14336, 28672, 53248.
- LLaMA: 7–65B, 1.0–1.4T public tokens, 2K context, 32K SentencePiece vocab; 13B beat GPT-3 on most benchmarks.
- Llama 2: 2T tokens, 4K context, GQA in 34B/70B, chat models via SFT + RLHF. Llama 3/3.1: 128K tiktoken-based vocab, 15T+ tokens, RoPE base 500,000, GQA everywhere, 8K → 128K context, 405B.
- The design goal is inference cost: train small models far past compute-optimal and shrink the KV cache with GQA.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
