# Pretraining and continued pretraining

> Pretraining fits next-token loss on trillions of filtered, deduplicated, mixed tokens. Chinchilla puts the compute optimum near 20 tokens per parameter (C ≈ 6ND). Continued pretraining resumes on new data with a re-warmed learning rate and a few percent replay to limit forgetting.

Source: https://www.mlfoundry.in/learn/pretraining-and-continued-pretraining  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 16 min read · updated 2026-10-13

**Pretraining:** is the first and most expensive stage of building a language model. A randomly initialised network is trained with the self-supervised next-token loss on a very large unlabeled corpus, and the result is a **base model**. **Continued pretraining** (CPT) resumes the same objective from a pre-trained checkpoint on different data: a new domain, a new language, fresher text or longer documents. Three papers define the modern practice. Kaplan et al. ("Scaling Laws for Neural Language Models", 2020) and Hoffmann et al. ("Training Compute-Optimal Large Language Models", NeurIPS 2022, the Chinchilla paper) set how big a model should be for a given budget. Gururangan et al. ("Don't Stop Pretraining: Adapt Language Models to Domains and Tasks", ACL 2020) showed that a second phase of pre-training on in-domain text pays off.

The corpora grew by four orders of magnitude in five years. GPT (2018) trained on BooksCorpus, about 7,000 unpublished books. GPT-2 (2019) used WebText, 40 GB of pages linked from Reddit posts with at least 3 karma. T5's C4 (2020) was about 750 GB of filtered Common Crawl, and The Pile (Gao et al., 2020) mixed 22 sources into 825 GiB. RefinedWeb (Penedo et al., 2023) showed that carefully filtered and deduplicated web data alone could match curated mixtures. FineWeb (Penedo et al., 2024) released 15T tokens from 96 Common Crawl snapshots. Llama 3 trained on about 15T tokens and Qwen2.5 on 18T. As corpora grew, **which** tokens to train on became as important as how many.

## The data pipeline

A modern pipeline turns raw crawl archives into a token stream in a fixed sequence of steps. **Extraction** pulls main text out of HTML: FineWeb found that re-extracting from WARC files with trafilatura beat the WET text that Common Crawl ships. **Language identification** keeps documents whose fastText language score passes a threshold. **Quality filtering** combines rules with classifiers. The Gopher rules (Rae et al., 2021) keep documents of 50 to 100,000 words with a mean word length of 3 to 10 characters, in which at least 80% of words contain a letter. They also drop documents with too many symbols, bullet lines or repeated lines. Classifier filters score each page against a reference of "good" text. GPT-3 used one trained to separate WebText, books and Wikipedia from raw Common Crawl. FineWeb-Edu had Llama-3-70B rate 450K pages for educational value, trained a small classifier on those ratings, and kept 1.3T tokens.

**Deduplication** follows. Lee et al. ("Deduplicating Training Data Makes Language Models Better", ACL 2022) found a single 61-word English sentence repeated over 60,000 times in C4. Removing exact repeated substrings (with a suffix array) and near-duplicate documents (with MinHash) gave models that emitted memorised text about ten times less often and needed fewer steps for the same accuracy. MinHash estimates the Jaccard similarity of two documents' sets of word $n$-grams, and locality-sensitive hashing turns it into a cheap candidate test:

$$
J(A,B)=\frac{|A\cap B|}{|A\cup B|},\qquad \Pr\big[\min h(A)=\min h(B)\big]=J(A,B),\qquad \Pr[\text{candidate}]=1-\big(1-s^{\,r}\big)^{b}
$$

- $A,\,B$: sets of word 5-grams ("shingles") of two documents
- $h$: a random hash; the chance that both sets share the minimum equals their Jaccard similarity
- $b,\ r$: $b$ bands of $r$ hashes; a pair becomes a candidate if all $r$ hashes agree in any band. FineWeb used 112 hashes as $b=14$, $r=8$
- $s$: true similarity: with 14 × 8, a pair at $s=0.75$ is caught with probability 0.772, at $s=0.5$ only 0.053

The last steps are **safety filters** (PII removal or masking, toxicity and adult-content filters, URL blocklists), **tokenization** (see **Byte-pair encoding (BPE)**), and **mixing**. Each source gets a sampling weight. Llama 1's mix was 67% CommonCrawl, 15% C4, 4.5% each GitHub, Wikipedia and books, 2.5% arXiv and 2% StackExchange. Llama 3's final mix was about 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual. Mixture weights can be learned. DoReMi (Xie et al., NeurIPS 2023) tunes them with a small proxy model and transfers them to a large one. Repetition is acceptable in moderation: Muennighoff et al. ("Scaling Data-Constrained Language Models", NeurIPS 2023) found up to about 4 epochs of repeated data almost as good as fresh data, with value decaying quickly after that.

> Figure: The full path from crawl to domain model. Steps 1–6 turn raw Common Crawl into clean tokens. Step 7 sets the domain mixture, shown here as Llama 3's final proportions. Step 8 trains the base model: linear warmup, then either a cosine decay to about 10% of the peak or a warmup-stable-decay (WSD) schedule, ending with an anneal on the best data. Step 9 continues training on a new domain. It re-warms the learning rate, re-decays it, and mixes a fraction r of the original data into each batch to limit forgetting.

## How big, how long: compute-optimal scaling

Training compute is about $C\approx6ND$ FLOPs for $N$ parameters and $D$ tokens (see **Decoder-only transformer**). For a fixed $C$, a model can be large and briefly trained or small and long-trained. Kaplan et al. (2020) fitted power laws and recommended spending most extra compute on size, with $N_{opt}\propto C^{0.73}$. GPT-3 (175B, 300B tokens) and Gopher (280B, 300B tokens) followed that advice. Hoffmann et al. retrained over 400 models from 70M to 16B parameters on 5B to 500B tokens. They found that $N$ and $D$ should grow in **equal proportion**, and traced part of the discrepancy to Kaplan's fixed learning-rate schedule: a cosine cycle much longer than the run leaves short runs under-trained. Their third method fits a parametric loss directly:

$$
L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},\qquad \min_{N,D}\,L\ \ \text{s.t.}\ 6ND=C\ \Rightarrow\ \frac{\alpha A}{N^{\alpha}}=\frac{\beta B}{D^{\beta}},\qquad N_{opt}=G\Big(\frac C6\Big)^{\frac{\beta}{\alpha+\beta}},\ D_{opt}=G^{-1}\Big(\frac C6\Big)^{\frac{\alpha}{\alpha+\beta}},\ G=\Big(\frac{\alpha A}{\beta B}\Big)^{\frac1{\alpha+\beta}}
$$

- $E$: irreducible loss, the entropy of text under this tokenizer and data
- $A/N^{\alpha}$: excess loss from finite model size
- $B/D^{\beta}$: excess loss from finite data
- $\alpha A/N^\alpha=\beta B/D^\beta$: at the optimum the two excess terms are in the ratio $\beta:\alpha$; substitute $D=C/6N$ and solve for $N$
- $\text{fit}$: published: $E=1.69$, $A=406.4$, $B=410.7$, $\alpha=0.34$, $\beta=0.28$. Besiroglu et al. (2024) refit the paper's data: $E=1.82$, $A=482.01$, $B=2085.43$, $\alpha=0.3478$, $\beta=0.3658$

Hoffmann et al.'s first two methods both gave exponents near 0.5 for $N$ and $D$ and about **20 tokens per parameter**. To test it they trained **Chinchilla**, 70B parameters on 1.4T tokens, with the compute of the 280B Gopher. It was better on almost every benchmark, for example 67.6% on MMLU against Gopher's 60.0%, and four times cheaper to serve. The published constants of the third method imply smaller models and more tokens (about 98 tokens per parameter at $10^{24}$ FLOPs). Besiroglu, Erdil, Barnett and You ("Chinchilla Scaling: A replication attempt", 2024) reconstructed the data from the paper's figures. They found that the reported fit did not match it, and that their refit agreed with the 20-tokens rule. Both fits are in the figure.

The Chinchilla optimum minimises loss per **training** FLOP. A deployed model also pays for every inference token, which favours a smaller model trained longer (Sardana and Frankle, "Beyond Chinchilla-Optimal", 2023). Llama 2 7B saw about 290 tokens per parameter and Llama 3 8B about 1,900. For its flagship Meta fitted its own isoFLOP curves and trained Llama 3.1 405B close to that optimum, at about $3.8\times10^{25}$ FLOPs.

## Learning-rate schedules

$$
\eta_t^{\cos}=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\Big(1+\cos\frac{\pi(t-t_w)}{T-t_w}\Big),\qquad \eta_t^{\text{WSD}}=\begin{cases}\eta_{\max}\,t/t_w & t<t_w\\ \eta_{\max} & t_w\le t<T-t_d\\ \text{decay to }\eta_{\min} & t\ge T-t_d\end{cases}
$$

- $t_w$: linear warmup steps; Llama 2 used 2,000
- $\eta_{\min}$: Llama 2 decays to 10% of the peak ($3\times10^{-4}$ for 7B and 13B, $1.5\times10^{-4}$ for 70B)
- $T$: total steps; a cosine schedule must be set to the planned run length in advance
- $t_d$: WSD cooldown length, typically 10–20% of the run

Cosine decay fixes the run length up front. Stopping early, or extending later, leaves the model at a sub-optimal point of its schedule. **Warmup-stable-decay** (WSD, introduced with MiniCPM by Hu et al., 2024) holds the peak rate for most of training and decays only in a short final phase. Any stable-phase checkpoint can be branched into a cooldown, so one run yields models for several token budgets. Hägele et al. (2024) found that a cooldown of about 20% of the run matches cosine. The end of training is also where data quality matters most. Llama 3 finishes with an **annealing** phase that decays the learning rate to zero on upsampled high-quality sources.

**Published pretraining setups**

- **Chinchilla:** 70B, 80 layers, $d=8192$, 64 heads; 1.4T MassiveText tokens; AdamW; cosine decay by $10\times$ over a cycle matched to the run
- **Llama 2:** 2T tokens; AdamW $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-5}$, weight decay 0.1, gradient clipping 1.0; 2,000 warmup steps, cosine to 10% of peak; batch 4M tokens
- **Llama 3.1 405B:** about 15.6T tokens; peak lr $8\times10^{-5}$, 8,000 warmup steps, cosine to $8\times10^{-7}$; context grown in stages from 8K to 128K; final annealing on high-quality data
- **FineWeb:** 15T tokens from 96 Common Crawl snapshots; trafilatura extraction, fastText English filter, Gopher and C4 rules, MinHash deduplication within each snapshot
- **Code Llama:** continued pretraining of Llama 2 on 500B tokens of code-heavy data, then long-context fine-tuning on 16K sequences with RoPE base raised from $10^4$ to $10^6$
- **DAPT (Gururangan et al.):** RoBERTa continued on four domains (biomedical papers, computer-science papers, news, reviews), evaluated on eight classification tasks

## Continued pretraining

A base model is a compressed snapshot of its corpus. When the target is a domain or language the corpus covered thinly, it is cheaper to continue pre-training than to start over. Gururangan et al. continued RoBERTa on unlabeled in-domain text (**DAPT**, domain-adaptive pretraining) and on the unlabeled text of the task itself (**TAPT**). DAPT improved results in all four domains. TAPT was far cheaper and also helped, and DAPT followed by TAPT was best. The gains were largest where the domain was most dissimilar to RoBERTa's training data.

Decoder-only LLMs use CPT the same way, at larger scale. Code Llama continued Llama 2 on 500B tokens of code. Llemma (Azerbayev et al., 2023) continued Code Llama on mathematical text. Long-context extension is also CPT, on long documents with a modified RoPE base (see **Rotary positional embeddings (RoPE)**, **Context window optimization**). After CPT come **Supervised fine-tuning (SFT)** and preference tuning, which use far fewer tokens.

The cost of CPT is **catastrophic forgetting** (McCloskey and Cohen, 1989). Gradients from the new domain move weights the old capabilities depend on, and nothing pulls them back. Two choices dominate. The **learning rate**: a pre-trained model ends at a tiny learning rate, and new data needs a larger one to make progress. Gupta et al. ("Continual Pre-Training of Large Language Models: How to (re)warm your model?", 2023) found that re-warming raises the loss on the original data at first, even when it improves adaptation. **Replay**: mix a fraction of the original data into every batch. Ibrahim et al. ("Simple and Scalable Strategies to Continually Pre-train Large Language Models", 2024) combined re-warming, re-decaying and replay. With as little as 5% replay for a mild distribution shift, and more for a strong one such as a new language, they matched retraining from scratch on all the data at a fraction of the cost.

$$
\mathcal L_{\text{CPT}}(\theta)=(1-r)\,\mathbb E_{x\sim\mathcal D_{\text{new}}}\big[-\log p_\theta(x)\big]+r\,\mathbb E_{x\sim\mathcal D_{\text{old}}}\big[-\log p_\theta(x)\big]
$$

- $r$: replay fraction: the share of each batch drawn from the original pre-training mix
- $\mathcal D_{\text{old}}$: the original data, or a proxy for it if it is not available (open web text of similar quality)
- $\theta_0$: initialised from the base model; the optimiser state is often reset, with the learning rate re-warmed to a peak below the original

> Figure: Two views. Compute-optimal (default) plots the Chinchilla loss with the Besiroglu et al. refit. At C = 10²⁴ FLOPs the optimum is N = 95.9B and D = 1.74T, which is 18.1 tokens per parameter, at L = 1.9625. A model 4× smaller trained on 4× the tokens (290 per parameter) loses 0.0176 nats, and an optimal run reaches that loss with 52.0% of the compute. A model 4× larger is about as wasteful (51.4%). A 2× error either way costs about 16% (84.3–84.4%), and 16× smaller costs 90% (9.5%). These shares do not depend on C. With the published approach-3 constants the optimum at 10²⁴ moves to 41.3B and 4.04T (97.7 tokens per parameter). Continued pre-training is a 4-weight toy with an exact solution for both domains; curves are smoothed and final losses are means over the last 40 of 400 steps. With no replay and a re-warmed peak of 0.3, the model forgets: old-domain excess loss ends at 0.4501 (smoothed worst 0.519) while the new domain reaches 0.0012. With 5% replay the old loss ends at 0.0143, and with 10% at 0.0017, at no cost to the new domain. A peak of 1.0 adapts and forgets faster and leaves a noise floor of 0.0075 (old) and 0.0083 (new) even with 5% replay. A peak of 0.01 forgets less (0.3607) but barely adapts (new 0.0247). At a peak of 0.03, 5% replay only brings the old loss to 0.3308: replay needs enough learning rate and steps to pull the weights back.

## Worked example: spending Gopher's budget

Gopher used $C=6\cdot280\cdot10^9\cdot300\cdot10^9=5.04\times10^{23}$ FLOPs by the $6ND$ rule. Here is how the refit loss would allocate the same budget, and where other well-known models fall. The loss values come from a fit to Chinchilla's own data and tokenizer, so only the comparisons between rows are meaningful. They are not predictions of these models' real losses.

Allocation under L(N, D) with the Besiroglu et al. refit (computed in Python and checked against the figure's code)

| model | N | D | D/N | C = 6ND | L(N, D) | optimal run, same L |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-3 | 175B | 300B | 1.7 | 3.15e23 | 2.0113 | 60.8% of C |
| Gopher | 280B | 300B | 1.1 | 5.04e23 | 2.0024 | 49.7% of C |
| Chinchilla | 70B | 1.4T | 20.0 | 5.88e23 | 1.9767 | 99.9% of C |
| Llama 2 7B | 7B | 2T | 285.7 | 8.40e22 | 2.0675 | 53.8% of C |
| Llama 3 8B | 8B | 15T | 1,875 | 7.20e23 | 2.0249 | 18.1% of C |
| Llama 3.1 405B | 405B | 15.6T | 38.5 | 3.79e25 | 1.8954 | 93.9% of C |

**Step by step**

1. Rule of thumb: $D=20N$ gives $C=120N^2$, so $N=\sqrt{5.04\times10^{23}/120}=64.8$B and $D=1.30$T.
2. Closed form with the refit: $G=(\alpha A/\beta B)^{1/(\alpha+\beta)}$ and $N_{opt}=G\,(C/6)^{0.5126}=67.5$B, $D_{opt}=1.245$T, $D/N=18.5$, $L=1.9810$.
3. Gopher's own split: $A/N^\alpha=0.0503$ and $B/D^\beta=0.1321$. The data term dominates, so the model is under-trained. At the optimum the terms are $0.0825$ and $0.0785$, in the ratio $\beta/\alpha=1.052$ that the optimality condition requires.
4. Gopher's loss $2.0024$ is reached by an optimally split run with $49.7\%$ of the compute: roughly half of the budget went into excess size.
5. Llama 3 8B reaches only $L=2.0249$ with $7.2\times10^{23}$ FLOPs, a loss an optimal run (about 81B parameters) would reach with $18.1\%$ of that compute. But that 81B model would cost about 10 times more per generated token for its whole deployed life.

```python
import re, zlib
import numpy as np

# ---- 1) Near-duplicate detection with MinHash + LSH banding ----
def shingles(text, n=5):
    w = re.findall(r"\w+", text.lower())
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

P = (1 << 61) - 1                                   # Mersenne prime for universal hashing
rs = np.random.default_rng(0)
A = rs.integers(1, P, size=112, dtype=np.uint64)    # 112 hash functions = 14 bands x 8 rows
B = rs.integers(0, P, size=112, dtype=np.uint64)

def minhash(text):
    h = np.array([zlib.crc32(s.encode()) for s in shingles(text)], dtype=np.uint64)
    # (a*h + b) mod P, computed in Python ints to avoid uint64 overflow
    return np.array([min((int(a) * int(x) + int(b)) % P for x in h) for a, b in zip(A, B)])

def lsh_candidate_prob(s, bands=14, rows=8):        # P(two docs with Jaccard s share a band)
    return 1 - (1 - s ** rows) ** bands

doc1 = "The quick brown fox jumps over the lazy dog near the river bank on a sunny afternoon in May"
doc2 = "The quick brown fox jumps over the lazy dog near the river bank on a rainy afternoon in May"
s1, s2 = shingles(doc1), shingles(doc2)
jac = len(s1 & s2) / len(s1 | s2)
est = np.mean(minhash(doc1) == minhash(doc2))
print(f"Jaccard {jac:.3f}, MinHash estimate {est:.3f}")    # Jaccard 0.579, MinHash estimate 0.580
for s in (0.5, 0.75, 0.9):
    print(f"s = {s}: candidate probability {lsh_candidate_prob(s):.4f}")   # 0.0533, 0.7716, 0.9996

# ---- 2) Compute-optimal allocation from L(N, D) = E + A/N^a + B/D^b ----
def chinchilla_optimal(C, E=1.82, A_=482.01, B_=2085.43, a=0.3478, b=0.3658):
    G = (a * A_ / (b * B_)) ** (1 / (a + b))
    N = G * (C / 6) ** (b / (a + b))
    D = C / 6 / N
    return N, D, E + A_ / N ** a + B_ / D ** b

for C in (1e21, 1e23, 1e25):
    N, D, L = chinchilla_optimal(C)
    print(f"C = {C:.0e}: N = {N / 1e9:7.2f}B, D = {D / 1e9:8.1f}B tokens, D/N = {D / N:4.1f}, L = {L:.3f}")
# C = 1e+21: N =    2.78B, D =     60.0B tokens, D/N = 21.6, L = 2.308
# C = 1e+23: N =   29.45B, D =    566.0B tokens, D/N = 19.2, L = 2.035
# C = 1e+25: N =  312.07B, D =   5340.7B tokens, D/N = 17.1, L = 1.915

# ---- 3) Warmup-stable-decay schedule (re-use it with a lower peak for continued pretraining) ----
def wsd(step, total, peak, warmup=2000, decay_frac=0.1, floor=0.0):
    if step < warmup:
        return peak * (step + 1) / warmup
    decay_start = int(total * (1 - decay_frac))
    if step < decay_start:
        return peak
    p = (step - decay_start) / (total - decay_start)
    return floor + (peak - floor) * (1 - p)          # linear cooldown

print([round(wsd(t, 100_000, 3e-4), 7) for t in (0, 1999, 50_000, 95_000, 99_999)])
# [1e-07, 0.0003, 0.0003, 0.00015, 0.0]
```

MinHash near-duplicate detection with LSH banding, the closed-form Chinchilla allocation, and a warmup-stable-decay schedule (numpy; run, outputs in comments)

> **A fixed cosine schedule biases scaling experiments:** If every run in a sweep uses a cosine schedule sized for the longest run, the short runs never decay and look worse than they are. This is one reason Kaplan et al. overestimated the value of model size. Size each schedule to its own run length, or use WSD and branch cooldowns from a shared stable phase.

> **Forgetting is drift that no data resists:** In the figure's toy, one weight vector fits both domains perfectly, so forgetting is not inevitable. New-domain gradients only act inside the new data's subspace. They change a coordinate the old domain relies on, and with no old data in the batch nothing pushes it back. A little replay supplies that restoring force. Replay matters more than its token share suggests: 5% of each batch removes almost all of the 0.45 forgetting at a moderate learning rate.

> **Deduplicate across splits, too:** Near-duplicates of benchmark items in the pre-training corpus inflate evaluation scores (see **LLM evaluation and benchmarking**). Lee et al. found C4's validation set overlapping its training set. Run the same MinHash pass between the training corpus and every evaluation set, and between the original corpus and any CPT data that will be replayed.

**Strengths**

- Compute-optimal scaling (≈ 20 tokens per parameter) gives the lowest loss per training FLOP and is easy to apply
- Filtering and deduplication improve both quality and memorisation behaviour at no inference cost
- WSD schedules allow branching checkpoints at several budgets from one run
- Continued pretraining adapts a strong base model to a domain or language for a small fraction of the original compute
- Replay of a few percent and a re-warmed, re-decayed learning rate largely prevent forgetting

**Limitations**

- Loss-optimal is not deployment-optimal: inference cost pushes models far past 20 tokens per parameter
- Scaling-law constants depend on data, tokenizer and architecture, and fitted exponents differ between studies
- Classifier filters inherit the biases of their reference text and can remove dialects or minority content
- CPT needs access to the original data or a good proxy for replay, and the peak learning rate trades adaptation against forgetting

- **Base models:** Every open LLM family (Llama, Qwen, Mistral, Gemma, DeepSeek) starts from a trillions-of-tokens pretraining run with this pipeline.
- **Domain models:** Code (Code Llama), mathematics (Llemma), biomedicine, law and finance models built by continued pretraining on domain corpora.
- **Language and context extension:** Adding languages, or extending the context window, with CPT on targeted data and replay of the original mix.

**Question:** You have 1e24 FLOPs. Under the Chinchilla refit, the optimum is about 96B parameters on 1.74T tokens. You train a 24B model on 7.0T tokens instead. What do you pay?

- A. Nothing: loss depends only on compute
- B. About 0.018 nats of loss; an optimal run would reach your loss with about 52% of the compute. You get a model 4× cheaper to serve
- C. The loss doubles
- D. The run diverges, because small models cannot use that many tokens

**Answer:** B. About 0.018 nats of loss; an optimal run would reach your loss with about 52% of the compute. You get a model 4× cheaper to serve — $L(N,D)$ is flat near its optimum along $6ND=C$. A 4× size error costs about half of the compute, a 2× error about 16%. That is why inference-heavy models deliberately sit to the left of the optimum.

**Question:** A team continues a base model on medical text with no replay and its original peak learning rate. General benchmarks drop sharply. Which change most directly addresses this?

- A. Train for more steps on the medical data
- B. Mix a few percent of general pre-training data into every batch and use a lower re-warmed peak learning rate, re-decayed to the end
- C. Remove warmup
- D. Increase the vocabulary size

**Answer:** B. Mix a few percent of general pre-training data into every batch and use a lower re-warmed peak learning rate, re-decayed to the end — Forgetting comes from new-domain updates moving weights the old data relied on. Replay adds back a restoring gradient, and a moderate peak limits how far each update moves. Ibrahim et al. (2024) matched full retraining with this combination. In the figure, 5% replay at peak 0.3 cuts the final old-domain loss from 0.4501 to 0.0143.

**Key points**

- Pipeline: extract → language ID → quality filter (rules + classifier) → deduplicate (exact + MinHash) → safety filters → tokenize → mix domains; data quality now matters as much as quantity.
- Chinchilla: $L=E+A/N^{\alpha}+B/D^{\beta}$ under $C\approx6ND$ gives $N_{opt},D_{opt}\propto C^{\approx0.5}$, about 20 tokens per parameter. Chinchilla 70B on 1.4T beat Gopher 280B on 300B at the same compute.
- Mis-sizing by 2× costs about 16% of compute and by 4× about half; inference-heavy models are deliberately over-trained (Llama 3 8B: about 1,900 tokens per parameter).
- Schedules: warmup then cosine to about 10% of peak, or warmup-stable-decay with a short cooldown, finishing with an anneal on high-quality data.
- Continued pretraining (DAPT, TAPT, code, maths, long context) resumes next-token training on new data. Re-warm and re-decay the learning rate and replay about 5% or more of the old data to limit catastrophic forgetting.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
