Pretraining and continued pretraining

Pretraining fits next-token loss on trillions of filtered, deduplicated, mixed tokens. Chinchilla puts the compute optimum near 20 tokens per parameter (C ≈ 6ND). Continued pretraining resumes on new data with a re-warmed learning rate and a few percent replay to limit forgetting.

16 min read1842 wordscore0 viewsupdated 2026-10-13

Definition

Pretraining is the first and most expensive stage of building a language model. A randomly initialised network is trained with the self-supervised next-token loss on a very large unlabeled corpus, and the result is a base model. Continued pretraining (CPT) resumes the same objective from a pre-trained checkpoint on different data: a new domain, a new language, fresher text or longer documents. Three papers define the modern practice. Kaplan et al. ("Scaling Laws for Neural Language Models", 2020) and Hoffmann et al. ("Training Compute-Optimal Large Language Models", NeurIPS 2022, the Chinchilla paper) set how big a model should be for a given budget. Gururangan et al. ("Don't Stop Pretraining: Adapt Language Models to Domains and Tasks", ACL 2020) showed that a second phase of pre-training on in-domain text pays off.

The corpora grew by four orders of magnitude in five years. GPT (2018) trained on BooksCorpus, about 7,000 unpublished books. GPT-2 (2019) used WebText, 40 GB of pages linked from Reddit posts with at least 3 karma. T5's C4 (2020) was about 750 GB of filtered Common Crawl, and The Pile (Gao et al., 2020) mixed 22 sources into 825 GiB. RefinedWeb (Penedo et al., 2023) showed that carefully filtered and deduplicated web data alone could match curated mixtures. FineWeb (Penedo et al., 2024) released 15T tokens from 96 Common Crawl snapshots. Llama 3 trained on about 15T tokens and Qwen2.5 on 18T. As corpora grew, which tokens to train on became as important as how many.

The data pipeline

A modern pipeline turns raw crawl archives into a token stream in a fixed sequence of steps. Extraction pulls main text out of HTML: FineWeb found that re-extracting from WARC files with trafilatura beat the WET text that Common Crawl ships. Language identification keeps documents whose fastText language score passes a threshold. Quality filtering combines rules with classifiers. The Gopher rules (Rae et al., 2021) keep documents of 50 to 100,000 words with a mean word length of 3 to 10 characters, in which at least 80% of words contain a letter. They also drop documents with too many symbols, bullet lines or repeated lines. Classifier filters score each page against a reference of "good" text. GPT-3 used one trained to separate WebText, books and Wikipedia from raw Common Crawl. FineWeb-Edu had Llama-3-70B rate 450K pages for educational value, trained a small classifier on those ratings, and kept 1.3T tokens.

Deduplication follows. Lee et al. ("Deduplicating Training Data Makes Language Models Better", ACL 2022) found a single 61-word English sentence repeated over 60,000 times in C4. Removing exact repeated substrings (with a suffix array) and near-duplicate documents (with MinHash) gave models that emitted memorised text about ten times less often and needed fewer steps for the same accuracy. MinHash estimates the Jaccard similarity of two documents' sets of word nn-grams, and locality-sensitive hashing turns it into a cheap candidate test:

J(A,B)=∣A∩B∣∣A∪B∣,Pr⁡[min⁡h(A)=min⁡h(B)]=J(A,B),Pr⁡[candidate]=1−(1−s r)bJ(A,B)=\frac{|A\cap B|}{|A\cup B|},\qquad \Pr\big[\min h(A)=\min h(B)\big]=J(A,B),\qquad \Pr[\text{candidate}]=1-\big(1-s^{\,r}\big)^{b}
A, BA,\,B
sets of word 5-grams ("shingles") of two documents
hh
a random hash; the chance that both sets share the minimum equals their Jaccard similarity
b, rb,\ r
bb bands of rr hashes; a pair becomes a candidate if all rr hashes agree in any band. FineWeb used 112 hashes as b=14b=14, r=8r=8
ss
true similarity: with 14 × 8, a pair at s=0.75s=0.75 is caught with probability 0.772, at s=0.5s=0.5 only 0.053

The last steps are safety filters (PII removal or masking, toxicity and adult-content filters, URL blocklists), tokenization (see Byte-pair encoding (BPE)), and mixing. Each source gets a sampling weight. Llama 1's mix was 67% CommonCrawl, 15% C4, 4.5% each GitHub, Wikipedia and books, 2.5% arXiv and 2% StackExchange. Llama 3's final mix was about 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual. Mixture weights can be learned. DoReMi (Xie et al., NeurIPS 2023) tunes them with a small proxy model and transfers them to a large one. Repetition is acceptable in moderation: Muennighoff et al. ("Scaling Data-Constrained Language Models", NeurIPS 2023) found up to about 4 epochs of repeated data almost as good as fresh data, with value decaying quickly after that.

raw crawl → curated token stream → base model → domain model1 extract textWARC/HTML → text2 language IDfastText classifier3 quality filterrules + classifier4 deduplicateURL, exact, MinHash5 PII, toxicityremove, mask, filter6 tokenizeBPE, 32K–256K vocab7 mix domains (Llama 3 final mix)■ general 50% ■ math 25%■ code 17% ■ multilingual 8%FineWeb: 15T tokens from 96 crawls8 pre-train: next-token loss, C ≈ 6ND, D ≈ 20N or morewarmupWSD: stable- - cosine to ~10% of peak→ decay / annealanneal on the highest-quality data at the endbase modele.g. Llama 2, 2T tokens9 continued pre-training on a new domainbatch = (1 − r) new domain + r replay of the original mixre-warm, then re-decayrisk: forgettingfix: replay ~5%+,moderate peak lrdomain modelCode Llama: +500Balso: DAPT, TAPT,long-context CPT
fig.The full path from crawl to domain model. Steps 1–6 turn raw Common Crawl into clean tokens. Step 7 sets the domain mixture, shown here as Llama 3's final proportions. Step 8 trains the base model: linear warmup, then either a cosine decay to about 10% of the peak or a warmup-stable-decay (WSD) schedule, ending with an anneal on the best data. Step 9 continues training on a new domain. It re-warms the learning rate, re-decays it, and mixes a fraction r of the original data into each batch to limit forgetting.

How big, how long: compute-optimal scaling

Training compute is about C≈6NDC\approx6ND FLOPs for NN parameters and DD tokens (see Decoder-only transformer). For a fixed CC, a model can be large and briefly trained or small and long-trained. Kaplan et al. (2020) fitted power laws and recommended spending most extra compute on size, with Nopt∝C0.73N_{opt}\propto C^{0.73}. GPT-3 (175B, 300B tokens) and Gopher (280B, 300B tokens) followed that advice. Hoffmann et al. retrained over 400 models from 70M to 16B parameters on 5B to 500B tokens. They found that NN and DD should grow in equal proportion, and traced part of the discrepancy to Kaplan's fixed learning-rate schedule: a cosine cycle much longer than the run leaves short runs under-trained. Their third method fits a parametric loss directly:

L(N,D)=E+ANα+BDβ,min⁡N,D L  s.t. 6ND=C ⇒ αANα=βBDβ,Nopt=G(C6)βα+β, Dopt=G−1(C6)αα+β, G=(αAβB)1α+βL(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},\qquad \min_{N,D}\,L\ \ \text{s.t.}\ 6ND=C\ \Rightarrow\ \frac{\alpha A}{N^{\alpha}}=\frac{\beta B}{D^{\beta}},\qquad N_{opt}=G\Big(\frac C6\Big)^{\frac{\beta}{\alpha+\beta}},\ D_{opt}=G^{-1}\Big(\frac C6\Big)^{\frac{\alpha}{\alpha+\beta}},\ G=\Big(\frac{\alpha A}{\beta B}\Big)^{\frac1{\alpha+\beta}}
EE
irreducible loss, the entropy of text under this tokenizer and data
A/NαA/N^{\alpha}
excess loss from finite model size
B/DβB/D^{\beta}
excess loss from finite data
αA/Nα=βB/Dβ\alpha A/N^\alpha=\beta B/D^\beta
at the optimum the two excess terms are in the ratio β:α\beta:\alpha; substitute D=C/6ND=C/6N and solve for NN
fit\text{fit}
published: E=1.69E=1.69, A=406.4A=406.4, B=410.7B=410.7, α=0.34\alpha=0.34, β=0.28\beta=0.28. Besiroglu et al. (2024) refit the paper's data: E=1.82E=1.82, A=482.01A=482.01, B=2085.43B=2085.43, α=0.3478\alpha=0.3478, β=0.3658\beta=0.3658

Hoffmann et al.'s first two methods both gave exponents near 0.5 for NN and DD and about 20 tokens per parameter. To test it they trained Chinchilla, 70B parameters on 1.4T tokens, with the compute of the 280B Gopher. It was better on almost every benchmark, for example 67.6% on MMLU against Gopher's 60.0%, and four times cheaper to serve. The published constants of the third method imply smaller models and more tokens (about 98 tokens per parameter at 102410^{24} FLOPs). Besiroglu, Erdil, Barnett and You ("Chinchilla Scaling: A replication attempt", 2024) reconstructed the data from the paper's figures. They found that the reported fit did not match it, and that their refit agreed with the 20-tokens rule. Both fits are in the figure.

The Chinchilla optimum minimises loss per training FLOP. A deployed model also pays for every inference token, which favours a smaller model trained longer (Sardana and Frankle, "Beyond Chinchilla-Optimal", 2023). Llama 2 7B saw about 290 tokens per parameter and Llama 3 8B about 1,900. For its flagship Meta fitted its own isoFLOP curves and trained Llama 3.1 405B close to that optimum, at about 3.8×10253.8\times10^{25} FLOPs.

Learning-rate schedules

ηtcos⁡=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡π(t−tw)T−tw),ηtWSD={ηmax⁡ t/twt<twηmax⁡tw≤t<T−tddecay to ηmin⁡t≥T−td\eta_t^{\cos}=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\Big(1+\cos\frac{\pi(t-t_w)}{T-t_w}\Big),\qquad \eta_t^{\text{WSD}}=\begin{cases}\eta_{\max}\,t/t_w & t<t_w\\ \eta_{\max} & t_w\le t<T-t_d\\ \text{decay to }\eta_{\min} & t\ge T-t_d\end{cases}
twt_w
linear warmup steps; Llama 2 used 2,000
ηmin⁡\eta_{\min}
Llama 2 decays to 10% of the peak (3×10−43\times10^{-4} for 7B and 13B, 1.5×10−41.5\times10^{-4} for 70B)
TT
total steps; a cosine schedule must be set to the planned run length in advance
tdt_d
WSD cooldown length, typically 10–20% of the run

Cosine decay fixes the run length up front. Stopping early, or extending later, leaves the model at a sub-optimal point of its schedule. Warmup-stable-decay (WSD, introduced with MiniCPM by Hu et al., 2024) holds the peak rate for most of training and decays only in a short final phase. Any stable-phase checkpoint can be branched into a cooldown, so one run yields models for several token budgets. Hägele et al. (2024) found that a cooldown of about 20% of the run matches cosine. The end of training is also where data quality matters most. Llama 3 finishes with an annealing phase that decays the learning rate to zero on upsampled high-quality sources.

Published pretraining setups

Chinchilla
70B, 80 layers, d=8192d=8192, 64 heads; 1.4T MassiveText tokens; AdamW; cosine decay by 10×10\times over a cycle matched to the run
Llama 2
2T tokens; AdamW β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, ϵ=10−5\epsilon=10^{-5}, weight decay 0.1, gradient clipping 1.0; 2,000 warmup steps, cosine to 10% of peak; batch 4M tokens
Llama 3.1 405B
about 15.6T tokens; peak lr 8×10−58\times10^{-5}, 8,000 warmup steps, cosine to 8×10−78\times10^{-7}; context grown in stages from 8K to 128K; final annealing on high-quality data
FineWeb
15T tokens from 96 Common Crawl snapshots; trafilatura extraction, fastText English filter, Gopher and C4 rules, MinHash deduplication within each snapshot
Code Llama
continued pretraining of Llama 2 on 500B tokens of code-heavy data, then long-context fine-tuning on 16K sequences with RoPE base raised from 10410^4 to 10610^6
DAPT (Gururangan et al.)
RoBERTa continued on four domains (biomedical papers, computer-science papers, news, reviews), evaluated on eight classification tasks

Continued pretraining

A base model is a compressed snapshot of its corpus. When the target is a domain or language the corpus covered thinly, it is cheaper to continue pre-training than to start over. Gururangan et al. continued RoBERTa on unlabeled in-domain text (DAPT, domain-adaptive pretraining) and on the unlabeled text of the task itself (TAPT). DAPT improved results in all four domains. TAPT was far cheaper and also helped, and DAPT followed by TAPT was best. The gains were largest where the domain was most dissimilar to RoBERTa's training data.

Decoder-only LLMs use CPT the same way, at larger scale. Code Llama continued Llama 2 on 500B tokens of code. Llemma (Azerbayev et al., 2023) continued Code Llama on mathematical text. Long-context extension is also CPT, on long documents with a modified RoPE base (see Rotary positional embeddings (RoPE), Context window optimization). After CPT come Supervised fine-tuning (SFT) and preference tuning, which use far fewer tokens.

The cost of CPT is catastrophic forgetting (McCloskey and Cohen, 1989). Gradients from the new domain move weights the old capabilities depend on, and nothing pulls them back. Two choices dominate. The learning rate: a pre-trained model ends at a tiny learning rate, and new data needs a larger one to make progress. Gupta et al. ("Continual Pre-Training of Large Language Models: How to (re)warm your model?", 2023) found that re-warming raises the loss on the original data at first, even when it improves adaptation. Replay: mix a fraction of the original data into every batch. Ibrahim et al. ("Simple and Scalable Strategies to Continually Pre-train Large Language Models", 2024) combined re-warming, re-decaying and replay. With as little as 5% replay for a mild distribution shift, and more for a strong one such as a new language, they matched retraining from scratch on all the data at a fraction of the cost.

LCPT(θ)=(1−r) Ex∼Dnew[−log⁡pθ(x)]+r Ex∼Dold[−log⁡pθ(x)]\mathcal L_{\text{CPT}}(\theta)=(1-r)\,\mathbb E_{x\sim\mathcal D_{\text{new}}}\big[-\log p_\theta(x)\big]+r\,\mathbb E_{x\sim\mathcal D_{\text{old}}}\big[-\log p_\theta(x)\big]
rr
replay fraction: the share of each batch drawn from the original pre-training mix
Dold\mathcal D_{\text{old}}
the original data, or a proxy for it if it is not available (open web text of similar quality)
θ0\theta_0
initialised from the base model; the optimiser state is often reset, with the learning rate re-warmed to a peak below the original
L(N, D) = 1.82 + 482.01/N^0.3478 + 2085.43/D^0.3658 · C = 6ND100M1B10B100B1T1B10B100B1T10T100Tparameters N →tokens D ↑GPT-3GopherChinchillaLlama 2 7BLlama 3 8BLlama 3.1 405Bloss at C = 1.00e24 FLOPs/64/8N_opt×8×642.161.96optimumN 95.9B · D 1.74TD/N = 18.1 · L = 1.9625your runN 95.9B · D 1.74TD/N = 18.1 · L = 1.9625penaltyΔL = +0.0000optimal: same L at 100.0% of C— compute-optimal path · - - D = 20N · grey: C = 1e20 … 1e26, dotted: chosen CGPT-3 2 · Gopher 1 · Chinchilla 20 · L2 7B 286 · L3 8B 1875 · L3.1 405B 39 tokens/paramcompute-optimal ratio here: 18.1 tokens per parameter, ∝ C^-0.025N_opt ∝ C^0.513, D_opt ∝ C^0.487
view
loss fit
fig.Two views. Compute-optimal (default) plots the Chinchilla loss with the Besiroglu et al. refit. At C = 10²⁴ FLOPs the optimum is N = 95.9B and D = 1.74T, which is 18.1 tokens per parameter, at L = 1.9625. A model 4× smaller trained on 4× the tokens (290 per parameter) loses 0.0176 nats, and an optimal run reaches that loss with 52.0% of the compute. A model 4× larger is about as wasteful (51.4%). A 2× error either way costs about 16% (84.3–84.4%), and 16× smaller costs 90% (9.5%). These shares do not depend on C. With the published approach-3 constants the optimum at 10²⁴ moves to 41.3B and 4.04T (97.7 tokens per parameter). Continued pre-training is a 4-weight toy with an exact solution for both domains; curves are smoothed and final losses are means over the last 40 of 400 steps. With no replay and a re-warmed peak of 0.3, the model forgets: old-domain excess loss ends at 0.4501 (smoothed worst 0.519) while the new domain reaches 0.0012. With 5% replay the old loss ends at 0.0143, and with 10% at 0.0017, at no cost to the new domain. A peak of 1.0 adapts and forgets faster and leaves a noise floor of 0.0075 (old) and 0.0083 (new) even with 5% replay. A peak of 0.01 forgets less (0.3607) but barely adapts (new 0.0247). At a peak of 0.03, 5% replay only brings the old loss to 0.3308: replay needs enough learning rate and steps to pull the weights back.

Worked example: spending Gopher's budget

Gopher used C=6⋅280⋅109⋅300⋅109=5.04×1023C=6\cdot280\cdot10^9\cdot300\cdot10^9=5.04\times10^{23} FLOPs by the 6ND6ND rule. Here is how the refit loss would allocate the same budget, and where other well-known models fall. The loss values come from a fit to Chinchilla's own data and tokenizer, so only the comparisons between rows are meaningful. They are not predictions of these models' real losses.

modelNDD/NC = 6NDL(N, D)optimal run, same L
GPT-3175B300B1.73.15e232.011360.8% of C
Gopher280B300B1.15.04e232.002449.7% of C
Chinchilla70B1.4T20.05.88e231.976799.9% of C
Llama 2 7B7B2T285.78.40e222.067553.8% of C
Llama 3 8B8B15T1,8757.20e232.024918.1% of C
Llama 3.1 405B405B15.6T38.53.79e251.895493.9% of C
Allocation under L(N, D) with the Besiroglu et al. refit (computed in Python and checked against the figure's code)

Step by step

  1. 1Rule of thumb: D=20ND=20N gives C=120N2C=120N^2, so N=5.04×1023/120=64.8N=\sqrt{5.04\times10^{23}/120}=64.8B and D=1.30D=1.30T.
  2. 2Closed form with the refit: G=(αA/βB)1/(α+β)G=(\alpha A/\beta B)^{1/(\alpha+\beta)} and Nopt=G (C/6)0.5126=67.5N_{opt}=G\,(C/6)^{0.5126}=67.5B, Dopt=1.245D_{opt}=1.245T, D/N=18.5D/N=18.5, L=1.9810L=1.9810.
  3. 3Gopher's own split: A/Nα=0.0503A/N^\alpha=0.0503 and B/Dβ=0.1321B/D^\beta=0.1321. The data term dominates, so the model is under-trained. At the optimum the terms are 0.08250.0825 and 0.07850.0785, in the ratio β/α=1.052\beta/\alpha=1.052 that the optimality condition requires.
  4. 4Gopher's loss 2.00242.0024 is reached by an optimally split run with 49.7%49.7\% of the compute: roughly half of the budget went into excess size.
  5. 5Llama 3 8B reaches only L=2.0249L=2.0249 with 7.2×10237.2\times10^{23} FLOPs, a loss an optimal run (about 81B parameters) would reach with 18.1%18.1\% of that compute. But that 81B model would cost about 10 times more per generated token for its whole deployed life.
python
1import re, zlib
2import numpy as np
3
4# ---- 1) Near-duplicate detection with MinHash + LSH banding ----
5def shingles(text, n=5):
6 w = re.findall(r"\w+", text.lower())
7 return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
8
9P = (1 << 61) - 1 # Mersenne prime for universal hashing
10rs = np.random.default_rng(0)
11A = rs.integers(1, P, size=112, dtype=np.uint64) # 112 hash functions = 14 bands x 8 rows
12B = rs.integers(0, P, size=112, dtype=np.uint64)
13
14def minhash(text):
15 h = np.array([zlib.crc32(s.encode()) for s in shingles(text)], dtype=np.uint64)
16 # (a*h + b) mod P, computed in Python ints to avoid uint64 overflow
17 return np.array([min((int(a) * int(x) + int(b)) % P for x in h) for a, b in zip(A, B)])
18
19def lsh_candidate_prob(s, bands=14, rows=8): # P(two docs with Jaccard s share a band)
20 return 1 - (1 - s ** rows) ** bands
21
22doc1 = "The quick brown fox jumps over the lazy dog near the river bank on a sunny afternoon in May"
23doc2 = "The quick brown fox jumps over the lazy dog near the river bank on a rainy afternoon in May"
24s1, s2 = shingles(doc1), shingles(doc2)
25jac = len(s1 & s2) / len(s1 | s2)
26est = np.mean(minhash(doc1) == minhash(doc2))
27print(f"Jaccard {jac:.3f}, MinHash estimate {est:.3f}") # Jaccard 0.579, MinHash estimate 0.580
28for s in (0.5, 0.75, 0.9):
29 print(f"s = {s}: candidate probability {lsh_candidate_prob(s):.4f}") # 0.0533, 0.7716, 0.9996
30
31# ---- 2) Compute-optimal allocation from L(N, D) = E + A/N^a + B/D^b ----
32def chinchilla_optimal(C, E=1.82, A_=482.01, B_=2085.43, a=0.3478, b=0.3658):
33 G = (a * A_ / (b * B_)) ** (1 / (a + b))
34 N = G * (C / 6) ** (b / (a + b))
35 D = C / 6 / N
36 return N, D, E + A_ / N ** a + B_ / D ** b
37
38for C in (1e21, 1e23, 1e25):
39 N, D, L = chinchilla_optimal(C)
40 print(f"C = {C:.0e}: N = {N / 1e9:7.2f}B, D = {D / 1e9:8.1f}B tokens, D/N = {D / N:4.1f}, L = {L:.3f}")
41# C = 1e+21: N = 2.78B, D = 60.0B tokens, D/N = 21.6, L = 2.308
42# C = 1e+23: N = 29.45B, D = 566.0B tokens, D/N = 19.2, L = 2.035
43# C = 1e+25: N = 312.07B, D = 5340.7B tokens, D/N = 17.1, L = 1.915
44
45# ---- 3) Warmup-stable-decay schedule (re-use it with a lower peak for continued pretraining) ----
46def wsd(step, total, peak, warmup=2000, decay_frac=0.1, floor=0.0):
47 if step < warmup:
48 return peak * (step + 1) / warmup
49 decay_start = int(total * (1 - decay_frac))
50 if step < decay_start:
51 return peak
52 p = (step - decay_start) / (total - decay_start)
53 return floor + (peak - floor) * (1 - p) # linear cooldown
54
55print([round(wsd(t, 100_000, 3e-4), 7) for t in (0, 1999, 50_000, 95_000, 99_999)])
56# [1e-07, 0.0003, 0.0003, 0.00015, 0.0]
MinHash near-duplicate detection with LSH banding, the closed-form Chinchilla allocation, and a warmup-stable-decay schedule (numpy; run, outputs in comments)

Use when

  • +Compute-optimal scaling (≈ 20 tokens per parameter) gives the lowest loss per training FLOP and is easy to apply
  • +Filtering and deduplication improve both quality and memorisation behaviour at no inference cost
  • +WSD schedules allow branching checkpoints at several budgets from one run
  • +Continued pretraining adapts a strong base model to a domain or language for a small fraction of the original compute
  • +Replay of a few percent and a re-warmed, re-decayed learning rate largely prevent forgetting

Avoid when

  • −Loss-optimal is not deployment-optimal: inference cost pushes models far past 20 tokens per parameter
  • −Scaling-law constants depend on data, tokenizer and architecture, and fitted exponents differ between studies
  • −Classifier filters inherit the biases of their reference text and can remove dialects or minority content
  • −CPT needs access to the original data or a good proxy for replay, and the peak learning rate trades adaptation against forgetting

Where it's used

Base models

Every open LLM family (Llama, Qwen, Mistral, Gemma, DeepSeek) starts from a trillions-of-tokens pretraining run with this pipeline.

Domain models

Code (Code Llama), mathematics (Llemma), biomedicine, law and finance models built by continued pretraining on domain corpora.

Language and context extension

Adding languages, or extending the context window, with CPT on targeted data and replay of the original mix.

Check yourself

You have 1e24 FLOPs. Under the Chinchilla refit, the optimum is about 96B parameters on 1.74T tokens. You train a 24B model on 7.0T tokens instead. What do you pay?

Check yourself

A team continues a base model on medical text with no replay and its original peak learning rate. General benchmarks drop sharply. Which change most directly addresses this?

Remember

  • ✓Pipeline: extract → language ID → quality filter (rules + classifier) → deduplicate (exact + MinHash) → safety filters → tokenize → mix domains; data quality now matters as much as quantity.
  • ✓Chinchilla: L=E+A/Nα+B/DβL=E+A/N^{\alpha}+B/D^{\beta} under C≈6NDC\approx6ND gives Nopt,Dopt∝C≈0.5N_{opt},D_{opt}\propto C^{\approx0.5}, about 20 tokens per parameter. Chinchilla 70B on 1.4T beat Gopher 280B on 300B at the same compute.
  • ✓Mis-sizing by 2× costs about 16% of compute and by 4× about half; inference-heavy models are deliberately over-trained (Llama 3 8B: about 1,900 tokens per parameter).
  • ✓Schedules: warmup then cosine to about 10% of peak, or warmup-stable-decay with a short cooldown, finishing with an anneal on high-quality data.
  • ✓Continued pretraining (DAPT, TAPT, code, maths, long context) resumes next-token training on new data. Re-warm and re-decay the learning rate and replay about 5% or more of the old data to limit catastrophic forgetting.
← back to Deep Learning roadmap