Definition
Pretraining is the first and most expensive stage of building a language model. A randomly initialised network is trained with the self-supervised next-token loss on a very large unlabeled corpus, and the result is a base model. Continued pretraining (CPT) resumes the same objective from a pre-trained checkpoint on different data: a new domain, a new language, fresher text or longer documents. Three papers define the modern practice. Kaplan et al. ("Scaling Laws for Neural Language Models", 2020) and Hoffmann et al. ("Training Compute-Optimal Large Language Models", NeurIPS 2022, the Chinchilla paper) set how big a model should be for a given budget. Gururangan et al. ("Don't Stop Pretraining: Adapt Language Models to Domains and Tasks", ACL 2020) showed that a second phase of pre-training on in-domain text pays off.
The corpora grew by four orders of magnitude in five years. GPT (2018) trained on BooksCorpus, about 7,000 unpublished books. GPT-2 (2019) used WebText, 40 GB of pages linked from Reddit posts with at least 3 karma. T5's C4 (2020) was about 750 GB of filtered Common Crawl, and The Pile (Gao et al., 2020) mixed 22 sources into 825 GiB. RefinedWeb (Penedo et al., 2023) showed that carefully filtered and deduplicated web data alone could match curated mixtures. FineWeb (Penedo et al., 2024) released 15T tokens from 96 Common Crawl snapshots. Llama 3 trained on about 15T tokens and Qwen2.5 on 18T. As corpora grew, which tokens to train on became as important as how many.
The data pipeline
A modern pipeline turns raw crawl archives into a token stream in a fixed sequence of steps. Extraction pulls main text out of HTML: FineWeb found that re-extracting from WARC files with trafilatura beat the WET text that Common Crawl ships. Language identification keeps documents whose fastText language score passes a threshold. Quality filtering combines rules with classifiers. The Gopher rules (Rae et al., 2021) keep documents of 50 to 100,000 words with a mean word length of 3 to 10 characters, in which at least 80% of words contain a letter. They also drop documents with too many symbols, bullet lines or repeated lines. Classifier filters score each page against a reference of "good" text. GPT-3 used one trained to separate WebText, books and Wikipedia from raw Common Crawl. FineWeb-Edu had Llama-3-70B rate 450K pages for educational value, trained a small classifier on those ratings, and kept 1.3T tokens.
Deduplication follows. Lee et al. ("Deduplicating Training Data Makes Language Models Better", ACL 2022) found a single 61-word English sentence repeated over 60,000 times in C4. Removing exact repeated substrings (with a suffix array) and near-duplicate documents (with MinHash) gave models that emitted memorised text about ten times less often and needed fewer steps for the same accuracy. MinHash estimates the Jaccard similarity of two documents' sets of word -grams, and locality-sensitive hashing turns it into a cheap candidate test:
- sets of word 5-grams ("shingles") of two documents
- a random hash; the chance that both sets share the minimum equals their Jaccard similarity
- bands of hashes; a pair becomes a candidate if all hashes agree in any band. FineWeb used 112 hashes as ,
- true similarity: with 14 × 8, a pair at is caught with probability 0.772, at only 0.053
The last steps are safety filters (PII removal or masking, toxicity and adult-content filters, URL blocklists), tokenization (see Byte-pair encoding (BPE)), and mixing. Each source gets a sampling weight. Llama 1's mix was 67% CommonCrawl, 15% C4, 4.5% each GitHub, Wikipedia and books, 2.5% arXiv and 2% StackExchange. Llama 3's final mix was about 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual. Mixture weights can be learned. DoReMi (Xie et al., NeurIPS 2023) tunes them with a small proxy model and transfers them to a large one. Repetition is acceptable in moderation: Muennighoff et al. ("Scaling Data-Constrained Language Models", NeurIPS 2023) found up to about 4 epochs of repeated data almost as good as fresh data, with value decaying quickly after that.
How big, how long: compute-optimal scaling
Training compute is about FLOPs for parameters and tokens (see Decoder-only transformer). For a fixed , a model can be large and briefly trained or small and long-trained. Kaplan et al. (2020) fitted power laws and recommended spending most extra compute on size, with . GPT-3 (175B, 300B tokens) and Gopher (280B, 300B tokens) followed that advice. Hoffmann et al. retrained over 400 models from 70M to 16B parameters on 5B to 500B tokens. They found that and should grow in equal proportion, and traced part of the discrepancy to Kaplan's fixed learning-rate schedule: a cosine cycle much longer than the run leaves short runs under-trained. Their third method fits a parametric loss directly:
- irreducible loss, the entropy of text under this tokenizer and data
- excess loss from finite model size
- excess loss from finite data
- at the optimum the two excess terms are in the ratio ; substitute and solve for
- published: , , , , . Besiroglu et al. (2024) refit the paper's data: , , , ,
Hoffmann et al.'s first two methods both gave exponents near 0.5 for and and about 20 tokens per parameter. To test it they trained Chinchilla, 70B parameters on 1.4T tokens, with the compute of the 280B Gopher. It was better on almost every benchmark, for example 67.6% on MMLU against Gopher's 60.0%, and four times cheaper to serve. The published constants of the third method imply smaller models and more tokens (about 98 tokens per parameter at FLOPs). Besiroglu, Erdil, Barnett and You ("Chinchilla Scaling: A replication attempt", 2024) reconstructed the data from the paper's figures. They found that the reported fit did not match it, and that their refit agreed with the 20-tokens rule. Both fits are in the figure.
The Chinchilla optimum minimises loss per training FLOP. A deployed model also pays for every inference token, which favours a smaller model trained longer (Sardana and Frankle, "Beyond Chinchilla-Optimal", 2023). Llama 2 7B saw about 290 tokens per parameter and Llama 3 8B about 1,900. For its flagship Meta fitted its own isoFLOP curves and trained Llama 3.1 405B close to that optimum, at about FLOPs.
Learning-rate schedules
- linear warmup steps; Llama 2 used 2,000
- Llama 2 decays to 10% of the peak ( for 7B and 13B, for 70B)
- total steps; a cosine schedule must be set to the planned run length in advance
- WSD cooldown length, typically 10–20% of the run
Cosine decay fixes the run length up front. Stopping early, or extending later, leaves the model at a sub-optimal point of its schedule. Warmup-stable-decay (WSD, introduced with MiniCPM by Hu et al., 2024) holds the peak rate for most of training and decays only in a short final phase. Any stable-phase checkpoint can be branched into a cooldown, so one run yields models for several token budgets. Hägele et al. (2024) found that a cooldown of about 20% of the run matches cosine. The end of training is also where data quality matters most. Llama 3 finishes with an annealing phase that decays the learning rate to zero on upsampled high-quality sources.
Published pretraining setups
- Chinchilla
- 70B, 80 layers, , 64 heads; 1.4T MassiveText tokens; AdamW; cosine decay by over a cycle matched to the run
- Llama 2
- 2T tokens; AdamW , , , weight decay 0.1, gradient clipping 1.0; 2,000 warmup steps, cosine to 10% of peak; batch 4M tokens
- Llama 3.1 405B
- about 15.6T tokens; peak lr , 8,000 warmup steps, cosine to ; context grown in stages from 8K to 128K; final annealing on high-quality data
- FineWeb
- 15T tokens from 96 Common Crawl snapshots; trafilatura extraction, fastText English filter, Gopher and C4 rules, MinHash deduplication within each snapshot
- Code Llama
- continued pretraining of Llama 2 on 500B tokens of code-heavy data, then long-context fine-tuning on 16K sequences with RoPE base raised from to
- DAPT (Gururangan et al.)
- RoBERTa continued on four domains (biomedical papers, computer-science papers, news, reviews), evaluated on eight classification tasks
Continued pretraining
A base model is a compressed snapshot of its corpus. When the target is a domain or language the corpus covered thinly, it is cheaper to continue pre-training than to start over. Gururangan et al. continued RoBERTa on unlabeled in-domain text (DAPT, domain-adaptive pretraining) and on the unlabeled text of the task itself (TAPT). DAPT improved results in all four domains. TAPT was far cheaper and also helped, and DAPT followed by TAPT was best. The gains were largest where the domain was most dissimilar to RoBERTa's training data.
Decoder-only LLMs use CPT the same way, at larger scale. Code Llama continued Llama 2 on 500B tokens of code. Llemma (Azerbayev et al., 2023) continued Code Llama on mathematical text. Long-context extension is also CPT, on long documents with a modified RoPE base (see Rotary positional embeddings (RoPE), Context window optimization). After CPT come Supervised fine-tuning (SFT) and preference tuning, which use far fewer tokens.
The cost of CPT is catastrophic forgetting (McCloskey and Cohen, 1989). Gradients from the new domain move weights the old capabilities depend on, and nothing pulls them back. Two choices dominate. The learning rate: a pre-trained model ends at a tiny learning rate, and new data needs a larger one to make progress. Gupta et al. ("Continual Pre-Training of Large Language Models: How to (re)warm your model?", 2023) found that re-warming raises the loss on the original data at first, even when it improves adaptation. Replay: mix a fraction of the original data into every batch. Ibrahim et al. ("Simple and Scalable Strategies to Continually Pre-train Large Language Models", 2024) combined re-warming, re-decaying and replay. With as little as 5% replay for a mild distribution shift, and more for a strong one such as a new language, they matched retraining from scratch on all the data at a fraction of the cost.
- replay fraction: the share of each batch drawn from the original pre-training mix
- the original data, or a proxy for it if it is not available (open web text of similar quality)
- initialised from the base model; the optimiser state is often reset, with the learning rate re-warmed to a peak below the original
Worked example: spending Gopher's budget
Gopher used FLOPs by the rule. Here is how the refit loss would allocate the same budget, and where other well-known models fall. The loss values come from a fit to Chinchilla's own data and tokenizer, so only the comparisons between rows are meaningful. They are not predictions of these models' real losses.
| model | N | D | D/N | C = 6ND | L(N, D) | optimal run, same L |
|---|---|---|---|---|---|---|
| GPT-3 | 175B | 300B | 1.7 | 3.15e23 | 2.0113 | 60.8% of C |
| Gopher | 280B | 300B | 1.1 | 5.04e23 | 2.0024 | 49.7% of C |
| Chinchilla | 70B | 1.4T | 20.0 | 5.88e23 | 1.9767 | 99.9% of C |
| Llama 2 7B | 7B | 2T | 285.7 | 8.40e22 | 2.0675 | 53.8% of C |
| Llama 3 8B | 8B | 15T | 1,875 | 7.20e23 | 2.0249 | 18.1% of C |
| Llama 3.1 405B | 405B | 15.6T | 38.5 | 3.79e25 | 1.8954 | 93.9% of C |
Step by step
- 1Rule of thumb: gives , so B and T.
- 2Closed form with the refit: and B, T, , .
- 3Gopher's own split: and . The data term dominates, so the model is under-trained. At the optimum the terms are and , in the ratio that the optimality condition requires.
- 4Gopher's loss is reached by an optimally split run with of the compute: roughly half of the budget went into excess size.
- 5Llama 3 8B reaches only with FLOPs, a loss an optimal run (about 81B parameters) would reach with of that compute. But that 81B model would cost about 10 times more per generated token for its whole deployed life.
1import re, zlib2import numpy as np34# ---- 1) Near-duplicate detection with MinHash + LSH banding ----5def shingles(text, n=5):6 w = re.findall(r"\w+", text.lower())7 return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}89P = (1 << 61) - 1 # Mersenne prime for universal hashing10rs = np.random.default_rng(0)11A = rs.integers(1, P, size=112, dtype=np.uint64) # 112 hash functions = 14 bands x 8 rows12B = rs.integers(0, P, size=112, dtype=np.uint64)1314def minhash(text):15 h = np.array([zlib.crc32(s.encode()) for s in shingles(text)], dtype=np.uint64)16 # (a*h + b) mod P, computed in Python ints to avoid uint64 overflow17 return np.array([min((int(a) * int(x) + int(b)) % P for x in h) for a, b in zip(A, B)])1819def lsh_candidate_prob(s, bands=14, rows=8): # P(two docs with Jaccard s share a band)20 return 1 - (1 - s ** rows) ** bands2122doc1 = "The quick brown fox jumps over the lazy dog near the river bank on a sunny afternoon in May"23doc2 = "The quick brown fox jumps over the lazy dog near the river bank on a rainy afternoon in May"24s1, s2 = shingles(doc1), shingles(doc2)25jac = len(s1 & s2) / len(s1 | s2)26est = np.mean(minhash(doc1) == minhash(doc2))27print(f"Jaccard {jac:.3f}, MinHash estimate {est:.3f}") # Jaccard 0.579, MinHash estimate 0.58028for s in (0.5, 0.75, 0.9):29 print(f"s = {s}: candidate probability {lsh_candidate_prob(s):.4f}") # 0.0533, 0.7716, 0.99963031# ---- 2) Compute-optimal allocation from L(N, D) = E + A/N^a + B/D^b ----32def chinchilla_optimal(C, E=1.82, A_=482.01, B_=2085.43, a=0.3478, b=0.3658):33 G = (a * A_ / (b * B_)) ** (1 / (a + b))34 N = G * (C / 6) ** (b / (a + b))35 D = C / 6 / N36 return N, D, E + A_ / N ** a + B_ / D ** b3738for C in (1e21, 1e23, 1e25):39 N, D, L = chinchilla_optimal(C)40 print(f"C = {C:.0e}: N = {N / 1e9:7.2f}B, D = {D / 1e9:8.1f}B tokens, D/N = {D / N:4.1f}, L = {L:.3f}")41# C = 1e+21: N = 2.78B, D = 60.0B tokens, D/N = 21.6, L = 2.30842# C = 1e+23: N = 29.45B, D = 566.0B tokens, D/N = 19.2, L = 2.03543# C = 1e+25: N = 312.07B, D = 5340.7B tokens, D/N = 17.1, L = 1.9154445# ---- 3) Warmup-stable-decay schedule (re-use it with a lower peak for continued pretraining) ----46def wsd(step, total, peak, warmup=2000, decay_frac=0.1, floor=0.0):47 if step < warmup:48 return peak * (step + 1) / warmup49 decay_start = int(total * (1 - decay_frac))50 if step < decay_start:51 return peak52 p = (step - decay_start) / (total - decay_start)53 return floor + (peak - floor) * (1 - p) # linear cooldown5455print([round(wsd(t, 100_000, 3e-4), 7) for t in (0, 1999, 50_000, 95_000, 99_999)])56# [1e-07, 0.0003, 0.0003, 0.00015, 0.0]Use when
- +Compute-optimal scaling (≈ 20 tokens per parameter) gives the lowest loss per training FLOP and is easy to apply
- +Filtering and deduplication improve both quality and memorisation behaviour at no inference cost
- +WSD schedules allow branching checkpoints at several budgets from one run
- +Continued pretraining adapts a strong base model to a domain or language for a small fraction of the original compute
- +Replay of a few percent and a re-warmed, re-decayed learning rate largely prevent forgetting
Avoid when
- −Loss-optimal is not deployment-optimal: inference cost pushes models far past 20 tokens per parameter
- −Scaling-law constants depend on data, tokenizer and architecture, and fitted exponents differ between studies
- −Classifier filters inherit the biases of their reference text and can remove dialects or minority content
- −CPT needs access to the original data or a good proxy for replay, and the peak learning rate trades adaptation against forgetting
Where it's used
Base models
Every open LLM family (Llama, Qwen, Mistral, Gemma, DeepSeek) starts from a trillions-of-tokens pretraining run with this pipeline.
Domain models
Code (Code Llama), mathematics (Llemma), biomedicine, law and finance models built by continued pretraining on domain corpora.
Language and context extension
Adding languages, or extending the context window, with CPT on targeted data and replay of the original mix.
Check yourself
You have 1e24 FLOPs. Under the Chinchilla refit, the optimum is about 96B parameters on 1.74T tokens. You train a 24B model on 7.0T tokens instead. What do you pay?
Check yourself
A team continues a base model on medical text with no replay and its original peak learning rate. General benchmarks drop sharply. Which change most directly addresses this?
Remember
- ✓Pipeline: extract → language ID → quality filter (rules + classifier) → deduplicate (exact + MinHash) → safety filters → tokenize → mix domains; data quality now matters as much as quantity.
- ✓Chinchilla: under gives , about 20 tokens per parameter. Chinchilla 70B on 1.4T beat Gopher 280B on 300B at the same compute.
- ✓Mis-sizing by 2× costs about 16% of compute and by 4× about half; inference-heavy models are deliberately over-trained (Llama 3 8B: about 1,900 tokens per parameter).
- ✓Schedules: warmup then cosine to about 10% of peak, or warmup-stable-decay with a short cooldown, finishing with an anneal on high-quality data.
- ✓Continued pretraining (DAPT, TAPT, code, maths, long context) resumes next-token training on new data. Re-warm and re-decay the learning rate and replay about 5% or more of the old data to limit catastrophic forgetting.