Definition
Foundation model is "any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks". The term comes from Bommasani et al., "On the Opportunities and Risks of Foundation Models" (Stanford CRFM, 2021). It names a way of building systems, not an architecture: BERT, GPT-3, CLIP and diffusion image models all qualify.
Before about 2018, each NLP or vision task usually got its own model, trained from scratch or from ImageNet features on a labelled dataset. Labels are expensive, so most tasks had thousands of examples, not billions.
A foundation model splits the work in two. Pretraining learns general structure from unlabelled data that is cheap and plentiful: web text, code, image–caption pairs. Adaptation turns that one model into many products with little or no labelled data. In a real system the foundation model is the expensive, rarely changed component. Prompts, retrieval indexes and adapters are the cheap parts you change every week.
Bommasani et al. name two consequences. Emergence: abilities such as in-context learning are induced by scale rather than designed (see Emergent abilities). Homogenisation: one model, and so one set of defects, sits under many applications.
Two stages, two objectives
- unlabelled pretraining corpus; the data is its own target (here next-token prediction, see Language modeling)
- the foundation model's weights, trained once
- task-specific parameters for task : a prompt, a probe head or a LoRA adapter; usually , and for pure prompting
- a small labelled set (or just a few in-context examples) for task
The pretraining objective varies by modality. Decoder LLMs predict the next token. BERT and MAE reconstruct masked tokens or patches. CLIP and SigLIP pull matching image–text pairs together with a contrastive loss. What they share is that the label comes from the data itself, so the corpus can grow to trillions of tokens.
Scale: compute, data and loss
- training FLOPs: about 2N per token for the forward pass and 4N for the backward pass
- parameters and training tokens
- fitted constants; Hoffmann et al. (2022) report , , , , for their models and data
Loss falls as a smooth power law in both and (Kaplan et al., 2020). For a fixed compute budget, Hoffmann et al. ("Chinchilla", 2022) found that and should grow roughly equally, about 20 tokens per parameter. Their 70B model trained on 1.4T tokens beat the 280B Gopher trained on 300B tokens at similar compute. Models meant for heavy use are now often trained far past that ratio, because a smaller model is cheaper to serve (see Pretraining and continued pretraining).
| Model | Paper | Objective | Size | Pretraining data | Note |
|---|---|---|---|---|---|
| BERT-large | Devlin et al., 2018 | masked tokens | 340M | BooksCorpus + English Wikipedia (3.3B words) | fine-tuned per task |
| GPT-3 | Brown et al., 2020 | next token | 175B | 300B tokens | 3.14 × 10²³ FLOPs; few-shot prompting |
| CLIP | Radford et al., 2021 | contrastive | ViT-L/14@336px | 400M image–text pairs | 76.2% zero-shot ImageNet |
| Chinchilla | Hoffmann et al., 2022 | next token | 70B | 1.4T tokens | beat 280B Gopher at similar compute |
| Llama 2 | Touvron et al., 2023 | next token | 7B–70B | 2T tokens | weights released; 7B took 184,320 A100-hours |
Adapting one backbone
| Method | Trained params | Labelled data | Where covered |
|---|---|---|---|
| Zero-/few-shot prompting | 0 | 0–a few examples in the prompt | Zero-shot prompting, Few-shot prompting |
| Retrieval | 0 (plus an index) | none; needs documents | RAG architecture |
| Linear probe on frozen features | one head | hundreds–thousands | this article's figure |
| LoRA / PEFT | ≈ 0.01–1% | thousands | LoRA, Parameter-efficient fine-tuning (PEFT) |
| Full fine-tuning | 100% | thousands–millions | Supervised fine-tuning (SFT) |
Hu et al. (2021) report that LoRA on GPT-3 175B cut trainable parameters by 10,000× and GPU memory by 3× compared with full fine-tuning, with similar quality. For choosing among these options, see Fine-tuning vs RAG vs prompting.
The figure shows both sides of homogenisation. A good backbone helps all six tasks at once, most when labels are scarce. A backbone that never saw factor 4 fails every task that needs it, and no amount of probe training recovers information the features dropped. Full fine-tuning can, in principle, because it changes .
Worked example: what pretraining costs
A 7B model on 2T tokens (the Llama 2 7B setting)
- 1Compute: FLOPs.
- 2Tokens per parameter: , about 14× the compute-optimal 20 (which would be 140B tokens).
- 3Hardware time: an A100 peaks at 312 TFLOP/s in BF16. At 40% utilisation, GPU-seconds GPU-hours.
- 4The Llama 2 paper reports 184,320 A100-hours for 7B, which implies about 40.6% utilisation.
- 5Adapting it with full fine-tuning on 10M tokens costs FLOPs, 1/200,000 of pretraining. That ratio is the economic case for foundation models.
- 6Loss check with the Chinchilla fit: versus . The smaller model trained on more data is predicted to be better, as observed.
1from transformers import AutoModelForCausalLM, AutoTokenizer2from peft import LoraConfig, get_peft_model34base = "meta-llama/Llama-2-7b-hf" # the foundation model, trained once5tok = AutoTokenizer.from_pretrained(base)6model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto")78# 1) adapt by prompting: no weights change9prompt = "Review: The battery died after a day.\nSentiment (positive/negative):"10ids = tok(prompt, return_tensors="pt")11print(tok.decode(model.generate(**ids, max_new_tokens=2)[0][ids.input_ids.shape[1]:]))1213# 2) adapt by LoRA: train small low-rank updates to two projections per layer14cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,15 target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")16model = get_peft_model(model, cfg)17model.print_trainable_parameters() # trainable% ≈ 0.062Use when
- +Pretraining cost is paid once and shared by every task built on the model.
- +Strong results with few or no labels: prompting needs none, probes and LoRA need little.
- +One model can serve many tasks and modalities, which simplifies serving and updates.
Avoid when
- −Pretraining needs 10²²–10²⁵ FLOPs and trillions of tokens, so few organisations can do it.
- −Homogenisation: defects, biases and a fixed knowledge cutoff reach every downstream use.
- −Behaviour is hard to predict or audit; abilities are discovered after training, not specified.
Check yourself
In the figure, the corpus under-represents factor 4. Why can't more labelled data for task 5 fix the linear probe?
Check yourself
Roughly how many FLOPs does it take to train a 70B-parameter model on 1.4T tokens?
Remember
- ✓Foundation model (Bommasani et al., 2021): trained on broad data, usually by self-supervision at scale, then adapted to many tasks.
- ✓Objectives: next token (GPT, Llama), masked tokens (BERT, MAE), contrastive pairs (CLIP, SigLIP).
- ✓Training compute C ≈ 6ND; Chinchilla: about 20 tokens per parameter is compute-optimal, but deployed models are often trained much longer.
- ✓Adaptation, cheapest first: prompting, retrieval, probes, LoRA/PEFT, full fine-tuning.
- ✓Homogenisation: one model's strengths and defects reach every system built on it, so evaluate each adapted system on its own task.