Foundation models

A foundation model (Bommasani et al., 2021) is trained once on broad data by self-supervision, at a cost of about 6·N·D FLOPs, then adapted to many tasks by prompting, retrieval, probes or fine-tuning; every task inherits its strengths and flaws.

8 min read715 wordscore0 viewsupdated 2026-10-21

Definition

Foundation model is "any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks". The term comes from Bommasani et al., "On the Opportunities and Risks of Foundation Models" (Stanford CRFM, 2021). It names a way of building systems, not an architecture: BERT, GPT-3, CLIP and diffusion image models all qualify.

Before about 2018, each NLP or vision task usually got its own model, trained from scratch or from ImageNet features on a labelled dataset. Labels are expensive, so most tasks had thousands of examples, not billions.

A foundation model splits the work in two. Pretraining learns general structure from unlabelled data that is cheap and plentiful: web text, code, image–caption pairs. Adaptation turns that one model into many products with little or no labelled data. In a real system the foundation model is the expensive, rarely changed component. Prompts, retrieval indexes and adapters are the cheap parts you change every week.

Bommasani et al. name two consequences. Emergence: abilities such as in-context learning are induced by scale rather than designed (see Emergent abilities). Homogenisation: one model, and so one set of defects, sits under many applications.

Two stages, two objectives

θ∗=arg⁡min⁡θ  Ex∼Dbroad[−∑tlog⁡pθ(xt∣x<t)],ϕk∗=arg⁡min⁡ϕ  E(x,y)∼Dk ℓ(fθ∗,ϕ(x),y)\theta^*=\arg\min_\theta\;\mathbb E_{x\sim\mathcal D_\text{broad}}\Big[-\sum_t\log p_\theta(x_t\mid x_{<t})\Big],\qquad \phi^*_k=\arg\min_{\phi}\;\mathbb E_{(x,y)\sim\mathcal D_k}\,\ell\big(f_{\theta^*,\phi}(x),y\big)
Dbroad\mathcal D_\text{broad}
unlabelled pretraining corpus; the data is its own target (here next-token prediction, see Language modeling)
θ∗\theta^*
the foundation model's weights, trained once
ϕk\phi_k
task-specific parameters for task kk: a prompt, a probe head or a LoRA adapter; usually ∣ϕk∣≪∣θ∣|\phi_k|\ll|\theta|, and ϕk=∅\phi_k=\varnothing for pure prompting
Dk\mathcal D_k
a small labelled set (or just a few in-context examples) for task kk

The pretraining objective varies by modality. Decoder LLMs predict the next token. BERT and MAE reconstruct masked tokens or patches. CLIP and SigLIP pull matching image–text pairs together with a contrastive loss. What they share is that the label comes from the data itself, so the corpus can grow to trillions of tokens.

pretrain once on broad data, adapt many times1 broad data2 self-supervised3 foundation model4 adaptweb texttrillions of tokenssource codepublic repositoriesbooks, paperslong-form textimage–text pairsCLIP: 400M pairsaudio, videospeech, captionsnext token−Σ log p(xₜ | x<ₜ)GPT, Llamamasked tokens−Σ log p(xₘ | rest)BERT, MAEcontrastiveInfoNCE on pairsCLIP, SigLIPno labels; the data isits own targetone set of weights θGPT-3 (2020):N = 175B paramsD = 300B tokensC ≈ 3.1×10²³ FLOPtrained oncecompute C ≈ 6·N·D6 × 175e9 × 300e9prompting0 params, in contextRAG0 params + an indexlinear probeone small headLoRA / PEFT≈ 0.01–1% of paramsfull fine-tune100% of paramscheapest firstdownstream taskschatcodesummariseclassifyextractsearchcaptionhomogenisation: every task inherits θa bias, knowledge gap, data leak or backdoor learned in pretrainingshows up in all adapted systems unless each adaptation corrects it
fig.Broad unlabelled data (left) trains one set of weights with a self-supervised objective. GPT-3 is the example: N = 175B parameters, D = 300B tokens, so C ≈ 6ND ≈ 3.1 × 10²³ FLOPs. The same weights are then adapted, from cheapest (a prompt, 0 trained parameters) to most expensive (full fine-tuning, 100%). Every downstream task inherits whatever the base model learned, good or bad.

Scale: compute, data and loss

C≈6ND,L(N,D)≈E+ANα+BDβC\approx 6ND,\qquad L(N,D)\approx E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}
CC
training FLOPs: about 2N per token for the forward pass and 4N for the backward pass
N, DN,\,D
parameters and training tokens
E,A,B,α,βE,A,B,\alpha,\beta
fitted constants; Hoffmann et al. (2022) report E=1.69E=1.69, A=406.4A=406.4, B=410.7B=410.7, α=0.34\alpha=0.34, β=0.28\beta=0.28 for their models and data

Loss falls as a smooth power law in both NN and DD (Kaplan et al., 2020). For a fixed compute budget, Hoffmann et al. ("Chinchilla", 2022) found that NN and DD should grow roughly equally, about 20 tokens per parameter. Their 70B model trained on 1.4T tokens beat the 280B Gopher trained on 300B tokens at similar compute. Models meant for heavy use are now often trained far past that ratio, because a smaller model is cheaper to serve (see Pretraining and continued pretraining).

ModelPaperObjectiveSizePretraining dataNote
BERT-largeDevlin et al., 2018masked tokens340MBooksCorpus + English Wikipedia (3.3B words)fine-tuned per task
GPT-3Brown et al., 2020next token175B300B tokens3.14 × 10²³ FLOPs; few-shot prompting
CLIPRadford et al., 2021contrastiveViT-L/14@336px400M image–text pairs76.2% zero-shot ImageNet
ChinchillaHoffmann et al., 2022next token70B1.4T tokensbeat 280B Gopher at similar compute
Llama 2Touvron et al., 2023next token7B–70B2T tokensweights released; 7B took 184,320 A100-hours
Landmark foundation models, as reported in their papers

Adapting one backbone

MethodTrained paramsLabelled dataWhere covered
Zero-/few-shot prompting00–a few examples in the promptZero-shot prompting, Few-shot prompting
Retrieval0 (plus an index)none; needs documentsRAG architecture
Linear probe on frozen featuresone headhundreds–thousandsthis article's figure
LoRA / PEFT≈ 0.01–1%thousandsLoRA, Parameter-efficient fine-tuning (PEFT)
Full fine-tuning100%thousands–millionsSupervised fine-tuning (SFT)
Adaptation options, cheapest first. Trained-parameter shares are typical, not fixed.

Hu et al. (2021) report that LoRA on GPT-3 175B cut trainable parameters by 10,000× and GPU memory by 3× compared with full fine-tuning, with similar quality. For choosing among these options, see Fine-tuning vs RAG vs prompting.

PCA backbone from M = 200 unlabelled inputs · k = 16 labels per taskbackbone keeps of each true factor: z1 93% z2 92% z3 91% z4 89% 5060708090481632641282565121024labels per task k (log scale)mean test accuracy over 6 tasks, %adapted (probe on 4 features) 81.6%from scratch (100-d input) 76.3%per task at k = 16 (axis 50–100%)task 1w₄ 0.007987task 2w₄ 0.007681task 3w₄ 0.198087task 4w₄ 0.587781task 5w₄ 0.987373task 6w₄ 0.447380x = U z + 0.4 noise (100-d, 4 factors) · y_t = sign(w_t · z) · ridge λ = 10 · 500 test ptspretraining never sees a label; one backbone serves all six tasks, so a flaw in it reaches all of them
pretraining corpus
fig.Six classification tasks on synthetic 100-d inputs built from 4 hidden factors. "Pretraining" is PCA on M unlabelled inputs, giving a frozen 4-feature backbone; each task is then solved with k labels either from scratch on the raw input or by a linear probe on the backbone. Defaults (M = 200, k = 16): the backbone keeps 89–93% of each factor and mean accuracy is 81.6% adapted vs 76.3% from scratch (72.6% vs 65.5% at k = 4). With M = 10 the backbone keeps only 19–56% and loses from k = 16 on (79.2% vs 84.4% at k = 1024). Make factor 4 rare in the corpus and the backbone keeps 1.6% of it: task 5 (w₄ = 0.98) falls to 53% at k = 16, against 73% from scratch.

The figure shows both sides of homogenisation. A good backbone helps all six tasks at once, most when labels are scarce. A backbone that never saw factor 4 fails every task that needs it, and no amount of probe training recovers information the features dropped. Full fine-tuning can, in principle, because it changes θ\theta.

Worked example: what pretraining costs

A 7B model on 2T tokens (the Llama 2 7B setting)

  1. 1Compute: C≈6ND=6×7×109×2×1012=8.4×1022C\approx6ND=6\times7\times10^{9}\times2\times10^{12}=8.4\times10^{22} FLOPs.
  2. 2Tokens per parameter: 2×1012/7×109≈2862\times10^{12}/7\times10^{9}\approx286, about 14× the compute-optimal 20 (which would be 140B tokens).
  3. 3Hardware time: an A100 peaks at 312 TFLOP/s in BF16. At 40% utilisation, 8.4×1022/(312×1012×0.4)=6.73×1088.4\times10^{22}/(312\times10^{12}\times0.4)=6.73\times10^{8} GPU-seconds ≈187,000\approx187{,}000 GPU-hours.
  4. 4The Llama 2 paper reports 184,320 A100-hours for 7B, which implies about 40.6% utilisation.
  5. 5Adapting it with full fine-tuning on 10M tokens costs 6×7×109×107=4.2×10176\times7\times10^{9}\times10^{7}=4.2\times10^{17} FLOPs, 1/200,000 of pretraining. That ratio is the economic case for foundation models.
  6. 6Loss check with the Chinchilla fit: L(70B,1.4T)=1.69+0.083+0.163=1.937L(70\text{B},1.4\text{T})=1.69+0.083+0.163=1.937 versus L(280B,300B)=1.69+0.052+0.251=1.993L(280\text{B},300\text{B})=1.69+0.052+0.251=1.993. The smaller model trained on more data is predicted to be better, as observed.
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import LoraConfig, get_peft_model
3
4base = "meta-llama/Llama-2-7b-hf" # the foundation model, trained once
5tok = AutoTokenizer.from_pretrained(base)
6model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto")
7
8# 1) adapt by prompting: no weights change
9prompt = "Review: The battery died after a day.\nSentiment (positive/negative):"
10ids = tok(prompt, return_tensors="pt")
11print(tok.decode(model.generate(**ids, max_new_tokens=2)[0][ids.input_ids.shape[1]:]))
12
13# 2) adapt by LoRA: train small low-rank updates to two projections per layer
14cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
15 target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
16model = get_peft_model(model, cfg)
17model.print_trainable_parameters() # trainable% ≈ 0.062
One backbone, two adaptations with Hugging Face transformers and peft (not run here: needs the libraries, model access and a GPU). For Llama-2-7B the LoRA config trains 4,194,304 of 6,742,609,920 parameters (0.062%).

Use when

  • +Pretraining cost is paid once and shared by every task built on the model.
  • +Strong results with few or no labels: prompting needs none, probes and LoRA need little.
  • +One model can serve many tasks and modalities, which simplifies serving and updates.

Avoid when

  • −Pretraining needs 10²²–10²⁵ FLOPs and trillions of tokens, so few organisations can do it.
  • −Homogenisation: defects, biases and a fixed knowledge cutoff reach every downstream use.
  • −Behaviour is hard to predict or audit; abilities are discovered after training, not specified.

Check yourself

In the figure, the corpus under-represents factor 4. Why can't more labelled data for task 5 fix the linear probe?

Check yourself

Roughly how many FLOPs does it take to train a 70B-parameter model on 1.4T tokens?

Remember

  • ✓Foundation model (Bommasani et al., 2021): trained on broad data, usually by self-supervision at scale, then adapted to many tasks.
  • ✓Objectives: next token (GPT, Llama), masked tokens (BERT, MAE), contrastive pairs (CLIP, SigLIP).
  • ✓Training compute C ≈ 6ND; Chinchilla: about 20 tokens per parameter is compute-optimal, but deployed models are often trained much longer.
  • ✓Adaptation, cheapest first: prompting, retrieval, probes, LoRA/PEFT, full fine-tuning.
  • ✓Homogenisation: one model's strengths and defects reach every system built on it, so evaluate each adapted system on its own task.
← back to Generative AI roadmap