# Foundation models

> A foundation model (Bommasani et al., 2021) is trained once on broad data by self-supervision, at a cost of about 6·N·D FLOPs, then adapted to many tasks by prompting, retrieval, probes or fine-tuning; every task inherits its strengths and flaws.

Source: https://www.mlfoundry.in/learn/foundation-models  
Section: Generative AI › Foundations of Generative AI  
Level: core · 8 min read · updated 2026-10-21

**Foundation model:** is "any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks". The term comes from Bommasani et al., "On the Opportunities and Risks of Foundation Models" (Stanford CRFM, 2021). It names a way of building systems, not an architecture: **BERT**, GPT-3, CLIP and diffusion image models all qualify.

Before about 2018, each NLP or vision task usually got its own model, trained from scratch or from ImageNet features on a labelled dataset. Labels are expensive, so most tasks had thousands of examples, not billions.

A foundation model splits the work in two. **Pretraining** learns general structure from unlabelled data that is cheap and plentiful: web text, code, image–caption pairs. **Adaptation** turns that one model into many products with little or no labelled data. In a real system the foundation model is the expensive, rarely changed component. Prompts, retrieval indexes and adapters are the cheap parts you change every week.

Bommasani et al. name two consequences. **Emergence**: abilities such as in-context learning are induced by scale rather than designed (see **Emergent abilities**). **Homogenisation**: one model, and so one set of defects, sits under many applications.

## Two stages, two objectives

$$
\theta^*=\arg\min_\theta\;\mathbb E_{x\sim\mathcal D_\text{broad}}\Big[-\sum_t\log p_\theta(x_t\mid x_{<t})\Big],\qquad \phi^*_k=\arg\min_{\phi}\;\mathbb E_{(x,y)\sim\mathcal D_k}\,\ell\big(f_{\theta^*,\phi}(x),y\big)
$$

- $\mathcal D_\text{broad}$: unlabelled pretraining corpus; the data is its own target (here next-token prediction, see **Language modeling**)
- $\theta^*$: the foundation model's weights, trained once
- $\phi_k$: task-specific parameters for task $k$: a prompt, a probe head or a LoRA adapter; usually $|\phi_k|\ll|\theta|$, and $\phi_k=\varnothing$ for pure prompting
- $\mathcal D_k$: a small labelled set (or just a few in-context examples) for task $k$

The pretraining objective varies by modality. Decoder LLMs predict the next token. **BERT** and MAE reconstruct masked tokens or patches. **CLIP** and **SigLIP** pull matching image–text pairs together with a contrastive loss. What they share is that the label comes from the data itself, so the corpus can grow to trillions of tokens.

> Figure: Broad unlabelled data (left) trains one set of weights with a self-supervised objective. GPT-3 is the example: N = 175B parameters, D = 300B tokens, so C ≈ 6ND ≈ 3.1 × 10²³ FLOPs. The same weights are then adapted, from cheapest (a prompt, 0 trained parameters) to most expensive (full fine-tuning, 100%). Every downstream task inherits whatever the base model learned, good or bad.

## Scale: compute, data and loss

$$
C\approx 6ND,\qquad L(N,D)\approx E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}
$$

- $C$: training FLOPs: about 2N per token for the forward pass and 4N for the backward pass
- $N,\,D$: parameters and training tokens
- $E,A,B,\alpha,\beta$: fitted constants; Hoffmann et al. (2022) report $E=1.69$, $A=406.4$, $B=410.7$, $\alpha=0.34$, $\beta=0.28$ for their models and data

Loss falls as a smooth power law in both $N$ and $D$ (Kaplan et al., 2020). For a fixed compute budget, Hoffmann et al. ("Chinchilla", 2022) found that $N$ and $D$ should grow roughly equally, about 20 tokens per parameter. Their 70B model trained on 1.4T tokens beat the 280B Gopher trained on 300B tokens at similar compute. Models meant for heavy use are now often trained far past that ratio, because a smaller model is cheaper to serve (see **Pretraining and continued pretraining**).

Landmark foundation models, as reported in their papers

| Model | Paper | Objective | Size | Pretraining data | Note |
| --- | --- | --- | --- | --- | --- |
| BERT-large | Devlin et al., 2018 | masked tokens | 340M | BooksCorpus + English Wikipedia (3.3B words) | fine-tuned per task |
| GPT-3 | Brown et al., 2020 | next token | 175B | 300B tokens | 3.14 × 10²³ FLOPs; few-shot prompting |
| CLIP | Radford et al., 2021 | contrastive | ViT-L/14@336px | 400M image–text pairs | 76.2% zero-shot ImageNet |
| Chinchilla | Hoffmann et al., 2022 | next token | 70B | 1.4T tokens | beat 280B Gopher at similar compute |
| Llama 2 | Touvron et al., 2023 | next token | 7B–70B | 2T tokens | weights released; 7B took 184,320 A100-hours |

## Adapting one backbone

Adaptation options, cheapest first. Trained-parameter shares are typical, not fixed.

| Method | Trained params | Labelled data | Where covered |
| --- | --- | --- | --- |
| Zero-/few-shot prompting | 0 | 0–a few examples in the prompt | **Zero-shot prompting**, **Few-shot prompting** |
| Retrieval | 0 (plus an index) | none; needs documents | **RAG architecture** |
| Linear probe on frozen features | one head | hundreds–thousands | this article's figure |
| LoRA / PEFT | ≈ 0.01–1% | thousands | **LoRA**, **Parameter-efficient fine-tuning (PEFT)** |
| Full fine-tuning | 100% | thousands–millions | **Supervised fine-tuning (SFT)** |

Hu et al. (2021) report that LoRA on GPT-3 175B cut trainable parameters by 10,000× and GPU memory by 3× compared with full fine-tuning, with similar quality. For choosing among these options, see **Fine-tuning vs RAG vs prompting**.

> Figure: Six classification tasks on synthetic 100-d inputs built from 4 hidden factors. "Pretraining" is PCA on M unlabelled inputs, giving a frozen 4-feature backbone; each task is then solved with k labels either from scratch on the raw input or by a linear probe on the backbone. Defaults (M = 200, k = 16): the backbone keeps 89–93% of each factor and mean accuracy is 81.6% adapted vs 76.3% from scratch (72.6% vs 65.5% at k = 4). With M = 10 the backbone keeps only 19–56% and loses from k = 16 on (79.2% vs 84.4% at k = 1024). Make factor 4 rare in the corpus and the backbone keeps 1.6% of it: task 5 (w₄ = 0.98) falls to 53% at k = 16, against 73% from scratch.

The figure shows both sides of homogenisation. A good backbone helps all six tasks at once, most when labels are scarce. A backbone that never saw factor 4 fails every task that needs it, and no amount of probe training recovers information the features dropped. Full fine-tuning can, in principle, because it changes $\theta$.

## Worked example: what pretraining costs

**A 7B model on 2T tokens (the Llama 2 7B setting)**

1. Compute: $C\approx6ND=6\times7\times10^{9}\times2\times10^{12}=8.4\times10^{22}$ FLOPs.
2. Tokens per parameter: $2\times10^{12}/7\times10^{9}\approx286$, about 14× the compute-optimal 20 (which would be 140B tokens).
3. Hardware time: an A100 peaks at 312 TFLOP/s in BF16. At 40% utilisation, $8.4\times10^{22}/(312\times10^{12}\times0.4)=6.73\times10^{8}$ GPU-seconds $\approx187{,}000$ GPU-hours.
4. The Llama 2 paper reports 184,320 A100-hours for 7B, which implies about 40.6% utilisation.
5. Adapting it with full fine-tuning on 10M tokens costs $6\times7\times10^{9}\times10^{7}=4.2\times10^{17}$ FLOPs, 1/200,000 of pretraining. That ratio is the economic case for foundation models.
6. Loss check with the Chinchilla fit: $L(70\text{B},1.4\text{T})=1.69+0.083+0.163=1.937$ versus $L(280\text{B},300\text{B})=1.69+0.052+0.251=1.993$. The smaller model trained on more data is predicted to be better, as observed.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

base = "meta-llama/Llama-2-7b-hf"            # the foundation model, trained once
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto")

# 1) adapt by prompting: no weights change
prompt = "Review: The battery died after a day.\nSentiment (positive/negative):"
ids = tok(prompt, return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=2)[0][ids.input_ids.shape[1]:]))

# 2) adapt by LoRA: train small low-rank updates to two projections per layer
cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
                 target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)
model.print_trainable_parameters()   # trainable% ≈ 0.062
```

One backbone, two adaptations with Hugging Face transformers and peft (not run here: needs the libraries, model access and a GPU). For Llama-2-7B the LoRA config trains 4,194,304 of 6,742,609,920 parameters (0.062%).

> **Inherited defects:** An adapter or prompt sits on top of the base model's knowledge, biases and blind spots. Bommasani et al. stress that a flaw in a widely used foundation model propagates to every product built on it, including memorised private data, a training-data cutoff and social biases (see **Bias and fairness in generative models**, **Privacy and data leakage**). Evaluate each adapted system on its own task. A good base-model benchmark score is not evidence for your use case.

> **Regulation uses the same idea:** The EU AI Act (Regulation (EU) 2024/1689) has separate obligations for "general-purpose AI models", and presumes systemic risk above $10^{25}$ training FLOPs. Thresholds and guidance change; this is not legal advice.

**Strengths**

- Pretraining cost is paid once and shared by every task built on the model.
- Strong results with few or no labels: prompting needs none, probes and LoRA need little.
- One model can serve many tasks and modalities, which simplifies serving and updates.

**Limitations**

- Pretraining needs 10²²–10²⁵ FLOPs and trillions of tokens, so few organisations can do it.
- Homogenisation: defects, biases and a fixed knowledge cutoff reach every downstream use.
- Behaviour is hard to predict or audit; abilities are discovered after training, not specified.

**Question:** In the figure, the corpus under-represents factor 4. Why can't more labelled data for task 5 fix the linear probe?

- A. The probe overfits as k grows
- B. The frozen features keep only 1.6% of factor 4, so the information the task needs is not in them
- C. PCA needs labels to find factor 4
- D. Task 5 is not linearly separable

**Answer:** B. The frozen features keep only 1.6% of factor 4, so the information the task needs is not in them — A probe can only recombine the backbone's features. If pretraining discarded a direction, no head on top can recover it. Changing $\theta$ (fine-tuning) or pretraining on better data can.

**Question:** Roughly how many FLOPs does it take to train a 70B-parameter model on 1.4T tokens?

- A. 5.9 × 10¹⁹
- B. 9.8 × 10²¹
- C. 5.9 × 10²³
- D. 1.4 × 10²⁵

**Answer:** C. 5.9 × 10²³ — $C\approx6ND=6\times7\times10^{10}\times1.4\times10^{12}=5.88\times10^{23}$. This is the Chinchilla setting.

**Key points**

- Foundation model (Bommasani et al., 2021): trained on broad data, usually by self-supervision at scale, then adapted to many tasks.
- Objectives: next token (GPT, Llama), masked tokens (BERT, MAE), contrastive pairs (CLIP, SigLIP).
- Training compute C ≈ 6ND; Chinchilla: about 20 tokens per parameter is compute-optimal, but deployed models are often trained much longer.
- Adaptation, cheapest first: prompting, retrieval, probes, LoRA/PEFT, full fine-tuning.
- Homogenisation: one model's strengths and defects reach every system built on it, so evaluate each adapted system on its own task.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
