Definition
Generative AI is the use of generative models, models of the data distribution or of a conditional , to produce new text, images, audio, video or code by sampling. A discriminative model instead learns : it labels inputs but cannot produce them. The distinction is classical (Ng and Jordan, "On Discriminative vs. Generative Classifiers", NIPS 2001). The term "generative AI" now usually means large pretrained models (Foundation models) steered by a prompt .
Both kinds of model can be trained on the same data, but they answer different questions. A spam filter needs . An email-drafting assistant needs , from which it can sample many plausible emails. The second problem is harder: the model must put probability on every detail of , not just on the few features that separate the classes.
In a product, the generative model is one component. Around it sit a prompt (the condition ), a decoding rule that turns probabilities into outputs, often retrieved documents or tools, and filters and evaluators on the way out. The rest of this branch covers those parts.
Two factorisations of the same joint
- what a discriminative model learns: labels given inputs
- what a generative model learns: the inputs themselves
- Bayes' rule turns a class-conditional generative model into a classifier
The arrow only goes one way. A model of and yields , and samples. A model of yields none of the others.
How generative models are trained
- the model distribution with parameters
- Kullback–Leibler divergence; the two objectives differ by the data entropy, a constant
Maximum likelihood needs . Autoregressive models get it exactly by the chain rule, which is how every GPT architecture model is trained (see Language modeling):
- the -th token (word piece, pixel, audio codec token)
- all earlier tokens, plus the prompt for a conditional model
The other families trade exactness for something else. A Variational autoencoder (VAE) and Diffusion models maximise a lower bound (the ELBO). A Generative adversarial network (GAN) drops the likelihood entirely and trains against a discriminator. Normalizing flows keep it exact by making the network invertible. Flow matching regresses a velocity field and samples by integrating an ODE.
| Family | Key paper | Training signal | log p(x) | Steps per sample |
|---|---|---|---|---|
| Autoregressive | PixelCNN (van den Oord et al., 2016); GPT (Radford et al., 2018) | next-token cross-entropy | exact | one per token |
| VAE | Kingma & Welling, 2013 (ICLR 2014) | ELBO: reconstruction + KL to N(0, I) | lower bound | 1 |
| GAN | Goodfellow et al., 2014 | minimax game with a discriminator | none | 1 |
| Normalizing flow | NICE (Dinh et al., 2014); RealNVP (2016) | exact log-likelihood via the Jacobian | exact | 1 |
| Diffusion | Sohl-Dickstein et al., 2015; DDPM (Ho et al., 2020) | predict the added noise (ELBO-derived) | lower bound | 1,000 in DDPM; ~10–50 with fast samplers |
| Flow matching | Lipman et al., 2022 (ICLR 2023) | regress a velocity field | via ODE, expensive | tens of ODE steps |
Conditioning is the interface
Applications almost always use a conditional model . For a chat model, is the conversation and the reply. For Text-to-image generation, is the prompt embedding and the image. Practitioners mostly cannot change , so they work on four levers: the condition (prompting, retrieval), the sampler (decoding), small changes to (fine-tuning) and checks on (evaluation and safety).
The figure shows the trade. The discriminative model spends all its capacity on the boundary and wins on error. The generative model must also get the shape of each class right. When its family is wrong (one Gaussian for a two-cluster class), its samples drift into empty space and its likelihood drops. Ng and Jordan found the same pattern for naive Bayes versus logistic regression: the generative classifier approaches its (higher) asymptotic error with fewer examples.
| Use | Inputs | Computation | Result |
|---|---|---|---|
| Classify via Bayes | p(cat) = 0.3, p(x | cat) = 0.2, p(x | dog) = 0.05 | 0.06 / (0.06 + 0.7 × 0.05) = 0.06 / 0.095 | p(cat | x) = 0.632 |
| Score a sentence | p(the) = 0.1, p(cat | the) = 0.05, p(sat | the cat) = 0.2 | 0.1 × 0.05 × 0.2 | p = 0.001; −log p = 6.908 nats |
| Per token | 3 tokens | 6.908 / 3 = 2.303 nats; e^2.303 | perplexity 10.0 |
The last row is the definition of Perplexity: the model is, on average, as uncertain as a uniform choice among 10 tokens.
1import numpy as np23rng = np.random.default_rng(0)4X0 = rng.normal([-0.7, -0.2], 0.6, size=(100, 2)) # class 05X1 = rng.normal([0.7, 0.2], 0.6, size=(100, 2)) # class 167# Generative: fit p(x | y) as one Gaussian per class, plus p(y)8fit = [(X.mean(0), np.cov(X.T)) for X in (X0, X1)]9prior = np.array([0.5, 0.5])1011def log_gauss(x, m, S):12 d = x - m13 return -0.5 * (d @ np.linalg.solve(S, d) + np.log(np.linalg.det(2 * np.pi * S)))1415def posterior(x): # Bayes: p(y | x)16 lp = np.array([log_gauss(x, m, S) for m, S in fit]) + np.log(prior)17 return np.exp(lp - np.logaddexp.reduce(lp))1819def log_px(x): # density of the input itself20 return np.logaddexp.reduce([log_gauss(x, m, S) + np.log(p) for (m, S), p in zip(fit, prior)])2122def sample(n): # ancestral sampling: y, then x | y23 ys = rng.choice(2, size=n, p=prior)24 return np.array([rng.multivariate_normal(*fit[y]) for y in ys]), ys2526x = np.array([0.5, 0.0])27print(np.round(posterior(x), 3)) # [0.113 0.887]28print(round(float(log_px(x)), 3)) # -1.50329print(sample(3)[0].shape) # (3, 2): new pointsWhere it's used
Text and code
Chat assistants, drafting, summarisation, Code generation, Text-to-SQL: autoregressive LLMs conditioned on a prompt.
Images and video
Text-to-image generation, Image editing and inpainting, Text-to-video generation: mostly diffusion or flow-matching models in a VAE latent space.
Multimodal
Multimodal LLMs read images and text; Any-to-any models also emit several modalities.
Data
Synthetic data generation for training or testing other models.
Check yourself
You have a trained logistic-regression spam filter. Which of these can it provide?
Check yourself
Which two families give an exact log-likelihood log p(x)?
Remember
- ✓Generative: model or and sample; discriminative: model only. Bayes' rule converts the first into the second, not the reverse.
- ✓Training is usually maximum likelihood, , or a bound or surrogate for it.
- ✓Families: autoregressive (exact, one step per token), VAE (bound), GAN (no likelihood), flows (exact), diffusion (bound, many steps), flow matching (ODE).
- ✓Applications use conditional models; the levers are the prompt, the sampler, fine-tuning and output checks.
- ✓A wrong model family shows up as samples in empty regions and a lower held-out likelihood; samples are plausible, not guaranteed true.