What is generative AI

A generative model learns p(x) or p(x | c) and can sample new data; a discriminative one learns only p(y | x). Autoregressive, VAE, GAN, flow, diffusion and flow-matching models differ in how they sample and whether log p(x) is exact.

7 min read654 wordscore1 viewsupdated 2026-10-21

Definition

Generative AI is the use of generative models, models of the data distribution p(x)p(x) or of a conditional p(x∣c)p(x\mid c), to produce new text, images, audio, video or code by sampling. A discriminative model instead learns p(y∣x)p(y\mid x): it labels inputs but cannot produce them. The distinction is classical (Ng and Jordan, "On Discriminative vs. Generative Classifiers", NIPS 2001). The term "generative AI" now usually means large pretrained models (Foundation models) steered by a prompt cc.

Both kinds of model can be trained on the same data, but they answer different questions. A spam filter needs p(spam∣email)p(\text{spam}\mid\text{email}). An email-drafting assistant needs p(email∣instructions)p(\text{email}\mid\text{instructions}), from which it can sample many plausible emails. The second problem is harder: the model must put probability on every detail of xx, not just on the few features that separate the classes.

In a product, the generative model is one component. Around it sit a prompt (the condition cc), a decoding rule that turns probabilities into outputs, often retrieved documents or tools, and filters and evaluators on the way out. The rest of this branch covers those parts.

Two factorisations of the same joint

p(x,y)=p(y∣x) p(x)=p(x∣y) p(y),p(y∣x)=p(x∣y) p(y)∑y′p(x∣y′) p(y′)p(x,y)=p(y\mid x)\,p(x)=p(x\mid y)\,p(y),\qquad p(y\mid x)=\frac{p(x\mid y)\,p(y)}{\sum_{y'}p(x\mid y')\,p(y')}
p(y∣x)p(y\mid x)
what a discriminative model learns: labels given inputs
p(x∣y), p(x)p(x\mid y),\,p(x)
what a generative model learns: the inputs themselves
∑y′\sum_{y'}
Bayes' rule turns a class-conditional generative model into a classifier

The arrow only goes one way. A model of p(x∣y)p(x\mid y) and p(y)p(y) yields p(y∣x)p(y\mid x), p(x)p(x) and samples. A model of p(y∣x)p(y\mid x) yields none of the others.

two questions about the same datadiscriminativelearns p(y | x)input xphoto of a catclassifiere.g. a ViTcat 0.93 · dog 0.06fox 0.01 (labels only)generativelearns p(x) or p(x | c)condition c + noise"a cat on a red sofa"generatore.g. diffusionnew x ~ p(x | c)an image never seenBayes: p(y | x) = p(x | y) p(y) / Σ p(x | y′) p(y′) so a p(x | y) model can also classifygenerative model familiesAutoregressivep(x) = Π p(xₜ | x<ₜ)sample: 1 pass per tokenlikelihood: exactVAE · Kingma & Welling 2013decode z ~ N(0, I)sample: 1 decoder passlikelihood: ELBO boundGAN · Goodfellow et al. 2014G(z) trained against Dsample: 1 generator passlikelihood: noneNormalizing flow · NICE 2014invertible x = f(z)sample: 1 passlikelihood: exact (Jacobian)Diffusion · DDPM 2020denoise from pure noisesample: ~10 to 1,000 stepslikelihood: ELBO boundFlow matching · 2022integrate a velocity ODEsample: ~10 to 100 stepslikelihood: via ODE, costlytext · codemostly autoregressive LLMsimages · videomostly diffusion, flow matchingspeech · musicAR over codec tokens, diffusion
fig.Top: a discriminative model maps an input to label probabilities; a generative model maps a condition plus random noise to a new sample from p(x | c). Bayes' rule links the two. Middle: six generative families, how each draws a sample and whether it gives an exact log-likelihood. Bottom: the families typically used for each modality (as of 2024).

How generative models are trained

θ∗=arg⁡max⁡θ  Ex∼pdata[log⁡pθ(x)]=arg⁡min⁡θ  KL(pdata ∥ pθ)\theta^*=\arg\max_\theta\;\mathbb E_{x\sim p_\text{data}}\big[\log p_\theta(x)\big]=\arg\min_\theta\;\mathrm{KL}\big(p_\text{data}\,\|\,p_\theta\big)
pθp_\theta
the model distribution with parameters θ\theta
KL\mathrm{KL}
Kullback–Leibler divergence; the two objectives differ by the data entropy, a constant

Maximum likelihood needs log⁡pθ(x)\log p_\theta(x). Autoregressive models get it exactly by the chain rule, which is how every GPT architecture model is trained (see Language modeling):

log⁡pθ(x1,…,xT)=∑t=1Tlog⁡pθ(xt∣x<t)\log p_\theta(x_1,\dots,x_T)=\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})
xtx_t
the tt-th token (word piece, pixel, audio codec token)
x<tx_{<t}
all earlier tokens, plus the prompt cc for a conditional model

The other families trade exactness for something else. A Variational autoencoder (VAE) and Diffusion models maximise a lower bound (the ELBO). A Generative adversarial network (GAN) drops the likelihood entirely and trains against a discriminator. Normalizing flows keep it exact by making the network invertible. Flow matching regresses a velocity field and samples by integrating an ODE.

FamilyKey paperTraining signallog p(x)Steps per sample
AutoregressivePixelCNN (van den Oord et al., 2016); GPT (Radford et al., 2018)next-token cross-entropyexactone per token
VAEKingma & Welling, 2013 (ICLR 2014)ELBO: reconstruction + KL to N(0, I)lower bound1
GANGoodfellow et al., 2014minimax game with a discriminatornone1
Normalizing flowNICE (Dinh et al., 2014); RealNVP (2016)exact log-likelihood via the Jacobianexact1
DiffusionSohl-Dickstein et al., 2015; DDPM (Ho et al., 2020)predict the added noise (ELBO-derived)lower bound1,000 in DDPM; ~10–50 with fast samplers
Flow matchingLipman et al., 2022 (ICLR 2023)regress a velocity fieldvia ODE, expensivetens of ODE steps
The main families. 'Steps' is the number of network evaluations per sample.

Conditioning is the interface

Applications almost always use a conditional model p(x∣c)p(x\mid c). For a chat model, cc is the conversation and xx the reply. For Text-to-image generation, cc is the prompt embedding and xx the image. Practitioners mostly cannot change θ\theta, so they work on four levers: the condition cc (prompting, retrieval), the sampler (decoding), small changes to θ\theta (fine-tuning) and checks on xx (evaluation and safety).

generative: p(x | y) = 1 Gaussian per class, p(y) = 0.50dots: training data · rings: 60 of the model's samplesn = 200 training pointstest set: 4,000 fresh pointslogistic regressionGaussian per classtest error9.5%log p(x) held outnot modelledcan sample new xnoanswersp(y | x)test error (Bayes)10.3%log p(x) held out-2.519 natscan sample new xyesanswersp(x), p(x|y), p(y|x)share in regions where p_data(x) < 0.01real test points3.5%1,000 model samples10.4%● class 0● class 1solid ellipse 1σ, dashed 2σ
view
data
p(x | y) model
fig.Logistic regression versus a class-conditional generative model on the same 2-D data. Defaults (n = 200, class 1 has a second cluster, 1 Gaussian per class): LR test error 9.5%, Gaussian Bayes classifier 10.3%. Only the generative model gives a held-out log p(x) (−2.519 nats) and samples, but 10.4% of its 1,000 samples land where the true density is below 0.01, against 3.5% of real points. Switch to the 2-Gaussian mixture: error 9.6%, log p(x) −2.408, low-density samples 3.6%. At n = 40 the mixture overfits: log p(x) −3.656, worse than one Gaussian's −2.608.

The figure shows the trade. The discriminative model spends all its capacity on the boundary and wins on error. The generative model must also get the shape of each class right. When its family is wrong (one Gaussian for a two-cluster class), its samples drift into empty space and its likelihood drops. Ng and Jordan found the same pattern for naive Bayes versus logistic regression: the generative classifier approaches its (higher) asymptotic error with fewer examples.

UseInputsComputationResult
Classify via Bayesp(cat) = 0.3, p(x | cat) = 0.2, p(x | dog) = 0.050.06 / (0.06 + 0.7 × 0.05) = 0.06 / 0.095p(cat | x) = 0.632
Score a sentencep(the) = 0.1, p(cat | the) = 0.05, p(sat | the cat) = 0.20.1 × 0.05 × 0.2p = 0.001; −log p = 6.908 nats
Per token3 tokens6.908 / 3 = 2.303 nats; e^2.303perplexity 10.0
Worked example: one generative model, two uses

The last row is the definition of Perplexity: the model is, on average, as uncertain as a uniform choice among 10 tokens.

python
1import numpy as np
2
3rng = np.random.default_rng(0)
4X0 = rng.normal([-0.7, -0.2], 0.6, size=(100, 2)) # class 0
5X1 = rng.normal([0.7, 0.2], 0.6, size=(100, 2)) # class 1
6
7# Generative: fit p(x | y) as one Gaussian per class, plus p(y)
8fit = [(X.mean(0), np.cov(X.T)) for X in (X0, X1)]
9prior = np.array([0.5, 0.5])
10
11def log_gauss(x, m, S):
12 d = x - m
13 return -0.5 * (d @ np.linalg.solve(S, d) + np.log(np.linalg.det(2 * np.pi * S)))
14
15def posterior(x): # Bayes: p(y | x)
16 lp = np.array([log_gauss(x, m, S) for m, S in fit]) + np.log(prior)
17 return np.exp(lp - np.logaddexp.reduce(lp))
18
19def log_px(x): # density of the input itself
20 return np.logaddexp.reduce([log_gauss(x, m, S) + np.log(p) for (m, S), p in zip(fit, prior)])
21
22def sample(n): # ancestral sampling: y, then x | y
23 ys = rng.choice(2, size=n, p=prior)
24 return np.array([rng.multivariate_normal(*fit[y]) for y in ys]), ys
25
26x = np.array([0.5, 0.0])
27print(np.round(posterior(x), 3)) # [0.113 0.887]
28print(round(float(log_px(x)), 3)) # -1.503
29print(sample(3)[0].shape) # (3, 2): new points
A class-conditional Gaussian model answers all three questions: p(y | x), log p(x) and new samples (numpy; runs as is)

Where it's used

Text and code

Chat assistants, drafting, summarisation, Code generation, Text-to-SQL: autoregressive LLMs conditioned on a prompt.

Images and video

Text-to-image generation, Image editing and inpainting, Text-to-video generation: mostly diffusion or flow-matching models in a VAE latent space.

Multimodal

Multimodal LLMs read images and text; Any-to-any models also emit several modalities.

Data

Synthetic data generation for training or testing other models.

Check yourself

You have a trained logistic-regression spam filter. Which of these can it provide?

Check yourself

Which two families give an exact log-likelihood log p(x)?

Remember

  • ✓Generative: model p(x)p(x) or p(x∣c)p(x\mid c) and sample; discriminative: model p(y∣x)p(y\mid x) only. Bayes' rule converts the first into the second, not the reverse.
  • ✓Training is usually maximum likelihood, min⁡KL(pdata∥pθ)\min\mathrm{KL}(p_\text{data}\|p_\theta), or a bound or surrogate for it.
  • ✓Families: autoregressive (exact, one step per token), VAE (bound), GAN (no likelihood), flows (exact), diffusion (bound, many steps), flow matching (ODE).
  • ✓Applications use conditional models; the levers are the prompt, the sampler, fine-tuning and output checks.
  • ✓A wrong model family shows up as samples in empty regions and a lower held-out likelihood; samples are plausible, not guaranteed true.
← back to Generative AI roadmap