# What is generative AI

> A generative model learns p(x) or p(x | c) and can sample new data; a discriminative one learns only p(y | x). Autoregressive, VAE, GAN, flow, diffusion and flow-matching models differ in how they sample and whether log p(x) is exact.

Source: https://www.mlfoundry.in/learn/what-is-generative-ai  
Section: Generative AI › Foundations of Generative AI  
Level: core · 7 min read · updated 2026-10-21

**Generative AI:** is the use of **generative models**, models of the data distribution $p(x)$ or of a conditional $p(x\mid c)$, to produce new text, images, audio, video or code by sampling. A **discriminative** model instead learns $p(y\mid x)$: it labels inputs but cannot produce them. The distinction is classical (Ng and Jordan, "On Discriminative vs. Generative Classifiers", NIPS 2001). The term "generative AI" now usually means large pretrained models (**Foundation models**) steered by a prompt $c$.

Both kinds of model can be trained on the same data, but they answer different questions. A spam filter needs $p(\text{spam}\mid\text{email})$. An email-drafting assistant needs $p(\text{email}\mid\text{instructions})$, from which it can sample many plausible emails. The second problem is harder: the model must put probability on every detail of $x$, not just on the few features that separate the classes.

In a product, the generative model is one component. Around it sit a prompt (the condition $c$), a decoding rule that turns probabilities into outputs, often retrieved documents or tools, and filters and evaluators on the way out. The rest of this branch covers those parts.

## Two factorisations of the same joint

$$
p(x,y)=p(y\mid x)\,p(x)=p(x\mid y)\,p(y),\qquad p(y\mid x)=\frac{p(x\mid y)\,p(y)}{\sum_{y'}p(x\mid y')\,p(y')}
$$

- $p(y\mid x)$: what a discriminative model learns: labels given inputs
- $p(x\mid y),\,p(x)$: what a generative model learns: the inputs themselves
- $\sum_{y'}$: Bayes' rule turns a class-conditional generative model into a classifier

The arrow only goes one way. A model of $p(x\mid y)$ and $p(y)$ yields $p(y\mid x)$, $p(x)$ and samples. A model of $p(y\mid x)$ yields none of the others.

> Figure: Top: a discriminative model maps an input to label probabilities; a generative model maps a condition plus random noise to a new sample from p(x | c). Bayes' rule links the two. Middle: six generative families, how each draws a sample and whether it gives an exact log-likelihood. Bottom: the families typically used for each modality (as of 2024).

## How generative models are trained

$$
\theta^*=\arg\max_\theta\;\mathbb E_{x\sim p_\text{data}}\big[\log p_\theta(x)\big]=\arg\min_\theta\;\mathrm{KL}\big(p_\text{data}\,\|\,p_\theta\big)
$$

- $p_\theta$: the model distribution with parameters $\theta$
- $\mathrm{KL}$: Kullback–Leibler divergence; the two objectives differ by the data entropy, a constant

Maximum likelihood needs $\log p_\theta(x)$. Autoregressive models get it exactly by the chain rule, which is how every **GPT architecture** model is trained (see **Language modeling**):

$$
\log p_\theta(x_1,\dots,x_T)=\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})
$$

- $x_t$: the $t$-th token (word piece, pixel, audio codec token)
- $x_{<t}$: all earlier tokens, plus the prompt $c$ for a conditional model

The other families trade exactness for something else. A **Variational autoencoder (VAE)** and **Diffusion models** maximise a lower bound (the ELBO). A **Generative adversarial network (GAN)** drops the likelihood entirely and trains against a discriminator. **Normalizing flows** keep it exact by making the network invertible. **Flow matching** regresses a velocity field and samples by integrating an ODE.

The main families. 'Steps' is the number of network evaluations per sample.

| Family | Key paper | Training signal | log p(x) | Steps per sample |
| --- | --- | --- | --- | --- |
| Autoregressive | PixelCNN (van den Oord et al., 2016); GPT (Radford et al., 2018) | next-token cross-entropy | exact | one per token |
| VAE | Kingma & Welling, 2013 (ICLR 2014) | ELBO: reconstruction + KL to N(0, I) | lower bound | 1 |
| GAN | Goodfellow et al., 2014 | minimax game with a discriminator | none | 1 |
| Normalizing flow | NICE (Dinh et al., 2014); RealNVP (2016) | exact log-likelihood via the Jacobian | exact | 1 |
| Diffusion | Sohl-Dickstein et al., 2015; DDPM (Ho et al., 2020) | predict the added noise (ELBO-derived) | lower bound | 1,000 in DDPM; ~10–50 with fast samplers |
| Flow matching | Lipman et al., 2022 (ICLR 2023) | regress a velocity field | via ODE, expensive | tens of ODE steps |

## Conditioning is the interface

Applications almost always use a conditional model $p(x\mid c)$. For a chat model, $c$ is the conversation and $x$ the reply. For **Text-to-image generation**, $c$ is the prompt embedding and $x$ the image. Practitioners mostly cannot change $\theta$, so they work on four levers: the condition $c$ (prompting, retrieval), the sampler (decoding), small changes to $\theta$ (fine-tuning) and checks on $x$ (evaluation and safety).

> Figure: Logistic regression versus a class-conditional generative model on the same 2-D data. Defaults (n = 200, class 1 has a second cluster, 1 Gaussian per class): LR test error 9.5%, Gaussian Bayes classifier 10.3%. Only the generative model gives a held-out log p(x) (−2.519 nats) and samples, but 10.4% of its 1,000 samples land where the true density is below 0.01, against 3.5% of real points. Switch to the 2-Gaussian mixture: error 9.6%, log p(x) −2.408, low-density samples 3.6%. At n = 40 the mixture overfits: log p(x) −3.656, worse than one Gaussian's −2.608.

The figure shows the trade. The discriminative model spends all its capacity on the boundary and wins on error. The generative model must also get the shape of each class right. When its family is wrong (one Gaussian for a two-cluster class), its samples drift into empty space and its likelihood drops. Ng and Jordan found the same pattern for naive Bayes versus logistic regression: the generative classifier approaches its (higher) asymptotic error with fewer examples.

Worked example: one generative model, two uses

| Use | Inputs | Computation | Result |
| --- | --- | --- | --- |
| Classify via Bayes | p(cat) = 0.3, p(x \| cat) = 0.2, p(x \| dog) = 0.05 | 0.06 / (0.06 + 0.7 × 0.05) = 0.06 / 0.095 | p(cat \| x) = 0.632 |
| Score a sentence | p(the) = 0.1, p(cat \| the) = 0.05, p(sat \| the cat) = 0.2 | 0.1 × 0.05 × 0.2 | p = 0.001; −log p = 6.908 nats |
| Per token | 3 tokens | 6.908 / 3 = 2.303 nats; e^2.303 | perplexity 10.0 |

The last row is the definition of **Perplexity**: the model is, on average, as uncertain as a uniform choice among 10 tokens.

```python
import numpy as np

rng = np.random.default_rng(0)
X0 = rng.normal([-0.7, -0.2], 0.6, size=(100, 2))      # class 0
X1 = rng.normal([0.7, 0.2], 0.6, size=(100, 2))        # class 1

# Generative: fit p(x | y) as one Gaussian per class, plus p(y)
fit = [(X.mean(0), np.cov(X.T)) for X in (X0, X1)]
prior = np.array([0.5, 0.5])

def log_gauss(x, m, S):
    d = x - m
    return -0.5 * (d @ np.linalg.solve(S, d) + np.log(np.linalg.det(2 * np.pi * S)))

def posterior(x):                                      # Bayes: p(y | x)
    lp = np.array([log_gauss(x, m, S) for m, S in fit]) + np.log(prior)
    return np.exp(lp - np.logaddexp.reduce(lp))

def log_px(x):                                         # density of the input itself
    return np.logaddexp.reduce([log_gauss(x, m, S) + np.log(p) for (m, S), p in zip(fit, prior)])

def sample(n):                                         # ancestral sampling: y, then x | y
    ys = rng.choice(2, size=n, p=prior)
    return np.array([rng.multivariate_normal(*fit[y]) for y in ys]), ys

x = np.array([0.5, 0.0])
print(np.round(posterior(x), 3))                       # [0.113 0.887]
print(round(float(log_px(x)), 3))                      # -1.503
print(sample(3)[0].shape)                              # (3, 2): new points
```

A class-conditional Gaussian model answers all three questions: p(y | x), log p(x) and new samples (numpy; runs as is)

> **Plausible is not true:** A generative model is trained to put mass where the data is, not to be correct. A sample from $p(x\mid c)$ is a fluent, typical continuation of $c$, so an LLM can produce a confident citation that does not exist. This is the root of **Hallucination**; grounding the condition with retrieved text (**RAG architecture**) and checking outputs (**Hallucination and groundedness evaluation**) are the standard mitigations.

> **The sampler is part of the system:** The same $p_\theta$ gives different behaviour under different decoding rules. Greedy decoding picks the arg-max token; temperature, top-k and top-p reshape the distribution before sampling. See **Greedy and beam search** and **Temperature sampling**.

- **Text and code:** Chat assistants, drafting, summarisation, **Code generation**, **Text-to-SQL**: autoregressive LLMs conditioned on a prompt.
- **Images and video:** **Text-to-image generation**, **Image editing and inpainting**, **Text-to-video generation**: mostly diffusion or flow-matching models in a VAE latent space.
- **Multimodal:** **Multimodal LLMs** read images and text; **Any-to-any models** also emit several modalities.
- **Data:** **Synthetic data generation** for training or testing other models.

**Question:** You have a trained logistic-regression spam filter. Which of these can it provide?

- A. p(spam | email)
- B. p(email), to flag unusual emails
- C. New example spam emails
- D. All three

**Answer:** A. p(spam | email) — Logistic regression models only $p(y\mid x)$. Scoring how typical an email is needs $p(x)$, and producing emails needs a sampler for $p(x\mid y)$; both require a generative model.

**Question:** Which two families give an exact log-likelihood log p(x)?

- A. GAN and VAE
- B. Autoregressive models and normalizing flows
- C. Diffusion and VAE
- D. GAN and diffusion

**Answer:** B. Autoregressive models and normalizing flows — Autoregressive models use the chain rule $\sum_t\log p(x_t\mid x_{<t})$; flows use the change-of-variables formula with the Jacobian determinant. VAEs and diffusion models optimise a lower bound, and GANs define no likelihood.

**Key points**

- Generative: model $p(x)$ or $p(x\mid c)$ and sample; discriminative: model $p(y\mid x)$ only. Bayes' rule converts the first into the second, not the reverse.
- Training is usually maximum likelihood, $\min\mathrm{KL}(p_\text{data}\|p_\theta)$, or a bound or surrogate for it.
- Families: autoregressive (exact, one step per token), VAE (bound), GAN (no likelihood), flows (exact), diffusion (bound, many steps), flow matching (ODE).
- Applications use conditional models; the levers are the prompt, the sampler, fine-tuning and output checks.
- A wrong model family shows up as samples in empty regions and a lower held-out likelihood; samples are plausible, not guaranteed true.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
