Definition
FID and Inception Score are sample-based quality metrics for image generators that score images with a frozen ImageNet Inception-v3 network. The Inception Score (IS) (Salimans et al., "Improved Techniques for Training GANs", NeurIPS 2016) uses its class probabilities on generated images only. The Fréchet Inception Distance (FID) (Heusel et al., "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", NeurIPS 2017) compares the distribution of its 2048-d pool features on generated images with that on real images.
Image generators such as Generative adversarial network (GAN) and Diffusion models give no usable likelihood, or one that tracks perceived quality poorly. Teams still need one number per checkpoint to pick a sampler, a guidance scale or a training run. IS and FID provide it from a few thousand samples and a fixed network, with no human raters.
In practice FID is the headline metric of image-generation papers and training dashboards. IS appears mainly in older GAN work and in class-conditional ImageNet results. For text-to-image models FID measures realism only. Prompt alignment needs a separate metric such as CLIP score.
Inception Score
- Inception class softmax for one generated image
- marginal label distribution over all generated images
- between 1 and the number of classes; higher is better
The KL term is large when each image is classified confidently (low entropy of , read as "sharp") and the labels are spread out (high entropy of , read as "diverse"). Equivalently, is the mutual information between image and label under the generator. It never touches real images, so it cannot tell whether the samples look like the training data, and it is blind to diversity within a class.
Fréchet Inception Distance
- mean and covariance of the 2048-d pool features of real images
- the same for generated images
- matrix square root; lower FID is better, 0 when the two Gaussians are equal
This is the squared 2-Wasserstein (Fréchet) distance between and . Only the first two moments of the features count. The mean term grows when generated features are shifted, the trace term when their spread or correlation differ. Dropping modes changes both.
Sample-size bias and KID
FID is computed from sample estimates of and , and the estimator is biased upward: two sets of samples from the same distribution give FID > 0, and the gap shrinks with . Chong and Forsyth (CVPR 2020) showed the bias is roughly linear in and that it depends on the generator, so a ranking at can differ from one at . They proposed extrapolating to . IS is biased too.
- Inception features of real and generated images,
- cubic polynomial kernel; it matches moments up to the third
- unbiased estimate of squared MMD; its expectation is 0 for identical distributions, so single estimates can be negative
The Kernel Inception Distance (Bińkowski et al., "Demystifying MMD GANs", ICLR 2018) uses the same features but has an unbiased estimator and makes no Gaussian assumption. It is the better choice when only a few thousand images are available, for example in few-shot or per-subject evaluation.
Conventions and known results
- IS protocol
- Salimans et al.: 50,000 samples, split into 10 groups, reported as mean ± std over the groups; real CIFAR-10 training images score 11.24 ± 0.12
- FID protocol
- pool3 features (2048-d) of Inception-v3; the authors recommend at least 10,000 samples, and 50,000 generated images against the training set is the common GAN convention
- Text-to-image
- zero-shot FID on 30,000 MS-COCO validation captions (FID-30K); Imagen (Saharia et al., 2022) reported 7.27
- Class-conditional
- BigGAN at 128×128 on ImageNet: IS 166.5 and FID 7.4 (Brock et al., ICLR 2019), up from the previous best of 52.52 and 18.65
- Resizing
- Parmar et al. (CVPR 2022, clean-fid): different resize filters and JPEG compression of the same images change FID enough to reorder models
- Class leakage
- Kynkäänniemi et al. (ICLR 2023): FID can be lowered by matching the ImageNet class histogram of the real set, without better images
Worked example
| Generated set | p(y|x) per image | p(y) | KL per image | IS |
|---|---|---|---|---|
| sharp, diverse | (.9,.05,.05) (.05,.9,.05) (.05,.05,.9) | (.333, .333, .333) | 0.704 each | exp(0.704) = 2.022 |
| one blurry image | (.9,.05,.05) (.05,.9,.05) (.34,.33,.33) | (.430, .427, .143) | 0.505, 0.512, 0.111 | exp(0.376) = 1.456 |
| collapsed | (.9,.05,.05) three times | (.9, .05, .05) | 0, 0, 0 | exp(0) = 1.000 |
The collapsed set is three confident images of one class: each one is sharp, yet IS is at its minimum, because . For FID, take the diagram's toy Gaussians: , , and with trace 3, so FID .
1import numpy as np23def psd_sqrt(a):4 w, v = np.linalg.eigh(a)5 return (v * np.sqrt(np.clip(w, 0, None))) @ v.T67def fid(feat_r, feat_g):8 """feat_*: (n, d) pool features of real and generated images."""9 mu_r, mu_g = feat_r.mean(0), feat_g.mean(0)10 s_r, s_g = np.cov(feat_r, rowvar=False), np.cov(feat_g, rowvar=False)11 r = psd_sqrt(s_r) # Tr((S_r S_g)^1/2) = Tr((r S_g r)^1/2), r = S_r^1/212 tr_sqrt = np.trace(psd_sqrt(r @ s_g @ r))13 return float(((mu_r - mu_g) ** 2).sum() + np.trace(s_r) + np.trace(s_g) - 2 * tr_sqrt)1415def kid(feat_r, feat_g):16 """Unbiased MMD^2 with the cubic polynomial kernel of Binkowski et al. (2018)."""17 d, m = feat_r.shape[1], len(feat_r)18 k = lambda a, b: (a @ b.T / d + 1) ** 319 kxx, kyy, kxy = k(feat_r, feat_r), k(feat_g, feat_g), k(feat_r, feat_g)20 off = lambda K: (K.sum() - np.trace(K)) / (m * (m - 1))21 return float(off(kxx) + off(kyy) - 2 * kxy.mean())2223rng = np.random.default_rng(0)24d = 64 # stand-in for 2048-d Inception features25real = rng.normal(size=(5000, d))26same = rng.normal(size=(5000, d)) # identical distribution27worse = 1.3 * rng.normal(size=(5000, d)) + 0.1 # wider and shifted28for n in (500, 5000):29 print(n, round(fid(real[:n], same[:n]), 3), round(fid(real[:n], worse[:n]), 3),30 round(kid(real[:n], same[:n]), 4), round(kid(real[:n], worse[:n]), 4))For real evaluations use a maintained implementation that fixes the network weights and resizing, such as pytorch-fid, clean-fid or torchmetrics, and compute the reference statistics once with the same tool.
Use when
- +FID: one number that responds to both fidelity and mode coverage, and is compared against real data
- +Cheap: one forward pass per image through a fixed network, no raters
- +KID: unbiased, so usable with a few thousand images
Avoid when
- −IS ignores real data and within-class diversity, and is capped by the 1,000 ImageNet classes
- −FID assumes Gaussian features, is biased at small N and depends on preprocessing
- −Neither measures prompt alignment, and both can be gamed by matching ImageNet statistics
Check yourself
A generator memorises one perfect image per ImageNet class and outputs only those 1,000 images. What happens to IS and FID?
Check yourself
You compute FID between two disjoint halves of the same real test set, 2,000 images each, and get a clearly positive value. Why?
Remember
- ✓IS on generated images only: high for confident, varied labels, between 1 and the class count.
- ✓FID on 2048-d Inception pool features; lower is better.
- ✓Both are biased by sample size: report N (50k is the GAN convention, FID-30K on COCO for text-to-image) and never mix N.
- ✓KID (cubic-kernel MMD) is unbiased and Gaussian-free, so it suits small evaluation sets.
- ✓Same weights, resizing and reference statistics or the numbers are not comparable; add precision/recall, CLIP score and human studies.