Definition
Bias in a generative model is a systematic difference in what the model produces, or how well it serves people, that depends on a social group (gender, race, age, dialect, religion, disability) when it should not. Barocas, Crawford and colleagues (2017) separated representational harms, such as stereotyping and erasure, from allocational harms, where resources or opportunities are distributed unequally. Blodgett et al. ("Language (Technology) is Power", ACL 2020) showed that many NLP bias papers never say which harm they measure.
The bias comes mostly from data. Web text and image–caption pairs over-represent some groups, pair occupations with genders, and contain slurs and stereotypes. Bolukbasi et al. (2016) found word2vec completing "man is to computer programmer as woman is to homemaker". A generative model samples from what it learned, so the same associations show up in stories, letters and pictures. Sampling can also amplify a skew. For example, if 70% of training captions for an occupation show men, a model that prefers the mode can produce men 90% of the time.
Fairness work in generative AI therefore has three jobs: decide which harm matters for the product, measure it with a probe that targets that harm, and re-measure after every fix. This sits next to Content moderation and Guardrails, which filter unacceptable outputs. Bias is usually subtler: every individual output can be acceptable while the distribution is skewed.
Measuring it
- attribute of output (e.g. perceived gender in an image), from a classifier or human annotators
- target share for group : uniform, a census or workforce statistic, or a product decision; it must be stated
- the same prompt template filled with group terms and (names, pronouns, dialect)
- a score of the output (sentiment, toxicity, refusal, competence words) and the model under test
The counterfactual gap is a paired comparison, so test it with a paired test (sign-flip permutation or paired bootstrap). Template-level variation then cancels out. Attribute classifiers are themselves biased, especially for perceived race and gender in images, so audit a sample by hand and report the classifier's error rates.
- BBQ bias score in : among answers that name a person, the share that matches the stereotype, rescaled
- bias score in ambiguous contexts, scaled by the error rate, since there the only correct answer is "unknown"
| Benchmark | Authors, year | What it probes | Size |
|---|---|---|---|
| WinoBias | Zhao et al., 2018 | gendered coreference with occupations, pro- vs anti-stereotypical | 3,160 sentences |
| WinoGender | Rudinger et al., 2018 | pronoun resolution for 60 occupations | 720 sentences |
| CrowS-Pairs | Nangia et al., 2020 | likelihood of stereotyping vs less stereotyping sentence, 9 bias types | 1,508 pairs |
| RealToxicityPrompts | Gehman et al., 2020 | toxicity of continuations of web-text prompts | 100k prompts |
| BBQ | Parrish et al., 2022 | QA with ambiguous and disambiguated contexts, 9 social dimensions | about 58k questions |
When the model decides
An LLM that screens CVs, triages support tickets or summarises loan files is a classifier, and the classic group-fairness metrics apply. Demographic parity compares selection rates. Equal opportunity compares true-positive rates; equalised odds (Hardt, Price and Srebro, 2016) compares both TPR and FPR. Predictive parity compares precision. When base rates differ between groups and the classifier is imperfect, these cannot all hold at once (Kleinberg, Mullainathan and Raghavan, 2016; Chouldechova, 2017). Choosing one is a policy decision.
Two lessons from the figure. First, per-group thresholds can equalise a rate, but they do not make the decision blind to the name: the individual counterfactual still fails. The better fix is to remove the dependence from the model or the prompt. Second, with unequal base rates, equalising TPR leaves unequal selection rates and unequal precision. Picking one metric is a choice, not an error.
Worked example: a screening audit
| Group | TP | FN | FP | TN | Selection rate | TPR | FPR | Precision |
|---|---|---|---|---|---|---|---|---|
| A | 40 | 10 | 10 | 40 | 50/100 = 0.50 | 40/50 = 0.80 | 10/50 = 0.20 | 40/50 = 0.80 |
| B | 28 | 22 | 4 | 46 | 32/100 = 0.32 | 28/50 = 0.56 | 4/50 = 0.08 | 28/32 = 0.875 |
Reading it
- 1Selection-rate ratio B/A = 0.32 / 0.50 = 0.64, below the 0.8 of the US "four-fifths" guideline for adverse impact.
- 2Equal opportunity gap: TPR 0.80 − 0.56 = 0.24; 12 more qualified B applicants are rejected than the A rate would predict (50 × 0.24).
- 3FPR gap 0.20 − 0.08 = 0.12: the model is also stricter with unqualified B applicants.
- 4Precision is higher for B (0.875 vs 0.80): a B candidate must look stronger to pass. Reporting precision alone would make B look favoured.
- 5BBQ-style check on the same model: 60 of 80 non-unknown answers are stereotyped → s_DIS = 2 × 0.75 − 1 = 0.5; ambiguous accuracy 0.6 → s_AMB = 0.4 × 0.5 = 0.2.
1import numpy as np23def counterfactual_gap(score_a, score_b, n_perm=10_000, seed=0):4 """Paired name-swap audit: mean difference and a two-sided sign-flip permutation p-value."""5 d = np.asarray(score_a) - np.asarray(score_b)6 rng = np.random.default_rng(seed)7 null = (rng.choice([-1.0, 1.0], size=(n_perm, d.size)) * d).mean(axis=1)8 p = (np.sum(np.abs(null) >= abs(d.mean())) + 1) / (n_perm + 1)9 return d.mean(), p1011def group_rates(y_true, y_pred, group):12 """Selection rate, TPR and FPR per group, for an LLM used as a classifier."""13 out = {}14 for g in np.unique(group):15 t, p = y_true[group == g], y_pred[group == g]16 out[str(g)] = {"sr": p.mean(), "tpr": p[t == 1].mean(), "fpr": p[t == 0].mean()}17 return out1819# 1) Sentiment of 200 generated reference letters, same template, name A vs name B.20# In practice score_a[i], score_b[i] come from a classifier run on the model's two outputs.21rng = np.random.default_rng(1)22quality = rng.normal(0.70, 0.10, 200) # template-level variation (shared)23score_a = quality + rng.normal(0, 0.05, 200)24score_b = quality - 0.03 + rng.normal(0, 0.05, 200) # small systematic penalty for name B25gap, p = counterfactual_gap(score_a, score_b)26print(f"gap {gap:.3f}, p = {p:.4f}") # gap 0.024, p = 0.00012728# 2) The worked example: 100 applicants per group, 50 qualified in each.29y = np.r_[np.ones(50), np.zeros(50)] # qualified first30y_true = np.r_[y, y]31y_pred = np.r_[np.r_[np.ones(40), np.zeros(10), np.ones(10), np.zeros(40)],32 np.r_[np.ones(28), np.zeros(22), np.ones(4), np.zeros(46)]]33group = np.array(["A"] * 100 + ["B"] * 100)34r = group_rates(y_true, y_pred, group)35print(round(r["B"]["sr"] / r["A"]["sr"], 2), round(r["A"]["tpr"] - r["B"]["tpr"], 2)) # 0.64 0.24Mitigation
Where it's used
Data
Filter slurs and demeaning content, rebalance or caption-augment under-represented groups, and deduplicate stereotyped templates. Filtering can also remove dialects and identity terms; check what a toxicity filter drops.
Training
Add counterfactual and balanced examples to Supervised fine-tuning (SFT) data; include bias probes in preference data for RLHF or Direct preference optimization (DPO), or in the principles of Constitutional AI.
Prompt and decoding
Prompt expansion that samples attributes for underspecified people in image prompts; instructions not to infer demographics; for screening, strip names and other proxies before the model sees the file.
Output and monitoring
Slice production metrics by group where it is lawful and consented; re-run the counterfactual suite on every model or prompt change, as part of Red teaming and regression tests.
Check yourself
You equalise selection rates by giving group B a lower threshold. A name-swap test still changes about 14% of B's decisions. What does this show?
Check yourself
On BBQ, a model answers 90% of ambiguous questions correctly ("unknown"), and among the rest it picks the stereotyped person every time. What is s_AMB?
Remember
- ✓Name the harm first: representational (stereotypes, erasure), quality-of-service, or allocational (decisions).
- ✓Representation: sample many outputs per neutral prompt and compare attribute shares to a stated target (e.g. TV distance).
- ✓Counterfactual pairs change only the group term or proxy; test the paired gap with a permutation or paired bootstrap.
- ✓BBQ: s_AMB = (1 − accuracy)·s_DIS; benchmarks are narrow and do not substitute for product-specific probes.
- ✓For decisions, report selection rate, TPR, FPR and precision per group; with unequal base rates they cannot all be equal.
- ✓Group thresholds fix rates, not individual dependence on names; remove the proxy, then re-measure every fix, watching for overcorrection.