# Bias and fairness in generative models

> Generative models reproduce and can amplify social bias; measure it with output-distribution audits, counterfactual prompt pairs and benchmarks like BBQ, and, where a model makes decisions, compare selection rate, TPR and FPR per group.

Source: https://www.mlfoundry.in/learn/bias-and-fairness-in-generative-models  
Section: Generative AI › Safety, Alignment and Responsible AI  
Level: core · 8 min read · updated 2026-10-21

**Bias in a generative model:** is a systematic difference in what the model produces, or how well it serves people, that depends on a social group (gender, race, age, dialect, religion, disability) when it should not. Barocas, Crawford and colleagues (2017) separated **representational harms**, such as stereotyping and erasure, from **allocational harms**, where resources or opportunities are distributed unequally. Blodgett et al. ("Language (Technology) is Power", ACL 2020) showed that many NLP bias papers never say which harm they measure.

The bias comes mostly from data. Web text and image–caption pairs over-represent some groups, pair occupations with genders, and contain slurs and stereotypes. Bolukbasi et al. (2016) found word2vec completing "man is to computer programmer as woman is to *homemaker*". A generative model samples from what it learned, so the same associations show up in stories, letters and pictures. Sampling can also *amplify* a skew. For example, if 70% of training captions for an occupation show men, a model that prefers the mode can produce men 90% of the time.

Fairness work in generative AI therefore has three jobs: decide which harm matters for the product, measure it with a probe that targets that harm, and re-measure after every fix. This sits next to **Content moderation** and **Guardrails**, which filter unacceptable outputs. Bias is usually subtler: every individual output can be acceptable while the distribution is skewed.

> Figure: Three probes. Representation: sample many images for a neutral prompt, label the depicted attributes, and compare shares with a stated target. Counterfactual: change only the name (or dialect, or pronoun) in a fixed prompt and compare paired outputs. Benchmark: ambiguous questions where the correct answer is "cannot be determined", as in BBQ. Each probe maps to a harm, and each fix (data, training, prompt, output) is re-checked with the same probes.

## Measuring it

$$
\hat p_g=\frac1N\sum_{i=1}^{N}\mathbb 1\big[a(y_i)=g\big],\qquad \mathrm{TV}=\tfrac12\sum_g\big|\hat p_g-t_g\big|,\qquad \Delta_{cf}=\frac1M\sum_{m=1}^{M}\Big[s\big(f(x_m^{(a)})\big)-s\big(f(x_m^{(b)})\big)\Big]
$$

- $a(y_i)$: attribute of output $i$ (e.g. perceived gender in an image), from a classifier or human annotators
- $t_g$: target share for group $g$: uniform, a census or workforce statistic, or a product decision; it must be stated
- $x_m^{(a)},x_m^{(b)}$: the same prompt template $m$ filled with group terms $a$ and $b$ (names, pronouns, dialect)
- $s,\;f$: a score of the output (sentiment, toxicity, refusal, competence words) and the model under test

The counterfactual gap is a paired comparison, so test it with a paired test (sign-flip permutation or paired bootstrap). Template-level variation then cancels out. Attribute classifiers are themselves biased, especially for perceived race and gender in images, so audit a sample by hand and report the classifier's error rates.

$$
s_{\text{DIS}}=2\,\frac{n_{\text{biased}}}{n_{\text{non-unknown}}}-1,\qquad s_{\text{AMB}}=(1-\text{acc}_{\text{amb}})\;s_{\text{DIS}}
$$

- $s_{\text{DIS}}$: BBQ bias score in $[-1,1]$: among answers that name a person, the share that matches the stereotype, rescaled
- $s_{\text{AMB}}$: bias score in ambiguous contexts, scaled by the error rate, since there the only correct answer is "unknown"

Common text bias benchmarks. They measure different constructs, and scores on one rarely predict another.

| Benchmark | Authors, year | What it probes | Size |
| --- | --- | --- | --- |
| WinoBias | Zhao et al., 2018 | gendered coreference with occupations, pro- vs anti-stereotypical | 3,160 sentences |
| WinoGender | Rudinger et al., 2018 | pronoun resolution for 60 occupations | 720 sentences |
| CrowS-Pairs | Nangia et al., 2020 | likelihood of stereotyping vs less stereotyping sentence, 9 bias types | 1,508 pairs |
| RealToxicityPrompts | Gehman et al., 2020 | toxicity of continuations of web-text prompts | 100k prompts |
| BBQ | Parrish et al., 2022 | QA with ambiguous and disambiguated contexts, 9 social dimensions | about 58k questions |

> **Benchmarks are narrow instruments:** Blodgett et al. ("Stereotyping Norwegian Salmon", ACL 2021) found many CrowS-Pairs and StereoSet items were ill-formed or did not test a stereotype at all. Likelihood-based scores on a base model also say little about what an instruction-tuned assistant writes. Use benchmarks to track regressions, and pair them with probes built for your own prompts and users.

## When the model decides

An LLM that screens CVs, triages support tickets or summarises loan files is a classifier, and the classic group-fairness metrics apply. **Demographic parity** compares selection rates. **Equal opportunity** compares true-positive rates; **equalised odds** (Hardt, Price and Srebro, 2016) compares both TPR and FPR. **Predictive parity** compares precision. When base rates differ between groups and the classifier is imperfect, these cannot all hold at once (Kleinberg, Mullainathan and Raghavan, 2016; Chouldechova, 2017). Choosing one is a policy decision.

> Figure: 400 synthetic candidates per group; the score is a logit with means ±0.9 and unit noise, and group B's name costs δ. Defaults (δ = 0.5, one threshold t = 0): A selects 49.0% with TPR 85.3%, B selects 34.0% with TPR 59.7%, so the ratio is 0.69 (below 0.8) and a name swap flips 14.2% of B's decisions. With δ = 0 noise alone leaves a 7.2-point TPR gap. Equal selection rate moves B's threshold to −0.51, yet 13.8% of B still flip. With B's base rate at 30%, equal TPR gives a ratio of 0.87 and precision 58.2% vs 80.1%.

Two lessons from the figure. First, per-group thresholds can equalise a rate, but they do not make the decision blind to the name: the individual counterfactual still fails. The better fix is to remove the dependence from the model or the prompt. Second, with unequal base rates, equalising TPR leaves unequal selection rates and unequal precision. Picking one metric is a choice, not an error.

## Worked example: a screening audit

100 applicants per group, 50 qualified in each; the LLM's "advance" decisions

| Group | TP | FN | FP | TN | Selection rate | TPR | FPR | Precision |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| A | 40 | 10 | 10 | 40 | 50/100 = 0.50 | 40/50 = 0.80 | 10/50 = 0.20 | 40/50 = 0.80 |
| B | 28 | 22 | 4 | 46 | 32/100 = 0.32 | 28/50 = 0.56 | 4/50 = 0.08 | 28/32 = 0.875 |

**Reading it**

1. Selection-rate ratio B/A = 0.32 / 0.50 = **0.64**, below the 0.8 of the US "four-fifths" guideline for adverse impact.
2. Equal opportunity gap: TPR 0.80 − 0.56 = **0.24**; 12 more qualified B applicants are rejected than the A rate would predict (50 × 0.24).
3. FPR gap 0.20 − 0.08 = 0.12: the model is also stricter with unqualified B applicants.
4. Precision is *higher* for B (0.875 vs 0.80): a B candidate must look stronger to pass. Reporting precision alone would make B look favoured.
5. BBQ-style check on the same model: 60 of 80 non-unknown answers are stereotyped → s_DIS = 2 × 0.75 − 1 = 0.5; ambiguous accuracy 0.6 → s_AMB = 0.4 × 0.5 = 0.2.

```python
import numpy as np

def counterfactual_gap(score_a, score_b, n_perm=10_000, seed=0):
    """Paired name-swap audit: mean difference and a two-sided sign-flip permutation p-value."""
    d = np.asarray(score_a) - np.asarray(score_b)
    rng = np.random.default_rng(seed)
    null = (rng.choice([-1.0, 1.0], size=(n_perm, d.size)) * d).mean(axis=1)
    p = (np.sum(np.abs(null) >= abs(d.mean())) + 1) / (n_perm + 1)
    return d.mean(), p

def group_rates(y_true, y_pred, group):
    """Selection rate, TPR and FPR per group, for an LLM used as a classifier."""
    out = {}
    for g in np.unique(group):
        t, p = y_true[group == g], y_pred[group == g]
        out[str(g)] = {"sr": p.mean(), "tpr": p[t == 1].mean(), "fpr": p[t == 0].mean()}
    return out

# 1) Sentiment of 200 generated reference letters, same template, name A vs name B.
#    In practice score_a[i], score_b[i] come from a classifier run on the model's two outputs.
rng = np.random.default_rng(1)
quality = rng.normal(0.70, 0.10, 200)               # template-level variation (shared)
score_a = quality + rng.normal(0, 0.05, 200)
score_b = quality - 0.03 + rng.normal(0, 0.05, 200)  # small systematic penalty for name B
gap, p = counterfactual_gap(score_a, score_b)
print(f"gap {gap:.3f}, p = {p:.4f}")             # gap 0.024, p = 0.0001

# 2) The worked example: 100 applicants per group, 50 qualified in each.
y = np.r_[np.ones(50), np.zeros(50)]                # qualified first
y_true = np.r_[y, y]
y_pred = np.r_[np.r_[np.ones(40), np.zeros(10), np.ones(10), np.zeros(40)],
               np.r_[np.ones(28), np.zeros(22), np.ones(4), np.zeros(46)]]
group = np.array(["A"] * 100 + ["B"] * 100)
r = group_rates(y_true, y_pred, group)
print(round(r["B"]["sr"] / r["A"]["sr"], 2), round(r["A"]["tpr"] - r["B"]["tpr"], 2))   # 0.64 0.24
```

A paired counterfactual audit and per-group rates in numpy (runs as is); in production score_a and score_b are classifier scores of the model's outputs for the two name variants

## Mitigation

- **Data:** Filter slurs and demeaning content, rebalance or caption-augment under-represented groups, and deduplicate stereotyped templates. Filtering can also remove dialects and identity terms; check what a toxicity filter drops.
- **Training:** Add counterfactual and balanced examples to **Supervised fine-tuning (SFT)** data; include bias probes in preference data for **RLHF** or **Direct preference optimization (DPO)**, or in the principles of **Constitutional AI**.
- **Prompt and decoding:** Prompt expansion that samples attributes for underspecified people in image prompts; instructions not to infer demographics; for screening, strip names and other proxies before the model sees the file.
- **Output and monitoring:** Slice production metrics by group where it is lawful and consented; re-run the counterfactual suite on every model or prompt change, as part of **Red teaming** and regression tests.

> **Overcorrection is also a failure:** Blanket prompt rewriting ignores context. In February 2024 Google paused Gemini's generation of images of people after its tuning to show a range of people produced historically inaccurate depictions. A fix should apply only when the prompt leaves the attribute unspecified, and must be evaluated on prompts where the attribute *is* specified.

> **Bias can be covert:** Models tuned to avoid overt stereotypes can still treat proxies differently. Hofmann et al. (Nature, 2024) found that LLMs judged speakers of African American English more negatively than speakers of Standard American English saying the same thing, in tasks such as assigning occupations, even when they produced positive statements about Black people when asked directly. Probe with proxies (dialect, names, neighbourhoods), not only explicit group terms.

> **Law, briefly (not legal advice):** In the US, the EEOC's Uniform Guidelines on Employee Selection Procedures (1978) treat a selection rate below four-fifths of the highest group's as evidence of adverse impact, and that applies to automated tools too. The EU AI Act (2024) classifies AI systems used in recruitment and in access to essential services as high-risk, with duties on data governance and bias monitoring. Requirements vary by jurisdiction and change; consult counsel for any deployment decision.

**Question:** You equalise selection rates by giving group B a lower threshold. A name-swap test still changes about 14% of B's decisions. What does this show?

- A. The threshold search has a bug
- B. Group-level parity does not imply individual counterfactual fairness: the score still depends on the name
- C. Selection-rate parity is impossible with unequal base rates
- D. The name-swap test is invalid once thresholds differ

**Answer:** B. Group-level parity does not imply individual counterfactual fairness: the score still depends on the name — Parity constrains a rate over the group; the counterfactual test checks whether each person's decision would change with only the name changed. Moving the threshold offsets the penalty on average, but it changes which individuals pass. The fix is to remove the dependence, for example by redacting names before the model sees the file.

**Question:** On BBQ, a model answers 90% of ambiguous questions correctly ("unknown"), and among the rest it picks the stereotyped person every time. What is s_AMB?

- A. 0.9
- B. 1.0
- C. 0.1
- D. 0

**Answer:** C. 0.1 — All non-unknown answers are biased, so $s_{\text{DIS}}=2\cdot1-1=1$. Then $s_{\text{AMB}}=(1-0.9)\cdot1=0.1$. The scaling by error rate means a model that mostly abstains correctly scores low even if its rare errors are all stereotyped.

**Key points**

- Name the harm first: representational (stereotypes, erasure), quality-of-service, or allocational (decisions).
- Representation: sample many outputs per neutral prompt and compare attribute shares to a stated target (e.g. TV distance).
- Counterfactual pairs change only the group term or proxy; test the paired gap with a permutation or paired bootstrap.
- BBQ: s_AMB = (1 − accuracy)·s_DIS; benchmarks are narrow and do not substitute for product-specific probes.
- For decisions, report selection rate, TPR, FPR and precision per group; with unequal base rates they cannot all be equal.
- Group thresholds fix rates, not individual dependence on names; remove the proxy, then re-measure every fix, watching for overcorrection.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
