# Content moderation

> Content moderation checks user and model content against a written policy: hash matching for known media, calibrated per-category classifiers, and two thresholds that remove, queue for human review or allow, set by error costs.

Source: https://www.mlfoundry.in/learn/content-moderation  
Section: Generative AI › Safety, Alignment and Responsible AI  
Level: core · 8 min read · updated 2026-10-21

**Content moderation:** is the process of deciding whether a piece of content (a user post, an uploaded image, a model's response) violates a written policy, and acting on it: remove, restrict, label, send to a human or allow. In generative AI it runs twice: on what users send to the model and on what the model produces. A widely cited design is OpenAI's "A Holistic Approach to Undesired Content Detection in the Real World" (Markov et al., AAAI 2023): a category taxonomy, careful labelling instructions, active learning on production traffic, and a classifier that returns one score per category.

Moderation is a classification problem with three features that ordinary classification lacks. Violations are rare, so precision depends heavily on the base rate. The categories are defined by policy text, not by nature, and the policy changes. And errors cost very different amounts in each direction and each category. A missed self-harm message is not the same as a wrongly removed joke.

This article is about classifying content against a policy at scale. Wiring checks around one application's LLM call (input, output and tool rails, fallbacks, latency) is covered in **Guardrails**. Measuring harms in model outputs by deliberate attack is **Red teaming**. Group disparities in what gets flagged are covered in **Bias and fairness in generative models**.

## The pipeline

> Figure: Cheap, exact checks run first: a perceptual hash of an image is matched against a list of known violating media. A multi-label classifier then scores the item per category; this one has harassment 0.62, above the review threshold 0.20 and below the removal threshold 0.90, so it goes to a human. Reviewer decisions and appeal outcomes become new labels for the hash list, retraining and recalibration.

**Stages, in order of cost**

1. **Policy taxonomy.** Write each category with a definition, inclusions, exclusions and borderline examples. Labellers, the classifier, the thresholds and appeals all follow this one document.
2. **Exact matching.** Compare perceptual hashes of images and video against curated lists of known violating media; PhotoDNA (Microsoft and Hany Farid, 2009) is the standard for known child sexual abuse material. Add blocklists for known spam URLs. This is cheap and precise, but it only catches what has been seen before.
3. **Classifiers.** Small, fast models score every item per category (hate, harassment, sexual, self-harm, violence and so on). An LLM given the policy text (for example Llama Guard, or a general model prompted as a judge) handles new policies and long context at a higher cost per item.
4. **Decision.** Per-category thresholds route each item to remove, human review or allow. Some categories get softer actions instead: a warning label, reduced distribution, an age gate, or support resources for self-harm.
5. **Human review and appeals.** Reviewers decide the uncertain band and escalations. Appeals go to a second reviewer. Their labels feed back into training data and threshold calibration.

## From a score to a decision

A classifier's score is only usable for thresholds if it is **calibrated**: among items scored 0.8, about 80% should truly violate. Calibration depends on prevalence. If a raw score $s$ has class-conditional densities $f_1$ (violating) and $f_0$ (benign), Bayes' rule gives

$$
p(s)=P(\text{violates}\mid s)=\frac{\pi f_1(s)}{\pi f_1(s)+(1-\pi)f_0(s)}
$$

- $\pi$: prevalence: share of traffic in this category that violates
- $f_1,\,f_0$: score distributions of violating and benign items

The same raw score means a lower probability when the category is rarer. A model calibrated on a balanced labelled set will overstate risk on production traffic. Recalibrate (Platt scaling or isotonic regression) on a random sample of real traffic, separately per category and per surface.

Given a calibrated $p$, a decision has expected cost $p\,C_{\text{fn}}$ if you allow, $(1-p)\,C_{\text{fp}}$ if you remove, and $C_r$ if you pay a human. Chow's reject rule (Chow, 1970) picks the cheapest of the three:

$$
\text{allow if } p<\frac{C_r}{C_{\text{fn}}},\qquad \text{remove if } p>1-\frac{C_r}{C_{\text{fp}}},\qquad \text{review otherwise}
$$

- $C_{\text{fn}}$: cost of leaving a violation up (harm, regulatory exposure)
- $C_{\text{fp}}$: cost of removing a legitimate item (user harm, appeals load, chilling effect)
- $C_r$: cost of one human review; a review band exists only if $C_r<C_{\text{fn}}C_{\text{fp}}/(C_{\text{fn}}+C_{\text{fp}})$

With no review option the single threshold is $p^*=C_{\text{fp}}/(C_{\text{fp}}+C_{\text{fn}})$. Categories with costly misses therefore get low thresholds, and in practice those are also the categories where human review capacity is spent.

Worked example: Chow's rule with C_fn = 10, C_fp = 1, C_r = 0.3 (arbitrary units)

| Item p | Allow cost p·C_fn | Remove cost (1 − p)·C_fp | Review cost | Decision |
| --- | --- | --- | --- | --- |
| 0.02 | 0.20 | 0.98 | 0.30 | allow |
| 0.62 (the diagram's item) | 6.20 | 0.38 | 0.30 | review |
| 0.95 | 9.50 | 0.05 | 0.30 | remove |
| thresholds | allow below 0.3/10 = 0.03 | remove above 1 − 0.3/1 = 0.70 | band 0.03–0.70 | no review: p* = 1/11 = 0.091 |

## Two thresholds on real traffic

> Figure: 100,000 items per category, raw scores Gaussian (benign N(0,1), violating N(d′,1)) and calibrated with the category's prevalence; reviewers are assumed 95% accurate at 60 items per hour. Defaults (harassment, prevalence 2%, d′ = 3, review from 0.20, remove from 0.90): 617 auto-removals (precision 96.7%), a queue of 1,834 items (30.6 reviewer-hours), overall recall 72.4%, precision 95.6% and 67 benign items removed. One threshold at 0.5 with no review: recall 58.0%, precision 82.1%, 253 benign removed. Review from 0.05: recall 85.3% but 4,548 items (75.8 hours).

The review band buys recall and precision with reviewer time. Lowering the review threshold catches more violations, but the queue grows faster than the catch, because benign items outnumber violating ones 49 to 1. Measured this way, moderation quality is a staffing question as much as a modelling one.

**Checkable specifics**

- **OpenAI moderation endpoint:** `POST /v1/moderations` returns `flagged`, a boolean per category and a `category_scores` value per category. As of the 2024 `omni-moderation` models it accepts text and images and covers harassment, hate, illicit, self-harm, sexual and violence, with subcategories such as `self-harm/intent` and `sexual/minors`.
- **Perspective API:** Jigsaw/Google. Its TOXICITY score estimates the probability that a reader would perceive the comment as toxic. Other attributes include SEVERE_TOXICITY, INSULT, PROFANITY, IDENTITY_ATTACK and THREAT.
- **Jigsaw Toxic Comment (Kaggle, 2018):** Wikipedia talk comments labelled toxic, severe_toxic, obscene, threat, insult and identity_hate; a common baseline dataset.
- **RealToxicityPrompts:** Gehman et al., Findings of EMNLP 2020: about 100K web-text prompts. Models are scored by expected maximum toxicity over 25 generations, with toxicity from Perspective and "toxic" meaning a score ≥ 0.5.
- **Identity-term bias:** Dixon et al. (AIES 2018) showed toxicity classifiers scoring neutral sentences containing identity terms (e.g. "gay") as toxic. Sap et al. (ACL 2019) found tweets in African-American English more likely to be labelled offensive.

## Measuring it

Precision is easy to measure: sample the removals and check them. Recall is hard, because the misses are buried in the allowed traffic. The standard answer is **prevalence**: label a uniform random sample of all content (or of all views) and count violations. With $n=2000$ sampled items and 2% found violating, the 95% interval is $0.02\pm1.96\sqrt{0.02\cdot0.98/2000}=2\%\pm0.61\%$. Rare categories need much larger samples, or stratified sampling over score bands with reweighting.

Track per category: prevalence, precision of automated actions, appeal rate and appeal overturn rate, review queue age, and the score drift of new traffic. A model update that changes the score distribution changes every threshold's meaning, so recalibrate before each rollout.

```python
# Response shape of a moderation API call (OpenAI /v1/moderations, abridged)
resp = {"results": [{
    "flagged": True,
    "category_scores": {"harassment": 0.62, "harassment/threatening": 0.04,
                        "hate": 0.08, "self-harm": 0.01, "sexual": 0.02,
                        "violence": 0.11},
}]}

# Per-category policy: (review_from, remove_from); stricter where misses cost more
POLICY = {"harassment": (0.20, 0.90), "harassment/threatening": (0.10, 0.80),
          "hate": (0.20, 0.90), "self-harm": (0.05, 0.95),   # self-harm: route to support
          "sexual": (0.30, 0.95), "violence": (0.30, 0.95)}

def route(scores):
    decision, reasons = "allow", []
    for cat, p in scores.items():
        lo, hi = POLICY[cat]
        if p >= hi:
            return "remove", [f"{cat}={p:.2f}>={hi}"]
        if p >= lo:
            decision = "review"
            reasons.append(f"{cat}={p:.2f}>={lo}")
    return decision, reasons

print(route(resp["results"][0]["category_scores"]))
# ('review', ['harassment=0.62>=0.2'])
```

Routing a moderation API response with per-category thresholds (runs as is on the example response; a live call needs an API key and the network)

> **The vendor's flag is not your policy:** A moderation API's boolean `flagged` reflects the vendor's thresholds and taxonomy, tuned for their traffic. A children's education app and a medical forum need different lines: self-harm discussion may be core content on one and a crisis signal on the other. Use the per-category scores, set your own thresholds on a labelled sample of your traffic, and add categories the vendor does not cover (spam, off-topic, competitor promotion) with your own classifier.

> **Moderating generated content:** Model outputs differ from user posts: they arrive in a stream, they follow from a prompt you can also score, and the "author" is your product. Score the prompt and the response together (Llama Guard takes the conversation as input). Treat a refused or blocked generation as data for **Red teaming** and for the safety-training set, not just as an incident.

- **User-generated content platforms:** Hash matching plus per-category classifiers on posts, comments and uploads, with large review teams for the uncertain band.
- **Chat products:** Scoring prompts and responses, with softer actions (refuse, redirect to support resources) rather than takedowns.
- **Training-data filtering:** Running toxicity and NSFW classifiers over pretraining or fine-tuning corpora; see **Dataset curation for fine-tuning**.

**Question:** A classifier is calibrated on a balanced labelled set (50% violating) and deployed where 1% of items violate. An item scores 0.8. What is its true probability of violating, roughly?

- A. 0.8
- B. Higher than 0.8
- C. Much lower than 0.8, about 0.04
- D. Exactly 0.01

**Answer:** C. Much lower than 0.8, about 0.04 — On the balanced set the likelihood ratio at that score is $0.8/0.2=4$. Production prior odds are $1/99$, so posterior odds are $4/99$ and $p=4/103\approx0.039$. Recalibrate on production traffic before you set thresholds.

**Question:** Under Chow's rule with C_fn = 10, C_fp = 1, what happens to the review band if a human review becomes more expensive, C_r rising from 0.3 to 0.9?

- A. It widens
- B. It narrows to 0.09–0.10
- C. It disappears entirely
- D. It is unaffected

**Answer:** B. It narrows to 0.09–0.10 — Allow below $0.9/10=0.09$ and remove above $1-0.9/1=0.10$, so only items with $0.09\le p\le0.10$ go to review. The band vanishes completely once $C_r\ge C_{\text{fn}}C_{\text{fp}}/(C_{\text{fn}}+C_{\text{fp}})=10/11$.

**Key points**

- One written policy taxonomy drives labels, classifiers, thresholds, reviewers and appeals.
- Run cheap exact checks first (perceptual hashes such as PhotoDNA, blocklists), then per-category classifiers, then humans.
- Calibrated probability depends on prevalence: $p=\pi f_1/(\pi f_1+(1-\pi)f_0)$. Recalibrate on real traffic per category.
- Chow's rule: allow below $C_r/C_{\text{fn}}$, remove above $1-C_r/C_{\text{fp}}$, review in between.
- The review band trades reviewer-hours for recall and precision; the benign mass makes the queue grow fast.
- Measure recall through prevalence on random samples; track appeal overturn rates and identity-term bias.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
