Content moderation

Content moderation checks user and model content against a written policy: hash matching for known media, calibrated per-category classifiers, and two thresholds that remove, queue for human review or allow, set by error costs.

8 min read930 wordscore0 viewsupdated 2026-10-21

Definition

Content moderation is the process of deciding whether a piece of content (a user post, an uploaded image, a model's response) violates a written policy, and acting on it: remove, restrict, label, send to a human or allow. In generative AI it runs twice: on what users send to the model and on what the model produces. A widely cited design is OpenAI's "A Holistic Approach to Undesired Content Detection in the Real World" (Markov et al., AAAI 2023): a category taxonomy, careful labelling instructions, active learning on production traffic, and a classifier that returns one score per category.

Moderation is a classification problem with three features that ordinary classification lacks. Violations are rare, so precision depends heavily on the base rate. The categories are defined by policy text, not by nature, and the policy changes. And errors cost very different amounts in each direction and each category. A missed self-harm message is not the same as a wrongly removed joke.

This article is about classifying content against a policy at scale. Wiring checks around one application's LLM call (input, output and tool rails, fallbacks, latency) is covered in Guardrails. Measuring harms in model outputs by deliberate attack is Red teaming. Group disparities in what gets flagged are covered in Bias and fairness in generative models.

The pipeline

moderation at scale: cheap exact checks first, then scores, then peopleuser poststext, imagesmodel outputbefore displayhash matchperceptualhash vs listof knownbad mediahit → removemulti-label classifierharassment0.62violence0.11hate0.08sexual0.02self-harm0.01auto-removep ≥ 0.90 in any categoryhuman review0.20 ≤ p < 0.90allowall p < 0.20this item: harassment 0.62reviewersapply written policyremove / allow / escalateappealssecond reviewer;overturn rate is a metricnew labels →hash list,retraining,recalibrationone written policy taxonomy (definitions + examples per category) drives the classifier'straining labels, the thresholds, the reviewers' decisions and the appeals
fig.Cheap, exact checks run first: a perceptual hash of an image is matched against a list of known violating media. A multi-label classifier then scores the item per category; this one has harassment 0.62, above the review threshold 0.20 and below the removal threshold 0.90, so it goes to a human. Reviewer decisions and appeal outcomes become new labels for the hash list, retraining and recalibration.

Stages, in order of cost

  1. 1Policy taxonomy. Write each category with a definition, inclusions, exclusions and borderline examples. Labellers, the classifier, the thresholds and appeals all follow this one document.
  2. 2Exact matching. Compare perceptual hashes of images and video against curated lists of known violating media; PhotoDNA (Microsoft and Hany Farid, 2009) is the standard for known child sexual abuse material. Add blocklists for known spam URLs. This is cheap and precise, but it only catches what has been seen before.
  3. 3Classifiers. Small, fast models score every item per category (hate, harassment, sexual, self-harm, violence and so on). An LLM given the policy text (for example Llama Guard, or a general model prompted as a judge) handles new policies and long context at a higher cost per item.
  4. 4Decision. Per-category thresholds route each item to remove, human review or allow. Some categories get softer actions instead: a warning label, reduced distribution, an age gate, or support resources for self-harm.
  5. 5Human review and appeals. Reviewers decide the uncertain band and escalations. Appeals go to a second reviewer. Their labels feed back into training data and threshold calibration.

From a score to a decision

A classifier's score is only usable for thresholds if it is calibrated: among items scored 0.8, about 80% should truly violate. Calibration depends on prevalence. If a raw score ss has class-conditional densities f1f_1 (violating) and f0f_0 (benign), Bayes' rule gives

p(s)=P(violates∣s)=πf1(s)πf1(s)+(1−π)f0(s)p(s)=P(\text{violates}\mid s)=\frac{\pi f_1(s)}{\pi f_1(s)+(1-\pi)f_0(s)}
π\pi
prevalence: share of traffic in this category that violates
f1, f0f_1,\,f_0
score distributions of violating and benign items

The same raw score means a lower probability when the category is rarer. A model calibrated on a balanced labelled set will overstate risk on production traffic. Recalibrate (Platt scaling or isotonic regression) on a random sample of real traffic, separately per category and per surface.

Given a calibrated pp, a decision has expected cost p Cfnp\,C_{\text{fn}} if you allow, (1−p) Cfp(1-p)\,C_{\text{fp}} if you remove, and CrC_r if you pay a human. Chow's reject rule (Chow, 1970) picks the cheapest of the three:

allow if p<CrCfn,remove if p>1−CrCfp,review otherwise\text{allow if } p<\frac{C_r}{C_{\text{fn}}},\qquad \text{remove if } p>1-\frac{C_r}{C_{\text{fp}}},\qquad \text{review otherwise}
CfnC_{\text{fn}}
cost of leaving a violation up (harm, regulatory exposure)
CfpC_{\text{fp}}
cost of removing a legitimate item (user harm, appeals load, chilling effect)
CrC_r
cost of one human review; a review band exists only if Cr<CfnCfp/(Cfn+Cfp)C_r<C_{\text{fn}}C_{\text{fp}}/(C_{\text{fn}}+C_{\text{fp}})

With no review option the single threshold is p∗=Cfp/(Cfp+Cfn)p^*=C_{\text{fp}}/(C_{\text{fp}}+C_{\text{fn}}). Categories with costly misses therefore get low thresholds, and in practice those are also the categories where human review capacity is spent.

Item pAllow cost p·C_fnRemove cost (1 − p)·C_fpReview costDecision
0.020.200.980.30allow
0.62 (the diagram's item)6.200.380.30review
0.959.500.050.30remove
thresholdsallow below 0.3/10 = 0.03remove above 1 − 0.3/1 = 0.70band 0.03–0.70no review: p* = 1/11 = 0.091
Worked example: Chow's rule with C_fn = 10, C_fp = 1, C_r = 0.3 (arbitrary units)

Two thresholds on real traffic

harassment: 2.0% of 100,000 items violate · classifier d′ = 3p = 0.20p = 0.90-20246raw classifier score z (p = 0.5 at z = 2.80)benign · violating (each normalised)calibrated p(z) at z = 2: 0.084auto-removep ≥ 0.90violating 596benign 20human review0.20 ≤ p < 0.90violating 898benign 937allowp < 0.20violating 506benign 97,043recall 72.4% · precision of removals 95.6%violations missed 551 · benign items removed 67review queue 1,834 items = 30.6 reviewer-hours at 60 items/hassumed: reviewers decide 95% of items correctly; scores calibrated to thecategory's prevalence, so the same p threshold sits at a different z per category
category
fig.100,000 items per category, raw scores Gaussian (benign N(0,1), violating N(d′,1)) and calibrated with the category's prevalence; reviewers are assumed 95% accurate at 60 items per hour. Defaults (harassment, prevalence 2%, d′ = 3, review from 0.20, remove from 0.90): 617 auto-removals (precision 96.7%), a queue of 1,834 items (30.6 reviewer-hours), overall recall 72.4%, precision 95.6% and 67 benign items removed. One threshold at 0.5 with no review: recall 58.0%, precision 82.1%, 253 benign removed. Review from 0.05: recall 85.3% but 4,548 items (75.8 hours).

The review band buys recall and precision with reviewer time. Lowering the review threshold catches more violations, but the queue grows faster than the catch, because benign items outnumber violating ones 49 to 1. Measured this way, moderation quality is a staffing question as much as a modelling one.

Checkable specifics

OpenAI moderation endpoint
POST /v1/moderations returns flagged, a boolean per category and a category_scores value per category. As of the 2024 omni-moderation models it accepts text and images and covers harassment, hate, illicit, self-harm, sexual and violence, with subcategories such as self-harm/intent and sexual/minors.
Perspective API
Jigsaw/Google. Its TOXICITY score estimates the probability that a reader would perceive the comment as toxic. Other attributes include SEVERE_TOXICITY, INSULT, PROFANITY, IDENTITY_ATTACK and THREAT.
Jigsaw Toxic Comment (Kaggle, 2018)
Wikipedia talk comments labelled toxic, severe_toxic, obscene, threat, insult and identity_hate; a common baseline dataset.
RealToxicityPrompts
Gehman et al., Findings of EMNLP 2020: about 100K web-text prompts. Models are scored by expected maximum toxicity over 25 generations, with toxicity from Perspective and "toxic" meaning a score ≥ 0.5.
Identity-term bias
Dixon et al. (AIES 2018) showed toxicity classifiers scoring neutral sentences containing identity terms (e.g. "gay") as toxic. Sap et al. (ACL 2019) found tweets in African-American English more likely to be labelled offensive.

Measuring it

Precision is easy to measure: sample the removals and check them. Recall is hard, because the misses are buried in the allowed traffic. The standard answer is prevalence: label a uniform random sample of all content (or of all views) and count violations. With n=2000n=2000 sampled items and 2% found violating, the 95% interval is 0.02±1.960.02⋅0.98/2000=2%±0.61%0.02\pm1.96\sqrt{0.02\cdot0.98/2000}=2\%\pm0.61\%. Rare categories need much larger samples, or stratified sampling over score bands with reweighting.

Track per category: prevalence, precision of automated actions, appeal rate and appeal overturn rate, review queue age, and the score drift of new traffic. A model update that changes the score distribution changes every threshold's meaning, so recalibrate before each rollout.

python
1# Response shape of a moderation API call (OpenAI /v1/moderations, abridged)
2resp = {"results": [{
3 "flagged": True,
4 "category_scores": {"harassment": 0.62, "harassment/threatening": 0.04,
5 "hate": 0.08, "self-harm": 0.01, "sexual": 0.02,
6 "violence": 0.11},
7}]}
8
9# Per-category policy: (review_from, remove_from); stricter where misses cost more
10POLICY = {"harassment": (0.20, 0.90), "harassment/threatening": (0.10, 0.80),
11 "hate": (0.20, 0.90), "self-harm": (0.05, 0.95), # self-harm: route to support
12 "sexual": (0.30, 0.95), "violence": (0.30, 0.95)}
13
14def route(scores):
15 decision, reasons = "allow", []
16 for cat, p in scores.items():
17 lo, hi = POLICY[cat]
18 if p >= hi:
19 return "remove", [f"{cat}={p:.2f}>={hi}"]
20 if p >= lo:
21 decision = "review"
22 reasons.append(f"{cat}={p:.2f}>={lo}")
23 return decision, reasons
24
25print(route(resp["results"][0]["category_scores"]))
26# ('review', ['harassment=0.62>=0.2'])
Routing a moderation API response with per-category thresholds (runs as is on the example response; a live call needs an API key and the network)

Where it's used

User-generated content platforms

Hash matching plus per-category classifiers on posts, comments and uploads, with large review teams for the uncertain band.

Chat products

Scoring prompts and responses, with softer actions (refuse, redirect to support resources) rather than takedowns.

Training-data filtering

Running toxicity and NSFW classifiers over pretraining or fine-tuning corpora; see Dataset curation for fine-tuning.

Check yourself

A classifier is calibrated on a balanced labelled set (50% violating) and deployed where 1% of items violate. An item scores 0.8. What is its true probability of violating, roughly?

Check yourself

Under Chow's rule with C_fn = 10, C_fp = 1, what happens to the review band if a human review becomes more expensive, C_r rising from 0.3 to 0.9?

Remember

  • ✓One written policy taxonomy drives labels, classifiers, thresholds, reviewers and appeals.
  • ✓Run cheap exact checks first (perceptual hashes such as PhotoDNA, blocklists), then per-category classifiers, then humans.
  • ✓Calibrated probability depends on prevalence: p=πf1/(πf1+(1−π)f0)p=\pi f_1/(\pi f_1+(1-\pi)f_0). Recalibrate on real traffic per category.
  • ✓Chow's rule: allow below Cr/CfnC_r/C_{\text{fn}}, remove above 1−Cr/Cfp1-C_r/C_{\text{fp}}, review in between.
  • ✓The review band trades reviewer-hours for recall and precision; the benign mass makes the queue grow fast.
  • ✓Measure recall through prevalence on random samples; track appeal overturn rates and identity-term bias.
← back to Generative AI roadmap