Definition
Content moderation is the process of deciding whether a piece of content (a user post, an uploaded image, a model's response) violates a written policy, and acting on it: remove, restrict, label, send to a human or allow. In generative AI it runs twice: on what users send to the model and on what the model produces. A widely cited design is OpenAI's "A Holistic Approach to Undesired Content Detection in the Real World" (Markov et al., AAAI 2023): a category taxonomy, careful labelling instructions, active learning on production traffic, and a classifier that returns one score per category.
Moderation is a classification problem with three features that ordinary classification lacks. Violations are rare, so precision depends heavily on the base rate. The categories are defined by policy text, not by nature, and the policy changes. And errors cost very different amounts in each direction and each category. A missed self-harm message is not the same as a wrongly removed joke.
This article is about classifying content against a policy at scale. Wiring checks around one application's LLM call (input, output and tool rails, fallbacks, latency) is covered in Guardrails. Measuring harms in model outputs by deliberate attack is Red teaming. Group disparities in what gets flagged are covered in Bias and fairness in generative models.
The pipeline
Stages, in order of cost
- 1Policy taxonomy. Write each category with a definition, inclusions, exclusions and borderline examples. Labellers, the classifier, the thresholds and appeals all follow this one document.
- 2Exact matching. Compare perceptual hashes of images and video against curated lists of known violating media; PhotoDNA (Microsoft and Hany Farid, 2009) is the standard for known child sexual abuse material. Add blocklists for known spam URLs. This is cheap and precise, but it only catches what has been seen before.
- 3Classifiers. Small, fast models score every item per category (hate, harassment, sexual, self-harm, violence and so on). An LLM given the policy text (for example Llama Guard, or a general model prompted as a judge) handles new policies and long context at a higher cost per item.
- 4Decision. Per-category thresholds route each item to remove, human review or allow. Some categories get softer actions instead: a warning label, reduced distribution, an age gate, or support resources for self-harm.
- 5Human review and appeals. Reviewers decide the uncertain band and escalations. Appeals go to a second reviewer. Their labels feed back into training data and threshold calibration.
From a score to a decision
A classifier's score is only usable for thresholds if it is calibrated: among items scored 0.8, about 80% should truly violate. Calibration depends on prevalence. If a raw score has class-conditional densities (violating) and (benign), Bayes' rule gives
- prevalence: share of traffic in this category that violates
- score distributions of violating and benign items
The same raw score means a lower probability when the category is rarer. A model calibrated on a balanced labelled set will overstate risk on production traffic. Recalibrate (Platt scaling or isotonic regression) on a random sample of real traffic, separately per category and per surface.
Given a calibrated , a decision has expected cost if you allow, if you remove, and if you pay a human. Chow's reject rule (Chow, 1970) picks the cheapest of the three:
- cost of leaving a violation up (harm, regulatory exposure)
- cost of removing a legitimate item (user harm, appeals load, chilling effect)
- cost of one human review; a review band exists only if
With no review option the single threshold is . Categories with costly misses therefore get low thresholds, and in practice those are also the categories where human review capacity is spent.
| Item p | Allow cost p·C_fn | Remove cost (1 − p)·C_fp | Review cost | Decision |
|---|---|---|---|---|
| 0.02 | 0.20 | 0.98 | 0.30 | allow |
| 0.62 (the diagram's item) | 6.20 | 0.38 | 0.30 | review |
| 0.95 | 9.50 | 0.05 | 0.30 | remove |
| thresholds | allow below 0.3/10 = 0.03 | remove above 1 − 0.3/1 = 0.70 | band 0.03–0.70 | no review: p* = 1/11 = 0.091 |
Two thresholds on real traffic
The review band buys recall and precision with reviewer time. Lowering the review threshold catches more violations, but the queue grows faster than the catch, because benign items outnumber violating ones 49 to 1. Measured this way, moderation quality is a staffing question as much as a modelling one.
Checkable specifics
- OpenAI moderation endpoint
POST /v1/moderationsreturnsflagged, a boolean per category and acategory_scoresvalue per category. As of the 2024omni-moderationmodels it accepts text and images and covers harassment, hate, illicit, self-harm, sexual and violence, with subcategories such asself-harm/intentandsexual/minors.- Perspective API
- Jigsaw/Google. Its TOXICITY score estimates the probability that a reader would perceive the comment as toxic. Other attributes include SEVERE_TOXICITY, INSULT, PROFANITY, IDENTITY_ATTACK and THREAT.
- Jigsaw Toxic Comment (Kaggle, 2018)
- Wikipedia talk comments labelled toxic, severe_toxic, obscene, threat, insult and identity_hate; a common baseline dataset.
- RealToxicityPrompts
- Gehman et al., Findings of EMNLP 2020: about 100K web-text prompts. Models are scored by expected maximum toxicity over 25 generations, with toxicity from Perspective and "toxic" meaning a score ≥ 0.5.
- Identity-term bias
- Dixon et al. (AIES 2018) showed toxicity classifiers scoring neutral sentences containing identity terms (e.g. "gay") as toxic. Sap et al. (ACL 2019) found tweets in African-American English more likely to be labelled offensive.
Measuring it
Precision is easy to measure: sample the removals and check them. Recall is hard, because the misses are buried in the allowed traffic. The standard answer is prevalence: label a uniform random sample of all content (or of all views) and count violations. With sampled items and 2% found violating, the 95% interval is . Rare categories need much larger samples, or stratified sampling over score bands with reweighting.
Track per category: prevalence, precision of automated actions, appeal rate and appeal overturn rate, review queue age, and the score drift of new traffic. A model update that changes the score distribution changes every threshold's meaning, so recalibrate before each rollout.
1# Response shape of a moderation API call (OpenAI /v1/moderations, abridged)2resp = {"results": [{3 "flagged": True,4 "category_scores": {"harassment": 0.62, "harassment/threatening": 0.04,5 "hate": 0.08, "self-harm": 0.01, "sexual": 0.02,6 "violence": 0.11},7}]}89# Per-category policy: (review_from, remove_from); stricter where misses cost more10POLICY = {"harassment": (0.20, 0.90), "harassment/threatening": (0.10, 0.80),11 "hate": (0.20, 0.90), "self-harm": (0.05, 0.95), # self-harm: route to support12 "sexual": (0.30, 0.95), "violence": (0.30, 0.95)}1314def route(scores):15 decision, reasons = "allow", []16 for cat, p in scores.items():17 lo, hi = POLICY[cat]18 if p >= hi:19 return "remove", [f"{cat}={p:.2f}>={hi}"]20 if p >= lo:21 decision = "review"22 reasons.append(f"{cat}={p:.2f}>={lo}")23 return decision, reasons2425print(route(resp["results"][0]["category_scores"]))26# ('review', ['harassment=0.62>=0.2'])Where it's used
User-generated content platforms
Hash matching plus per-category classifiers on posts, comments and uploads, with large review teams for the uncertain band.
Chat products
Scoring prompts and responses, with softer actions (refuse, redirect to support resources) rather than takedowns.
Training-data filtering
Running toxicity and NSFW classifiers over pretraining or fine-tuning corpora; see Dataset curation for fine-tuning.
Check yourself
A classifier is calibrated on a balanced labelled set (50% violating) and deployed where 1% of items violate. An item scores 0.8. What is its true probability of violating, roughly?
Check yourself
Under Chow's rule with C_fn = 10, C_fp = 1, what happens to the review band if a human review becomes more expensive, C_r rising from 0.3 to 0.9?
Remember
- ✓One written policy taxonomy drives labels, classifiers, thresholds, reviewers and appeals.
- ✓Run cheap exact checks first (perceptual hashes such as PhotoDNA, blocklists), then per-category classifiers, then humans.
- ✓Calibrated probability depends on prevalence: . Recalibrate on real traffic per category.
- ✓Chow's rule: allow below , remove above , review in between.
- ✓The review band trades reviewer-hours for recall and precision; the benign mass makes the queue grow fast.
- ✓Measure recall through prevalence on random samples; track appeal overturn rates and identity-term bias.