AI Content Moderation System Design

When models could only answer questions, content risk was at most "answering wrongly"; now that models can generate images, write copy, post on users' behalf, and connect to tools, content risk becomes violations, illegality, and harm to real users. Content moderation isn't a seatbelt bolted on after launch—it's the admission ticket that determines whether an AI product can reach production. China's Interim Measures for the Management of Generative Artificial Intelligence Services explicitly require compliance across three layers—training data, generated content, and user behavior—which turns moderation from an "optional feature" into a hard obligation.

1. Why AI applications must do content moderation

Four reasons, in priority order:

1. Compliance baseline: generated content touching politics, terrorism, pornography, or violence leads at minimum to takedowns and remediation, and at worst to revocation of filing. This is a survival issue.
2. Brand trust: once a model outputs harmful content that gets screenshotted and spread, the brand damage is irreversible. Users don't distinguish between "the model said it" and "the platform said it."
3. User experience: conversations polluted by prompt injection and communities spammed by malicious users drive normal users away quickly.
4. Abuse prevention: mass-generated spam, quota bypassing, AI-generated scam scripts, or phishing emails all need to be intercepted at the moderation layer.

The key insight: moderation isn't a single-point feature, but a cross-cutting concern that runs through the request lifecycle. It must fire multiple times—before the request enters the model, after the model outputs, before content is persisted, and before content is published externally.

2. First distinguish what's being moderated: input, output, UGC

Many teams moderate only the input, or call Moderation once and call it done. In reality, risk comes from three different objects:

• User input: directly violating requests ("teach me to make drugs"), prompt injection, jailbreaks, and gradual grooming disguised as normal conversation.
• Model output: harmful content the model generates on its own, infringement or defamation caused by hallucination, and violating answers elicited by injection.
• User-generated UGC: content users generate with AI and then publish—mass astroturfing copy, violating images, scam scripts.

The most easily overlooked fact: clean input doesn't mean clean output. In multi-turn dialogue the model can be gradually led into violating territory, and a single-turn check will miss it. Output moderation must therefore be done independently and combined with the full conversation history.

3. Multi-layer defense architecture: rules, classification models, LLM review

No single approach is fast, accurate, and cheap all at once. Industrial-grade solutions use defense in depth, with three layers each doing its own job:

Request ──► [Rules layer] ──► [Classification model] ──► [LLM review (hard cases)] ──► Business
              │            │              │
              ▼            ▼              ▼
        Reject on hit   Confidence grading   Semantic judgment
         (<1ms)       (tens of ms)      (seconds, side path)

Rules layer: keywords, regex, blacklists, variant word lists. Latency under 1ms, interpretable, zero marginal cost. The downside is coarseness—homophones, character-splitting ("氵去 土拍"), Unicode variants, and puns can all bypass it. Its role is the first gate: blocking the most obvious violations and keeping 90% of low-level attacks away from the expensive model layer.

Classification model: the OpenAI Moderation API, or a self-trained small model (fine-tuned from Chinese-RoBERTa, Qwen, etc.). Latency in the tens of milliseconds; can identify categories like hate, violence, sexual content, self-harm, and illegality with a confidence score. This is the highest-ROI workhorse layer.

LLM review: use a stronger model as a "reviewer" for complex cases the classifier can't decide—intent recognition (is "how not to get caught" asking about technique or about crime?), context-dependent judgments, and coherence review of long text. It understands semantics and can give reasons, but is slow (seconds), expensive, and has its own misses; it's only suitable as a hard-case appeal and secondary confirmation at low risk thresholds.

A skeleton for layered invocation:

async function moderate(text: string, ctx: ModerationContext) {
  // 1. Rules layer: fastest, reject directly on high-severity hit
  const ruleHit = ruleEngine.match(text);
  if (ruleHit.severity === "block") return reject(ruleHit);

  // 2. Classification model: main grading
  const cls = await classifier.predict(text);
  if (cls.maxScore > thresholds[cls.category]) {
    // 3. Hand the gray zone to LLM review instead of rejecting outright
    if (cls.maxScore < hardBlockLine) {
      return await llmReview(text, ctx);
    }
    return reject(cls);
  }
  return pass();
}

4. Balancing false positives and missed detections

This is the real hard part of a moderation system, and the watershed between "usable" and "good." Raise the threshold and you get more missed detections; lower it and you get more false positives—both hurt the product.

A few practical principles:

• Tier thresholds by category, not one global value. Use low thresholds for politics and terrorism (accept false positives), and high thresholds for abuse and mild edge cases (tolerate some).
• Gray-list mechanism: don't outright reject uncertain content—after entering the gray list, degrade handling (rate-limit, require rewriting, delay display) to give users a chance to self-correct.
• Context-aware pass-through: the same "how to relieve pain" is legitimate in a medical-consultation context but violating in a drug context; "human anatomy" is normal in art and medicine but may be violating in other contexts. In implementation, feed scene tags, user profiles, and conversation history as additional inputs to the review model, or maintain whitelisted contexts.

The most expensive one-line summary: the biggest cost of a moderation system isn't missed detections, but the user churn and brand damage caused by false positives. One wrongly banned active user is more dangerous than ten missed violating pieces of content.

5. Handling strategy: reject, rewrite, warn, flag—don't one-size-fits-all

The worst approach is to reject everything that hits a rule and ban every account. The right approach is to handle by severity gradient:

When rejecting, give the user a compliance-appropriate, explainable reason—"this request involves the manufacture of prohibited items and cannot be processed"—not a cold "community guidelines violated." The latter only angers legitimate users.

6. Appeals and human review

Wrongly blocked users must have a remedy channel; otherwise a false positive is terminal harm.

• Moderation results must be explainable: record which category was hit, the confidence score, and which rule fired, so appeals have a basis.
• Provide an appeal channel with a human-review SLA; high-risk actions (account bans, content deletion, external publication) must have a second human confirmation.
• Feeding false positives back: every human override is a high-quality training sample that continuously corrects the classifier.
• Establish moderation-quality metrics: false-positive rate, missed-detection rate (estimated via online sampling), appeal pass rate, human override rate.

Pure AI or pure human isn't an industrial-grade solution—"AI first-pass + human backstop" is. AI handles 99% of the volume; humans focus on the 1% of hard cases and all appeals.

7. Selection: OpenAI Moderation, self-built, third-party

The three aren't mutually exclusive; mature teams usually use a hybrid architecture.

OpenAI Moderation: free, fast, good in English scenarios; integration takes a single API call. The shortfalls are weak Chinese coverage (especially domestically specific sensitive terms, homophone puns, and internet slang) and that data leaving the country is a compliance issue domestically. Suitable for English products or first-pass filtering on non-critical paths.

Self-built classifier: controllable, optimizable for Chinese and industry (e.g. finance, healthcare), with no data leaving the country. The costs come from labeled data, model training and maintenance, and continuous updates against variant words. Suitable for teams operating at scale, with special compliance requirements, and willing to invest long-term. Choose Chinese-RoBERTa or the Qwen series as the base for fine-tuning; it's usually more accurate than generic Moderation in Chinese scenarios.

Third-party services (Alibaba Cloud Content Security, NetEase Yidun, Tencent Tianyu, etc.): the top choice for domestic compliance—complete category coverage, human-review backstop, usage-based billing, fast integration. The downsides are black-box (hard to customize), long-term costs rising with volume, and weak customization.

Practical advice: don't all-in moderation on a single vendor. Use a third-party or official API as the baseline backstop, self-built models to strengthen specific scenarios, and LLM review for hard cases. That way, if any one link fails, the other layers still hold.

Summary

• Moderation is a cross-cutting concern; input, output, and UGC must all be covered—single-point checks will inevitably miss things.
• Multi-layer defense, each with its own job: rules are fast, classifiers are accurate, LLM review handles hard cases—all are indispensable.
• False positives hurt the product more than missed detections—tier thresholds, use gray lists, and context-aware-release legitimate contexts.
• Gradient handling, keep appeal channels, human backstop—don't one-size-fits-all ban accounts.
• Hybrid selection: self-built for differentiation, third-party for compliance backstop, LLM for edge cases.

The goal of AI content moderation is never to train the model to be more obedient, but to use engineered multi-layer defense so that harmful content is consumed layer by layer before reaching users—while letting legitimate content pass through cleanly.