Moderation
Filtering inputs and outputs against a policy.
Automated classification of user input and model output for policy violations — harassment, self-harm, sexual content, violence — usually via a dedicated classifier, separate from the generation model.
Keep moderation as a distinct step: it is easier to test, tune and audit than policy embedded in a prompt, and it produces a signal you can log and appeal.
Architecturally it is usually a separate classifier rather than the generation model, for a practical reason: you want it to be cheap, fast, independent and consistently calibrated, and a prompt-based check inherits the same context distractions as everything else in the request.
Moderating input and output separately
A support assistant where customers describe distressing situations.
Input moderation here causes harm by blocking legitimate requests. Output moderation catches the narrower case you actually care about. Applying both indiscriminately produces a system that refuses people who need help, which is its own failure mode.
- docsSafety best practices ↗
Application-level controls: adversarial testing, human review for high-stakes and code output, moderation, bounded inputs, and telling users what the model cannot do.
- standardAI Risk Management Framework: Generative AI Profile (NIST AI 600-1) ↗
The official record for the cross-sector GenAI companion to the AI RMF. Use it as the governance vocabulary; the PDF itself was not machine-readable in this research pass, so do not quote definitions from here.
Guardrails
Constraints on what the system will do, enforced in code.
Content Policy
The written rules about what the system may produce.
Bias · Fairness
Systematic skew in outputs, usually inherited from data.
Observability
Being able to see what the system actually did.