Add guardrails with a moderation API

By Ryze AI Team

This tutorial adds input and output moderation to an LLM call using the OpenAI moderation endpoint, with a local fallback option.

1. Moderate input

from openai import OpenAI
client = OpenAI()
def check_input(text):
    r = client.moderations.create(input=text)
    return r.results[0].flagged, r.results[0].categories

If flagged, return a safe refusal message instead of calling the LLM.

2. Moderate output

After generating a response, run the same check on the output. If flagged, regenerate or return a fallback message.

def safe_generate(prompt):
    flagged, _ = check_input(prompt)
    if flagged:
        return "I can't help with that."
    out = call_llm(prompt)
    flagged, _ = check_input(out)
    if flagged:
        return "I can't provide that response."
    return out

3. Fallback message

Keep the fallback generic and non-judgmental. Log the event with the category for review, but do not echo the flagged content back to the user.

4. Local option

For offline apps, use a small toxicity classifier:

from transformers import pipeline
clf = pipeline("text-classification", model="unitary/toxic-bert")
result = clf("some text")[0]
# result: {'label': 'toxic', 'score': 0.95}

5. Tips

Moderation adds latency; batch or run in parallel with generation when possible. Tune thresholds to your tolerance for false positives. And remember that moderation is one layer: combine it with system prompt hardening and output validation.

Resources: OpenAI moderation.