By Ryze AI Team
This tutorial adds input and output moderation to an LLM call using the OpenAI moderation endpoint, with a local fallback option.
from openai import OpenAI
client = OpenAI()
def check_input(text):
r = client.moderations.create(input=text)
return r.results[0].flagged, r.results[0].categories
If flagged, return a safe refusal message instead of calling the LLM.
After generating a response, run the same check on the output. If flagged, regenerate or return a fallback message.
def safe_generate(prompt):
flagged, _ = check_input(prompt)
if flagged:
return "I can't help with that."
out = call_llm(prompt)
flagged, _ = check_input(out)
if flagged:
return "I can't provide that response."
return out
Keep the fallback generic and non-judgmental. Log the event with the category for review, but do not echo the flagged content back to the user.
For offline apps, use a small toxicity classifier:
from transformers import pipeline
clf = pipeline("text-classification", model="unitary/toxic-bert")
result = clf("some text")[0]
# result: {'label': 'toxic', 'score': 0.95}
Moderation adds latency; batch or run in parallel with generation when possible. Tune thresholds to your tolerance for false positives. And remember that moderation is one layer: combine it with system prompt hardening and output validation.
Resources: OpenAI moderation.