By Ryze AI Team
This tutorial builds a small but complete evaluation harness in Python, the kind you can run in CI on every commit.
Create evals.jsonl with one JSON object per line:
{"input": "What is 2+2?", "expected": "4"}
{"input": "Summarize: The cat sat on the mat.", "expected": "A cat rested on a mat."}
Use real user queries where possible. Aim for 30 to 100 cases covering easy, hard, and adversarial inputs.
import json
from openai import OpenAI
client = OpenAI()
cases = [json.loads(l) for l in open("evals.jsonl")]
scores = []
for c in cases:
r = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":c["input"]}])
out = r.choices[0].message.content
score = 1 if c["expected"].lower() in out.lower() else 0
scores.append((c["input"], score, out))
accuracy = sum(s for _, s, _ in scores) / len(scores)
print(f"accuracy: {accuracy:.2%}")
For open-ended answers, use a stronger model to score 0 to 5 against a rubric:
judge_prompt = f"Score this answer 0-5. Rubric: accurate and concise. Answer: {out}"
r = client.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":judge_prompt}])
Parse the score from the response. Calibrate the judge against human labels on a subset first.
Save scores with a git commit hash and prompt version. Run in CI and fail the build if accuracy drops below a threshold. Plot trends to catch slow regressions.
Keep evals fast enough to run on every commit. Add failures from production to the set. Version the eval data alongside the code. And never tune on the test set; keep a separate held-out set for final validation.