Evaluate an LLM with a test harness

By Ryze AI Team

This tutorial builds a small but complete evaluation harness in Python, the kind you can run in CI on every commit.

1. Create the eval data

Create evals.jsonl with one JSON object per line:

{"input": "What is 2+2?", "expected": "4"}
{"input": "Summarize: The cat sat on the mat.", "expected": "A cat rested on a mat."}

Use real user queries where possible. Aim for 30 to 100 cases covering easy, hard, and adversarial inputs.

2. Write the runner

import json
from openai import OpenAI
client = OpenAI()
cases = [json.loads(l) for l in open("evals.jsonl")]
scores = []
for c in cases:
    r = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":c["input"]}])
    out = r.choices[0].message.content
    score = 1 if c["expected"].lower() in out.lower() else 0
    scores.append((c["input"], score, out))
accuracy = sum(s for _, s, _ in scores) / len(scores)
print(f"accuracy: {accuracy:.2%}")

3. LLM as judge

For open-ended answers, use a stronger model to score 0 to 5 against a rubric:

judge_prompt = f"Score this answer 0-5. Rubric: accurate and concise. Answer: {out}"
r = client.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":judge_prompt}])

Parse the score from the response. Calibrate the judge against human labels on a subset first.

4. Track over time

Save scores with a git commit hash and prompt version. Run in CI and fail the build if accuracy drops below a threshold. Plot trends to catch slow regressions.

5. Tips

Keep evals fast enough to run on every commit. Add failures from production to the set. Version the eval data alongside the code. And never tune on the test set; keep a separate held-out set for final validation.

Resources: DeepEval, promptfoo.