Measure latency and cost per request

By Ryze AI Team

Instrumenting your LLM calls gives you the data to optimize. This tutorial adds timing, token counting, and cost tracking to a Python LLM call.

1. Timing

import time
start = time.perf_counter()
r = client.chat.completions.create(model="gpt-4o-mini", messages=messages, stream=True)
first_token_time = None
chunks = []
for c in r:
    if first_token_time is None and c.choices[0].delta.content:
        first_token_time = time.perf_counter() - start
    chunks.append(c.choices[0].delta.content or "")
total_time = time.perf_counter() - start
print("TTFT:", first_token_time, "total:", total_time)

2. Tokens

Read usage from the final chunk or response: r.usage.prompt_tokens and r.usage.completion_tokens. For streaming, the usage is in the last chunk. For local models, count with the tokenizer.

3. Cost

PRICES = {"gpt-4o-mini": {"input": 0.15, "output": 0.60}}  # per 1M tokens
cost = (prompt_tokens * PRICES[model]["input"] + completion_tokens * PRICES[model]["output"]) / 1e6

4. Log with context

Store each request's metrics with a feature tag, user ID, and prompt version. This lets you attribute spend and find regressions.

5. Dashboard and alerts

Aggregate by feature and user. Alert on p95 latency and daily cost spikes. Track cache hit rate alongside. Review weekly.

6. Local models

For local inference, track tokens per second, memory peak, and time-to-first-token. Optimize quantization and context size if throughput is low.