By Ryze AI Team
Instrumenting your LLM calls gives you the data to optimize. This tutorial adds timing, token counting, and cost tracking to a Python LLM call.
import time
start = time.perf_counter()
r = client.chat.completions.create(model="gpt-4o-mini", messages=messages, stream=True)
first_token_time = None
chunks = []
for c in r:
if first_token_time is None and c.choices[0].delta.content:
first_token_time = time.perf_counter() - start
chunks.append(c.choices[0].delta.content or "")
total_time = time.perf_counter() - start
print("TTFT:", first_token_time, "total:", total_time)
Read usage from the final chunk or response: r.usage.prompt_tokens and r.usage.completion_tokens. For streaming, the usage is in the last chunk. For local models, count with the tokenizer.
PRICES = {"gpt-4o-mini": {"input": 0.15, "output": 0.60}} # per 1M tokens
cost = (prompt_tokens * PRICES[model]["input"] + completion_tokens * PRICES[model]["output"]) / 1e6
Store each request's metrics with a feature tag, user ID, and prompt version. This lets you attribute spend and find regressions.
Aggregate by feature and user. Alert on p95 latency and daily cost spikes. Track cache hit rate alongside. Review weekly.
For local inference, track tokens per second, memory peak, and time-to-first-token. Optimize quantization and context size if throughput is low.