By Ryze AI Team
This tutorial adds an exact-match cache in front of LLM calls using Redis, cutting cost and latency for repeated requests.
pip install redis. Run Redis locally with Docker: docker run -p 6379:6379 redis, or use a managed instance.
Hash the model name, normalized prompt, and parameters:
import hashlib, json
def cache_key(model, prompt, params):
payload = model + "\n" + prompt.strip().lower() + "\n" + json.dumps(params, sort_keys=True)
return "llm:" + hashlib.sha256(payload.encode()).hexdigest()
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
def get_or_generate(model, prompt, params, ttl=3600):
key = cache_key(model, prompt, params)
if (cached := r.get(key)):
return json.loads(cached)
out = call_llm(model, prompt, params)
r.setex(key, ttl, json.dumps(out))
return out
Include a prompt version in the key so that updating the prompt invalidates old entries automatically. Include the model version too, so a model upgrade does not serve stale answers.
Cache deterministic, non-personalized, non-time-sensitive responses. Do not cache failures, personalized content, or high-stakes answers where staleness is costly. Set TTLs based on how fresh the answer needs to be.
Track hit rate. A low hit rate means your cache key is too specific or your traffic is too diverse. A high hit rate means the cache is paying for itself.