Cache LLM responses with Redis

By Ryze AI Team

This tutorial adds an exact-match cache in front of LLM calls using Redis, cutting cost and latency for repeated requests.

1. Install and run Redis

pip install redis. Run Redis locally with Docker: docker run -p 6379:6379 redis, or use a managed instance.

2. Build a cache key

Hash the model name, normalized prompt, and parameters:

import hashlib, json
def cache_key(model, prompt, params):
    payload = model + "\n" + prompt.strip().lower() + "\n" + json.dumps(params, sort_keys=True)
    return "llm:" + hashlib.sha256(payload.encode()).hexdigest()

3. Get or generate

import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
def get_or_generate(model, prompt, params, ttl=3600):
    key = cache_key(model, prompt, params)
    if (cached := r.get(key)):
        return json.loads(cached)
    out = call_llm(model, prompt, params)
    r.setex(key, ttl, json.dumps(out))
    return out

4. Versioning

Include a prompt version in the key so that updating the prompt invalidates old entries automatically. Include the model version too, so a model upgrade does not serve stale answers.

5. What to cache and what not to

Cache deterministic, non-personalized, non-time-sensitive responses. Do not cache failures, personalized content, or high-stakes answers where staleness is costly. Set TTLs based on how fresh the answer needs to be.

6. Monitoring

Track hit rate. A low hit rate means your cache key is too specific or your traffic is too diverse. A high hit rate means the cache is paying for itself.