How to Implement Model Caching in Redis for AI Apps

How to Implement Model Caching in Redis for AI Apps

AI applications are powerful, but they are also expensive and slow. Every time a user asks a question to a large language model or runs an inference on a vision model, you pay in latency, compute cost, and API fees. Many of those requests are redundant — users frequently ask the same or very similar questions. Model caching is the practice of storing the results of previous model calls so that subsequent identical or similar requests can be served instantly from a fast in-memory store instead of re-running the model. Redis, with its sub-millisecond read performance and rich data structures, is an ideal backend for this pattern.

What Is Model Caching?

Model caching is a specialized form of response caching where the "computation" being cached is an AI model inference. When a request comes in — say, a prompt sent to an LLM — the application first checks the cache for a stored response keyed by the prompt (and relevant parameters). If a hit is found, the cached response is returned immediately. If not, the model is invoked, and the result is written back to the cache for future use.

This differs from generic HTTP caching in a few important ways. First, cache keys often need to incorporate model-specific parameters like temperature, model version, and system prompts. Second, cached values can be large (full text completions, embeddings, or JSON payloads). Third, semantic similarity matters — you may want to return a cached answer for a prompt that is not identical but close enough. Redis handles all of these concerns well.

Why Model Caching Matters

• Cost reduction: Cached responses avoid paid API calls to providers like OpenAI, Anthropic, or your own GPU-hosted models.
• Latency reduction: Redis reads return in under a millisecond, compared to several seconds for a typical LLM call.
• Throughput scaling: Serving cached responses frees your model infrastructure to handle genuinely novel requests.
• Consistency: Identical prompts always return the same answer, which is useful for deterministic testing and user trust.
• Rate limit relief: Fewer calls to external APIs means you stay further away from rate limits and quotas.

Setting Up Redis

You can run Redis locally with Docker or use a managed service like Redis Cloud. For local development, the fastest path is:

docker run -d --name redis-cache -p 6379:6379 redis:latest redis-server --save "" --appendonly no

The --save "" and --appendonly no flags disable persistence, which is fine for a pure cache use case where data loss on restart is acceptable. In production, you may want persistence or replication depending on your tolerance for cache misses.

Install the Python client:

pip install redis openai

Basic Exact-Match Caching

The simplest implementation caches exact prompt strings. You hash the prompt and parameters into a stable key, check Redis, and either return the cached value or call the model and store the result.

import redis
import json
import hashlib
from openai import OpenAI

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
client = OpenAI()

def cache_key(prompt, model="gpt-4o", temperature=0):
    raw = f"{model}:{temperature}:{prompt}"
    return "llm:" + hashlib.sha256(raw.encode()).hexdigest()

def cached_completion(prompt, model="gpt-4o", temperature=0, ttl=86400):
    key = cache_key(prompt, model, temperature)
    cached = r.get(key)
    if cached:
        return json.loads(cached)["response"]

    response = client.chat.completions.create(
        model=model,
        temperature=temperature,
        messages=[{"role": "user", "content": prompt}],
    )
    text = response.choices[0].message.content
    r.setex(key, ttl, json.dumps({"response": text, "model": model}))
    return text

print(cached_completion("What is the capital of France?"))

The setex command sets a value with an expiration time in seconds. Setting a TTL is critical — model outputs can become stale, and you do not want your cache to grow unbounded. A 24-hour TTL is a reasonable default for many applications.

Caching Embeddings

Embeddings are even better candidates for caching than text completions because they are deterministic for a given model and input, and they are often recomputed many times for the same documents.

def embedding_key(text, model="text-embedding-3-small"):
    raw = f"{model}:{text}"
    return "emb:" + hashlib.sha256(raw.encode()).hexdigest()

def cached_embedding(text, model="text-embedding-3-small", ttl=604800):
    key = embedding_key(text, model)
    cached = r.get(key)
    if cached:
        return json.loads(cached)

    response = client.embeddings.create(model=model, input=text)
    vec = response.data[0].embedding
    r.setex(key, ttl, json.dumps(vec))
    return vec

Since embeddings rarely change for the same input, a longer TTL of a week or more is appropriate. For document corpora, you might even skip the TTL entirely and invalidate manually when documents are updated.

Using Redis Hashes for Structured Caching

When you want to store metadata alongside the cached response — such as token counts, timestamps, or the original prompt — a Redis hash is cleaner than a JSON blob.

def cached_completion_hash(prompt, model="gpt-4o", temperature=0, ttl=86400):
    key = cache_key(prompt, model, temperature)

    if r.exists(key):
        return r.hget(key, "response")

    response = client.chat.completions.create(
        model=model,
        temperature=temperature,
        messages=[{"role": "user", "content": prompt}],
    )
    text = response.choices[0].message.content
    r.hset(key, mapping={
        "response": text,
        "model": model,
        "prompt": prompt,
        "tokens": response.usage.total_tokens,
    })
    r.expire(key, ttl)
    return text

This approach makes it easy to inspect the cache during debugging with redis-cli HGETALL <key> and to selectively read individual fields without deserializing the whole payload.

Semantic Caching with Redis Vector Search

Exact-match caching misses opportunities when users phrase the same question differently. "What is the capital of France?" and "What's France's capital?" should return the same cached answer. Redis Stack includes vector search capabilities that enable semantic caching: you store the embedding of each prompt alongside its response, and on a new request you search for the nearest cached prompt. If it is within a similarity threshold, you return that cached response.

import numpy as np

# Create a vector index once
def ensure_index():
    try:
        r.ft("semantic_cache").create_index([
            r.ft("semantic_cache").TextField("response"),
            r.ft("semantic_cache").VectorField(
                "embedding",
                "FLAT",
                {"TYPE": "FLOAT32", "DIM": 1536, "DISTANCE_METRIC": "COSINE"},
            ),
        ])
    except Exception:
        pass  # Index already exists

def semantic_completion(prompt, model="gpt-4o", threshold=0.15):
    ensure_index()
    query_vec = np.array(cached_embedding(prompt), dtype=np.float32).tobytes()

    results = r.ft("semantic_cache").search(
        f"*=>[KNN 1 @embedding $vec AS score]",
        query_params={"vec": query_vec},
    )

    if results.docs:
        score = float(results.docs[0].score)
        if score < threshold:
            return results.docs[0].response

    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    text = response.choices[0].message.content

    r.hset(f"sem:{hashlib.sha256(prompt.encode()).hexdigest()}",
           mapping={"response": text, "embedding": query_vec})
    return text

The threshold controls how aggressive the cache is. A lower threshold requires closer matches and reduces false positives but increases cache misses. A value around 0.1 to 0.2 cosine distance works well for short prompts, but you should tune it based on your specific use case and evaluate the quality of returned answers.

Cache Invalidation Strategies

Caching is easy; invalidation is famously hard. For AI applications, consider these strategies:

• TTL-based expiration: The simplest and most common approach. Set a TTL that matches how stale your data can tolerate.
• Model version keys: Include the model version in the cache key so upgrading the model automatically invalidates old entries.
• Manual invalidation: When underlying data changes (for example, a document in your RAG pipeline is updated), delete the affected cache entries explicitly.
• Namespace flushing: Use key prefixes to group related caches, and flush an entire namespace with SCAN and DEL when needed.

def flush_namespace(prefix):
    cursor = 0
    while True:
        cursor, keys = r.scan(cursor, match=f"{prefix}*", count=500)
        if keys:
            r.delete(*keys)
        if cursor == 0:
            break

flush_namespace("llm:")

Best Practices

• Always set a TTL. Unbounded caches lead to memory exhaustion and stale data.
• Normalize prompts before keying. Strip whitespace, lowercase if case-insensitive, and remove irrelevant metadata to increase hit rates.
• Include all relevant parameters in the key. Temperature, system prompt, max tokens, and model version all affect output and should be part of the key.
• Monitor hit rates. Track cache hits and misses to understand effectiveness. A low hit rate suggests your keys are too specific or your traffic is too diverse.
• Use connection pooling. Create a single Redis client and reuse it across requests rather than opening a new connection per call.
• Handle cache failures gracefully. If Redis is unavailable, fall back to calling the model directly rather than failing the request.
• Be cautious with semantic caching for factual queries. Similar prompts do not always warrant identical answers, especially when small wording changes alter the meaning.
• Size your Redis instance appropriately. LLM responses can be several kilobytes each. Estimate memory usage as average_value_size * expected_entries and add headroom.

Adding Resilience with Fallbacks

Production systems should never let a cache outage take down the application. Wrap your cache access in try/except blocks and degrade gracefully.

def safe_cached_completion(prompt, model="gpt-4o", temperature=0, ttl=86400):
    key = cache_key(prompt, model, temperature)
    try:
        cached = r.get(key)
        if cached:
            return json.loads(cached)["response"]
    except redis.RedisError:
        pass  # Cache unavailable, proceed to model call

    response = client.chat.completions.create(
        model=model,
        temperature=temperature,
        messages=[{"role": "user", "content": prompt}],
    )
    text = response.choices[0].message.content

    try:
        r.setex(key, ttl, json.dumps({"response": text}))
    except redis.RedisError:
        pass  # Cache write failed, non-fatal

    return text

Measuring Cache Effectiveness

To justify and tune your caching layer, instrument it with metrics. At minimum, track hit count, miss count, and average response time for cached versus uncached requests.

from collections import Counter

metrics = Counter()

def instrumented_completion(prompt, model="gpt-4o", temperature=0, ttl=86400):
    key = cache_key(prompt, model, temperature)
    cached = r.get(key)
    if cached:
        metrics["hits"] += 1
        return json.loads(cached)["response"]

    metrics["misses"] += 1
    response = client.chat.completions.create(
        model=model,
        temperature=temperature,
        messages=[{"role": "user", "content": prompt}],
    )
    text = response.choices[0].message.content
    r.setex(key, ttl, json.dumps({"response": text}))
    return text

def hit_rate():
    total = metrics["hits"] + metrics["misses"]
    return metrics["hits"] / total if total else 0

In a real deployment, export these counters to Prometheus or your preferred monitoring system and visualize hit rate over time. A healthy cache for an AI assistant typically achieves a 30% to 70% hit rate depending on user behavior patterns.

Conclusion

Model caching in Redis is one of the highest-leverage optimizations you can apply to an AI application. It reduces latency from seconds to milliseconds, slashes API costs, and improves throughput — all with relatively little code. Start with exact-match caching for deterministic outputs like embeddings and completions, then graduate to semantic caching with Redis vector search once you understand your traffic patterns. Pair your cache with sensible TTLs, robust fallbacks, and ongoing metrics, and you will have a fast, cost-efficient AI backend that scales gracefully as your user base grows.