← Back to DevBytes
Developer Tutorial August 18, 2026 8 min read

How to Reduce LLM Latency with Prefix Caching

How to Reduce LLM Latency with Prefix Caching

Latency is one of the most persistent challenges when building applications on top of Large Language Models (LLMs). Whether you're powering a chatbot, a coding assistant, or a retrieval-augmented generation (RAG) pipeline, the time it takes for a model to produce its first token — known as Time To First Token (TTFT) — directly shapes the user experience. One of the most effective and underutilized techniques for cutting that latency is prefix caching.

In this tutorial, you'll learn what prefix caching is, why it matters, how it works under the hood, and how to implement it in your own applications using popular serving frameworks and APIs.

What Is Prefix Caching?

When an LLM processes a prompt, it doesn't just read the text — it computes a series of intermediate representations called the Key-Value (KV) cache. These key-value tensors are generated for every token in the input and are essential for the model to generate subsequent tokens efficiently. Normally, this computation happens from scratch every time you send a request, even if much of the prompt is identical to a previous request.

Prefix caching is an optimization that stores the KV cache for the shared beginning (the "prefix") of a prompt so that subsequent requests with the same prefix can skip recomputing those tokens. Instead of reprocessing the entire prompt, the model reuses the cached computations and only processes the new, unique portion.

This is especially powerful in scenarios where many requests share a common structure — for example, a long system prompt, a fixed set of few-shot examples, or a large retrieved context in a RAG system.

Why Prefix Caching Matters

The benefits of prefix caching fall into three main categories:

For interactive applications like coding assistants or chatbots, the latency reduction alone can be the difference between a product that feels instant and one that feels sluggish.

How Prefix Caching Works

Under the hood, prefix caching relies on the fact that transformer attention is causal — each token only attends to tokens that come before it. This means the KV cache for the first N tokens of a prompt is identical regardless of what comes after token N. If you've already computed the KV cache for a prefix, you can reuse it for any new prompt that starts with that same prefix.

The serving system typically maintains a cache keyed by a hash of the token sequence. When a new request arrives, it checks whether a prefix of the request matches a cached entry. If it does, it loads the cached KV tensors and begins computation from the first diverging token. If not, it computes everything from scratch and stores the result for future reuse.

Most implementations use a block-based approach, where the cache is stored in fixed-size chunks (for example, 16 or 32 tokens). This allows partial matches — if 1000 tokens match and the 1001st differs, you still get the benefit of the first 1000.

Using Prefix Caching with vLLM

vLLM is one of the most popular open-source inference engines, and it supports prefix caching out of the box. Here's how to enable it when serving a model:

# Start the vLLM OpenAI-compatible server with prefix caching enabled
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --port 8000

Once the server is running, you interact with it like any OpenAI-compatible API. The caching happens automatically based on the prompt content:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

# A long shared system prompt
SYSTEM_PROMPT = (
    "You are an expert software engineer. You have access to the following "
    "codebase context:\n\n" + ("..." * 4000)  # imagine a large context here
)

# First request — this populates the cache
response1 = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "Explain what the auth module does."},
    ],
)

# Second request — the system prompt prefix is reused from cache
response2 = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "How does the database connection pool work?"},
    ],
)

print(response2.choices[0].message.content)

In this example, the second request will have a significantly lower TTFT because the KV cache for the long system prompt is already computed and stored. You can verify this by checking the server logs, which will report cache hit statistics.

Using Prefix Caching with the Anthropic API

Managed API providers like Anthropic have built prefix caching directly into their API. With Anthropic, you explicitly mark which parts of the prompt should be cached using the cache_control parameter. Here's an example:

import anthropic

client = anthropic.Anthropic()

LARGE_CONTEXT = "..." * 8000  # A large document or knowledge base

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a helpful assistant. Answer questions based on the provided context.",
        },
        {
            "type": "text",
            "text": f"<context>\n{LARGE_CONTEXT}\n</context>",
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[
        {"role": "user", "content": "What is the return policy described in the context?"}
    ],
)

print(response.content[0].text)

# Check usage to confirm cache was written
print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")

The first call writes the cached prefix (you'll see cache_creation_input_tokens populated). Subsequent calls within the cache's time-to-live (typically 5 minutes) will read from the cache, reflected in cache_read_input_tokens, and will be billed at a significantly lower rate.

Using Prefix Caching with OpenAI

OpenAI provides automatic prefix caching for certain models, including the GPT-4o family. The caching is fully automatic — you don't need to change your API calls. However, you should structure your prompts so that the static portion comes first:

from openai import OpenAI

client = OpenAI()

# Static prefix first, variable part last
SYSTEM = "You are a legal document analyzer. " * 200  # long static prefix

questions = [
    "Summarize clause 3.2",
    "What are the termination conditions?",
    "Identify any liability limitations",
]

for q in questions:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": q},
        ],
        max_tokens=500,
    )
    # The system prompt prefix is automatically cached after the first call
    print(response.choices[0].message.content)
    print(f"Prompt tokens cached: {response.usage.prompt_tokens_details.cached_tokens}")

The cached_tokens field in the usage response tells you how many tokens were served from cache. If you see zero on the first call and a large number on subsequent calls, caching is working as expected.

Best Practices for Prefix Caching

To get the most out of prefix caching, keep these principles in mind:

Common Pitfalls to Avoid

Even with a good understanding of prefix caching, there are several mistakes that can silently undermine your cache hit rate:

Measuring the Impact

To quantify the benefit of prefix caching, you should measure TTFT and total latency before and after enabling it. Here's a simple benchmarking script you can adapt:

import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

LONG_PREFIX = "You are an expert assistant. " * 500  # ~2000 tokens

def measure_latency(user_msg):
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": LONG_PREFIX},
            {"role": "user", "content": user_msg},
        ],
        max_tokens=50,
        stream=True,
        stream_options={"include_usage": True},
    )
    first_token_time = None
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter() - start
            print(chunk.choices[0].delta.content, end="")
    total_time = time.perf_counter() - start
    print(f"\nTTFT: {first_token_time:.3f}s, Total: {total_time:.3f}s")

# First call — cache miss
measure_latency("What is 2 + 2?")

# Second call — cache hit
measure_latency("What is the capital of France?")

Run this script and compare the TTFT between the two calls. You should see a dramatic improvement on the second call, demonstrating the real-world impact of prefix caching.

Conclusion

Prefix caching is one of the highest-leverage optimizations available for reducing LLM latency. By reusing the KV cache computed for shared prompt prefixes, you can dramatically cut time-to-first-token, increase throughput, and lower your API costs — all without changing your model or your application logic. The key to success is structuring your prompts so that static content comes first and variable content comes last, then monitoring your cache hit rates to ensure the optimization is actually taking effect. Whether you're using an open-source serving engine like vLLM or a managed API from OpenAI or Anthropic, prefix caching is a technique worth integrating into every production LLM application where latency and cost matter.

— Ad —

Google AdSense will appear here after approval

← Back to all articles