Introduction to Prompt Caching in vLLM
Prompt caching is one of the most impactful optimizations available to developers running large language models in production. When you repeatedly send similar prompts to an LLM — such as system instructions, few-shot examples, or long context documents — the model recomputes the same token representations every single time. This wastes GPU resources, increases latency, and inflates costs. vLLM, the high-throughput inference engine, addresses this problem through its automatic prefix caching feature, which stores and reuses computed KV (key-value) caches across requests that share common prefixes.
In this guide, you will learn what prompt caching is, why it matters for cost reduction, how to enable and configure it in vLLM, and the best practices that will help you extract maximum value from this feature in production environments.
What Is Prompt Caching?
At its core, prompt caching exploits the observation that many LLM workloads share common prompt prefixes. Consider a customer support chatbot: every request includes the same system prompt defining the assistant's persona, guidelines, and tool descriptions. Without caching, the model recomputes the attention key-value tensors for these tokens on every single request. With caching, those tensors are stored in memory and reused, skipping redundant computation entirely.
Technically, prompt caching works at the KV cache level. When the transformer processes input tokens, each layer produces key and value tensors that are stored for use during attention computation. If a new request arrives with an identical prefix, vLLM can detect the match and reuse the previously computed KV cache blocks for those tokens. Only the new, divergent tokens need to be processed from scratch.
Prefix Matching vs. Full Prompt Matching
vLLM's implementation is based on prefix matching, not full prompt matching. This means the cache is reused as long as the beginning of the prompt matches a previously cached prompt. If two requests share the first 500 tokens but diverge after that, the first 500 tokens' KV cache is reused, and only the remaining tokens are computed fresh. This granular approach maximizes cache hit rates across diverse workloads.
Why Prompt Caching Matters for Cost Reduction
The financial impact of prompt caching can be substantial. To understand why, consider how LLM inference costs scale. Most inference providers and self-hosted deployments price computation based on the number of tokens processed. When you cache a prefix, you avoid recomputing those tokens, which directly reduces the GPU time consumed per request.
- Reduced compute cost: Cached tokens skip the expensive forward pass through transformer layers, cutting GPU utilization per request.
- Lower latency: Time-to-first-token drops significantly when large prefixes are cached, improving user experience.
- Higher throughput: Freed GPU cycles can serve additional requests, increasing overall system capacity without additional hardware.
- Energy savings: Less computation means lower power consumption, which matters at scale for both cost and sustainability goals.
In practice, workloads with large shared system prompts — sometimes several thousand tokens — can see compute cost reductions of 50% to 90% for the cached portion of requests. The exact savings depend on cache hit rates, prefix lengths, and traffic patterns.
How vLLM Implements Prompt Caching
vLLM uses a paged attention mechanism inspired by operating system virtual memory. The KV cache is divided into fixed-size blocks, and each block is identified by a hash of its token content and the hash of the preceding block. This block-level hashing enables efficient prefix detection: when a new request arrives, vLLM computes block hashes and checks whether matching blocks already exist in the cache.
This design means that prefix caching in vLLM is automatic and transparent. You do not need to manually tag or identify cacheable prefixes. The engine handles block matching, eviction, and reuse internally. Your responsibility as a developer is to structure your prompts so that shared content appears at the beginning, maximizing the chance of cache hits.
Automatic Prefix Caching vs. External Caching
It is important to distinguish vLLM's built-in prefix caching from application-level caching strategies. Application-level caching might store full responses and return them for identical queries, but this only helps when requests are exactly identical. vLLM's prefix caching operates at a finer granularity, helping even when requests differ but share common prefixes. The two approaches are complementary and can be used together.
Enabling Prompt Caching in vLLM
Enabling prefix caching in vLLM is straightforward. You can activate it through the server CLI flag or through the engine constructor parameter. Below are the two primary ways to enable it.
Option 1: Using the vLLM OpenAI-Compatible Server
If you run vLLM as an OpenAI-compatible API server, pass the --enable-prefix-caching flag at startup:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--enable-prefix-caching \
--max-model-len 8192 \
--gpu-memory-utilization 0.9
Once enabled, the server automatically caches prefixes across all incoming requests. No changes to client code are required — you simply send requests as usual, and the caching happens behind the scenes.
Option 2: Using the vLLM Python API
If you embed vLLM directly in a Python application, enable prefix caching through the LLM constructor:
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
enable_prefix_caching=True,
max_model_len=8192,
gpu_memory_utilization=0.9,
)
sampling_params = SamplingParams(
temperature=0.7,
max_tokens=256,
)
system_prompt = (
"You are a helpful customer support assistant for Acme Corp. "
"Always be polite, concise, and accurate. Refer to the following "
"product documentation when answering questions: "
+ long_product_documentation
)
prompts = [
f"{system_prompt}\n\nUser: How do I reset my password?\nAssistant:",
f"{system_prompt}\n\nUser: What is your return policy?\nAssistant:",
f"{system_prompt}\n\nUser: Do you ship internationally?\nAssistant:",
]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.outputs[0].text)
In this example, the system_prompt (which includes the product documentation) is identical across all three prompts. On the first request, vLLM computes and caches the KV blocks for the shared prefix. On the second and third requests, those blocks are reused, and only the unique user query tokens are processed fresh.
Measuring Cache Hit Rates and Cost Savings
To verify that prefix caching is working and to quantify your savings, you need to measure cache hit rates. vLLM exposes metrics through its server that report cache performance. When using the OpenAI-compatible server, you can scrape Prometheus metrics from the /metrics endpoint.
curl http://localhost:8000/metrics | grep vllm
Key metrics to monitor include:
vllm:time_to_first_token_seconds— lower values indicate effective caching.vllm:num_preemption— high values may indicate memory pressure affecting cache retention.- Request-level timing — compare TTFT for cached vs. uncached requests to estimate savings.
For programmatic measurement using the Python API, you can time requests and compare them against a baseline without caching:
import time
from vllm import LLM, SamplingParams
# Baseline without caching
llm_no_cache = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
enable_prefix_caching=False,
max_model_len=8192,
)
# With caching
llm_cached = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
enable_prefix_caching=True,
max_model_len=8192,
)
long_prefix = "You are an expert assistant. " + ("context " * 2000)
prompts = [f"{long_prefix}\n\nQuestion: What is 2+2?\nAnswer:" for _ in range(10)]
params = SamplingParams(temperature=0, max_tokens=10)
# Warm up
llm_no_cache.generate(prompts[:1], params)
llm_cached.generate(prompts[:1], params)
# Measure without caching
start = time.time()
llm_no_cache.generate(prompts, params)
no_cache_time = time.time() - start
# Measure with caching
start = time.time()
llm_cached.generate(prompts, params)
cache_time = time.time() - start
print(f"Without caching: {no_cache_time:.2f}s")
print(f"With caching: {cache_time:.2f}s")
print(f"Speedup: {no_cache_time / cache_time:.2f}x")
This benchmark gives you a concrete speedup factor, which translates directly to cost savings. If caching makes inference 3x faster for your workload, your effective cost per request drops by approximately two-thirds for the cached portion.
Using the OpenAI-Compatible API with Cached Prompts
When using vLLM's OpenAI-compatible server, you can take advantage of prefix caching by structuring your chat completions requests to share common prefixes. The most effective pattern is to place static content — system messages, tool definitions, and few-shot examples — at the beginning of the message list.
import openai
client = openai.Client(base_url="http://localhost:8000/v1", api_key="dummy")
SYSTEM_PROMPT = (
"You are a legal document analyzer. Given a contract clause, "
"identify potential risks and suggest improvements. "
"Always respond in structured JSON format with 'risk_level' "
"and 'recommendation' fields."
)
# Reference document included in every request — this is the cacheable prefix
REFERENCE_TEXT = open("standard_contract_terms.txt").read()
def analyze_clause(clause_text):
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "system", "content": f"Reference terms:\n{REFERENCE_TEXT}"},
{"role": "user", "content": f"Analyze this clause:\n{clause_text}"},
],
temperature=0.3,
max_tokens=512,
)
return response.choices[0].message.content
# First call: full prefix is computed and cached
result1 = analyze_clause("The vendor shall not be liable for indirect damages.")
# Second call: prefix (SYSTEM_PROMPT + REFERENCE_TEXT) is served from cache
result2 = analyze_clause("Payment shall be due within 30 days of invoice.")
# Third call: same cached prefix, only the new clause is processed
result3 = analyze_clause("Either party may terminate with 60 days notice.")
Notice how the REFERENCE_TEXT and SYSTEM_PROMPT are identical across all calls. Only the user's clause text changes. This structure ensures maximum cache reuse.
Best Practices for Maximizing Cache Efficiency
1. Structure Prompts with Static Content First
The single most important practice is to place all static, shared content at the very beginning of your prompt. vLLM caches based on prefix matching, so any variation early in the prompt will break the cache for everything that follows. Put system instructions, documentation, and few-shot examples before any dynamic content.
# GOOD: Static prefix first, dynamic content last
prompt = f"{SYSTEM_PROMPT}\n{DOCUMENTATION}\n{FEW_SHOT_EXAMPLES}\n\nUser query: {user_input}"
# BAD: Dynamic content early breaks the cache
prompt = f"Conversation ID: {conv_id}\n{SYSTEM_PROMPT}\n{DOCUMENTATION}\n\nUser query: {user_input}"
In the bad example, the unique conv_id at the start means every request has a different prefix, so nothing gets cached.
2. Keep Dynamic Content at the End
Any content that changes between requests — user queries, timestamps, session IDs, conversation history — should appear as late as possible in the prompt. This maximizes the length of the shared prefix that can be cached.
3. Avoid Unnecessary Variation in Shared Content
Be careful about subtle sources of variation in what should be static content. For example, if your system prompt includes a timestamp or a random session token, it will never cache. Review your prompt templates to ensure truly static content is genuinely identical across requests.
4. Monitor Cache Eviction Under Memory Pressure
vLLM's KV cache has finite size. Under heavy load or with very long prompts, cached blocks may be evicted to make room for new requests. Monitor preemption metrics and consider increasing gpu_memory_utilization or using a model with a smaller per-token KV cache footprint if eviction is frequent.
5. Warm Up the Cache for Predictable Workloads
If you have known common prefixes — such as a standard system prompt or a frequently referenced document — you can warm up the cache at startup by sending a dummy request with that prefix. This ensures the first real user request benefits from caching rather than paying the full computation cost.
def warm_up_cache(llm, system_prompt, reference_docs):
"""Send a warm-up request to populate the prefix cache."""
warmup_prompt = f"{system_prompt}\n{reference_docs}\n\nWarm-up: ready."
llm.generate([warmup_prompt], SamplingParams(temperature=0, max_tokens=1))
print("Prefix cache warmed up.")
# Call this during application startup
warm_up_cache(llm, SYSTEM_PROMPT, REFERENCE_TEXT)
6. Batch Requests with Shared Prefixes
When processing multiple requests that share a prefix, submit them as a batch. vLLM will compute the shared prefix once and reuse it for all requests in the batch, which is more efficient than processing them sequentially.
7. Consider Cache When Choosing Model Context Length
Longer shared prefixes mean more tokens cached, but they also consume more KV cache memory. Balance the benefit of caching large reference documents against the memory cost. If your GPU memory is limited, you may need to reduce the reference document size or use a model with grouped-query attention, which has a smaller KV cache footprint.
Common Pitfalls and How to Avoid Them
Tokenization Mismatches
Prefix caching operates at the token level, not the character level. Two prompts that look identical in text may tokenize differently if there are subtle whitespace or encoding differences. Always construct prompts programmatically from the same string variables rather than concatenating manually, which can introduce invisible characters.
Chat Template Variations
When using chat models, vLLM applies a chat template that wraps your messages with special tokens. The template output must be identical for the prefix to match. If you mix different message structures — for example, sometimes including a system message and sometimes not — the template output changes and caching breaks. Be consistent with your message structure.
Overestimating Savings
Cache hit rates depend on traffic patterns. If your workload has highly diverse prompts with little shared content, savings will be minimal. Profile your actual traffic to understand what fraction of tokens are in shared prefixes before projecting cost savings.
Conclusion
Prompt caching with vLLM is a powerful, low-effort optimization that can dramatically reduce inference costs and latency for workloads with shared prompt prefixes. By enabling prefix caching with a single flag or constructor parameter, structuring your prompts to place static content first, and following the best practices outlined in this guide, you can achieve significant cost reductions — often cutting compute costs by half or more for the cached portions of your requests. The key to success is intentional prompt design: treat your shared prefixes as a cacheable asset, monitor your hit rates, and continuously refine your prompt structure to maximize reuse. As LLM workloads continue to grow in scale and complexity, prompt caching will remain one of the highest-leverage optimizations available to any team running inference in production.