Introduction to Prompt Caching in llama.cpp
Prompt caching is one of the most impactful optimizations available to developers running local large language models. When you repeatedly send the same system prompt, documentation context, or few-shot examples to a model, the model normally reprocesses every token from scratch on each request. Prompt caching eliminates this redundant work by storing the intermediate computation — specifically the key-value (KV) cache — so that subsequent requests with a matching prefix can skip directly to the new tokens.
In llama.cpp, the popular C/C++ inference engine for GGUF models, prompt caching is built directly into the runtime. This guide explains how it works under the hood, why it matters for cost and latency, and how to integrate it into your own applications using both the command-line interface and the built-in HTTP server.
What Is Prompt Caching?
Transformer-based language models process input tokens through multiple layers, and at each layer they produce two tensors per token: a key and a value. Together these form the KV cache. During generation, every new token attends to the cached keys and values of all previous tokens, which is why autoregressive decoding is efficient once the prompt has been processed.
The expensive part is the prefill phase — the initial forward pass over the entire prompt. For a long prompt of, say, 8,000 tokens, prefill dominates the total inference time and consumes the bulk of compute resources. If your application sends the same long prefix repeatedly — a system prompt, a tool schema, a retrieved document — re-running prefill each time is wasteful.
Prompt caching stores the KV cache produced during prefill so it can be reused. When a new request arrives whose prompt begins with the same token sequence, the engine loads the cached state and only computes the KV entries for the new, divergent tokens. The result is dramatically lower latency and, in hosted or metered environments, significantly reduced cost.
Prompt Caching vs. Quantization
It is worth distinguishing prompt caching from other optimizations. Quantization reduces the precision of model weights to lower memory usage and increase throughput. Prompt caching, by contrast, does not alter the model at all — it simply avoids recomputing work that has already been done. The two techniques are fully complementary and are typically used together.
Why Prompt Caching Matters
The benefits of prompt caching fall into three categories:
- Latency reduction. Prefill is often 5–20x slower per token than decoding. Skipping prefill for a cached prefix can reduce time-to-first-token from seconds to milliseconds.
- Throughput improvement. Because the GPU or CPU is not busy reprocessing the prompt, it can serve more concurrent requests or generate more output tokens per second.
- Cost reduction. In cloud-hosted scenarios where you pay per input token, cached tokens are typically billed at a steep discount — often 10% of the standard rate. Even in fully local deployments, the cost is measured in electricity, hardware wear, and developer time, all of which benefit from reduced compute.
For agentic workflows that issue many calls with shared context — retrieval-augmented generation, multi-turn tool use, code analysis over a large repository — prompt caching can cut total inference cost by 80% or more.
How Prompt Caching Works in llama.cpp
llama.cpp exposes prompt caching through two primary mechanisms:
1. The --prompt-cache CLI Flag
When you run the llama-cli (formerly main) binary, you can pass a file path to --prompt-cache. On the first invocation, llama.cpp writes the KV cache for your prompt to that file. On subsequent invocations with the same or extended prompt, it loads the cache and only processes the new tokens.
# First run: processes the full prompt and saves the cache
llama-cli \
-m models/llama-3.1-8b-instruct.q4_k_m.gguf \
--prompt-cache ./cache/my_session.bin \
-p "You are a helpful assistant. Answer questions about the following document: [LONG DOCUMENT]" \
-n 256
# Second run: reuses the cached prefix, only new tokens are processed
llama-cli \
-m models/llama-3.1-8b-instruct.q4_k_m.gguf \
--prompt-cache ./cache/my_session.bin \
-p "You are a helpful assistant. Answer questions about the following document: [LONG DOCUMENT] What is the main thesis?" \
-n 256
The cache file is keyed to the exact token sequence. If the prefix changes by even one token, the engine detects the divergence point and recomputes from there. This makes the feature safe to use without manual invalidation logic.
2. Automatic Prefix Caching in Server Mode
The llama-server binary provides a more sophisticated caching system. When started with the --cache-reuse flag (or its default behavior in recent versions), the server maintains an in-memory pool of KV cache slots. Each incoming request is matched against existing slots by prefix, and the longest matching prefix is reused automatically.
llama-server \
-m models/llama-3.1-8b-instruct.q4_k_m.gguf \
--port 8080 \
--cache-reuse 256 \
-c 8192 \
-np 4
The --cache-reuse argument specifies the minimum number of tokens that must match before the cache is reused, which prevents tiny overlaps from triggering expensive slot-copy operations. The -np flag controls the number of parallel slots available for concurrent requests.
Using the Server API with Prompt Caching
When you run llama-server, prompt caching is transparent to the client. You simply send requests as normal, and the server handles cache matching internally. However, you can structure your prompts to maximize cache hits.
Structuring Prompts for Cache Hits
The key principle is to put stable content at the beginning of the prompt and variable content at the end. Consider this Python client:
import requests
SERVER_URL = "http://localhost:8080/v1/chat/completions"
# Stable prefix: system prompt and retrieved documents
SYSTEM_PROMPT = "You are a technical documentation assistant."
DOCUMENTS = """
[Document 1: API reference for the authentication module...]
[Document 2: Configuration options for the deployment system...]
[Document 3: Troubleshooting guide for common runtime errors...]
"""
def ask_question(question: str) -> str:
payload = {
"model": "llama-3.1-8b-instruct",
"messages": [
{"role": "system", "content": SYSTEM_PROMPT + "\n\n" + DOCUMENTS},
{"role": "user", "content": question},
],
"max_tokens": 512,
"temperature": 0.3,
}
response = requests.post(SERVER_URL, json=payload)
response.raise_for_status()
data = response.json()
# Recent llama-server versions report cache hit statistics
timings = data.get("timings", {})
prompt_n = timings.get("prompt_n", 0)
cached_n = timings.get("cached_n", 0)
print(f"Prompt tokens: {prompt_n}, Cached tokens: {cached_n}")
return data["choices"][0]["message"]["content"]
# First call: full prefill, nothing cached
print(ask_question("How do I configure authentication?"))
# Second call: system prompt + documents are cached, only the new question is processed
print(ask_question("What are common deployment errors?"))
In the response payload, the timings object includes fields like prompt_n (total prompt tokens), prompt_ms (time spent on prefill), and cached_n (tokens served from cache). Monitoring these values helps you verify that caching is working as expected.
Measuring the Impact
Here is a small benchmarking script that compares cached versus uncached performance:
import requests
import time
URL = "http://localhost:8080/v1/chat/completions"
LONG_CONTEXT = "Context: " + ("The quick brown fox jumps over the lazy dog. " * 500)
def timed_request(user_msg: str) -> dict:
payload = {
"messages": [
{"role": "system", "content": LONG_CONTEXT},
{"role": "user", "content": user_msg},
],
"max_tokens": 32,
}
start = time.perf_counter()
r = requests.post(URL, json=payload)
elapsed = time.perf_counter() - start
data = r.json()
timings = data.get("timings", {})
return {
"wall_time_s": round(elapsed, 3),
"prompt_ms": timings.get("prompt_ms", 0),
"cached_n": timings.get("cached_n", 0),
"prompt_n": timings.get("prompt_n", 0),
}
print("First request (cold):", timed_request("Summarize the context."))
print("Second request (warm):", timed_request("What animal is mentioned?"))
print("Third request (warm):", timed_request("Is there a dog?"))
Typical output on an 8B quantized model running on a mid-range GPU shows the first request taking several seconds for prefill, while subsequent requests with the same prefix complete in a fraction of the time, with cached_n matching the length of the shared context.
Programmatic Caching with the C API
For developers embedding llama.cpp directly into a C or C++ application, the caching API gives you fine-grained control. The core functions are llama_kv_cache_seq_rm, llama_kv_cache_seq_cp, and llama_kv_cache_seq_keep, which let you manipulate cache sequences by ID.
#include "llama.h"
#include <string.h>
#include <stdio.h>
int main(void) {
llama_backend_init();
llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = 99;
llama_model * model = llama_load_model_from_file("models/llama-3.1-8b-instruct.q4_k_m.gguf", model_params);
llama_context_params ctx_params = llama_context_default_params();
ctx_params.n_ctx = 8192;
ctx_params.n_batch = 512;
llama_context * ctx = llama_new_context_with_model(model, ctx_params);
const char * stable_prefix = "You are a helpful assistant. ";
llama_tokenize(model, (const uint8_t *)stable_prefix, strlen(stable_prefix),
NULL, 0, true, true);
/* In a full implementation you would:
1. Tokenize the stable prefix and evaluate it into the KV cache.
2. Save the KV cache state using llama_kv_cache_seq_cp to a reserved slot.
3. For each new request, restore from the saved slot via llama_kv_cache_seq_cp,
then evaluate only the new user tokens.
4. Clear the working sequence with llama_kv_cache_seq_rm between requests.
*/
llama_free(ctx);
llama_free_model(model);
llama_backend_free();
return 0;
}
The sequence-based API lets you maintain multiple cached prefixes simultaneously. For example, a chat application might keep one cache slot for each active conversation, restoring the appropriate slot when a user sends a new message.
Best Practices
Order Your Content Strategically
Always place the most stable content first. A good template is: system instructions, tool definitions, retrieved documents, conversation history, and finally the current user message. This maximizes the prefix that can be reused across requests.
Avoid Dynamic Prefixes
If your system prompt includes a timestamp, a random ID, or a per-request nonce, it will break cache matching. Move dynamic metadata to the end of the prompt or include it in the user message instead.
Choose an Appropriate Context Window
The KV cache consumes memory proportional to the context length. A 32K context with a 70B model can require several gigabytes of KV cache alone. Set -c to the smallest value that comfortably fits your use case, and use --cache-reuse to avoid wasting effort on trivial overlaps.
Monitor Cache Hit Rates
Use the timings field in server responses to track your cache hit ratio over time. A healthy RAG application should see 70–95% of prompt tokens served from cache on repeat queries. If the ratio is low, inspect your prompt structure for unintended variation.
Handle Cache Invalidation Gracefully
When using --prompt-cache files, delete or overwrite the cache file when the model or prompt template changes. llama.cpp validates the cache against the current model, but it is good practice to version your cache files alongside your model files.
Use Slot Management for Concurrent Users
In server mode, each parallel slot has its own KV cache. If you have many concurrent users with different contexts, increase -np to give each user a dedicated slot. This prevents cache thrashing, where one user's context evicts another's.
Common Pitfalls
- Chat templates and caching. The chat template applied by the server wraps your messages in special tokens. Two requests that look identical at the message level may differ after templating if roles or formatting vary. Keep your message structure consistent.
- Cache file corruption. If a process is killed mid-write, the cache file may be truncated. Always write to a temporary file and rename atomically if you are managing cache files yourself.
- Quantization mismatches. A cache file generated with one quantization format is not valid for another. Do not share cache files across different GGUF files.
- Ignoring memory overhead. Cached slots stay resident in VRAM or RAM. On memory-constrained systems, aggressive caching can cause OOM errors. Balance
-npand-cagainst your available memory.
Conclusion
Prompt caching is a straightforward but powerful optimization that can transform the economics of running local language models. By reusing the KV cache across requests with shared prefixes, llama.cpp lets you reduce time-to-first-token from seconds to milliseconds and cut effective compute cost by an order of magnitude or more. Whether you use the --prompt-cache flag for batch workflows, the server's automatic prefix matching for interactive applications, or the C API for embedded systems, the principles are the same: keep stable content at the front of your prompt, monitor your hit rates, and size your context window and slot count to match your workload. With these practices in place, you can build responsive, cost-effective LLM applications that scale gracefully as your user base and context lengths grow.