Caching Mechanisms for Local LLM Inference

Caching Mechanisms for Local LLM Inference

Running large language models locally is powerful, but it comes with a steep computational cost. Every token generated requires memory bandwidth, compute cycles, and time. When you repeatedly process the same prompts or prompt prefixes, you waste resources recomputing identical results. Caching mechanisms solve this problem by storing intermediate computations so they can be reused, dramatically reducing latency and resource consumption.

What Is LLM Inference Caching?

LLM inference caching refers to the practice of storing intermediate or final results of model computation so that subsequent requests can skip redundant work. There are several layers where caching can be applied, each targeting a different bottleneck in the inference pipeline.

The most impactful form of caching for autoregressive models is the Key-Value (KV) cache. During generation, the transformer attention mechanism computes key and value tensors for every token in the context. Without caching, these tensors would be recomputed for every new token generated. The KV cache stores them so only the new token's keys and values need to be computed at each step.

Beyond KV caching, developers can implement higher-level caches such as prompt prefix caching, response caching, and disk-based model weight caches. Each operates at a different granularity and serves different use cases.

Why Caching Matters

Local LLM inference is constrained by hardware. On consumer GPUs, memory bandwidth is often the primary bottleneck rather than raw compute. Caching directly addresses this by reducing the amount of work the model must perform.

• Reduced latency: KV caching turns generation from O(n²) per token into O(n), making streaming responses feel instantaneous.
• Lower compute cost: Prefix caching avoids recomputing shared system prompts across multiple requests.
• Better throughput: When serving multiple users, cached prefixes free up compute for additional concurrent requests.
• Energy efficiency: Less computation means less heat and power consumption, which matters for always-on local deployments.
• Improved user experience: Faster first-token times make local models feel competitive with cloud APIs.

Understanding the KV Cache

The KV cache is the foundational caching mechanism in transformer-based LLMs. To understand why it works, consider how self-attention operates. For each token, the model computes query, key, and value vectors. The attention output for a token depends on its query and the keys and values of all preceding tokens.

During autoregressive generation, when producing token N+1, the keys and values for tokens 1 through N have already been computed and do not change. Storing them in a cache means the model only needs to compute the key and value for the new token, then attend over the full cached sequence.

Most inference engines handle KV caching automatically. Here is a simple example using Hugging Face Transformers, which enables KV caching by default:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.2-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

prompt = "Explain how transformers process input sequences."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# use_cache=True is the default, shown here for clarity
outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    use_cache=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

To demonstrate the performance difference, you can disable caching and compare:

import time

# With KV cache
start = time.time()
_ = model.generate(**inputs, max_new_tokens=200, use_cache=True)
print(f"With cache:    {time.time() - start:.2f}s")

# Without KV cache
start = time.time()
_ = model.generate(**inputs, max_new_tokens=200, use_cache=False)
print(f"Without cache: {time.time() - start:.2f}s")

On most hardware, the cached version will be significantly faster, with the gap widening as the generated sequence grows longer.

Prompt Prefix Caching

Many applications send repeated prompts with a shared prefix. A common pattern is a long system prompt followed by a short user query. Without prefix caching, the model recomputes the KV cache for the entire system prompt on every request. Prefix caching stores the KV tensors for shared prefixes so they can be reused.

Several inference engines support prefix caching natively. Here is an example using vLLM, which enables automatic prefix caching:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.2-1B",
    enable_prefix_caching=True
)

system_prompt = (
    "You are a helpful coding assistant. "
    "Always provide concise, accurate answers with code examples. "
    "Follow best practices and include error handling where appropriate."
)

queries = [
    "How do I read a file in Python?",
    "How do I write JSON to a file in Python?",
    "How do I list files in a directory in Python?"
]

prompts = [f"{system_prompt}\n\nUser: {q}\nAssistant:" for q in queries]

sampling_params = SamplingParams(temperature=0.7, max_tokens=200)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)
    print("---")

With prefix caching enabled, the KV cache for the system prompt is computed once and reused across all three queries. This can reduce prefill latency by a large margin when the shared prefix is long relative to the variable portion.

Implementing a Response Cache

Sometimes the entire response can be cached. If a user asks the exact same question twice, there is no need to run inference again. A response cache stores prompt-to-response mappings, typically using a hash of the input as the key.

Here is a practical implementation using Python's functools.lru_cache pattern adapted for LLM calls:

import hashlib
import json
import sqlite3
from pathlib import Path

class ResponseCache:
    def __init__(self, db_path="llm_cache.db"):
        self.db_path = db_path
        self._init_db()

    def _init_db(self):
        with sqlite3.connect(self.db_path) as conn:
            conn.execute("""
                CREATE TABLE IF NOT EXISTS cache (
                    key TEXT PRIMARY KEY,
                    response TEXT,
                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                )
            """)

    def _make_key(self, prompt, model_id, params):
        payload = json.dumps({
            "prompt": prompt,
            "model": model_id,
            "params": params
        }, sort_keys=True)
        return hashlib.sha256(payload.encode()).hexdigest()

    def get(self, prompt, model_id, params):
        key = self._make_key(prompt, model_id, params)
        with sqlite3.connect(self.db_path) as conn:
            row = conn.execute(
                "SELECT response FROM cache WHERE key = ?", (key,)
            ).fetchone()
        return row[0] if row else None

    def set(self, prompt, model_id, params, response):
        key = self._make_key(prompt, model_id, params)
        with sqlite3.connect(self.db_path) as conn:
            conn.execute(
                "INSERT OR REPLACE INTO cache (key, response) VALUES (?, ?)",
                (key, response)
            )

# Usage with a local model
cache = ResponseCache()

def generate_with_cache(prompt, model, tokenizer, params=None):
    params = params or {"temperature": 0.7, "max_new_tokens": 200}
    model_id = model.config._name_or_path

    cached = cache.get(prompt, model_id, params)
    if cached is not None:
        print("[Cache hit]")
        return cached

    print("[Cache miss]")
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=params["max_new_tokens"],
        temperature=params["temperature"],
        do_sample=True
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    cache.set(prompt, model_id, params, response)
    return response

This approach is most effective when using low or zero temperature for deterministic outputs. With high temperature, cache hits become rare because users expect varied responses.

Model Weight Caching on Disk

Loading model weights from disk into GPU memory is slow, especially for multi-gigabyte models. Inference engines like llama.cpp and Hugging Face Transformers cache downloaded weights on disk, but you can go further by keeping models resident in memory or using memory-mapped files.

With llama.cpp, you can use the --mlock flag to lock model weights in RAM, preventing the operating system from swapping them out:

# Run a server with model locked in memory
./llama-server \
  --model models/llama-3.2-1b-q4_k_m.gguf \
  --mlock \
  --ctx-size 4096 \
  --port 8080

For Python-based workflows, loading the model once and reusing it across requests is the simplest form of weight caching:

# model_server.py - keep model loaded across requests
from transformers import AutoModelForCausalLM, AutoTokenizer
from fastapi import FastAPI

app = FastAPI()

# Load once at startup - weights stay in GPU memory
model_id = "meta-llama/Llama-3.2-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

@app.post("/generate")
def generate(prompt: str, max_tokens: int = 100):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=max_tokens)
    return {"response": tokenizer.decode(outputs[0], skip_special_tokens=True)}

Best Practices

• Always enable KV caching. It is the single most impactful optimization for autoregressive generation. Disabling it should only happen for debugging or research purposes.
• Structure prompts for prefix reuse. Place stable content like system instructions at the beginning of the prompt so prefix caching can maximize reuse. Put variable content at the end.
• Monitor cache memory usage. KV caches grow with context length. For long conversations, implement cache eviction or use sliding window attention to bound memory.
• Use deterministic settings for response caching. Set temperature to 0 or a very low value if you intend to cache full responses. Stochastic outputs make cache hits meaningless.
• Include all relevant parameters in cache keys. Model version, temperature, max tokens, and any sampling parameters should be part of the key to avoid returning stale or incorrect cached responses.
• Set cache expiration policies. Disk-based response caches should have TTLs or size limits to prevent unbounded growth, especially in production environments.
• Profile before optimizing. Use tools like torch.profiler or engine-specific metrics to identify whether your bottleneck is prefill, decode, or memory transfer before adding caching layers.
• Consider quantization alongside caching. A quantized model with KV caching will use less memory for both weights and activations, allowing larger context windows on limited hardware.

Conclusion

Caching is one of the most effective ways to make local LLM inference practical. The KV cache is essential and should always be enabled, while prefix caching and response caching provide additional gains depending on your workload patterns. By structuring prompts for reuse, keeping models resident in memory, and implementing sensible cache eviction policies, you can achieve response times that rival cloud-based APIs while retaining full control over your data and infrastructure. Start with the built-in caching provided by your inference engine, measure the impact, and layer in application-level caches where your usage patterns warrant them.