Prompt Caching for Cost Reduction with LlamaIndex: Complete Guide

Prompt Caching for Cost Reduction with LlamaIndex: Complete Guide

As LLM applications scale, prompt costs can spiral out of control—especially when you're repeatedly sending the same large context, system instructions, or retrieved documents. Prompt caching is a powerful optimization that lets you reuse previously processed prompt prefixes, dramatically reducing both latency and cost. In this guide, you'll learn how to leverage prompt caching within LlamaIndex to build cheaper, faster, and more efficient LLM applications.

What Is Prompt Caching?

Prompt caching is a feature offered by LLM providers (such as Anthropic and OpenAI) that stores the intermediate computation of a prompt prefix on the provider's side. When a subsequent request shares that same prefix, the provider reuses the cached computation instead of reprocessing the entire input from scratch.

Think of it as a server-side memoization layer for your prompts. Instead of the model re-reading and re-embedding a 50,000-token document on every single call, the provider caches the KV (key-value) states from the attention mechanism and serves them instantly on the next request.

Key Characteristics

• Prefix-based: Caching works on exact prefix matches. The beginning of your prompt must be identical across requests.
• Time-limited: Cached entries expire after a provider-defined TTL (typically 5 minutes for Anthropic, with recent extensions up to 1 hour).
• Cost-discounted: Cache writes cost slightly more than standard input tokens, but cache reads are heavily discounted (often 90% off for Anthropic, 50% off for OpenAI).
• Latency-reducing: Cache hits skip the prefill phase, reducing time-to-first-token significantly.

Why Prompt Caching Matters

For production LLM applications, prompt caching is not just a nice-to-have—it can be the difference between a sustainable product and one that bleeds money. Here's why it matters:

1. Dramatic Cost Reduction

Consider a RAG application that retrieves a 30,000-token document and appends a short user query. Without caching, every single user interaction reprocesses those 30,000 tokens at full price. With caching, the document is processed once (cache write), and subsequent queries hit the cache (cache read) at a fraction of the cost.

For Anthropic's Claude, cache reads are priced at roughly 0.1x the standard input token rate. If your application makes 100 queries against the same document, you could save over 80% on token costs.

2. Lower Latency

Cache hits skip the expensive prefill computation. Anthropic reports that prompt caching can reduce time-to-first-token by up to 85% for long prompts. This is critical for conversational agents and real-time applications where users expect snappy responses.

3. Enables Richer Context

Without caching, there's a strong economic pressure to keep prompts short. With caching, you can afford to include extensive system prompts, few-shot examples, tool definitions, and large retrieved contexts—unlocking better model performance without breaking the bank.

How Prompt Caching Works

The caching mechanism operates at the provider level. When you send a request, the provider checks if the prefix of your prompt has been recently processed. The workflow looks like this:

1. First request: The provider processes the full prompt, stores the KV cache, and charges a cache write premium (e.g., 1.25x for Anthropic).
2. Subsequent requests: If the prompt shares the same prefix, the provider loads the cached KV states and only processes the new suffix. You're charged the discounted cache read rate.
3. Expiration: If no request uses the cache within the TTL window, the cache is evicted and the next request must perform a fresh cache write.

The critical constraint is that caching is prefix-based. This means the cached portion must appear at the very beginning of the prompt, and any change—even a single character—breaks the cache for everything after that point.

Setting Up LlamaIndex with Prompt Caching

LlamaIndex provides built-in support for prompt caching through its LLM integration layer. Let's walk through setting up both Anthropic and OpenAI caching.

Prerequisites

First, install the required packages:

pip install llama-index llama-index-llms-anthropic llama-index-llms-openai

Set your API keys as environment variables:

import os

os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."
os.environ["OPENAI_API_KEY"] = "sk-..."

Using Anthropic Prompt Caching in LlamaIndex

Anthropic offers the most mature prompt caching implementation, with explicit cache control breakpoints. LlamaIndex's Anthropic integration supports this through the Anthropic LLM class.

Basic Setup with Caching

from llama_index.llms.anthropic import Anthropic

llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    temperature=0,
    max_tokens=512,
    # Enable prompt caching
    cache_control={"type": "ephemeral"},
)

By passing cache_control, LlamaIndex will automatically insert cache breakpoints into the system prompt and any large context blocks sent to the model.

Caching a Large System Prompt

One of the most effective caching strategies is to cache a large, static system prompt that contains instructions, persona definitions, and few-shot examples:

from llama_index.llms.anthropic import Anthropic

SYSTEM_PROMPT = """You are an expert financial analyst assistant.

You have deep knowledge of:
- Corporate financial statements (10-K, 10-Q, 8-K filings)
- Valuation methodologies (DCF, comparable company analysis, precedent transactions)
- Macroeconomic indicators and their impact on markets
- Regulatory frameworks (SEC, GAAP, IFRS)

When answering questions:
1. Always cite specific data points
2. Distinguish between facts and opinions
3. Note any assumptions you make
4. If you are uncertain, say so explicitly

Here are some example interactions:

User: What is a P/E ratio?
Assistant: The Price-to-Earnings (P/E) ratio is a valuation metric...

User: How do you calculate free cash flow?
Assistant: Free cash flow is calculated as...

[... many more few-shot examples ...]
"""

llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    temperature=0,
    max_tokens=1024,
    system_prompt=SYSTEM_PROMPT,
    cache_control={"type": "ephemeral"},
)

# First call - cache write (slightly more expensive)
response1 = llm.complete("What are the key differences between GAAP and IFRS?")

# Second call - cache hit (heavily discounted)
response2 = llm.complete("Explain the concept of deferred tax liabilities.")

Caching Retrieved Documents in a RAG Pipeline

For RAG applications, the most impactful use of prompt caching is caching the retrieved context. Since the same documents may be relevant across multiple queries, caching them can yield massive savings:

from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.anthropic import Anthropic
from llama_index.core import SimpleDirectoryReader

# Configure LlamaIndex with Anthropic and caching
llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    temperature=0,
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
)

Settings.llm = llm

# Load and index documents
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Create query engine
query_engine = index.as_query_engine(similarity_top_k=5)

# First query - processes and caches retrieved context
response1 = query_engine.query("What is the company's revenue growth strategy?")
print(response1)

# If the same documents are retrieved, this hits the cache
response2 = query_engine.query("What are the main risk factors mentioned?")
print(response2)

Manual Cache Control with Chat Messages

For fine-grained control, you can manually specify cache breakpoints at the message level:

from llama_index.core.llms import ChatMessage, MessageRole
from llama_index.llms.anthropic import Anthropic

llm = Anthropic(model="claude-3-5-sonnet-20241022", temperature=0)

large_context = """[... 20,000 tokens of document content ...]"""

messages = [
    ChatMessage(
        role=MessageRole.SYSTEM,
        content="You are a helpful assistant that answers questions based on the provided context.",
    ),
    ChatMessage(
        role=MessageRole.USER,
        content=large_context,
        additional_kwargs={"cache_control": {"type": "ephemeral"}},
    ),
    ChatMessage(
        role=MessageRole.USER,
        content="Summarize the key points from the context above.",
    ),
]

response = llm.chat(messages)
print(response)

Using OpenAI Prompt Caching in LlamaIndex

OpenAI's prompt caching is automatic and requires no explicit cache control parameters. When you send a prompt with a prefix that has been seen recently, OpenAI automatically applies the cache discount. However, there are some important differences from Anthropic's approach:

• Caching is automatic—no explicit cache breakpoints needed.
• The minimum cacheable prefix is 1,024 tokens.
• Cache reads are discounted at 50% (vs. 90% for Anthropic).
• The cache TTL is 5-10 minutes of inactivity.

OpenAI Setup

from llama_index.llms.openai import OpenAI
from llama_index.core import Settings

llm = OpenAI(
    model="gpt-4o",
    temperature=0,
    max_tokens=1024,
    # Prompt caching is automatic for OpenAI
    # Just ensure your prompts have a consistent prefix
)

Settings.llm = llm

# Large system prompt will be cached automatically after first call
SYSTEM_PROMPT = """You are an expert code reviewer with 20 years of experience.

[... extensive instructions and examples ...]
"""

response = llm.complete("Review this function for potential issues: def foo(): ...")

Verifying Cache Usage

OpenAI returns cache usage information in the response. You can inspect this to verify caching is working:

from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage, MessageRole

llm = OpenAI(model="gpt-4o", temperature=0)

large_prefix = "You are an expert assistant. " + ("Here is context. " * 500)

messages = [
    ChatMessage(role=MessageRole.SYSTEM, content=large_prefix),
    ChatMessage(role=MessageRole.USER, content="What is 2+2?"),
]

response = llm.chat(messages)

# Access raw response to check cache metrics
raw = response.raw
if hasattr(raw, "usage"):
    print(f"Prompt tokens: {raw.usage.prompt_tokens}")
    print(f"Cached tokens: {raw.usage.prompt_tokens_details.cached_tokens}")

Advanced Caching Patterns

Multi-Turn Conversations with Cached History

In multi-turn conversations, you can cache the conversation history so that each new turn only pays for the new message:

from llama_index.core.llms import ChatMessage, MessageRole
from llama_index.llms.anthropic import Anthropic

llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    temperature=0,
    max_tokens=1024,
)

# Build up conversation with cache breakpoints
conversation_history = [
    ChatMessage(
        role=MessageRole.SYSTEM,
        content="You are a helpful coding tutor. " * 200,  # Large system prompt
        additional_kwargs={"cache_control": {"type": "ephemeral"}},
    ),
]

# Turn 1
conversation_history.append(
    ChatMessage(role=MessageRole.USER, content="Explain recursion in Python.")
)
conversation_history.append(
    ChatMessage(role=MessageRole.ASSISTANT, content="Recursion is when a function calls itself...")
)

# Turn 2 - previous messages are cached
conversation_history.append(
    ChatMessage(role=MessageRole.USER, content="Show me an example with factorial.")
)

# Add cache control to the last assistant message to cache everything up to this point
conversation_history[-2] = ChatMessage(
    role=MessageRole.ASSISTANT,
    content=conversation_history[-2].content,
    additional_kwargs={"cache_control": {"type": "ephemeral"}},
)

response = llm.chat(conversation_history)
print(response)

Caching Tool Definitions

If your application uses many tools with lengthy schemas, caching the tool definitions can save significant tokens:

from llama_index.core.tools import FunctionTool
from llama_index.llms.anthropic import Anthropic

def search_database(query: str) -> str:
    """Search the product database for items matching the query.
    
    Args:
        query: Natural language search query describing the product.
    
    Returns:
        A JSON string with matching product results.
    """
    # Implementation here
    return '{"results": []}'

def calculate_shipping(weight: float, destination: str, method: str) -> float:
    """Calculate shipping cost based on package weight, destination, and method.
    
    Args:
        weight: Package weight in kilograms.
        destination: Shipping destination country code.
        method: Shipping method - 'standard', 'express', or 'overnight'.
    
    Returns:
        Shipping cost in USD.
    """
    # Implementation here
    return 15.99

# ... many more tool definitions ...

tools = [search_database, calculate_shipping]  # + 20 more tools

llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    temperature=0,
    cache_control={"type": "ephemeral"},  # Caches tool definitions
)

# The tool schemas are cached and reused across calls
response = llm.chat_with_tools(
    tools,
    user_msg="How much would it cost to ship a 2kg package to Canada via express?"
)
print(response)

Caching with Custom LLM Implementation

If you're building a custom LLM wrapper, you can integrate prompt caching by passing the appropriate parameters to the provider API:

from llama_index.core.llms.custom import CustomLLM
from llama_index.core.llms import CompletionResponse, LLMMetadata
from typing import Any
import anthropic

class CachedAnthropicLLM(CustomLLM):
    model: str = "claude-3-5-sonnet-20241022"
    client: Any = None
    
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.client = anthropic.Anthropic()
    
    @property
    def metadata(self) -> LLMMetadata:
        return LLMMetadata(
            model_name=self.model,
            context_window=200000,
            num_output=4096,
        )
    
    def complete(self, prompt: str, **kwargs) -> CompletionResponse:
        response = self.client.messages.create(
            model=self.model,
            max_tokens=1024,
            system=[
                {
                    "type": "text",
                    "text": "You are a helpful assistant.",
                    "cache_control": {"type": "ephemeral"},
                }
            ],
            messages=[{"role": "user", "content": prompt}],
        )
        
        # Log cache usage for monitoring
        usage = response.usage
        print(f"Cache read tokens: {usage.cache_read_input_tokens}")
        print(f"Cache write tokens: {usage.cache_creation_input_tokens}")
        
        return CompletionResponse(text=response.content[0].text)

Monitoring and Measuring Cache Performance

To verify that prompt caching is actually saving you money, you need to monitor cache hit rates and token usage. Here's a utility wrapper that tracks caching metrics:

from llama_index.llms.anthropic import Anthropic
from llama_index.core.llms import CompletionResponse
from dataclasses import dataclass, field
from typing import List

@dataclass
class CacheMetrics:
    total_calls: int = 0
    cache_read_tokens: int = 0
    cache_write_tokens: int = 0
    input_tokens: int = 0
    output_tokens: int = 0
    
    @property
    def cache_hit_rate(self) -> float:
        if self.total_calls == 0:
            return 0.0
        cached_calls = sum(1 for _ in range(self.total_calls) if self.cache_read_tokens > 0)
        return cached_calls / self.total_calls
    
    def summary(self) -> str:
        return (
            f"Total calls: {self.total_calls}\n"
            f"Cache read tokens: {self.cache_read_tokens}\n"
            f"Cache write tokens: {self.cache_write_tokens}\n"
            f"Regular input tokens: {self.input_tokens}\n"
            f"Output tokens: {self.output_tokens}\n"
        )

metrics = CacheMetrics()

llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    temperature=0,
    cache_control={"type": "ephemeral"},
)

# Wrap calls to track metrics
def tracked_complete(prompt: str) -> str:
    response = llm.complete(prompt)
    raw = response.raw
    
    metrics.total_calls += 1
    if hasattr(raw, "usage"):
        metrics.cache_read_tokens += getattr(raw.usage, "cache_read_input_tokens", 0)
        metrics.cache_write_tokens += getattr(raw.usage, "cache_creation_input_tokens", 0)
        metrics.input_tokens += getattr(raw.usage, "input_tokens", 0)
        metrics.output_tokens += getattr(raw.usage, "output_tokens", 0)
    
    return response.text

# Run a series of queries
queries = [
    "What is the capital of France?",
    "What is the capital of Germany?",
    "What is the capital of Japan?",
]

for q in queries:
    result = tracked_complete(q)
    print(f"Q: {q}\nA: {result[:100]}...\n")

print("\n--- Cache Metrics ---")
print(metrics.summary())

Best Practices for Prompt Caching

1. Structure Prompts for Cacheability

Place static content at the beginning of your prompt and dynamic content at the end. This maximizes the cacheable prefix:

# GOOD: Static content first, dynamic content last
prompt = f"""{LARGE_STATIC_SYSTEM_PROMPT}

{RETRIEVED_DOCUMENTS}

User question: {user_question}
"""

# BAD: Dynamic content breaks the cache early
prompt = f"""User: {user_question}

{LARGE_STATIC_SYSTEM_PROMPT}

{RETRIEVED_DOCUMENTS}
"""

2. Keep the Cache Warm

Cache entries expire after a period of inactivity. If your application has bursty traffic, consider implementing a background job that sends a minimal request every few minutes to keep the cache alive:

import asyncio
from llama_index.llms.anthropic import Anthropic

llm = Anthropic(
    model="claude-3-5-sonnet-20241022",
    cache_control={"type": "ephemeral"},
)

async def keep_cache_warm(interval_seconds: int = 240):
    """Send periodic requests to keep the prompt cache alive."""
    while True:
        try:
            # Minimal request that hits the cached prefix
            llm.complete("ping")
            print(f"[{asyncio.get_event_loop().time()}] Cache warmed")
        except Exception as e:
            print(f"Cache warm-up failed: {e}")
        await asyncio.sleep(interval_seconds)

# Run as a background task in your application
# asyncio.create_task(keep_cache_warm())

3. Use the Right Cache Breakpoint Placement

Anthropic allows up to 4 cache breakpoints. Place them strategically at natural boundaries:

• After the system prompt
• After tool definitions
• After retrieved documents/context
• After conversation history (for multi-turn)

4. Batch Similar Queries

If you have multiple queries that share the same context, batch them together to maximize cache utilization:

from llama_index.llms.anthropic import Anthropic
from llama_index.core.llms import ChatMessage, MessageRole

llm = Anthropic(model="claude-3-5-sonnet-20241022", temperature=0)

shared_context = "[... large document ...]"

queries = [
    "What is the main thesis of this document?",
    "What evidence supports the main argument?",
    "What are the limitations mentioned?",
]

# All queries share the cached context
for query in queries:
    messages = [
        ChatMessage(
            role=MessageRole.SYSTEM,
            content="Answer questions based on the provided context.",
        ),
        ChatMessage(
            role=MessageRole.USER,
            content=shared_context,
            additional_kwargs={"cache_control": {"type": "ephemeral"}},
        ),
        ChatMessage(role=MessageRole.USER, content=query),
    ]
    
    response = llm.chat(messages)
    print(f"Q: {query}\nA: {response.text}\n")

5. Monitor Cache Hit Rates

Regularly check your cache hit rates. If you see low hit rates, it usually means your prompt prefixes aren't consistent enough. Common causes include:

• Timestamps or random IDs in the system prompt
• Different ordering of retrieved documents
• Varying whitespace or formatting
• User-specific content placed too early in the prompt

6. Consider the Cache Write Premium

Cache writes cost more than standard input tokens (1.25x for Anthropic). If a cache entry is only used once before expiring, you actually lose money. A good rule of thumb: caching is worthwhile when you expect at least 2-3 cache hits per cache write.

7. Version Your Prompts

When you update your system prompt, all existing cache entries become invalid. To manage this, consider versioning your prompts and rolling out changes gradually:

import hashlib

PROMPT_VERSION = "v2.3"
SYSTEM_PROMPT = f"""[Version: {PROMPT_VERSION}]
You are an expert assistant...
"""

# The version tag ensures you can track which cache entries
# correspond to which prompt version
print(f"Prompt hash: {hashlib.md5(SYSTEM_PROMPT.encode()).hexdigest()}")

Cost Calculation Example

Let's quantify the savings. Suppose you have a RAG application with the following characteristics:

# Cost calculation example
# Anthropic Claude 3.5 Sonnet pricing (as of 2024)

STANDARD_INPUT_PRICE = 3.00 / 1_000_000  # $3 per million tokens
CACHE_WRITE_PRICE = 3.75 / 1_000_000     # $3.75 per million tokens (1.25x)
CACHE_READ_PRICE = 0.30 / 1_000_000      # $0.30 per million tokens (0.1x)
OUTPUT_PRICE = 15.00 / 1_000_000         # $15 per million tokens

CONTEXT_TOKENS = 30_000
QUERY_TOKENS = 50
OUTPUT_TOKENS = 500
NUM_QUERIES = 100

# Without caching
cost_without_caching = NUM_QUERIES * (
    (CONTEXT_TOKENS + QUERY_TOKENS) * STANDARD_INPUT_PRICE +
    OUTPUT_TOKENS * OUTPUT_PRICE
)

# With caching (1 cache write + 99 cache reads)
cost_with_caching = (
    (CONTEXT_TOKENS + QUERY_TOKENS) * CACHE_WRITE_PRICE +  # First query: cache write
    99 * CONTEXT_TOKENS * CACHE_READ_PRICE +               # 99 queries: cache read
    100 * QUERY_TOKENS * STANDARD_INPUT_PRICE +            # Query tokens (not cached)
    100 * OUTPUT_TOKENS * OUTPUT_PRICE                     # Output tokens
)

print(f"Cost without caching: ${cost_without_caching:.4f}")
print(f"Cost with caching:    ${cost_with_caching:.4f}")
print(f"Savings:              ${cost_without_caching - cost_with_caching:.4f}")
print(f"Savings percentage:   {((cost_without_caching - cost_with_caching) / cost_without_caching) * 100:.1f}%")

Running this calculation shows that for 100 queries against a 30,000-token context, prompt caching can reduce your input token costs by over 85%. At scale, this translates to thousands of dollars saved per month.

Common Pitfalls and How to Avoid Them

Pitfall 1: Breaking the Cache with Dynamic Content

Even a single changing character in the cached prefix invalidates the entire cache. Watch out for:

# BAD: Timestamp breaks the cache every call
import datetime
system_prompt = f"Current time: {datetime.datetime.now()}\nYou are a helpful assistant..."

# GOOD: Move dynamic content to the end
system_prompt = "You are a helpful assistant..."
user_message = f"Current time: {datetime.datetime.now()}\n\nQuestion: ..."

Pitfall 2: Not Enough Tokens for Caching

Both providers have minimum token thresholds for caching. Anthropic requires at least 1,024 tokens for Claude 3.5 Sonnet (and 2,048 for some models), while OpenAI requires 1,024 tokens. If your cached prefix is too short, caching won't activate.

Pitfall 3: Over-Caching Small Prompts

If your prompt is only 200 tokens, the cache write premium may exceed the savings from cache reads. Only cache prefixes that are large enough to justify the overhead.

Conclusion

Prompt caching is one of the highest-impact optimizations available for production LLM applications. By structuring your prompts to maximize cacheable prefixes, strategically placing cache breakpoints, and keeping caches warm, you can reduce token costs by up to 90% while simultaneously cutting latency. LlamaIndex's integration with both Anthropic and OpenAI makes it straightforward to enable caching in your existing pipelines—whether you're building RAG systems, conversational agents, or tool-augmented applications. Start by identifying the largest static portions of your prompts, enable caching, and monitor your cache hit rates to continuously refine your approach. The savings compound quickly, making prompt caching an essential technique for any team serious about building cost-efficient LLM applications at scale.