Introduction to Prompt Caching with Pydantic AI
As LLM-powered applications scale, token costs quickly become one of the largest line items in any engineering budget. Every API call re-sends the same system prompts, tool definitions, and context documents — and you pay for those tokens every single time. Prompt caching flips this model on its head by allowing providers to store and reuse the processed prefix of your prompt, charging you a fraction of the standard input-token price on subsequent calls.
Pydantic AI, a popular agent framework built around type-safe outputs and dependency injection, integrates cleanly with provider-side caching features. In this guide, we'll explore what prompt caching is, why it matters, how to leverage it inside Pydantic AI agents, and the best practices that will keep your bills predictable.
What Is Prompt Caching?
Prompt caching is a capability offered by model providers (such as Anthropic's Claude, OpenAI, and Google's Gemini) where the KV-cache state produced while processing the prefix of a prompt is stored server-side for a bounded time window. When a subsequent request arrives whose prefix matches the cached one, the provider skips recomputation and bills the cached tokens at a steep discount — typically 10% of the standard input price for cache reads, while cache writes cost slightly more than a normal input token.
The key constraint is prefix matching. The cache only hits when the beginning of your prompt is byte-for-byte identical to a previously cached prefix. This means the structure of your prompt matters enormously: static content must come first, dynamic content must come last.
How Providers Handle Caching
- Anthropic Claude: Explicit cache control via
cache_controlmarkers. You mark up to four breakpoints in your messages. Cache TTL is 5 minutes by default, extendable to 1 hour. - OpenAI: Automatic prefix caching for prompts longer than 1024 tokens. No explicit markers required, but prompt structure still determines hit rate.
- Google Gemini: Explicit context caching via a separate API to create cached content resources with a configurable TTL.
Why Prompt Caching Matters
Consider a customer support agent that includes a 20,000-token knowledge base, a 2,000-token system prompt, and a 1,000-token tool schema. Without caching, every single user message costs you 23,000 input tokens before the user's actual question is even processed. At Claude 3.5 Sonnet pricing ($3 per million input tokens), that's roughly $0.069 per message just for context. At 10,000 messages per day, you're spending $690 daily on context alone.
With caching, those 23,000 tokens are billed at the cached rate ($0.30 per million for Claude 3.5 Sonnet cache reads), dropping the per-message context cost to about $0.0069 — a 10x reduction. The same 10,000 daily messages now cost $69 instead of $690. That's the power of prompt caching: it transforms fixed per-call overhead into a near-negligible recurring cost.
Beyond cost, caching also reduces time-to-first-token. Skipping prefix recomputation can shave hundreds of milliseconds off latency, which compounds in agentic loops that make many sequential calls.
Setting Up Pydantic AI with Caching
Pydantic AI abstracts provider differences behind model client classes. To use prompt caching, you need a recent version of Pydantic AI and the appropriate provider package installed.
pip install pydantic-ai pydantic-ai-slim[anthropic,openai]
Let's start with a basic agent and then layer in caching. Here's the scaffold:
from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
model = AnthropicModel(
"claude-3-5-sonnet-latest",
provider=AnthropicProvider(),
)
support_agent = Agent(
model=model,
system_prompt="You are a helpful customer support assistant.",
)
This agent works, but every call reprocesses the system prompt from scratch. Let's fix that.
Using Anthropic Explicit Cache Control
Anthropic requires you to explicitly mark cache breakpoints. Pydantic AI exposes this through the system_prompt parameter and through message parts that support cache control. The cleanest approach is to split your static context into a cached system prompt and keep dynamic instructions separate.
Structuring the System Prompt for Caching
The trick is to put large, stable content first and mark it as cacheable. Pydantic AI supports passing a list of system prompt parts, some of which can carry cache control directives.
from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
from pydantic_ai.messages import ModelMessage, ModelRequest, UserPromptPart
KNOWLEDGE_BASE = """
<product_docs>
... 18,000 tokens of product documentation ...
</product_docs>
<faq>
... 2,000 tokens of frequently asked questions ...
</faq>
"""
SYSTEM_INSTRUCTIONS = """
You are a senior customer support agent for Acme Corp.
Always cite the product documentation when answering.
If you don't know the answer, say so and escalate.
"""
model = AnthropicModel(
"claude-3-5-sonnet-latest",
provider=AnthropicProvider(),
)
agent = Agent(
model=model,
system_prompt=[KNOWLEDGE_BASE, SYSTEM_INSTRUCTIONS],
)
By default, Pydantic AI's Anthropic integration applies a cache_control marker to the end of the system prompt when it exceeds the provider's minimum cacheable length. You can also configure this explicitly using the AnthropicModelSettings or by passing cache control hints in your prompt parts.
Explicit Cache Markers with Message Parts
For fine-grained control, you can construct messages with explicit cache breakpoints. This is useful when you have multi-turn conversations where you want to cache the conversation history up to a certain point.
from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
from pydantic_ai.messages import (
ModelRequest,
UserPromptPart,
SystemPromptPart,
)
model = AnthropicModel(
"claude-3-5-sonnet-latest",
provider=AnthropicProvider(),
)
agent = Agent(model=model)
async def cached_run(user_question: str, history: list[ModelRequest]):
# Build the message list with a cache breakpoint
# after the system prompt and history, before the new question
messages = list(history)
messages.append(
ModelRequest(parts=[
UserPromptPart(content=user_question),
])
)
result = await agent.run(user_question, message_history=history)
return result.data
When you pass message_history, Pydantic AI sends the prior messages to the provider. Anthropic will cache the prefix automatically if it's long enough and stable. The key is that the history must be identical across calls — don't mutate old messages.
OpenAI Automatic Prefix Caching
OpenAI's caching is automatic and requires no code changes, but it only works if your prompt prefix is stable. With Pydantic AI, this means being disciplined about prompt ordering.
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
model = OpenAIModel(
"gpt-4o",
provider=OpenAIProvider(api_key="sk-..."),
)
STATIC_CONTEXT = """
<company_policies>
... large block of policy text ...
</company_policies>
"""
agent = Agent(
model=model,
system_prompt=[STATIC_CONTEXT, "Answer questions based on company policies."],
)
# Because STATIC_CONTEXT comes first and never changes,
# OpenAI will cache it automatically after the first call.
async def ask(question: str):
result = await agent.run(question)
return result.data
OpenAI's minimum cacheable prefix is 1024 tokens. Anything shorter won't be cached. Keep your static prefix well above that threshold to ensure hits.
Building a Cached Multi-Turn Agent
Let's put it all together with a realistic multi-turn support agent that maximizes cache hits across a conversation. We'll use Anthropic's explicit caching and track token usage to verify savings.
import asyncio
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
from pydantic_ai.messages import ModelMessage
from pydantic_ai.usage import Usage
PRODUCT_DOCS = """
<product_catalog>
Acme Widget Pro: A professional-grade widget with titanium casing...
... (thousands of tokens of product details) ...
</product_catalog>
<troubleshooting_guide>
Issue: Widget won't power on.
Step 1: Verify the battery is charged...
... (thousands of tokens) ...
</troubleshooting_guide>
"""
SYSTEM_PROMPT = """
You are Acme Corp's tier-2 support agent.
Rules:
1. Always reference the product catalog or troubleshooting guide.
2. If the issue requires hardware replacement, escalate to RMA.
3. Never invent features not in the catalog.
"""
@dataclass
class SupportDependencies:
user_id: str
account_tier: str
model = AnthropicModel(
"claude-3-5-sonnet-latest",
provider=AnthropicProvider(),
)
support_agent = Agent(
model=model,
system_prompt=[PRODUCT_DOCS, SYSTEM_PROMPT],
deps_type=SupportDependencies,
result_type=str,
)
async def run_support_session():
deps = SupportDependencies(user_id="u12345", account_tier="enterprise")
history: list[ModelMessage] = []
total_input = 0
total_cached = 0
questions = [
"My Widget Pro won't turn on, what should I do?",
"I tried step 1, the battery is charged. What's next?",
"Okay, I'll request an RMA. What information do I need?",
]
for q in questions:
result = await support_agent.run(q, deps=deps, message_history=history)
history = result.all_messages()
usage: Usage = result.usage()
print(f"Q: {q}")
print(f"A: {result.data}\n")
print(f" Input tokens: {usage.request_tokens}")
print(f" Cached tokens: {getattr(usage, 'cached_tokens', 0)}")
print(f" Output tokens: {usage.response_tokens}\n")
total_input += usage.request_tokens or 0
total_cached += getattr(usage, 'cached_tokens', 0) or 0
print(f"--- Session totals ---")
print(f"Total input tokens billed: {total_input}")
print(f"Total cached tokens: {total_cached}")
if total_input > 0:
print(f"Cache hit rate: {total_cached / (total_input + total_cached) * 100:.1f}%")
asyncio.run(run_support_session())
In this example, the first call writes the cache (paying a small premium on those tokens), and every subsequent call in the conversation reads from it. By the third question, you should see the vast majority of input tokens marked as cached.
Best Practices for Maximizing Cache Hits
1. Order Your Prompt: Static First, Dynamic Last
The single most important rule is prompt ordering. Anything that changes between calls — timestamps, user-specific data, random examples — must go at the end. Anything stable — system instructions, documentation, tool schemas — goes at the beginning.
# GOOD: static context first, dynamic question last
system_prompt=[LARGE_DOCS, STATIC_RULES]
# BAD: dynamic content mixed into the cached prefix
system_prompt=[f"Current time: {datetime.now()}", LARGE_DOCS, STATIC_RULES]
2. Avoid Timestamps and Randomness in the Prefix
A common mistake is injecting datetime.now() or a random session ID into the system prompt. This invalidates the cache on every call. If you need temporal context, pass it as part of the user message instead.
3. Keep Tool Definitions Stable
Pydantic AI serializes your tool schemas into the request. If your tool definitions change between calls (for example, dynamically generated docstrings), the prefix breaks. Define tools statically at agent creation time.
from pydantic_ai import Agent, RunContext
agent = Agent(model=model, system_prompt=[LARGE_DOCS])
@agent.tool
async def search_docs(ctx: RunContext[SupportDependencies], query: str) -> str:
"""Search the product documentation for a query."""
# Tool schema is fixed at definition time — cache-safe
return await ctx.deps.search(query)
4. Use message_history Correctly
When building multi-turn agents, always pass the full prior message history via message_history. Never reconstruct or paraphrase old messages — even minor text changes break the prefix match. Pydantic AI's result.all_messages() gives you the exact message list to forward.
5. Monitor Cache Hit Rates
Pydantic AI exposes token usage through result.usage(). Log these metrics to track your actual savings. A healthy cached agent should see 70-95% cache hit rates on input tokens after the first call in a session.
result = await agent.run(question, message_history=history)
usage = result.usage()
metrics = {
"input_tokens": usage.request_tokens,
"output_tokens": usage.response_tokens,
"cached_tokens": getattr(usage, "cached_tokens", 0),
}
# Send to your observability platform (Datadog, Grafana, etc.)
6. Mind the TTL
Anthropic's default cache TTL is 5 minutes. If your agent has low traffic, the cache may expire between calls. For workloads with gaps longer than 5 minutes, consider Anthropic's 1-hour TTL extension (available via the anthropic-beta header) or accept the cache-write cost on the first call of each burst.
7. Batch Dynamic Content
If you have multiple dynamic elements (user ID, current date, session context), concatenate them into a single user message at the end rather than scattering them throughout the prompt. This keeps the cacheable prefix as long as possible.
Common Pitfalls and How to Avoid Them
Pitfall: Mutating Message History
If you modify messages returned by all_messages() — for example, trimming or reformatting them — the next call's prefix won't match the cache. Always pass messages through unmodified.
Pitfall: Provider-Specific Settings Not Propagated
Some caching features require provider-specific model settings. Make sure you're using the correct model class (AnthropicModel vs OpenAIModel) and that any beta headers are configured on the provider.
from pydantic_ai.providers.anthropic import AnthropicProvider
provider = AnthropicProvider(
api_key="sk-ant-...",
# Enable 1-hour cache TTL beta if needed
anthropic_client_kwargs={"default_headers": {"anthropic-beta": "extended-cache-ttl-2025-04-11"}},
)
Pitfall: Over-Caching Short Prompts
If your prompt is under the provider's minimum cacheable length, caching adds overhead without benefit. For short prompts, rely on automatic caching (OpenAI) or skip explicit markers (Anthropic). Focus your caching effort on prompts with substantial static context.
Measuring ROI: A Cost Comparison
Let's quantify the savings with a concrete example. Suppose you have a documentation-heavy agent with 30,000 tokens of static context, processing 50,000 queries per month, with an average of 3 turns per conversation.
# Without caching (Claude 3.5 Sonnet pricing)
turns_per_month = 50_000 * 3 # 150,000 turns
input_tokens_per_turn = 30_000
total_input = turns_per_month * input_tokens_per_turn # 4.5 billion tokens
cost_without_cache = total_input / 1_000_000 * 3.00 # $13,500/month
# With caching (90% cache hit rate after first turn)
cache_writes = 50_000 * 30_000 # first turn of each conversation
cache_reads = (turns_per_month - 50_000) * 30_000 # subsequent turns
uncached = 0 # assuming all context is cached
cost_cache_writes = cache_writes / 1_000_000 * 3.75 # $5,625
cost_cache_reads = cache_reads / 1_000_000 * 0.30 # $1,350
cost_with_cache = cost_cache_writes + cost_cache_reads # $6,975/month
savings = cost_without_cache - cost_with_cache
print(f"Monthly cost without caching: ${cost_without_cache:,.0f}")
print(f"Monthly cost with caching: ${cost_with_cache:,.0f}")
print(f"Monthly savings: ${savings:,.0f}")
print(f"Reduction: {savings/cost_without_cache*100:.0f}%")
That's roughly a 48% reduction in this scenario. The savings scale even more dramatically for agents with longer static contexts or higher cache hit rates.
Conclusion
Prompt caching is one of the highest-leverage optimizations available for production LLM applications, and Pydantic AI's provider abstractions make it straightforward to adopt. By structuring your prompts with static content first, leveraging message_history for multi-turn conversations, and monitoring cache hit rates through the usage API, you can cut input token costs by 50-90% while simultaneously reducing latency. The investment in getting your prompt architecture right pays dividends on every single API call your application makes — start with the ordering principles outlined here, measure your hit rates, and iterate until caching is working as hard for your budget as your agents are for your users.