Prompt Caching for Cost Reduction with OpenAI Agents SDK: Complete Guide

Prompt Caching for Cost Reduction with OpenAI Agents SDK: Complete Guide

As AI agents grow more sophisticated, they tend to consume increasingly large context windows. System prompts, tool definitions, retrieved documents, and conversation history all pile up token by token. If you're running agents in production, you've probably noticed that the bulk of your API bill comes from repeatedly sending the same expensive prefix on every request. Prompt caching is OpenAI's answer to this problem, and the OpenAI Agents SDK makes it straightforward to leverage. In this guide, we'll explore what prompt caching is, why it matters for agentic workloads, how to implement it with the Agents SDK, and the best practices that will keep your costs under control.

What Is Prompt Caching?

Prompt caching is a feature that allows OpenAI models to store the intermediate computation (KV cache) of a prompt prefix on the server side. When a subsequent request shares that same prefix, the model skips reprocessing those tokens. The result is a significant reduction in both latency and cost. Cached input tokens are billed at a steep discount compared to uncached input tokens — typically around 50% off for most models, and even more for some.

The key constraint is that caching works on a prefix basis. This means the beginning of your prompt must be identical across requests for the cache to hit. If even a single character changes in the cached region, the cache breaks and you pay full price for everything after the divergence point.

Why It Matters for Agentic Workflows

Agents are the perfect candidates for prompt caching because they naturally produce long, stable prefixes. Consider a typical agent invocation:

• A detailed system prompt defining the agent's persona and rules (often 500–2000 tokens)
• Tool definitions in JSON schema format (easily 1000+ tokens)
• RAG context or retrieved documents (potentially thousands of tokens)
• Conversation history from previous turns

In a multi-turn agent loop, the system prompt and tool definitions never change. The conversation history only grows. Without caching, you re-send and re-pay for all of that context on every single model call. With caching, the stable prefix is billed at the discounted rate, and only the genuinely new tokens incur full input pricing. For a complex agent making 10–20 model calls per user interaction, the savings can be dramatic — often 40–60% on input costs alone.

How Prompt Caching Works in the Agents SDK

The OpenAI Agents SDK (formerly known as the Swarm framework, now the official agents Python package) is built on top of the OpenAI Python SDK. Prompt caching is enabled by default on supported models — you don't need to pass a special flag. However, the SDK does expose configuration options that influence caching behavior, and understanding how your prompt structure affects cache hit rates is essential.

Supported models include gpt-4o, gpt-4o-mini, o1, o3-mini, and other recent models. The minimum cacheable prefix is 1024 tokens for most models. Requests are cached automatically for 5 to 10 minutes of inactivity, with the cache refreshed on each hit.

Setting Up the Agents SDK

First, install the SDK and ensure you have your API key configured:

pip install openai-agents

import os
os.environ["OPENAI_API_KEY"] = "sk-your-api-key-here"

Now let's create a basic agent with a substantial system prompt and some tools. This will be our foundation for exploring caching behavior.

from agents import Agent, Runner, function_tool

@function_tool
def get_weather(city: str) -> str:
    """Get the current weather for a given city."""
    # Simulated weather lookup
    return f"The weather in {city} is sunny and 72°F."

@function_tool
def search_knowledge_base(query: str) -> str:
    """Search the internal knowledge base for relevant documents."""
    # Simulated KB search
    return f"Found 3 documents matching '{query}'. Document 1: ..."

SYSTEM_PROMPT = """You are a helpful customer support agent for Acme Corp.
Your responsibilities include:
1. Answering questions about Acme products and services
2. Looking up weather information when requested
3. Searching the internal knowledge base for technical documentation
4. Escalating complex issues to human agents when appropriate

Always be polite, concise, and accurate. If you don't know something,
say so rather than guessing. Use the available tools when they can
help you provide a better answer.

Company policies:
- Refunds are available within 30 days of purchase
- Technical support is available 24/7
- Premium customers get priority routing
- Never share internal pricing details with customers
"""

agent = Agent(
    name="SupportAgent",
    instructions=SYSTEM_PROMPT,
    model="gpt-4o",
    tools=[get_weather, search_knowledge_base],
)

Running the Agent and Observing Cache Behavior

When you run the agent, the SDK handles the model calls internally. Each call includes the system prompt, tool definitions, and conversation history as the prefix. Let's run a simple interaction and inspect the usage data:

import asyncio
from agents import Runner

async def main():
    result = await Runner.run(
        agent,
        "What's the weather like in San Francisco?"
    )
    print(result.final_output)
    
    # Access usage information from the result
    if result.context_wrapper and hasattr(result, 'last_agent'):
        print(f"New response: {result.final_output}")

asyncio.run(main())

To get detailed usage statistics including cache hit information, you can use the lower-level response object. The Agents SDK's Runner returns results that include usage data. Let's look at a more detailed approach using the raw model response:

from openai import AsyncOpenAI
import asyncio

client = AsyncOpenAI()

async def check_cache_behavior():
    system_prompt = "You are a helpful assistant. " * 200  # Long stable prefix
    
    # First call - should populate the cache
    response1 = await client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": "Say hello."}
        ]
    )
    
    usage1 = response1.usage
    print(f"Call 1 - Prompt tokens: {usage1.prompt_tokens}")
    print(f"Call 1 - Cached tokens: {getattr(usage1, 'prompt_tokens_details', None)}")
    
    # Second call - same prefix, should hit cache
    response2 = await client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": "Say goodbye."}
        ]
    )
    
    usage2 = response2.usage
    print(f"Call 2 - Prompt tokens: {usage2.prompt_tokens}")
    print(f"Call 2 - Cached tokens: {getattr(usage2, 'prompt_tokens_details', None)}")

asyncio.run(check_cache_behavior())

Structuring Your Agent for Maximum Cache Hits

The most important principle is to put stable content first and dynamic content last. The Agents SDK already does this correctly by placing the system prompt and tool definitions before the conversation history. But you can optimize further by being intentional about what goes into your system prompt versus what you pass as dynamic context.

Here's an example of a well-structured agent that maximizes cache hits across a multi-turn conversation:

from agents import Agent, Runner, function_tool
from dataclasses import dataclass

@dataclass
class UserContext:
    user_id: str
    name: str
    tier: str  # "free" or "premium"

@function_tool
def get_account_details(user_id: str) -> str:
    """Retrieve account details for a user."""
    return f"User {user_id}: Premium tier, member since 2023."

# Stable system prompt - this will be cached across all calls
BASE_SYSTEM_PROMPT = """You are Acme Corp's customer support agent.

## Your Role
You help customers with product questions, billing issues, and 
technical support. You have access to tools for looking up 
account information and searching the knowledge base.

## Guidelines
- Always greet the customer by name
- Be empathetic and professional
- Use tools to verify information before answering
- Escalate to human agents for refund requests over $500
- Never make promises about timelines without checking

## Product Catalog
- Acme Basic: $9.99/month - 10GB storage, email support
- Acme Pro: $29.99/month - 100GB storage, priority support
- Acme Enterprise: $99.99/month - unlimited storage, dedicated support

## Common Issues and Solutions
1. Login problems: Reset password via /forgot-password
2. Billing errors: Verify in account dashboard, escalate if unresolved
3. Feature requests: Log in feedback system, notify product team
4. Data export: Available in Settings > Data > Export

## Knowledge Base Topics
- API integration and webhooks
- SSO configuration
- Data migration guides
- Security best practices
- Compliance documentation (SOC2, GDPR, HIPAA)
"""

agent = Agent(
    name="SupportAgent",
    instructions=BASE_SYSTEM_PROMPT,
    model="gpt-4o",
    tools=[get_account_details],
)

Notice how all the stable reference material — product catalog, common issues, knowledge base topics — lives in the system prompt. This content will be cached. Dynamic information like the user's name and specific question comes through the conversation messages, which are appended after the cached prefix.

Handling Dynamic Context Without Breaking the Cache

A common mistake is injecting dynamic content into the system prompt, which breaks the cache on every request. Instead, pass dynamic context through the conversation or through a context object. Here's the wrong way and the right way:

# WRONG: Dynamic content in system prompt breaks cache every time
import datetime

def make_agent_with_bad_prompt(user_name, current_time):
    # This system prompt changes every call - cache will NEVER hit
    bad_prompt = f"""You are a support agent.
Current time: {current_time}
Customer name: {user_name}
Today is {datetime.date.today().strftime('%A, %B %d')}
...rest of instructions...
"""
    return Agent(name="BadAgent", instructions=bad_prompt, model="gpt-4o")


# RIGHT: Stable system prompt, dynamic info in user message
STABLE_PROMPT = """You are a support agent.
...rest of instructions (never changes)...
"""

agent = Agent(name="GoodAgent", instructions=STABLE_PROMPT, model="gpt-4o")

async def run_with_context(user_name, current_time):
    # Dynamic info goes in the user message, after the cached prefix
    user_message = f"""[Context: Customer name is {user_name}, 
current time is {current_time}]

Hello, I need help with my account."""
    
    result = await Runner.run(agent, user_message)
    return result.final_output

Using the ModelSettings for Cache Control

The Agents SDK provides a ModelSettings class that lets you configure model behavior. While caching is automatic, you can use these settings to ensure consistent behavior across your agent runs:

from agents import Agent, ModelSettings

agent = Agent(
    name="CachedAgent",
    instructions=STABLE_PROMPT,
    model="gpt-4o",
    model_settings=ModelSettings(
        temperature=0.7,
        max_tokens=1000,
        # The SDK handles caching automatically, but keeping
        # settings consistent helps with cache stability
    ),
    tools=[get_account_details],
)

Building a Multi-Turn Agent Loop with Caching

In a real application, your agent will have multi-turn conversations. The Agents SDK's Runner manages the conversation loop internally, but for custom loops, you need to maintain the conversation history yourself. Here's a complete example of a multi-turn agent that benefits from prompt caching:

from agents import Agent, Runner, function_tool
import asyncio

@function_tool
def calculate(expression: str) -> str:
    """Safely evaluate a mathematical expression."""
    try:
        allowed = set("0123456789+-*/.() ")
        if not all(c in allowed for c in expression):
            return "Invalid expression"
        result = eval(expression)  # Safe due to input validation
        return str(result)
    except Exception as e:
        return f"Error: {e}"

@function_tool
def lookup_product(product_id: str) -> str:
    """Look up product information by ID."""
    products = {
        "P001": "Widget Pro - $19.99 - In stock",
        "P002": "Gadget Max - $39.99 - 2 left",
        "P003": "Tool Kit - $59.99 - Out of stock",
    }
    return products.get(product_id, "Product not found")

SYSTEM_PROMPT = """You are a shopping assistant for Acme Store.
Help customers find products, check prices, and calculate totals.
Use the calculate tool for math and lookup_product for product info.
Be friendly and helpful. Always confirm order details before 
finalizing recommendations.

Store policies:
- Free shipping on orders over $50
- 30-day return policy on all items
- Price matching available on request
"""

agent = Agent(
    name="ShoppingAssistant",
    instructions=SYSTEM_PROMPT,
    model="gpt-4o",
    tools=[calculate, lookup_product],
)

async def multi_turn_conversation():
    conversation_history = []
    
    # Simulate a multi-turn conversation
    user_messages = [
        "Do you have any widgets in stock?",
        "How much would 3 of those cost?",
        "What about the Gadget Max? Is there a discount if I buy both?",
    ]
    
    for user_msg in user_messages:
        print(f"\nUser: {user_msg}")
        
        # Each call includes the full history, but the system prompt
        # and tool definitions (the prefix) are cached from the first call
        result = await Runner.run(agent, user_msg)
        print(f"Agent: {result.final_output}")
        
        # In a production app, you'd maintain the conversation
        # history and pass it to maintain context across turns

asyncio.run(multi_turn_conversation())

Measuring and Monitoring Cache Performance

To verify that caching is working, you should monitor the usage data returned by the API. The OpenAI API returns prompt_tokens_details which includes cached_tokens — the number of tokens that were served from cache. Here's a utility function to track this:

from openai import AsyncOpenAI
import asyncio

client = AsyncOpenAI()

async def measure_cache_efficiency():
    """Measure how many tokens are cached across multiple calls."""
    long_system_prompt = (
        "You are an expert software engineer with deep knowledge of "
        "Python, JavaScript, Go, Rust, and Java. You provide detailed, "
        "accurate code reviews and suggestions. " * 50
    )
    
    questions = [
        "How do I handle exceptions in Python?",
        "What's the difference between async/await in JS and Python?",
        "Explain Rust's ownership model.",
        "How do goroutines work in Go?",
    ]
    
    total_prompt_tokens = 0
    total_cached_tokens = 0
    
    for i, question in enumerate(questions):
        response = await client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": long_system_prompt},
                {"role": "user", "content": question}
            ]
        )
        
        usage = response.usage
        cached = 0
        if hasattr(usage, 'prompt_tokens_details') and usage.prompt_tokens_details:
            cached = usage.prompt_tokens_details.cached_tokens or 0
        
        total_prompt_tokens += usage.prompt_tokens
        total_cached_tokens += cached
        
        hit_rate = (cached / usage.prompt_tokens * 100) if usage.prompt_tokens else 0
        print(f"Call {i+1}: {usage.prompt_tokens} prompt tokens, "
              f"{cached} cached ({hit_rate:.1f}% hit rate)")
    
    overall = (total_cached_tokens / total_prompt_tokens * 100) if total_prompt_tokens else 0
    print(f"\nOverall: {total_prompt_tokens} total prompt tokens, "
          f"{total_cached_tokens} cached ({overall:.1f}% overall hit rate)")
    print(f"Estimated savings: ~{overall * 0.5:.1f}% on input costs")

asyncio.run(measure_cache_efficiency())

Best Practices for Prompt Caching with Agents

Now that we've covered the mechanics, let's consolidate the best practices that will maximize your cache hit rates and minimize your costs:

• Keep system prompts stable. Any change to the system prompt invalidates the entire cache. Avoid injecting timestamps, random IDs, or user-specific data into the system prompt.
• Order matters. Put the most stable content first (system prompt, tool definitions) and the most dynamic content last (latest user message). The Agents SDK handles this ordering for you, but be mindful if you customize the message structure.
• Minimize tool definition changes. Tool schemas are part of the cached prefix. If you dynamically add or remove tools between calls, you break the cache. Define your full tool set once and keep it consistent.
• Use context objects for dynamic data. The Agents SDK supports context objects that pass through to tool functions without being part of the prompt. Use these for user-specific data instead of embedding it in messages.
• Batch related queries. If you're making multiple independent calls that share the same system prompt, make them within the cache window (5–10 minutes) to benefit from the cached prefix.
• Monitor cache hit rates. Use the prompt_tokens_details.cached_tokens field to track how effectively your prompts are being cached. A sudden drop in cache hits often indicates an unintended change in your prompt structure.
• Consider prompt length carefully. Caching only kicks in after the minimum prefix threshold (typically 1024 tokens). If your system prompt is very short, you won't see caching benefits. However, don't artificially inflate your prompt — the discount on cached tokens still doesn't make unnecessary tokens free.
• Be aware of cache eviction. Caches are evicted after a period of inactivity. If your agent has infrequent interactions (more than 10 minutes apart), consider whether a "warm-up" call is worth it for high-volume scenarios.

Advanced Pattern: Caching with RAG Context

Retrieval-augmented generation (RAG) presents a caching challenge because retrieved documents change per query. However, you can still benefit from caching by structuring your prompt so that the stable parts come first and retrieved context is appended later. Here's how to do this with the Agents SDK:

from agents import Agent, Runner, function_tool
import asyncio

@function_tool
def search_documents(query: str) -> str:
    """Search internal documents and return relevant excerpts."""
    # In production, this would query a vector database
    return "Document excerpt: The return policy allows 30 days..."

# The system prompt and tool definitions are stable and will be cached
RAG_SYSTEM_PROMPT = """You are a documentation assistant for Acme Corp.

Your job is to answer user questions based on retrieved documents.
Always cite the source document when providing information. If the
retrieved documents don't contain enough information, say so clearly.

Response format:
1. Direct answer to the question
2. Supporting details from the documents
3. Source citation

Guidelines:
- Be precise and factual
- Quote directly when accuracy matters
- Note any limitations or caveats in the source material
- Suggest follow-up questions when appropriate
"""

rag_agent = Agent(
    name="RAGAssistant",
    instructions=RAG_SYSTEM_PROMPT,
    model="gpt-4o",
    tools=[search_documents],
)

async def rag_query(user_question: str):
    # The system prompt + tool definitions form the cached prefix.
    # The user question is the dynamic part that changes each call.
    # The tool call results are also dynamic but come after the prefix.
    result = await Runner.run(rag_agent, user_question)
    return result.final_output

# Run multiple queries - the system prompt stays cached
async def demo():
    questions = [
        "What is the return policy?",
        "How do I configure SSO?",
        "What are the security compliance certifications?",
    ]
    for q in questions:
        answer = await rag_query(q)
        print(f"Q: {q}\nA: {answer}\n")

asyncio.run(demo())

Cost Estimation Example

Let's put some numbers on the savings. Suppose you have an agent with a 2000-token system prompt and 1500 tokens of tool definitions, making 15 model calls per user session. Without caching, you pay for 3500 input tokens × 15 calls = 52,500 tokens at full price. With caching, the first call pays full price (3500 tokens) and subsequent calls pay the discounted rate on the cached prefix. Here's a rough calculation:

# Cost estimation for a 15-call agent session
# Using gpt-4o pricing as an example

system_prompt_tokens = 2000
tool_def_tokens = 1500
cached_prefix = system_prompt_tokens + tool_def_tokens  # 3500 tokens
new_tokens_per_call = 200  # average new user message + tool results
num_calls = 15

# gpt-4o input pricing (example rates)
full_input_price_per_1m = 2.50  # $2.50 per 1M tokens
cached_input_price_per_1m = 1.25  # $1.25 per 1M tokens (50% discount)

# Without caching
total_tokens_no_cache = (cached_prefix + new_tokens_per_call) * num_calls
cost_no_cache = (total_tokens_no_cache / 1_000_000) * full_input_price_per_1m

# With caching (first call full price, rest cached on prefix)
first_call_cost = (cached_prefix / 1_000_000) * full_input_price_per_1m
cached_calls_cost = (
    (cached_prefix / 1_000_000) * cached_input_price_per_1m * (num_calls - 1)
)
new_tokens_cost = (
    (new_tokens_per_call * num_calls / 1_000_000) * full_input_price_per_1m
)
cost_with_cache = first_call_cost + cached_calls_cost + new_tokens_cost

print(f"Without caching: ${cost_no_cache:.4f}")
print(f"With caching:    ${cost_with_cache:.4f}")
print(f"Savings:         ${cost_no_cache - cost_with_cache:.4f} "
      f"({(1 - cost_with_cache/cost_no_cache)*100:.1f}%)")
print(f"Per session savings scale linearly across thousands of sessions")

At scale, these savings compound. An application handling 100,000 sessions per month could save hundreds or thousands of dollars depending on prompt size and call frequency.

Conclusion

Prompt caching is one of the highest-leverage optimizations available for production AI agents. The OpenAI Agents SDK handles the underlying mechanics automatically, but the real savings come from how you structure your prompts and manage dynamic context. By keeping system prompts stable, ordering content from most stable to most dynamic, using context objects for user-specific data, and monitoring cache hit rates, you can cut input token costs by 40–60% or more. As agent workflows grow more complex with longer system prompts, richer tool definitions, and larger context windows, prompt caching becomes not just a nice-to-have optimization but a fundamental requirement for sustainable agent deployment. Start by auditing your current agent's prompt structure, measure your cache hit rates, and iterate on your prompt organization to maximize the cached prefix. Your API bill will thank you.