How to Cache Tool Outputs for Cost-Effective Agents

How to Cache Tool Outputs for Cost-Effective Agents

AI agents that call external tools—search APIs, database queries, code interpreters, web scrapers—can quickly rack up costs and latency. Many of those tool calls return identical or near-identical results across runs, especially during development, evaluation, and iterative prompting. Caching tool outputs is one of the highest-leverage optimizations available: it reduces token spend, shortens response times, and makes agents more predictable. This tutorial walks through what tool output caching is, why it matters, how to implement it, and the best practices that separate a toy cache from a production-grade one.

What Is Tool Output Caching?

Tool output caching is the practice of storing the results of tool calls keyed by their inputs, so that when an agent requests the same operation again, the cached result is returned instead of re-executing the tool. Conceptually, it is a memoization layer sitting between the agent's reasoning loop and the actual tool implementation.

A cache entry typically contains four things:

• Key: A deterministic hash of the tool name plus its serialized arguments.
• Value: The serialized output returned by the tool.
• Metadata: Timestamp, TTL, hit count, and optionally the model that triggered the call.
• Invalidation rules: Conditions under which the entry should be discarded.

When the agent decides to call a tool, the runtime first checks the cache. On a hit, it injects the stored result back into the conversation as if the tool had just run. On a miss, it executes the tool, stores the result, and then proceeds.

Why It Matters

The economics of agentic systems are dominated by three factors: token cost, latency, and API rate limits. Caching attacks all three:

• Cost: If a search tool returns a 2,000-token result and the agent calls it ten times with the same query during a session, caching saves 18,000 tokens of tool output tokens—and often the reasoning tokens that would have been spent re-processing them.
• Latency: A cache hit returns in milliseconds; a live web search or database round-trip can take seconds. For multi-step agents that call tools in a loop, this compounds dramatically.
• Rate limits: External APIs enforce quotas. Caching absorbs repeated identical requests, keeping you under the limit.
• Reproducibility: During evaluation and debugging, deterministic cached outputs let you isolate changes in the model's reasoning from changes in the tool's behavior.

In practice, teams report 40–80% reductions in tool-related costs after introducing caching, with the biggest wins in research agents, retrieval-augmented generation pipelines, and any agent that re-runs the same workflow multiple times.

How to Use It: A Practical Implementation

Let's build a minimal but realistic caching layer in Python. We'll start with an in-memory cache, then extend it to disk and Redis. The design goal is to make caching transparent: tools should not need to know caching exists.

Step 1: Define the Cache Interface

import hashlib
import json
import time
from abc import ABC, abstractmethod
from typing import Any, Optional


class ToolCache(ABC):
    @abstractmethod
    def get(self, key: str) -> Optional[Any]:
        ...

    @abstractmethod
    def set(self, key: str, value: Any, ttl: int) -> None:
        ...

    @abstractmethod
    def delete(self, key: str) -> None:
        ...

    @staticmethod
    def make_key(tool_name: str, args: dict, kwargs: dict) -> str:
        payload = json.dumps(
            {"tool": tool_name, "args": args, "kwargs": kwargs},
            sort_keys=True,
            default=str,
        )
        return hashlib.sha256(payload.encode()).hexdigest()

The make_key method is critical. It serializes the tool name and arguments in a canonical, sorted form so that {"q": "weather", "units": "metric"} and {"units": "metric", "q": "weather"} produce the same key.

Step 2: Implement an In-Memory Cache

class InMemoryCache(ToolCache):
    def __init__(self):
        self._store: dict[str, tuple[Any, float]] = {}

    def get(self, key: str) -> Optional[Any]:
        entry = self._store.get(key)
        if entry is None:
            return None
        value, expires_at = entry
        if expires_at and time.time() > expires_at:
            del self._store[key]
            return None
        return value

    def set(self, key: str, value: Any, ttl: int) -> None:
        expires_at = time.time() + ttl if ttl > 0 else 0
        self._store[key] = (value, expires_at)

    def delete(self, key: str) -> None:
        self._store.pop(key, None)

Step 3: Wrap Tools with a Caching Decorator

import functools

def cached_tool(cache: ToolCache, ttl: int = 3600):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            key = ToolCache.make_key(func.__name__, args, kwargs)
            cached = cache.get(key)
            if cached is not None:
                print(f"[cache hit] {func.__name__}")
                return cached
            print(f"[cache miss] {func.__name__}")
            result = func(*args, **kwargs)
            cache.set(key, result, ttl)
            return result
        return wrapper
    return decorator

Step 4: Apply It to a Real Tool

import requests

cache = InMemoryCache()

@cached_tool(cache, ttl=1800)
def web_search(query: str, max_results: int = 5) -> list[dict]:
    response = requests.get(
        "https://api.example-search.com/v1/search",
        params={"q": query, "limit": max_results},
        timeout=10,
    )
    response.raise_for_status()
    return response.json()["results"]

# First call: executes the HTTP request
results_a = web_search("python asyncio tutorial")

# Second call: returns instantly from cache
results_b = web_search("python asyncio tutorial")

assert results_a is results_b  # same object, zero network cost

Step 5: Integrate with an Agent Loop

Most agent frameworks expose a hook where tool calls are dispatched. You can intercept that hook to check the cache before execution. Here is a simplified agent loop that demonstrates the pattern:

class CachingAgent:
    def __init__(self, model, tools: dict, cache: ToolCache, ttl: int = 3600):
        self.model = model
        self.tools = tools
        self.cache = cache
        self.ttl = ttl

    def run(self, user_input: str, max_steps: int = 10) -> str:
        messages = [{"role": "user", "content": user_input}]
        for _ in range(max_steps):
            response = self.model.chat(messages, tools=list(self.tools))
            messages.append(response.message)
            if not response.tool_calls:
                return response.message["content"]
            for call in response.tool_calls:
                result = self._execute_tool(call)
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": json.dumps(result),
                })
        return "Max steps reached."

    def _execute_tool(self, call) -> Any:
        tool = self.tools[call.name]
        key = ToolCache.make_key(call.name, call.arguments, {})
        cached = self.cache.get(key)
        if cached is not None:
            return cached
        result = tool(**call.arguments)
        self.cache.set(key, result, self.ttl)
        return result

Step 6: Scale with Redis

In-memory caches do not survive restarts and cannot be shared across processes. For production, use Redis:

import pickle
import redis

class RedisCache(ToolCache):
    def __init__(self, host="localhost", port=6379, db=0, prefix="tool:"):
        self.client = redis.Redis(host=host, port=port, db=db)
        self.prefix = prefix

    def get(self, key: str) -> Optional[Any]:
        raw = self.client.get(self.prefix + key)
        if raw is None:
            return None
        return pickle.loads(raw)

    def set(self, key: str, value: Any, ttl: int) -> None:
        raw = pickle.dumps(value)
        if ttl > 0:
            self.client.setex(self.prefix + key, ttl, raw)
        else:
            self.client.set(self.prefix + key, raw)

    def delete(self, key: str) -> None:
        self.client.delete(self.prefix + key)

Swap InMemoryCache for RedisCache and nothing else in your agent code changes. That is the value of the abstraction.

Best Practices

Choose TTLs Based on Data Volatility

Not all tools deserve the same TTL. Stock prices change every second; a Wikipedia article's content is stable for days. Set per-tool TTLs:

• Real-time data (prices, weather, news): 30–300 seconds.
• Reference data (documentation, encyclopedias): 24 hours or more.
• Computed results (code execution, math): effectively permanent, since deterministic inputs yield deterministic outputs.

Normalize Keys Aggressively

Small variations in arguments cause unnecessary cache misses. Normalize before hashing:

def normalize_search_args(args: dict) -> dict:
    normalized = dict(args)
    if "query" in normalized:
        normalized["query"] = normalized["query"].strip().lower()
    if "max_results" in normalized:
        normalized["max_results"] = int(normalized["max_results"])
    # Drop volatile fields that should not affect the key
    normalized.pop("request_id", None)
    return normalized

Never Cache Non-Deterministic Tools Blindly

Tools that depend on time, random numbers, user-specific state, or mutable external systems are dangerous to cache. Either exclude them, or include the volatile dimension in the key. For example, a "get current weather" tool should include the timestamp rounded to the nearest five minutes in its key, not just the city name.

Bound Cache Size

Unbounded caches cause memory leaks and Redis bloat. Use an LRU eviction policy in Redis, or implement a max-size check in your in-memory cache:

from collections import OrderedDict

class LRUCache(ToolCache):
    def __init__(self, max_size: int = 1000):
        self.max_size = max_size
        self._store: OrderedDict[str, tuple[Any, float]] = OrderedDict()

    def get(self, key: str) -> Optional[Any]:
        entry = self._store.get(key)
        if entry is None:
            return None
        value, expires_at = entry
        if expires_at and time.time() > expires_at:
            del self._store[key]
            return None
        self._store.move_to_end(key)
        return value

    def set(self, key: str, value: Any, ttl: int) -> None:
        expires_at = time.time() + ttl if ttl > 0 else 0
        self._store[key] = (value, expires_at)
        self._store.move_to_end(key)
        while len(self._store) > self.max_size:
            self._store.popitem(last=False)

    def delete(self, key: str) -> None:
        self._store.pop(key, None)

Log Cache Hits and Misses

Instrument your cache so you can measure hit rate, latency savings, and cost avoidance. A simple counter-based approach:

class InstrumentedCache(ToolCache):
    def __init__(self, inner: ToolCache):
        self.inner = inner
        self.hits = 0
        self.misses = 0

    def get(self, key: str) -> Optional[Any]:
        value = self.inner.get(key)
        if value is not None:
            self.hits += 1
        else:
            self.misses += 1
        return value

    def set(self, key: str, value: Any, ttl: int) -> None:
        self.inner.set(key, value, ttl)

    def delete(self, key: str) -> None:
        self.inner.delete(key)

    @property
    def hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total else 0.0

Provide Manual Invalidation

Sometimes you know a tool's underlying data has changed—a document was updated, a database row was modified. Expose an invalidation API so callers can evict specific entries:

def invalidate_tool(cache: ToolCache, tool_name: str, **kwargs):
    key = ToolCache.make_key(tool_name, (), kwargs)
    cache.delete(key)

Consider Semantic Caching for Fuzzy Matches

Exact-match caching misses near-duplicates like "What is the capital of France?" versus "What's France's capital?" Semantic caching uses embeddings to find similar past queries and reuse their results. This is more complex and introduces a similarity threshold you must tune carefully to avoid returning stale or incorrect results, but for high-volume retrieval agents it can double your effective hit rate.

Cache at the Right Granularity

Caching entire tool responses is the simplest approach, but sometimes you should cache sub-results. If a search tool returns ten results and the agent only uses three, consider caching each result individually so partial overlaps across queries can be reused. The trade-off is complexity: finer granularity means more cache entries and more key management.

Handle Serialization Safely

Tool outputs must be serializable to survive disk or Redis storage. Prefer JSON-serializable return types. If you must return complex objects, use pickle with awareness of its security implications—never unpickle data from untrusted sources. Document the expected return type of every cached tool so downstream consumers know what they are getting.

Test Cache Behavior Explicitly

Add tests that verify cache hits return identical results, TTLs expire correctly, and invalidation works. A common bug is a tool whose output contains a timestamp or request ID that changes on every call, defeating the cache entirely. Tests catch this early.

def test_cache_returns_same_result():
    cache = InMemoryCache()

    @cached_tool(cache, ttl=60)
    def add(a, b):
        return a + b

    first = add(2, 3)
    second = add(2, 3)
    assert first == second
    assert cache._store  # entry was stored

def test_cache_expires():
    cache = InMemoryCache()

    @cached_tool(cache, ttl=1)
    def echo(x):
        return x

    echo("hello")
    time.sleep(2)
    assert cache.get(ToolCache.make_key("echo", ("hello",), {})) is None

Conclusion

Caching tool outputs is a straightforward technique with outsized impact on agent cost, latency, and reliability. By inserting a thin memoization layer between your agent's reasoning loop and its tools, you eliminate redundant work, smooth out API rate limits, and gain reproducibility for evaluation. The implementation does not need to be elaborate—a decorator, a key-derivation function, and a TTL-aware store are enough to start. As your agents scale, graduate to Redis-backed caches with per-tool TTLs, instrumentation, and explicit invalidation. Treat caching as a first-class concern in your agent architecture, not an afterthought, and you will see immediate returns in both performance and operating cost.