What is AI Agent Logging?
AI agent logging is the systematic practice of recording the internal operations, decisions, and interactions of autonomous AI agents as they execute tasks. Unlike traditional application logging—which primarily tracks request/response cycles and errors—agent logging must capture a far richer set of signals: reasoning traces, tool selections, intermediate outputs, retry logic, and the agent's evolving internal state across multi-step workflows.
An AI agent doesn't simply process a single input and return a single output. It thinks, plans, acts, observes results, and adjusts its course. Logging gives developers visibility into this dynamic, often non-deterministic process. Without it, debugging an agent that took a wrong turn three steps into a ten-step chain becomes nearly impossible—like trying to debug a program with no stack trace.
At its core, agent logging answers four critical questions at every step of execution:
- What did the agent decide to do, and why?
- What external tools or APIs did it invoke, and what came back?
- How much did each step cost in tokens and time?
- Where did things go wrong, and how did the agent recover?
Why Logging Matters for AI Agents
The stakes for agent logging are significantly higher than for conventional software for several reasons:
Non-Deterministic Behavior
A traditional function call with the same arguments produces the same result every time. An LLM-powered agent, given the same prompt, may choose a different tool, produce a different reasoning path, or hallucinate entirely. Logging every decision point allows you to replay and understand why a particular path was chosen on a specific run.
Cost Accountability
Every token consumed by the agent—whether in the planning phase, tool-calling phase, or final synthesis—translates directly to dollars spent on API calls. Without granular logs, you're flying blind on cost attribution. Detailed token logging lets you pinpoint expensive runs, optimize prompts, and set budget guardrails.
Multi-Step Failure Cascades
When an agent fails on step 7 of a 12-step workflow, the root cause often lies in a subtle error on step 3—a malformed tool response, a truncated context window, or a misrouted output. Comprehensive step-by-step logs create a forensic trail that makes root-cause analysis tractable.
Compliance and Audit Trails
In regulated industries, every action an agent takes—especially those affecting external systems, customer data, or financial records—must be auditable. Structured agent logs provide the evidence trail needed to satisfy compliance requirements and internal review processes.
What to Log: The Essential Signals
Not everything deserves a log line. The art of agent logging lies in capturing the right signals at the right granularity. Here are the categories you should always log:
1. Agent Lifecycle Events
Log the start and end of every agent run, including a unique correlation ID, the input query or task description, the model used, and hyperparameters like temperature and max tokens. On completion, log the final outcome, total steps taken, total tokens consumed, and total wall-clock time.
{
"event": "agent_run_start",
"correlation_id": "run_8f2a9c",
"timestamp": "2025-03-21T14:32:01.123Z",
"agent_id": "customer_support_agent_v2",
"input": "User query: refund policy for international orders",
"model": "gpt-4o",
"temperature": 0.2,
"max_iterations": 10
}2. Reasoning Traces / Chain-of-Thought
If your agent emits a reasoning step—either as a dedicated "think" phase or as part of a ReAct-style loop—capture it verbatim. This is the single most valuable artifact for debugging unexpected behavior. Store it at DEBUG level so it can be toggled on during investigation without overwhelming production log volume.
{
"event": "agent_reasoning",
"correlation_id": "run_8f2a9c",
"iteration": 2,
"thought": "The user asked about international refunds. I need to check two things: the standard refund policy and any country-specific exceptions. I'll start by querying the policy database, then cross-reference with the user's detected region (UK).",
"next_action": "query_policy_db",
"action_input": {"query": "refund_policy international"}
}3. Tool Calls and Responses
Every external tool invocation—database queries, API calls, file operations, code execution—should be logged with the tool name, input parameters, the raw response, and whether it succeeded or failed. Sanitize any secrets or PII in the parameters before logging.
{
"event": "tool_call",
"correlation_id": "run_8f2a9c",
"iteration": 2,
"tool_name": "query_policy_db",
"tool_input": {"query": "refund_policy international", "limit": 5},
"tool_output": {
"success": true,
"results": [
{"region": "default", "policy": "30-day refund window"},
{"region": "UK", "policy": "14-day right to cancel under CCRs"}
],
"row_count": 2
},
"duration_ms": 142,
"retry_count": 0
}4. Model Calls (LLM Completions)
Log every call to the underlying language model: the system prompt (or its hash for brevity), the messages array, the completion response, token counts (prompt tokens, completion tokens, total), finish reason, and latency. This is essential for cost tracking and prompt debugging.
{
"event": "llm_call",
"correlation_id": "run_8f2a9c",
"iteration": 3,
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "You are a helpful support agent..."},
{"role": "user", "content": "What's the refund policy for UK customers?"},
{"role": "assistant", "content": null, "tool_calls": [...]}
],
"usage": {
"prompt_tokens": 1240,
"completion_tokens": 85,
"total_tokens": 1325
},
"finish_reason": "tool_calls",
"latency_ms": 2100,
"estimated_cost_usd": 0.0042
}5. Errors and Retries
Log every error with full context: the step where it occurred, the exception type and message, the stack trace (in development/staging), the agent's recovery action (retry, fallback, escalate), and the outcome of that recovery. Structured error logs make it possible to build dashboards for error rate, top failure modes, and recovery success rate.
{
"event": "agent_error",
"correlation_id": "run_8f2a9c",
"iteration": 4,
"error_type": "ToolTimeoutError",
"error_message": "query_policy_db timed out after 5000ms",
"recovery_action": "retry_with_backoff",
"recovery_attempt": 1,
"max_retries": 3,
"backoff_ms": 1000
}6. Token Budget and Cost Snapshots
At the end of each iteration, emit a running tally of tokens consumed and estimated cost. This helps spot runs that are spiraling in cost before they exhaust your budget.
{
"event": "token_budget_check",
"correlation_id": "run_8f2a9c",
"iteration": 5,
"cumulative_prompt_tokens": 6100,
"cumulative_completion_tokens": 420,
"cumulative_cost_usd": 0.031,
"budget_limit_usd": 0.50,
"budget_remaining_pct": 93.8
}What to Skip: Noise to Avoid
Over-logging is just as harmful as under-logging. It bloats storage, slows down agents, drowns signal in noise, and can inadvertently leak sensitive data. Here's what you should deliberately skip or aggressively downsample:
Raw LLM Prompt/Response Payloads in Production at INFO Level
Full message arrays can be enormous—tens of thousands of tokens—and logging them at INFO level will overwhelm your log pipeline. Instead, log a content hash (SHA-256) of prompts for deduplication, and store full payloads only at DEBUG level or in a separate tracing system designed for large blobs.
# Bad: logging the entire messages array at INFO
logger.info("LLM call", messages=messages, response=response)
# Good: log metadata + hash at INFO, full payload at DEBUG
logger.info("LLM call", model=model, prompt_hash=sha256(messages), token_count=total_tokens)
logger.debug("LLM call detail", messages=messages, response=response)Duplicate or Redundant Intermediate States
If your agent's state hasn't changed between iterations, don't log it again. Implement a state diff mechanism: log only what changed since the last step. This is especially important for agents with large context windows or accumulated memory.
# Bad: logging the entire accumulated state every iteration
logger.info("Agent state", full_state={"history": [...200 messages...], "scratchpad": "..."})
# Good: log only the delta
logger.info("Agent state delta", added_messages=2, scratchpad_changed=True, tool_results_added=1)Successful Heartbeats and No-Op Polls
If your agent runs a polling loop waiting for an external event, logging every successful poll creates a flood of identical log lines. Log only state transitions—when the poll result changes or when an error occurs.
Sensitive Data (PII, Secrets, API Keys)
Never log personally identifiable information (names, emails, phone numbers, addresses), authentication tokens, API keys, or passwords. Implement a sanitization layer that redacts or hashes these values before they reach the logger. This should be automatic, not reliant on developer discipline.
# Example sanitizer
import re
def sanitize_for_logging(data):
sensitive_patterns = {
"email": r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}",
"api_key": r"sk-[a-zA-Z0-9]{32,}",
"phone": r"\+?\d{7,15}",
}
for field, pattern in sensitive_patterns.items():
data = re.sub(pattern, f"[REDACTED_{field}]", str(data))
return dataDeterministic Internal Calculations
Don't log the intermediate results of purely mathematical operations or data transformations that are fully reproducible from inputs. If you can recompute it, you don't need to log it—log only the inputs and the final result.
How to Implement Agent Logging
Let's walk through a practical implementation using Python's structlog library, which provides structured, JSON-oriented logging ideal for agent observability.
Setting Up Structured Logging
# agent_logging_setup.py
import structlog
import logging
import uuid
import time
import hashlib
import json
from typing import Any, Dict, Optional
# Configure structlog for JSON output
structlog.configure(
processors=[
structlog.stdlib.filter_by_level, # Respect log levels
structlog.stdlib.add_log_level, # Add 'level' field
structlog.processors.TimeStamper(fmt="iso"), # ISO timestamps
structlog.processors.JSONRenderer(indent=2) # JSON output
],
logger_factory=structlog.stdlib.LoggerFactory(),
cache_logger_on_first_use=True,
)
logger = structlog.get_logger("ai_agent")Building a Logging Wrapper for Agent Runs
# agent_logger.py
class AgentRunLogger:
"""Wraps an agent execution with structured logging at every step."""
def __init__(self, agent_id: str, model: str, max_iterations: int = 10):
self.agent_id = agent_id
self.model = model
self.max_iterations = max_iterations
self.correlation_id = None
self.iteration = 0
self.cumulative_tokens = 0
self.cumulative_cost = 0.0
def start_run(self, task_input: str, metadata: Optional[Dict] = None):
self.correlation_id = f"run_{uuid.uuid4().hex[:12]}"
self.start_time = time.time()
logger.info(
"agent_run_start",
correlation_id=self.correlation_id,
agent_id=self.agent_id,
model=self.model,
input=task_input,
max_iterations=self.max_iterations,
metadata=metadata or {},
)
return self.correlation_id
def log_reasoning(self, thought: str, next_action: str, action_input: Any):
self.iteration += 1
logger.info(
"agent_reasoning",
correlation_id=self.correlation_id,
iteration=self.iteration,
thought=thought,
next_action=next_action,
action_input=action_input,
)
def log_tool_call(self, tool_name: str, tool_input: Dict,
tool_output: Dict, duration_ms: float, retry_count: int = 0):
logger.info(
"tool_call",
correlation_id=self.correlation_id,
iteration=self.iteration,
tool_name=tool_name,
tool_input=tool_input,
tool_output=tool_output,
duration_ms=round(duration_ms, 2),
retry_count=retry_count,
)
def log_llm_call(self, messages: list, response: Any, latency_ms: float):
prompt_tokens = getattr(response.usage, 'prompt_tokens', 0)
completion_tokens = getattr(response.usage, 'completion_tokens', 0)
total_tokens = prompt_tokens + completion_tokens
self.cumulative_tokens += total_tokens
# Estimate cost (adjust per-model rates as needed)
cost_per_1k_prompt = 0.01 # Example for GPT-4o
cost_per_1k_completion = 0.03
estimated_cost = (prompt_tokens / 1000 * cost_per_1k_prompt) + \
(completion_tokens / 1000 * cost_per_1k_completion)
self.cumulative_cost += estimated_cost
# Hash messages for deduplication
messages_hash = hashlib.sha256(
json.dumps(messages, default=str).encode()
).hexdigest()[:16]
logger.info(
"llm_call",
correlation_id=self.correlation_id,
iteration=self.iteration,
model=self.model,
messages_hash=messages_hash,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=total_tokens,
finish_reason=getattr(response, 'finish_reason', 'unknown'),
latency_ms=round(latency_ms, 2),
estimated_cost_usd=round(estimated_cost, 6),
)
# Full payload only at DEBUG
logger.debug(
"llm_call_detail",
correlation_id=self.correlation_id,
messages=messages,
response=str(response),
)
def log_error(self, error_type: str, error_message: str,
recovery_action: str, attempt: int, max_retries: int):
logger.error(
"agent_error",
correlation_id=self.correlation_id,
iteration=self.iteration,
error_type=error_type,
error_message=error_message,
recovery_action=recovery_action,
recovery_attempt=attempt,
max_retries=max_retries,
)
def log_budget_check(self, budget_limit_usd: float):
remaining_pct = max(0, (1 - self.cumulative_cost / budget_limit_usd) * 100)
logger.info(
"token_budget_check",
correlation_id=self.correlation_id,
iteration=self.iteration,
cumulative_tokens=self.cumulative_tokens,
cumulative_cost_usd=round(self.cumulative_cost, 6),
budget_limit_usd=budget_limit_usd,
budget_remaining_pct=round(remaining_pct, 1),
)
def end_run(self, final_output: Any, success: bool):
elapsed_ms = (time.time() - self.start_time) * 1000
logger.info(
"agent_run_end",
correlation_id=self.correlation_id,
total_iterations=self.iteration,
total_tokens=self.cumulative_tokens,
total_cost_usd=round(self.cumulative_cost, 6),
total_duration_ms=round(elapsed_ms, 2),
success=success,
final_output_preview=str(final_output)[:500], # Truncate for log sanity
)Integrating with an Agent Loop
# agent_runner.py
class SimpleAgent:
"""A minimal ReAct-style agent that uses the logging wrapper."""
def __init__(self, model: str = "gpt-4o"):
self.model = model
self.run_logger = None
def run(self, task: str) -> str:
self.run_logger = AgentRunLogger(
agent_id="simple_react_agent_v1",
model=self.model,
max_iterations=10
)
correlation_id = self.run_logger.start_run(task)
try:
# Simulate an agent loop
for i in range(10):
# Reasoning step
thought = f"Processing step {i+1} for task: {task}"
self.run_logger.log_reasoning(
thought=thought,
next_action="mock_tool",
action_input={"step": i+1}
)
# Simulate tool call
tool_start = time.time()
tool_result = {"output": f"result_for_step_{i+1}", "status": "ok"}
tool_duration = (time.time() - tool_start) * 1000
self.run_logger.log_tool_call(
tool_name="mock_tool",
tool_input={"step": i+1},
tool_output=tool_result,
duration_ms=tool_duration,
)
# Simulate LLM call
llm_start = time.time()
# In real code, this would call the actual LLM API
llm_latency = (time.time() - llm_start) * 1000 + 500 # mock latency
self.run_logger.log_llm_call(
messages=[{"role": "user", "content": task}],
response=type('Response', (), {
'usage': type('Usage', (), {
'prompt_tokens': 200,
'completion_tokens': 50
})(),
'finish_reason': 'stop'
})(),
latency_ms=llm_latency,
)
# Budget check every 3 iterations
if i % 3 == 0:
self.run_logger.log_budget_check(budget_limit_usd=0.50)
# Break condition
if i >= 3:
break
final_output = "Task completed successfully: synthesized result"
self.run_logger.end_run(final_output, success=True)
return final_output
except Exception as e:
self.run_logger.log_error(
error_type=type(e).__name__,
error_message=str(e),
recovery_action="fail_run",
attempt=1,
max_retries=1,
)
self.run_logger.end_run(f"Failed: {str(e)}", success=False)
raiseUsing Log Levels Effectively
# log_level_guide.py
"""
Recommended log level usage for AI agents:
TRACE - Raw LLM request/response payloads, full context dumps
DEBUG - Detailed reasoning steps, tool response bodies, state diffs
INFO - Lifecycle events (start/end), tool call summaries, token counts,
iteration boundaries, budget checks (DEFAULT PRODUCTION LEVEL)
WARN - Retry attempts, degraded tool responses, approaching budget limits
ERROR - Tool failures, exceeded retry limits, model errors, exceptions
CRITICAL - Agent run aborted, budget exhausted, security violations
In production, set the root level to INFO and selectively enable DEBUG
or TRACE for specific correlation IDs when debugging.
"""
import logging
# Production config
logging.basicConfig(level=logging.INFO)
# Per-run debug toggle via environment variable or flag
import os
if os.environ.get("AGENT_DEBUG_RUN_ID") == correlation_id:
logging.getLogger("ai_agent").setLevel(logging.DEBUG)Best Practices for AI Agent Logging
1. Always Include a Correlation ID
Every log line from a single agent run must share a unique correlation ID. This allows you to reconstruct the entire execution trace in log aggregation tools like Elasticsearch, Datadog, or Grafana Loki with a single query. Generate it at the start of the run and thread it through every logger call.
2. Use Structured Logging (JSON)
Plain-text log lines like "Step 3 completed successfully" are nearly useless for analysis. Structured JSON logs let you query by field: find all runs where tool_name="query_policy_db" and success=false, or aggregate total_cost_usd across all runs in the last hour. The examples above all emit JSON, which can be ingested directly into modern log platforms.
3. Separate Operational Logs from Debug Traces
Operational logs (INFO level: lifecycle events, tool calls, errors, cost snapshots) should be concise, always-on, and aggregated for dashboards. Debug traces (DEBUG/TRACE: full prompts, reasoning chains, state dumps) should be voluminous but toggled on only when needed. Store debug traces in a dedicated system—like a vector database or blob store—so they don't overwhelm your operational log pipeline.
4. Implement Log Sampling for High-Volume Agents
If your agent processes thousands of items in a loop, log every Nth iteration or use reservoir sampling. For example, log every tool call for the first 10 iterations, then every 10th iteration thereafter. This keeps volume manageable while preserving a representative sample.
def should_log_iteration(iteration: int, sample_rate: int = 10) -> bool:
"""Log all iterations up to 10, then sample every Nth."""
if iteration <= 10:
return True
return iteration % sample_rate == 05. Hash Large Payloads for Deduplication
When logging prompts or tool inputs that may repeat across runs, include a SHA-256 hash. This lets you identify duplicate patterns without storing or indexing the full content. Store the full payload in a cheap object store keyed by the hash, and reference it in logs by hash only.
6. Build Alerts on Log Patterns
Don't just collect logs—act on them. Set up alerts for:
- Cost per run exceeding a threshold (e.g.,
total_cost_usd > 1.00) - Error rate spikes (more than 5% of runs failing in a 5-minute window)
- Tool timeout frequency increasing (indicates upstream degradation)
- Retry storms (same tool retried more than 3 times across many runs)
- Budget exhaustion warnings (runs approaching their cost limit)
7. Sanitize at the Logging Boundary, Not in Application Code
Build sanitization into your logger wrapper so that every log call automatically scrubs PII, secrets, and credentials. Developers shouldn't need to remember to sanitize each log line individually—make it the default.
class SanitizingLogger:
def __init__(self, base_logger):
self.base_logger = base_logger
self.pii_patterns = [
(re.compile(r'\b[\w.-]+@[\w.-]+\.\w{2,}\b'), '[REDACTED_EMAIL]'),
(re.compile(r'\b\d{3}-\d{2}-\d{4}\b'), '[REDACTED_SSN]'),
(re.compile(r'sk-[a-zA-Z0-9]{32,}'), '[REDACTED_API_KEY]'),
]
def _sanitize(self, data):
if isinstance(data, str):
for pattern, replacement in self.pii_patterns:
data = pattern.sub(replacement, data)
elif isinstance(data, dict):
return {k: self._sanitize(v) for k, v in data.items()}
elif isinstance(data, list):
return [self._sanitize(item) for item in data]
return data
def info(self, event, **kwargs):
sanitized = {k: self._sanitize(v) for k, v in kwargs.items()}
self.base_logger.info(event, **sanitized)
# ... repeat for debug, error, warn, etc.8. Rotate and Archive Logs Strategically
Agent logs can grow quickly. Implement a tiered retention policy:
- Hot storage (7 days): Full operational logs at INFO+ level for real-time querying and alerting
- Warm storage (30 days): Aggregated metrics and sampled traces for trend analysis
- Cold storage (90+ days): Compressed full logs for compliance and post-mortem investigations
9. Include Context for the Human Operator
When an agent fails, the on-call engineer needs context fast. Include in your error logs: the task the user originally asked for, the step that failed, what the agent was trying to accomplish at that step, and a link to the full debug trace if available. A log line that says "ToolTimeoutError in query_policy_db" without any surrounding context is frustratingly opaque at 3 AM.
Conclusion
Effective AI agent logging is a deliberate engineering discipline, not an afterthought. By capturing the right signals—lifecycle events, reasoning traces, tool calls with their responses, token consumption, and structured errors—you create an observability surface that makes agents debuggable, cost-transparent, and production-ready. Equally important is knowing what to skip: raw payloads at the wrong log level, redundant states, no-op polls, and anything containing sensitive data. Implement logging as a first-class layer in your agent architecture with structured JSON, correlation IDs, sanitization, and tiered storage. When done well, agent logging transforms opaque, non-deterministic black boxes into inspectable, trustworthy systems that you can confidently deploy, monitor, and improve at scale.