Introduction to Observability and Tracing with LlamaIndex
Building applications with Large Language Models (LLMs) introduces a unique set of challenges that traditional software observability tools were never designed to handle. When a user submits a query to your RAG (Retrieval-Augmented Generation) pipeline, a complex chain of events unfolds: query rewriting, embedding generation, vector database lookups, document reranking, prompt construction, and finally LLM generation. If the final answer is wrong, slow, or unexpectedly expensive, how do you pinpoint the culprit? This is where observability and tracing become essential.
LlamaIndex, one of the most popular frameworks for building LLM applications, provides robust built-in observability capabilities through its callback system and integrations with leading observability platforms. This guide walks you through everything you need to know to instrument, monitor, and debug your LlamaIndex applications like a professional.
What Is Observability and Tracing in the Context of LLMs?
Observability is the ability to understand the internal state of a system based on its external outputs. In the LLM world, this means having visibility into every step your application takes — from the moment a query enters to the moment a response is returned. Tracing is a specific subset of observability that records the causal chain of operations, typically represented as a tree of spans where each span represents a unit of work.
Key Concepts
- Trace: A complete execution path through your application, usually triggered by a single user request.
- Span: A single operation within a trace, such as an LLM call, an embedding generation, or a retrieval step. Spans have start times, end times, durations, inputs, outputs, and metadata.
- Event: A discrete point-in-time occurrence, such as a token being streamed or a chunk being selected.
- Metrics: Aggregated numerical data such as token counts, latency percentiles, error rates, and cost estimates.
Why LLM Observability Is Different
Traditional web applications are largely deterministic: the same input produces the same output, and performance bottlenecks are usually tied to database queries or network latency. LLM applications are fundamentally different. They are probabilistic, token-based, and often involve multiple model calls with varying latencies. A single user query might trigger dozens of internal operations, each of which can fail independently. Furthermore, the quality of an LLM response is subjective and difficult to assess with traditional metrics like HTTP status codes.
Why Observability Matters for LlamaIndex Applications
Without observability, debugging a LlamaIndex application is like flying blind. You might notice that responses are slow or inaccurate, but you have no way to determine why. Here are the concrete problems observability solves:
Performance Debugging
A RAG pipeline might take 8 seconds to respond. Is the bottleneck the embedding model, the vector database query, the reranker, or the LLM itself? Tracing breaks down the total latency into individual spans, so you can immediately see that, for example, the reranker is consuming 5 of those 8 seconds.
Cost Management
LLM APIs charge per token. If your application is making redundant LLM calls or sending unnecessarily large contexts, costs can spiral out of control. Observability platforms track token usage per span, allowing you to identify and eliminate wasteful patterns.
Quality Assurance
When a user reports a bad answer, you need to inspect what documents were retrieved, what prompt was sent to the LLM, and what the raw model output was. Tracing captures all of this, enabling post-hoc debugging and systematic quality improvement.
Production Monitoring
In production, you need to monitor error rates, latency distributions, and usage patterns over time. Observability platforms provide dashboards and alerting so you can detect regressions before they impact users.
LlamaIndex's Built-in Observability Architecture
LlamaIndex provides a callback handler system that serves as the foundation for all observability. Every major operation in LlamaIndex — LLM calls, embedding generation, retrieval, query engine execution, and agent steps — emits callback events. By attaching one or more callback handlers, you can capture these events and route them to any destination.
The Callback Event Flow
When you execute a query in LlamaIndex, the framework emits a structured sequence of events. A typical RAG query produces events like the following:
CBEventType.EMBEDDING— fired when text is converted to vectorsCBEventType.RETRIEVE— fired when documents are fetched from an indexCBEventType.SYNTHESIZE— fired when the LLM generates the final answerCBEventType.LLM— fired for each individual LLM API callCBEventType.CHUNKING— fired when text is split into chunksCBEventType.TEMPLATING— fired when prompts are constructed from templates
Each event includes a payload with the inputs, outputs, and timing information. Callback handlers receive these events and can process them in any way they choose.
Getting Started: Basic Tracing with LlamaIndex Debug Logging
Before integrating with external platforms, let's start with LlamaIndex's simplest built-in observability tool: the LlamaDebugHandler. This handler captures all callback events and lets you inspect them programmatically.
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler
# Create a debug handler that captures all events
debug_handler = LlamaDebugHandler(print_trace_on_end=True)
# Attach it to the global callback manager
Settings.callback_manager = CallbackManager([debug_handler])
# Load documents and build an index
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
# Create a query engine and run a query
query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of the documents?")
print(response)
# Inspect the captured events programmatically
events = debug_handler.get_events()
for event in events:
print(f"Event type: {event.type}")
print(f"Duration: {event.end_time - event.start_time:.2f}s")
print("---")
The LlamaDebugHandler is excellent for local development and quick debugging. It prints a trace of all events to the console and stores them in memory for programmatic access. However, for production applications, you need a more robust solution.
Integrating with Arize Phoenix
Arize Phoenix is an open-source LLM observability platform that works seamlessly with LlamaIndex. It provides a local web UI for inspecting traces, comparing responses, and analyzing token usage. It is the recommended starting point for most developers.
Installation and Setup
pip install arize-phoenix llama-index-callbacks-arize
Basic Configuration
import phoenix as px
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
# Launch Phoenix in the background (opens a local server on port 6060)
px.launch_app()
# Configure LlamaIndex to send traces to Phoenix
import llama_index.core
from llama_index.callbacks.arize import ArizeCallbackHandler
from llama_index.core.callbacks import CallbackManager
arize_handler = ArizeCallbackHandler()
llama_index.core.Settings.callback_manager = CallbackManager([arize_handler])
# Build and query your index as usual
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Explain the key concepts in the documents.")
print(response)
# Open your browser to http://localhost:6060 to view traces
Once configured, every query you run will appear in the Phoenix UI with a full breakdown of spans, inputs, outputs, token counts, and latencies. You can click into any span to see the exact prompt sent to the LLM and the raw response received.
Integrating with Langfuse
Langfuse is another popular open-source observability platform that offers both self-hosted and cloud-hosted options. It provides excellent support for LlamaIndex through a dedicated callback handler.
Installation
pip install langfuse llama-index-callbacks-langfuse
Configuration
import os
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.callbacks import CallbackManager
from llama_index.callbacks.langfuse import LangfuseCallbackHandler
# Set your Langfuse credentials (get these from your Langfuse dashboard)
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-your-public-key"
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-your-secret-key"
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
# Create and attach the handler
langfuse_handler = LangfuseCallbackHandler()
Settings.callback_manager = CallbackManager([langfuse_handler])
# Use your application normally
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the key findings.")
print(response)
Adding User and Session Metadata
One of Langfuse's strengths is the ability to attach metadata to traces, which is critical for production applications where you need to correlate traces with specific users or sessions.
from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager
from llama_index.callbacks.langfuse import LangfuseCallbackHandler
# Create handler with user and session metadata
langfuse_handler = LangfuseCallbackHandler(
user_id="user-12345",
session_id="session-abc-def",
tags=["production", "rag-pipeline"],
metadata={
"app_version": "2.1.0",
"environment": "production",
"feature_flag": "advanced_retrieval"
}
)
Settings.callback_manager = CallbackManager([langfuse_handler])
Integrating with OpenTelemetry
For organizations that already use OpenTelemetry for their broader observability stack, LlamaIndex supports OTLP-compatible exporters. This allows you to send LlamaIndex traces alongside your existing application traces, giving you a unified view of your entire system.
Using OpenLLMetry
pip install opentelemetry-sdk opentelemetry-exporter-otlp
pip install openllmetry
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
# Set up the OpenTelemetry tracer provider
resource = Resource.create({"service.name": "llamaindex-app"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
)
)
trace.set_tracer_provider(provider)
# Initialize OpenLLMetry instrumentation
from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
# Now use LlamaIndex normally — traces are automatically captured
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the main recommendations?")
print(response)
Integrating with OpenLIT
OpenLIT is an open-source observability platform built on OpenTelemetry that provides a beautiful UI specifically designed for LLM applications. It offers a straightforward integration with LlamaIndex.
pip install openlit
import openlit
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
# Initialize OpenLIT — this automatically instruments LlamaIndex
openlit.init(
collector_endpoint="http://localhost:4318",
application_name="my-llamaindex-app",
environment="development"
)
# Use LlamaIndex normally
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What is the conclusion of the report?")
print(response)
Tracing LlamaIndex Agents and Multi-Step Workflows
Agents are where observability becomes truly indispensable. A LlamaIndex agent might make dozens of LLM calls, invoke multiple tools, and iterate through complex reasoning loops. Without tracing, it is nearly impossible to understand what an agent did or why it produced a particular result.
Tracing a ReAct Agent
from llama_index.core import Settings
from llama_index.core.agent import ReActAgent
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI
from llama_index.callbacks.arize import ArizeCallbackHandler
from llama_index.core.callbacks import CallbackManager
import phoenix as px
# Set up observability
px.launch_app()
arize_handler = ArizeCallbackHandler()
Settings.callback_manager = CallbackManager([arize_handler])
Settings.llm = OpenAI(model="gpt-4o")
# Define tools
def multiply(a: float, b: float) -> float:
"""Multiply two numbers and return the result."""
return a * b
def add(a: float, b: float) -> float:
"""Add two numbers and return the result."""
return a + b
multiply_tool = FunctionTool.from_defaults(fn=multiply)
add_tool = FunctionTool.from_defaults(fn=add)
# Create the agent
agent = ReActAgent.from_tools(
[multiply_tool, add_tool],
verbose=True,
max_iterations=10
)
# Run a query that requires multiple tool calls
response = agent.chat("What is (15 + 27) * 3? Think step by step.")
print(response)
When you inspect this trace in Phoenix or any other observability platform, you will see a hierarchical view showing each reasoning step, each tool call, the inputs and outputs of each tool, and the LLM calls that drove the decision-making process. This level of detail is invaluable for debugging agent behavior and optimizing prompts.
Tracing Custom Workflows
LlamaIndex's Workflow feature allows you to build complex, event-driven pipelines. These workflows benefit enormously from tracing because they often involve branching logic and parallel execution.
from llama_index.core.workflow import Workflow, step, Event, Context
from llama_index.core import Settings
from llama_index.callbacks.arize import ArizeCallbackHandler
from llama_index.core.callbacks import CallbackManager
from llama_index.llms.openai import OpenAI
import phoenix as px
# Set up observability
px.launch_app()
Settings.callback_manager = CallbackManager([ArizeCallbackHandler()])
Settings.llm = OpenAI(model="gpt-4o")
class ResearchEvent(Event):
query: str
class SynthesisEvent(Event):
research_results: str
class ResearchWorkflow(Workflow):
@step
async def research_step(self, ctx: Context, ev: ResearchEvent) -> SynthesisEvent:
# Simulate research
response = await Settings.llm.acomplete(
f"Research the following topic and provide key findings: {ev.query}"
)
return SynthesisEvent(research_results=str(response))
@step
async def synthesis_step(self, ctx: Context, ev: SynthesisEvent) -> str:
response = await Settings.llm.acomplete(
f"Synthesize these findings into a concise summary: {ev.research_results}"
)
return str(response)
# Run the workflow
workflow = ResearchWorkflow(timeout=60, verbose=True)
result = await workflow.run(ResearchEvent(query="The impact of AI on software development"))
print(result)
Adding Custom Spans and Metadata
Sometimes the built-in callback events are not enough. You may want to add custom spans to track business logic that wraps around your LlamaIndex calls, or attach metadata that helps you filter and search traces later.
Using the Callback Manager Directly
from llama_index.core.callbacks import CallbackManager, CBEventType
from llama_index.core import Settings
callback_manager = Settings.callback_manager
# Start a custom trace span
with callback_manager.event(
CBEventType.EMBEDDING,
payload={"description": "Custom embedding step for user profile"}
) as event:
# Your custom logic here
import time
time.sleep(0.5) # Simulate work
# Add payload data to the span
event.on_end(payload={
"custom_metric": 42,
"user_segment": "premium"
})
Adding Metadata to Existing Traces
from llama_index.core import Settings
# You can attach metadata to the global settings
# which will be included in all subsequent traces
Settings.metadata = {
"app_name": "customer-support-bot",
"version": "3.2.1",
"deployment": "us-east-1"
}
# Or pass metadata per-query
query_engine = index.as_query_engine()
response = query_engine.query(
"How do I reset my password?",
# Metadata can be passed through the query kwargs
# depending on your observability platform
)
Best Practices for LLM Observability
1. Instrument from Day One
Do not treat observability as an afterthought. Add tracing to your LlamaIndex application from the very first prototype. The cost of instrumentation is minimal, and the insights you gain during development will shape your architecture decisions. Retrofitting observability into a mature application is significantly harder.
2. Capture Full Inputs and Outputs
Ensure your observability platform is configured to record the complete inputs and outputs of every LLM call, retrieval step, and tool invocation. Without this data, traces are nearly useless for debugging. Most platforms support this by default, but some may require explicit configuration to avoid truncating large payloads.
3. Be Mindful of Sensitive Data
LLM applications often process sensitive user data — personal information, medical records, financial data, or proprietary business information. Before sending traces to a third-party observability platform, ensure you understand what data is being captured and implement redaction where necessary.
import re
from llama_index.core.callbacks import CallbackManager
from llama_index.core.callbacks.base_handler import BaseCallbackHandler
class RedactingCallbackHandler(BaseCallbackHandler):
"""A callback handler that redacts sensitive data before forwarding."""
def __init__(self, inner_handler):
self.inner_handler = inner_handler
self.sensitive_patterns = [
(re.compile(r'\b\d{3}-\d{2}-\d{4}\b'), '[SSN REDACTED]'),
(re.compile(r'\b\d{16}\b'), '[CARD REDACTED]'),
(re.compile(r'\b[\w.]+@[\w.]+\b'), '[EMAIL REDACTED]'),
]
def _redact(self, text):
if not isinstance(text, str):
return text
for pattern, replacement in self.sensitive_patterns:
text = pattern.sub(replacement, text)
return text
def start_trace(self, trace_id, span_id, parent_id=None):
self.inner_handler.start_trace(trace_id, span_id, parent_id)
def end_trace(self, trace_id, span_id=None):
self.inner_handler.end_trace(trace_id, span_id)
def on_event_start(self, event_type, payload, **kwargs):
if payload and 'serialized' in payload:
payload['serialized'] = self._redact(str(payload['serialized']))
return self.inner_handler.on_event_start(event_type, payload, **kwargs)
def on_event_end(self, event_type, payload, **kwargs):
if payload:
for key in ['response', 'prompt', 'completion']:
if key in payload:
payload[key] = self._redact(str(payload[key]))
return self.inner_handler.on_event_end(event_type, payload, **kwargs)
4. Set Up Alerting on Key Metrics
In production, configure alerts for the following conditions:
- LLM call error rate exceeds a threshold (e.g., 5%)
- P95 latency exceeds your service level objective
- Token usage per query spikes unexpectedly (may indicate prompt injection or context bloat)
- Retrieval returns zero documents (may indicate index corruption or query mismatch)
- Cost per day exceeds a budget threshold
5. Use Evaluation Traces
Beyond runtime tracing, use observability platforms to record evaluation runs. When you test your RAG pipeline against a golden dataset, capture those traces too. This allows you to compare pipeline performance across code changes and model upgrades over time.
from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings
from llama_index.callbacks.arize import ArizeCallbackHandler
from llama_index.core.callbacks import CallbackManager
import phoenix as px
# Set up observability
px.launch_app()
Settings.callback_manager = CallbackManager([ArizeCallbackHandler()])
Settings.llm = OpenAI(model="gpt-4o")
# Build index
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
# Define evaluation questions
eval_questions = [
"What is the main argument of the document?",
"What evidence supports the conclusion?",
"What are the limitations mentioned?",
]
# Run evaluations — each will be traced
faithfulness_evaluator = FaithfulnessEvaluator()
relevancy_evaluator = RelevancyEvaluator()
for question in eval_questions:
response = query_engine.query(question)
faith_result = faithfulness_evaluator.evaluate_response(
query=question, response=response
)
relev_result = relevancy_evaluator.evaluate_response(
query=question, response=response
)
print(f"Q: {question}")
print(f" Faithfulness: {faith_result.passing} (score: {faith_result.score})")
print(f" Relevancy: {relev_result.passing} (score: {relev_result.score})")
6. Version Your Prompts and Configurations
Attach version information to your traces so you can correlate performance changes with code or prompt modifications. When you update a prompt template, change a chunk size, or swap a model, the version metadata in your traces will help you understand the impact.
from llama_index.core import Settings
# Attach version info that will be included in traces
Settings.global_context = {
"prompt_version": "v2.3",
"chunk_size": 512,
"chunk_overlap": 50,
"similarity_top_k": 5,
"model": "gpt-4o",
"embedding_model": "text-embedding-3-small"
}
7. Sample in High-Traffic Production
If your application handles thousands of queries per minute, sending every trace to your observability platform may be cost-prohibitive. Most platforms support head-based or tail-based sampling. Configure sampling rates so you capture enough traces for meaningful analysis without overwhelming your infrastructure.
import random
from llama_index.core.callbacks import CallbackManager
from llama_index.callbacks.arize import ArizeCallbackHandler
class SamplingCallbackHandler:
"""Wraps a handler and only forwards a percentage of traces."""
def __init__(self, handler, sample_rate=0.1):
self.handler = handler
self.sample_rate = sample_rate
self._active = False
def start_trace(self, trace_id, span_id=None, parent_id=None):
self._active = random.random() < self.sample_rate
if self._active:
self.handler.start_trace(trace_id, span_id, parent_id)
def end_trace(self, trace_id, span_id=None):
if self._active:
self.handler.end_trace(trace_id, span_id)
def on_event_start(self, *args, **kwargs):
if self._active:
return self.handler.on_event_start(*args, **kwargs)
def on_event_end(self, *args, **kwargs):
if self._active:
self.handler.on_event_end(*args, **kwargs)
# Use 10% sampling in production
arize_handler = ArizeCallbackHandler()
sampling_handler = SamplingCallbackHandler(arize_handler, sample_rate=0.1)
Comparing Observability Platforms
Choosing the right observability platform depends on your team's needs, budget, and existing infrastructure. Here is a quick comparison of the most popular options for LlamaIndex:
- Arize Phoenix: Open-source, local-first, excellent for development and small teams. Easy setup, great UI for inspecting individual traces. Best for getting started quickly.
- Langfuse: Open-source with cloud option. Strong support for user/session metadata, prompt management, and evaluation. Good for teams that need production features without heavy infrastructure.
- OpenLIT: Open-source, built on OpenTelemetry. Good for teams already invested in OTel. Provides GPU monitoring alongside LLM tracing.
- LangSmith: Commercial offering from LangChain. Works with LlamaIndex through OpenTelemetry export. Excellent evaluation and dataset management features.
- Custom OpenTelemetry: For organizations with existing OTel infrastructure (Jaeger, Tempo, Datadog, Honeycomb). Maximum flexibility but requires more setup effort.
Conclusion
Observability and tracing are not optional luxuries for production LLM applications — they are fundamental requirements. LlamaIndex's callback architecture makes it straightforward to instrument your applications with minimal code changes, and the ecosystem of observability platforms ensures there is a solution for every team and budget. By starting with a tool like Arize Phoenix during development, graduating to Langfuse or a full OpenTelemetry stack for production, and following best practices around data redaction, alerting, and evaluation tracing, you will be well-equipped to build reliable, performant, and cost-effective LLM applications. The investment you make in observability will pay dividends every time you debug an issue, optimize a prompt, or scale your application to serve more users.