Introduction to OpenTelemetry for LLMs
Large Language Models (LLMs) have become a foundational component of modern applications, powering chatbots, copilots, retrieval-augmented generation (RAG) pipelines, and autonomous agents. However, as these systems grow in complexity, so does the difficulty of understanding what is happening inside them. Traditional application monitoring tools were not designed to capture the nuances of model invocations, token usage, prompt construction, or multi-step reasoning chains.
OpenTelemetry (OTel) has emerged as the de facto open standard for observability, providing a vendor-neutral way to instrument, generate, collect, and export telemetry data such as traces, metrics, and logs. By applying OpenTelemetry to LLM workloads, developers gain a unified, standardized approach to observing AI systems alongside the rest of their infrastructure.
In this tutorial, you will learn what OpenTelemetry for LLMs entails, why it matters, how to implement it in practice, and which best practices to follow when instrumenting AI-powered applications.
What Is OpenTelemetry for LLMs?
OpenTelemetry for LLMs refers to the application of the OpenTelemetry specification and its instrumentation libraries to observe the behavior, performance, and cost of large language model interactions. This includes capturing spans for each model call, recording attributes such as prompt tokens, completion tokens, model names, temperature settings, and latency, and correlating these spans with broader distributed traces.
The OpenTelemetry community, through the GenAI semantic conventions working group, has been developing a standardized set of attributes specifically for generative AI workloads. These semantic conventions ensure that regardless of which provider you use — OpenAI, Anthropic, Hugging Face, or a self-hosted model — the telemetry data follows a consistent schema.
Key Telemetry Signals
- Traces: Distributed traces capture the end-to-end journey of a request through your application, including LLM calls, vector database queries, tool invocations, and downstream service calls.
- Metrics: Metrics aggregate quantitative data such as total token consumption, request counts, error rates, and average response latency over time intervals.
- Logs: Structured logs provide detailed records of individual events, including the full prompt and completion text (when appropriate), error messages, and system events.
GenAI Semantic Conventions
The GenAI semantic conventions define a standardized attribute namespace under the gen_ai prefix. Some of the most important attributes include:
gen_ai.system— The AI system or provider (e.g.,openai,anthropic).gen_ai.request.model— The model name being invoked (e.g.,gpt-4o).gen_ai.request.temperature— The sampling temperature.gen_ai.request.max_tokens— The maximum number of tokens to generate.gen_ai.usage.input_tokens— The number of tokens in the prompt.gen_ai.usage.output_tokens— The number of tokens in the completion.gen_ai.response.finish_reasons— Why the model stopped generating (e.g.,stop,length).
Why Observability Matters for LLM Applications
LLM applications introduce unique observability challenges that traditional web services do not face. Understanding these challenges clarifies why a standardized observability layer is essential.
Cost Management
Most LLM providers charge per token. Without granular visibility into token consumption at the request, user, or feature level, costs can spiral out of control. OpenTelemetry spans that record input and output token counts allow teams to build dashboards that attribute costs to specific features, tenants, or users.
Latency and Performance
LLM calls are inherently slow compared to traditional database queries or cache lookups. A single user request might trigger multiple model calls — for example, a RAG pipeline that embeds a query, retrieves documents, and then generates a response. Distributed tracing reveals where time is spent and helps identify bottlenecks such as slow vector searches or unnecessarily long prompts.
Quality and Reliability
LLMs are non-deterministic. The same prompt can produce different outputs on different runs, and models can hallucinate, refuse to answer, or return errors. By capturing the full request and response in spans (or linked logs), teams can debug unexpected behavior, evaluate output quality, and build feedback loops for improvement.
Security and Compliance
LLM applications often process sensitive user data. Observability tooling must capture enough context to be useful for debugging while respecting data privacy requirements. OpenTelemetry's attribute-based approach allows teams to selectively record or redact sensitive information.
Vendor Portability
Because OpenTelemetry is vendor-neutral, instrumenting your LLM application with OTel means you are not locked into a single observability backend. You can export traces to Jaeger, Tempo, Datadog, Honeycomb, New Relic, or any other OTLP-compatible backend, and switch providers without re-instrumenting your code.
Setting Up OpenTelemetry in a Python Application
Let's walk through a practical implementation. We will use Python, the most common language for LLM application development, and instrument an OpenAI call with OpenTelemetry.
Installing Dependencies
First, install the required packages. You will need the OpenTelemetry SDK, an exporter to send data to a backend, and the OpenAI client library.
pip install opentelemetry-api \
opentelemetry-sdk \
opentelemetry-exporter-otlp \
openai
Initializing the Tracer Provider
Before making any LLM calls, you must configure the OpenTelemetry SDK. This involves setting up a tracer provider, configuring a span processor, and registering an exporter. The example below uses the OTLP exporter, which sends data to any OTLP-compatible collector.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
# Define the service identity
resource = Resource.create({
"service.name": "llm-chatbot",
"service.version": "1.0.0",
"deployment.environment": "production",
})
# Set up the tracer provider
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(
endpoint="http://localhost:4318/v1/traces",
)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
# Register as the global tracer provider
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-chatbot")
Manually Instrumenting an LLM Call
With the tracer configured, you can now wrap LLM calls in spans. The following example creates a span for an OpenAI chat completion, populates it with GenAI semantic convention attributes, and records token usage from the response.
import os
from opentelemetry import trace
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
tracer = trace.get_tracer("llm-chatbot")
def chat_completion(user_message: str, model: str = "gpt-4o") -> str:
with tracer.start_as_current_span("chat.completion") as span:
# Record GenAI semantic convention attributes
span.set_attribute("gen_ai.system", "openai")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.request.temperature", 0.7)
span.set_attribute("gen_ai.request.max_tokens", 512)
try:
response = client.chat.completions.create(
model=model,
temperature=0.7,
max_tokens=512,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
],
)
# Record response attributes
span.set_attribute(
"gen_ai.usage.input_tokens",
response.usage.prompt_tokens
)
span.set_attribute(
"gen_ai.usage.output_tokens",
response.usage.completion_tokens
)
span.set_attribute(
"gen_ai.response.finish_reasons",
[response.choices[0].finish_reason]
)
span.set_attribute("gen_ai.response.id", response.id)
return response.choices[0].message.content
except Exception as e:
span.record_exception(e)
span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
raise
Capturing Prompt and Completion Content
For debugging and evaluation, you may want to record the actual prompt and completion text. The GenAI semantic conventions provide events for this purpose. You can attach events to the span to capture individual messages without polluting the span attributes themselves.
def chat_completion_with_events(user_message: str, model: str = "gpt-4o") -> str:
with tracer.start_as_current_span("chat.completion") as span:
span.set_attribute("gen_ai.system", "openai")
span.set_attribute("gen_ai.request.model", model)
# Record the user prompt as a span event
span.add_event("gen_ai.content.prompt", {
"gen_ai.prompt": user_message,
})
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
],
)
completion_text = response.choices[0].message.content
# Record the completion as a span event
span.add_event("gen_ai.content.completion", {
"gen_ai.completion": completion_text,
})
span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
return completion_text
Using Auto-Instrumentation Libraries
Manually instrumenting every LLM call is tedious and error-prone. Fortunately, the community has developed auto-instrumentation libraries that automatically wrap popular LLM SDKs with OpenTelemetry spans. One such library is opentelemetry-instrumentation-openai, part of the OpenLLMetry project by Traceloop, and the broader openinference instrumentation packages from Arize AI.
Instrumenting with OpenLLMetry
pip install traceloop-sdk
from traceloop.sdk import Traceloop
from traceloop.sdk.instruments import Instruments
from openai import OpenAI
# Initialize Traceloop with OpenTelemetry export
Traceloop.init(
app_name="llm-chatbot",
instruments=[Instruments.OPENAI],
exporter="otlp_http",
endpoint="http://localhost:4318/v1/traces",
)
client = OpenAI()
# This call is now automatically instrumented
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Explain quantum computing in one sentence."}],
)
print(response.choices[0].message.content)
With this setup, every OpenAI call automatically generates a span populated with GenAI semantic convention attributes, including model name, token usage, and finish reason. You do not need to modify your application logic at all.
Instrumenting with OpenInference
Arize AI's OpenInference project provides another approach, offering instrumentation packages that are fully compatible with OpenTelemetry and the GenAI semantic conventions.
pip install openinference-instrumentation-openai \
opentelemetry-exporter-otlp
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from openai import OpenAI
# Configure OpenTelemetry
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
)
)
# Auto-instrument the OpenAI client
OpenAIInstrumentor().instrument()
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is observability?"}],
)
print(response.choices[0].message.content)
Tracing Multi-Step LLM Workflows
Real LLM applications rarely make a single model call. A RAG pipeline, for example, involves embedding a query, searching a vector database, constructing a prompt, and generating a response. OpenTelemetry's distributed tracing model is ideal for representing these multi-step workflows as a tree of spans.
Instrumenting a RAG Pipeline
from opentelemetry import trace
tracer = trace.get_tracer("rag-pipeline")
def rag_query(question: str, vector_store, model: str = "gpt-4o") -> str:
with tracer.start_as_current_span("rag.query") as root_span:
root_span.set_attribute("gen_ai.system", "openai")
root_span.set_attribute("user.question", question)
# Step 1: Embed the query
with tracer.start_as_current_span("vector.embed_query") as embed_span:
embed_span.set_attribute("gen_ai.request.model", "text-embedding-3-small")
query_embedding = embed_text(question)
embed_span.set_attribute("embedding.dimensions", len(query_embedding))
# Step 2: Retrieve relevant documents
with tracer.start_as_current_span("vector.search") as search_span:
documents = vector_store.search(query_embedding, top_k=5)
search_span.set_attribute("vector.results_count", len(documents))
search_span.set_attribute("vector.top_k", 5)
# Step 3: Construct the augmented prompt
with tracer.start_as_current_span("prompt.construct") as prompt_span:
context = "\n\n".join([doc.text for doc in documents])
augmented_prompt = f"Context:\n{context}\n\nQuestion: {question}"
prompt_span.set_attribute("prompt.length_chars", len(augmented_prompt))
# Step 4: Generate the response
with tracer.start_as_current_span("chat.completion") as completion_span:
completion_span.set_attribute("gen_ai.request.model", model)
completion_span.set_attribute("gen_ai.request.temperature", 0.3)
response = client.chat.completions.create(
model=model,
temperature=0.3,
messages=[
{"role": "system", "content": "Answer based on the provided context."},
{"role": "user", "content": augmented_prompt},
],
)
completion_span.set_attribute(
"gen_ai.usage.input_tokens",
response.usage.prompt_tokens
)
completion_span.set_attribute(
"gen_ai.usage.output_tokens",
response.usage.completion_tokens
)
return response.choices[0].message.content
When you view this trace in a backend like Jaeger or Tempo, you will see a hierarchical view: the root rag.query span contains child spans for embedding, vector search, prompt construction, and the final LLM completion. This makes it immediately clear which step is the bottleneck.
Instrumenting LLM Agents
Agent-based architectures add another layer of complexity. An agent might make multiple LLM calls, invoke external tools, and loop through reasoning steps. Each of these actions should be captured as a span within a single trace, allowing you to reconstruct the agent's full decision-making process.
import json
from opentelemetry import trace
tracer = trace.get_tracer("agent")
def run_agent(user_task: str, max_iterations: int = 10) -> str:
with tracer.start_as_current_span("agent.run") as agent_span:
agent_span.set_attribute("agent.max_iterations", max_iterations)
agent_span.set_attribute("agent.task", user_task)
messages = [
{"role": "system", "content": "You are a helpful agent with access to tools."},
{"role": "user", "content": user_task},
]
for i in range(max_iterations):
with tracer.start_as_current_span("agent.iteration") as iter_span:
iter_span.set_attribute("agent.iteration_number", i)
# LLM decides the next action
with tracer.start_as_current_span("agent.llm_call") as llm_span:
llm_span.set_attribute("gen_ai.system", "openai")
llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=get_tool_definitions(),
)
llm_span.set_attribute(
"gen_ai.usage.input_tokens",
response.usage.prompt_tokens
)
llm_span.set_attribute(
"gen_ai.usage.output_tokens",
response.usage.completion_tokens
)
message = response.choices[0].message
messages.append(message)
# Check if the agent wants to call a tool
if message.tool_calls:
for tool_call in message.tool_calls:
with tracer.start_as_current_span("agent.tool_call") as tool_span:
tool_span.set_attribute("tool.name", tool_call.function.name)
tool_span.set_attribute(
"tool.arguments",
tool_call.function.arguments
)
try:
result = execute_tool(
tool_call.function.name,
json.loads(tool_call.function.arguments)
)
tool_span.set_attribute("tool.result", str(result))
except Exception as e:
tool_span.record_exception(e)
tool_span.set_status(
trace.Status(trace.StatusCode.ERROR, str(e))
)
result = f"Error: {e}"
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result),
})
else:
# No tool call means the agent is done
agent_span.set_attribute("agent.iterations_completed", i + 1)
return message.content
agent_span.set_attribute("agent.iterations_completed", max_iterations)
return "Max iterations reached without completion."
Exporting Telemetry to a Backend
Once your application is instrumented, you need to export the telemetry data to a backend for storage, visualization, and analysis. The most common approach is to run the OpenTelemetry Collector as an intermediary between your application and the backend.
Running the OpenTelemetry Collector
The Collector receives telemetry via OTLP, processes it (filtering, batching, attribute enrichment), and exports it to one or more backends. Here is a minimal Collector configuration:
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
timeout: 5s
send_batch_size: 1000
exporters:
# Export to Jaeger for trace visualization
otlp/jaeger:
endpoint: jaeger:4317
tls:
insecure: true
# Export to Prometheus for metrics
prometheus:
endpoint: 0.0.0.0:8889
# Export to stdout for debugging
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/jaeger, debug]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
You can run the Collector using Docker:
docker run -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
-p 4317:4317 -p 4318:4318 -p 8889:8889 \
otel/opentelemetry-collector-contrib:latest
Best Practices for LLM Observability with OpenTelemetry
1. Adopt Semantic Conventions Early
Use the GenAI semantic conventions from the start. Even if your initial implementation only uses a few attributes, following the standard naming scheme ensures compatibility with community tooling and makes it easier to adopt new conventions as they stabilize.
2. Be Deliberate About Recording Prompt Content
Prompt and completion text can contain sensitive user data. Avoid recording full prompt content in span attributes by default. Instead, use span events for content capture, and implement redaction logic for PII, API keys, or other secrets before they are exported. Consider making content recording a configurable feature that can be toggled per environment.
3. Use Consistent Service and Span Names
Name your spans consistently and hierarchically. Use dot notation (e.g., chat.completion, vector.search, agent.tool_call) to create clear groupings in trace visualizations. Set meaningful service.name and service.version resource attributes so you can filter traces by deployment.
4. Capture Cost-Relevant Attributes
Always record gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. These attributes are the foundation for cost analysis. You can build dashboards that multiply token counts by provider pricing to estimate spend per feature, per user, or per tenant.
5. Propagate Context Across Service Boundaries
If your LLM application spans multiple services (for example, a frontend API that calls a separate inference service), ensure that trace context is propagated across service boundaries. OpenTelemetry supports context propagation via W3C TraceContext headers, and most HTTP frameworks have auto-instrumentation that handles this automatically.
6. Use the BatchSpanProcessor for Production
In production, use the BatchSpanProcessor to batch and asynchronously export spans, minimizing the performance impact on your application. For development and debugging, the SimpleSpanProcessor exports spans synchronously, which can help you see data immediately but adds latency to each request.
7. Monitor Your Observability Pipeline
Instrumentation itself can fail. Monitor your OpenTelemetry exporter for dropped spans, connection errors, or export latency. The Collector exposes its own metrics that you can scrape with Prometheus to ensure your observability pipeline is healthy.
8. Correlate Traces with User Feedback
When users provide feedback (thumbs up/down, ratings, or corrections), attach that feedback to the corresponding trace. You can do this by recording a span event or by storing the trace ID alongside the feedback in your database. This creates a powerful dataset for evaluating and improving model performance over time.
9. Version Your Prompts and Models
Record the prompt template version and model version as span attributes. This allows you to compare the performance and quality of different prompt versions or model upgrades side by side in your observability backend.
10. Leverage Auto-Instrumentation Where Possible
Prefer auto-instrumentation libraries over manual instrumentation for standard operations. They reduce boilerplate, ensure consistency, and are maintained by the community to stay current with semantic convention changes. Reserve manual instrumentation for custom logic that auto-instrumentation cannot cover.
Conclusion
OpenTelemetry provides a powerful, vendor-neutral foundation for observing LLM applications. By adopting the GenAI semantic conventions, instrumenting both individual model calls and multi-step workflows, and exporting telemetry to a backend through the OpenTelemetry Collector, teams gain deep visibility into the performance, cost, and quality of their AI systems. As the LLM ecosystem continues to evolve, standardized observability will be essential for building reliable, cost-effective, and trustworthy AI applications. Start with the practices outlined in this tutorial, iterate on your instrumentation as your application grows, and you will be well-positioned to operate LLM workloads with the same rigor and confidence as any other critical service in your infrastructure.