Datadog for LLMs: Setting Up AI Monitoring

As large language models (LLMs) move from experimental prototypes into production systems, observability becomes a critical concern. Latency spikes, hallucinations, prompt injection attempts, and runaway token costs can all erode user trust and inflate budgets. Datadog's LLM Observability product gives engineering teams a unified view of model performance, quality, cost, and safety across their AI stack. This tutorial walks through what LLM monitoring in Datadog is, why it matters, and how to instrument your applications end to end.

What Is Datadog LLM Observability?

Datadog LLM Observability is a specialized layer within the Datadog platform designed to trace, evaluate, and monitor generative AI workloads. It captures every LLM call as a span within a trace, enriches it with token usage, cost estimates, prompt and completion payloads, and quality evaluations, and then surfaces anomalies through dashboards, monitors, and out-of-the-box views.

Unlike generic APM, LLM Observability understands the semantics of AI calls. It distinguishes between embeddings, chat completions, tool calls, and retrieval steps. It also supports quality evaluations — both rule-based and LLM-as-a-judge — so you can track metrics like answer relevance, faithfulness, and toxicity over time.

Why It Matters

Cost control: Token usage can balloon quickly with long contexts or chatty agents. Datadog attributes token spend per user, feature, or model.
Quality assurance: Production prompts drift. Evaluations catch regressions before users do.
Latency debugging: A single RAG query may chain multiple model calls. Distributed tracing pinpoints the slow link.
Security: PII detection and prompt injection flags help meet compliance requirements.
Vendor portability: Compare OpenAI, Anthropic, and self-hosted models side by side to inform routing decisions.

Prerequisites

A Datadog account with LLM Observability enabled (add the LLM Observability entitlement in your org settings).
A Datadog API key and application key.
Python 3.9 or newer (this tutorial uses Python, but Node.js and Java SDKs are also available).
An LLM provider account — here we use OpenAI, but the patterns apply broadly.

Step 1: Install the SDK

Datadog provides the ddtrace library and the dedicated llmobs module. Install both, along with the OpenAI client:

pip install ddtrace openai datadog-api-client

Verify the installation:

python -c "import ddtrace; from ddtrace.llmobs import LLMObs; print('OK')"

Step 2: Configure Environment Variables

The tracer reads configuration from environment variables. Set these in your shell or your deployment's env configuration:

export DD_API_KEY="your-datadog-api-key"
export DD_APP_KEY="your-datadog-app-key"
export DD_SITE="datadoghq.com"   # use eu, us3, us5, or ap1 if applicable
export DD_ENV="production"
export DD_SERVICE="support-agent"
export DD_VERSION="1.4.0"
export DD_LLMOBS_ENABLED=1
export DD_LLMOBS_ML_APP="customer-support-bot"

The DD_LLMOBS_ML_APP variable groups your traces into a logical application inside the LLM Observability UI. Choose a name that reflects the product surface, not the model.

Step 3: Initialize the Tracer in Code

While environment variables enable the integration, initializing LLMObs explicitly gives you control over evaluation and span decoration:

from ddtrace.llmobs import LLMObs
import openai
import os

# Enable automatic instrumentation for supported providers
LLMObs.enable(
    ml_app=os.environ["DD_LLMOBS_ML_APP"],
    agentless_enabled=True,
)

openai.api_key = os.environ["OPENAI_API_KEY"]

With agentless_enabled=True, traces ship directly to Datadog's intake without requiring the Datadog Agent on the host. This is ideal for serverless deployments. If you already run the Agent, leave it as False to route traffic locally.

Step 4: Make a Traced LLM Call

The OpenAI integration is patched automatically once LLMObs.enable() runs. A standard chat completion becomes a fully traced span:

from openai import OpenAI

client = OpenAI()

def answer_question(question: str, context: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a helpful support agent. Use only the provided context."},
            {"role": "user", "content": f"Context: {context}\n\nQuestion: {question}"},
        ],
        temperature=0.2,
        max_tokens=512,
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    ctx = "Datadog LLM Observability supports OpenAI, Anthropic, and LangChain integrations."
    print(answer_question("Does Datadog support LangChain?", ctx))

Run the script and navigate to LLM Observability → Traces in the Datadog UI. You should see a span tagged with the model name, input/output text, token counts, and an estimated cost.

Step 5: Trace Multi-Step RAG Pipelines

Real applications rarely make a single call. A retrieval-augmented generation flow might embed a query, search a vector store, then generate an answer. Use manual spans to capture the full pipeline:

from ddtrace.llmobs import LLMObs
import time

def rag_pipeline(question: str, vector_store) -> str:
    with LLMObs.span(kind="workflow", name="rag.query") as span:
        span.set_tag_str("user_question", question)

        # Step 1: Embedding
        with LLMObs.span(kind="embedding", name="embed_query"):
            query_embedding = embed(question)

        # Step 2: Retrieval
        with LLMObs.span(kind="retrieval", name="vector_search") as ret:
            docs = vector_store.search(query_embedding, k=4)
            ret.set_tag_str("retrieved_count", str(len(docs)))
            for i, doc in enumerate(docs):
                ret.set_tag_str(f"doc.{i}.source", doc.metadata.get("source", "unknown"))

        # Step 3: Generation
        context = "\n".join(d.text for d in docs)
        answer = answer_question(question, context)

        # Attach the final answer for evaluation
        span.set_tag_str("final_answer", answer)
        return answer

The kind parameter controls how Datadog categorizes the span: workflow, task, llm, embedding, retrieval, or tool. Using the correct kind ensures your spans appear in the right dashboards and that aggregate metrics are computed correctly.

Step 6: Add Quality Evaluations

Tracing tells you what happened; evaluations tell you how good it was. Datadog supports two evaluation types: custom (rule-based or heuristic) and LLM-as-a-judge. Both are attached to spans via the LLMObs.evaluate API.

from ddtrace.llmobs import LLMObs

def evaluate_response(span, question: str, answer: str, context: str):
    # Rule-based: check the answer is grounded in context
    grounded = any(sentence.strip() in context for sentence in answer.split(".") if len(sentence) > 20)
    LLMObs.evaluate(
        span=span,
        label="groundedness",
        metric_type="categorical",
        value="pass" if grounded else "fail",
    )

    # Heuristic: response length sanity check
    LLMObs.evaluate(
        span=span,
        label="response_length",
        metric_type="score",
        value=min(len(answer) / 1000.0, 1.0),
    )

For LLM-as-a-judge, call a second model to score the primary response, then attach the result:

def llm_judge_relevance(question: str, answer: str) -> float:
    judge_prompt = f"""Rate the relevance of the answer to the question on a scale of 0.0 to 1.0.
    Question: {question}
    Answer: {answer}
    Respond with only a number."""
    result = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": judge_prompt}],
    )
    return float(result.choices[0].message.content.strip())

# Inside your workflow span:
score = llm_judge_relevance(question, answer)
LLMObs.evaluate(
    span=span,
    label="relevance",
    metric_type="score",
    value=score,
)

Evaluations appear as columns in the trace table and can power monitor alerts — for example, notify on-call when the rolling average relevance score drops below 0.7.

Step 7: Build Dashboards and Monitors

Once traces flow in, create a dashboard to track the metrics that matter. Recommended widgets include:

Token cost per request (sum of input + output tokens × model price).
p95 latency for the rag.query workflow span.
Error rate by model.
Average relevance and groundedness scores over time.
Top retrieved document sources by frequency.

To alert on cost anomalies, create a monitor using the LLM Observability metrics namespace:

# Monitor: alert when hourly token spend exceeds $50
@llmobs.token.cost{env:production,ml_app:customer-support-bot}.sum_over(hour) > 50

For quality regressions, use an evaluation-based monitor:

# Monitor: alert when average relevance drops below 0.7 over 15 minutes
avg(last_15m):avg:llmobs.eval.relevance{env:production} < 0.7

Best Practices

Tag everything: Add user_id, session_id, feature, and experiment_version tags to spans. They unlock per-cohort analysis and A/B comparisons.
Sample judiciously: LLM-as-a-judge evaluations are expensive. Evaluate a random 10–20% of production traffic rather than every call.
Scrub PII: Enable the DD_LLMOBS_SANITIZE_PII option or use a custom processor to redact sensitive fields before payloads leave your network.
Version your prompts: Store prompt templates in source control and tag spans with a prompt_version. When quality drops, you can correlate it to a specific change.
Set token budgets: Use max_tokens and context truncation aggressively. Monitor the tokens.input metric to detect context bloat.
Separate dev and prod ML apps: Use distinct DD_LLMOBS_ML_APP values per environment so experiments don't pollute production dashboards.
Monitor fallback paths: If your app falls back from GPT-4o to GPT-4o-mini on rate limits, tag the span with fallback: true so you can track how often it happens.

Conclusion

Datadog LLM Observability brings the same rigor to generative AI that traditional APM brought to microservices. By instrumenting every model call, chaining spans across RAG pipelines, attaching quality evaluations, and alerting on cost and relevance, you turn a opaque LLM into a measurable, debuggable, and trustworthy production component. Start with a single high-traffic endpoint, get comfortable with the trace and evaluation workflow, then expand coverage to your full AI surface area. The investment pays off the first time a dashboard catches a prompt regression before a customer ever notices.