Phoenix Arize: Visualizing LLM Traces and Evaluations

Introduction to Phoenix Arize

Phoenix, developed by Arize AI, is an open-source observability and evaluation library designed specifically for Large Language Model (LLM) applications. As LLM-powered applications grow in complexity—often chaining multiple prompts, tool calls, retrieval steps, and reasoning passes—understanding what happens inside each request becomes critical. Phoenix gives developers a local, self-hosted UI to visualize LLM traces, inspect spans, and run evaluations on datasets of responses.

Unlike black-box monitoring tools, Phoenix integrates directly into your application code using OpenTelemetry-based instrumentation. This means every prompt, every model call, every retrieval, and every tool invocation can be captured as a structured trace. You can then inspect these traces in a browser-based interface, replay them, and evaluate their quality using both programmatic and LLM-as-a-judge techniques.

Why LLM Tracing Matters

Traditional application monitoring focuses on latency, error rates, and throughput. LLM applications require a deeper level of introspection because the behavior of the system is non-deterministic and heavily dependent on prompt content, retrieved context, and model reasoning. A request might return a 200 status code but still produce a hallucinated, irrelevant, or unsafe answer.

Key Problems Phoenix Solves

• Debugging unexpected outputs: Trace the exact prompt, context, and model parameters that produced a given response.
• Latency attribution: Identify which span in a multi-step chain (retrieval, embedding, generation) is the bottleneck.
• Evaluation at scale: Run evaluators across datasets of traces to measure accuracy, relevance, toxicity, and more.
• Cost tracking: Capture token counts and estimate cost per request.
• Experiment comparison: Compare traces from different prompts, models, or configurations side by side.

Installation and Setup

Phoenix is distributed as a Python package and also offers a TypeScript/JavaScript client for frontend and Node.js applications. The simplest way to get started is with pip.

pip install arize-phoenix openinference-instrumentation-openai openai

The openinference-instrumentation-openai package provides automatic instrumentation for the OpenAI client. Phoenix supports instrumentations for many providers including Anthropic, Cohere, Bedrock, LangChain, LlamaIndex, and others.

Starting the Phoenix Server

Phoenix can run as an in-process server within your Python application, or as a standalone Docker container. For development, the in-process approach is simplest.

import phoenix as px

# Start the Phoenix server in the background
px.launch_app()

# This prints a local URL, typically http://localhost:6006

Once running, you can open the URL in your browser to access the trace UI. The server persists traces in a local SQLite database by default, so traces remain available across restarts.

Instrumenting Your LLM Application

Instrumentation is the process of wrapping your LLM calls so that Phoenix can capture structured trace data. The OpenInference project provides auto-instrumentation packages that require minimal code changes.

Auto-Instrumenting OpenAI Calls

import phoenix as px
from openinference.instrumentation.openai import OpenAIInstrumentor
from openai import OpenAI
import os

# Start Phoenix
px.launch_app()

# Instrument the OpenAI client globally
OpenAIInstrumentor().instrument()

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain what a decorator is in Python."}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)

After running this script, open the Phoenix UI and you will see a trace representing the OpenAI call. The trace includes the full prompt, the response, token counts, latency, and model metadata.

Manual Span Creation

For custom application logic that is not covered by auto-instrumentation, you can create spans manually using OpenTelemetry. This is useful for wrapping retrieval steps, custom tool calls, or business logic.

from openinference.semconv.trace import SpanAttributes
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def retrieve_documents(query: str) -> list[str]:
    with tracer.start_as_current_span("retrieve_documents") as span:
        span.set_attribute(SpanAttributes.OPENINFERENCE_SPAN_KIND, "RETRIEVER")
        span.set_attribute(SpanAttributes.INPUT_VALUE, query)
        
        # Simulated retrieval logic
        documents = [
            "Python decorators wrap functions to modify behavior.",
            "Use @decorator_name syntax above a function definition."
        ]
        
        span.set_attribute(SpanAttributes.OUTPUT_VALUE, str(documents))
        return documents

def generate_answer(query: str, context: list[str]) -> str:
    with tracer.start_as_current_span("generate_answer") as span:
        span.set_attribute(SpanAttributes.OPENINFERENCE_SPAN_KIND, "LLM")
        span.set_attribute(SpanAttributes.INPUT_VALUE, query)
        
        prompt = f"Context: {context}\n\nQuestion: {query}\nAnswer:"
        # Call your LLM here
        answer = "A decorator is a callable that takes a function and returns a modified function."
        
        span.set_attribute(SpanAttributes.OUTPUT_VALUE, answer)
        return answer

def rag_pipeline(query: str) -> str:
    with tracer.start_as_current_span("rag_pipeline") as span:
        span.set_attribute(SpanAttributes.OPENINFERENCE_SPAN_KIND, "CHAIN")
        span.set_attribute(SpanAttributes.INPUT_VALUE, query)
        
        context = retrieve_documents(query)
        answer = generate_answer(query, context)
        
        span.set_attribute(SpanAttributes.OUTPUT_VALUE, answer)
        return answer

Each manually created span is nested under its parent, producing a hierarchical trace that mirrors your application's execution flow. The OPENINFERENCE_SPAN_KIND attribute tells Phoenix how to categorize and render the span in the UI.

Understanding the Trace UI

The Phoenix UI provides several views for inspecting traces. Understanding these views helps you navigate and debug effectively.

Projects View

Traces are organized into projects. By default, all traces go into a project called default. You can create separate projects for different environments (development, staging, production) or different features of your application.

Traces List

This view shows a table of all traces in the selected project. Each row includes the trace name, latency, token count, status, and timestamp. You can filter by status code, span kind, or custom attributes. Clicking a trace opens the detailed trace view.

Trace Detail View

The trace detail view is the most powerful part of Phoenix. It displays a waterfall timeline of all spans within the trace, showing their nesting relationships and durations. Below the timeline, each span can be expanded to reveal its full input, output, attributes, and events. This is where you inspect the exact prompts and responses that flowed through your system.

Evaluating LLM Outputs

Visualization is only half the story. Phoenix also provides an evaluation framework that lets you score traces programmatically. Evaluations help you answer questions like: "Is this response relevant to the question?" or "Does this response contain hallucinated information?"

Built-in Evaluators

Phoenix ships with several evaluators that use LLM-as-a-judge techniques. These evaluators send the trace data to an LLM and ask it to score the response based on a rubric.

import phoenix as px
from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents
from phoenix.evals import (
    HallucinationEvaluator,
    QAEvaluator,
    RelevanceEvaluator,
    OpenAIModel
)

# Connect to a running Phoenix instance
client = px.Client()

# Define the model used for evaluation
eval_model = OpenAIModel(model="gpt-4o")

# Create evaluators
hallucination_evaluator = HallucinationEvaluator(eval_model)
qa_evaluator = QAEvaluator(eval_model)
relevance_evaluator = RelevanceEvaluator(eval_model)

# Get data from Phoenix spans
# For a QA pipeline, extract query-reference pairs
qa_dataframe = get_qa_with_reference(client, project_name="default")

# For retrieval, extract query-document pairs
retrieved_dataframe = get_retrieved_documents(client, project_name="default")

# Run hallucination evaluation
hallucination_evals = hallucination_evaluator.evaluate(qa_dataframe)
print(hallucination_evals.head())

# Run QA correctness evaluation
qa_evals = qa_evaluator.evaluate(qa_dataframe)
print(qa_evals.head())

# Run relevance evaluation on retrieved documents
relevance_evals = relevance_evaluator.evaluate(retrieved_dataframe)
print(relevance_evals.head())

Each evaluator returns a dataframe with columns for the score, explanation, and metadata. The scores are then uploaded back to Phoenix and displayed alongside the corresponding traces in the UI.

Uploading Evaluations to Phoenix

from phoenix.trace import SpanEvaluations

# Create SpanEvaluations objects from the evaluator output
hallucination_span_evals = SpanEvaluations(
    eval_name="hallucination",
    dataframe=hallucination_evals
)

qa_span_evals = SpanEvaluations(
    eval_name="qa_correctness",
    dataframe=qa_evals
)

# Upload to Phoenix
client.log_evaluations(hallucination_span_evals, qa_span_evals)

Once uploaded, the evaluations appear in the trace detail view. You can filter traces by evaluation score, making it easy to find traces where the model hallucinated or provided incorrect answers.

Custom Evaluators

Beyond the built-in evaluators, you can define custom evaluation logic. This is useful for domain-specific checks such as compliance, format validation, or business rules.

import pandas as pd
from phoenix.evals import llm_classify

# Define a custom template for checking response tone
tone_template = """
You are an evaluation assistant. Determine if the following response
maintains a professional and helpful tone.

Input Question: {question}
Response: {response}

Respond with only one word: "professional" or "unprofessional".
"""

# Define the classification labels
tone_labels = ["professional", "unprofessional"]

# Prepare your dataframe with question and response columns
df = pd.DataFrame({
    "question": ["How do I reset my password?"],
    "response": ["Just go click the thing and figure it out, it's not hard."]
})

# Run the custom classification
tone_results = llm_classify(
    dataframe=df,
    template=tone_template,
    model=OpenAIModel(model="gpt-4o"),
    labels=tone_labels
)

print(tone_results)

Working with Datasets

Phoenix lets you curate datasets from production traces. This is valuable for building evaluation sets, regression tests, and fine-tuning datasets from real-world data.

from phoenix import Client

client = Client()

# Create a dataset from a filtered set of traces
# First, export traces matching certain criteria
spans_df = client.get_spans_dataframe(project_name="default")

# Filter for successful LLM spans with non-empty responses
filtered = spans_df[
    (spans_df["status_code"] == "OK") &
    (spans_df["output.value"].notna())
]

# Create a dataset from the filtered spans
dataset = client.upload_dataset(
    dataset_name="production_qa_examples",
    dataframe=filtered,
    input_keys=["input.value"],
    output_keys=["output.value"]
)

print(f"Created dataset with {len(filtered)} examples")

Datasets can then be used to run experiments, compare model versions, or serve as golden test sets for CI/CD pipelines.

Running Experiments

Experiments let you test different prompts, models, or configurations against a dataset and compare results in the Phoenix UI.

from phoenix.experiments import run_experiment
from phoenix.client import Client

client = Client()

# Load a dataset
dataset = client.get_dataset(name="production_qa_examples")

# Define a task function that processes each example
def my_task(example):
    # Your LLM call here
    from openai import OpenAI
    openai_client = OpenAI()
    
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer concisely."},
            {"role": "user", "content": example["input"]["value"]}
        ]
    )
    return response.choices[0].message.content

# Define an evaluator for experiment results
def correctness_eval(output, expected):
    return 1.0 if output.strip() == expected.strip() else 0.0

# Run the experiment
experiment = run_experiment(
    dataset,
    my_task,
    evaluators=[correctness_eval],
    experiment_name="gpt-4o-baseline"
)

Each experiment run is tracked in Phoenix, and you can compare multiple experiments side by side to see which configuration performs best on your evaluation set.

Best Practices

Structure Your Spans Meaningfully

Use appropriate span kinds (CHAIN, LLM, RETRIEVER, TOOL, EMBEDDING) so Phoenix can render them correctly. Give spans descriptive names that reflect the business logic, not just the function name.

Capture Rich Attributes

Beyond inputs and outputs, set custom attributes for metadata that matters to your application: user IDs, session IDs, feature flags, model versions, and prompt template versions. These attributes become filterable dimensions in the UI.

from openinference.semconv.trace import SpanAttributes

with tracer.start_as_current_span("chat_completion") as span:
    span.set_attribute(SpanAttributes.OPENINFERENCE_SPAN_KIND, "LLM")
    span.set_attribute(SpanAttributes.LLM_MODEL_NAME, "gpt-4o")
    span.set_attribute(SpanAttributes.LLM_TOKEN_COUNT_PROMPT, prompt_tokens)
    span.set_attribute("user.id", user_id)
    span.set_attribute("feature.flag_version", "v2.1")
    span.set_attribute("prompt.template_version", "v3")

Use Projects to Separate Environments

Do not mix development, staging, and production traces in the same project. Use separate projects to keep your evaluation datasets clean and your dashboards focused.

Run Evaluations Asynchronously

LLM-as-a-judge evaluations are expensive and slow. Run them as a background job rather than inline with user requests. Schedule evaluation runs on a nightly batch or trigger them from a CI pipeline.

Version Your Prompts

Tag every trace with the prompt template version used. This lets you correlate quality regressions with prompt changes and roll back confidently.

Set Retention Policies

Trace data grows quickly. For production deployments, configure retention policies to prune old traces and keep storage costs manageable. Phoenix supports exporting traces to object storage for long-term archival.

Deploying Phoenix in Production

For team use, run Phoenix as a containerized service rather than in-process. This allows multiple developers and services to send traces to a shared instance.

# Run Phoenix with Docker
docker run -p 6006:6006 \
  -v /path/to/data:/data \
  arizephoenix/phoenix:latest

For larger deployments, Phoenix supports PostgreSQL as a backend instead of SQLite, and can be deployed on Kubernetes with horizontal scaling. Configure environment variables for authentication, storage, and telemetry export as needed.

# Environment variables for production configuration
PHOENIX_SQL_DATABASE_URL=postgresql://user:pass@db:5432/phoenix
PHOENIX_ENABLE_AUTH=true
PHOENIX_SECRET_KEY=your-secret-key
PHOENIX_WORKING_DIR=/data

Conclusion

Phoenix by Arize provides a practical, open-source foundation for observing and evaluating LLM applications. By instrumenting your code with OpenInference spans, you gain visibility into every prompt, retrieval, and model call. The evaluation framework lets you move beyond manual inspection to systematic quality measurement using LLM-as-a-judge and custom evaluators. Combined with datasets and experiments, Phoenix enables an iterative development workflow where you can confidently change prompts, swap models, and ship improvements backed by data. Whether you are debugging a single failing request or running regression tests across thousands of traces, Phoenix gives you the tools to understand and improve your LLM application with precision.