← Back to DevBytes
Developer Tutorial August 17, 2026 12 min read

Observability and Tracing with vLLM: Complete Guide

Introduction to Observability and Tracing with vLLM

vLLM is a high-throughput, memory-efficient inference engine for Large Language Models (LLMs). While it excels at serving models with impressive performance, running it in production requires deep visibility into its behavior. Observability — the combination of metrics, logs, and traces — gives you the ability to understand request flow, diagnose bottlenecks, and ensure reliability at scale.

This guide walks through everything you need to know about instrumenting vLLM for observability: from enabling built-in metrics endpoints to integrating distributed tracing with OpenTelemetry, exporting to backends like Jaeger, Prometheus, and Grafana, and applying best practices for production deployments.

Why Observability Matters for LLM Serving

LLM inference differs from traditional web services in several ways. Requests are long-lived, token generation is streaming, GPU memory is a scarce resource, and batch scheduling introduces complex timing dynamics. Without observability, you are flying blind when issues arise.

Key Questions Observability Answers

Tracing is especially valuable because a single LLM request passes through multiple stages: tokenization, scheduling, prefill, decode, and detokenization. A trace lets you see the time spent in each stage, making it possible to pinpoint exactly where latency is introduced.

vLLM's Built-in Observability Features

vLLM ships with several observability features out of the box. You do not need to write custom instrumentation to get started — you just need to enable and configure them.

Metrics Endpoint

vLLM exposes a Prometheus-compatible metrics endpoint when you start the server. By default, this is available at /metrics on the same port as the API server.

# Start vLLM with metrics enabled (enabled by default)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-chat-hf \
  --port 8000

# In another terminal, scrape the metrics
curl http://localhost:8000/metrics

The metrics endpoint exposes a rich set of counters and histograms. Some of the most important ones include:

Disabling or Customizing Metrics

If you want to disable metrics for testing or reduce overhead, you can pass the --disable-log-stats flag. You can also configure the metrics namespace to avoid collisions when running multiple vLLM instances behind the same scrape target.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-chat-hf \
  --port 8000 \
  --disable-log-stats

Setting Up Prometheus and Grafana

The metrics endpoint is only useful if you scrape and visualize it. Prometheus collects the metrics, and Grafana displays them on dashboards.

Prometheus Configuration

Create a prometheus.yml file that tells Prometheus to scrape your vLLM instance:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'vllm'
    scrape_interval: 5s
    static_configs:
      - targets: ['vllm-host:8000']
        labels:
          model: 'llama-2-7b'
          instance: 'vllm-prod-1'

Start Prometheus with this configuration:

prometheus --config.file=prometheus.yml --storage.tsdb.path=/prometheus

Useful PromQL Queries

Once Prometheus is scraping vLLM, you can write queries to monitor performance. Here are some practical examples:

# Average time-to-first-token over the last 5 minutes
histogram_quantile(0.95, 
  sum(rate(vllm_time_to_first_token_seconds_bucket[5m])) by (le)
)

# Current GPU cache usage percentage
vllm_gpu_cache_usage_perc

# Requests waiting in queue (instantaneous)
vllm_num_requests_waiting

# Preemption rate per minute
rate(vllm_num_preemption[1m])

# Tokens generated per second across all requests
sum(rate(vllm_request_success_total[1m]))

Grafana Dashboard Setup

Point Grafana at your Prometheus data source and create panels using the queries above. A well-structured dashboard for vLLM typically includes:

Distributed Tracing with OpenTelemetry

While metrics tell you what is happening in aggregate, traces tell you why a specific request was slow. vLLM supports OpenTelemetry (OTel) tracing, which allows you to instrument the full request lifecycle and export spans to any OTel-compatible backend.

How vLLM Tracing Works

When tracing is enabled, vLLM creates spans for the major phases of request processing. Each span captures timing, attributes, and status. The spans are nested, forming a trace tree that shows the parent-child relationship between operations.

A typical trace for a single completion request looks like this:

Enabling Tracing

vLLM uses the OpenTelemetry Python SDK. You need to install the required packages and configure an exporter before starting the server.

pip install opentelemetry-api \
            opentelemetry-sdk \
            opentelemetry-exporter-otlp \
            opentelemetry-instrumentation

Next, create a tracing setup script that configures the OTel SDK and points it at your collector:

# tracing_setup.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

def setup_tracing(service_name="vllm", endpoint="http://localhost:4317"):
    resource = Resource.create({
        "service.name": service_name,
        "service.version": "1.0.0",
    })

    provider = TracerProvider(resource=resource)
    exporter = OTLPSpanExporter(endpoint=endpoint, insecure=True)
    processor = BatchSpanProcessor(exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)
    return provider

Now start vLLM with tracing enabled. vLLM accepts the --otlp-traces-endpoint flag to configure where spans are sent:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-chat-hf \
  --port 8000 \
  --otlp-traces-endpoint http://localhost:4317

This flag tells vLLM to export traces via OTLP gRPC to the specified endpoint. You can also use HTTP by specifying the HTTP port (typically 4318).

Running Jaeger as a Trace Backend

Jaeger is a popular open-source distributed tracing backend. The easiest way to run it is with Docker, using the all-in-one image that includes both the collector and the UI:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Once Jaeger is running and vLLM is exporting traces, open http://localhost:16686 to browse traces. You will see a service named vllm (or whatever you configured as the service name). Clicking on a trace reveals the full span tree with timing breakdowns.

Running Everything with Docker Compose

For a complete local observability stack, use Docker Compose to run vLLM, Prometheus, Jaeger, and Grafana together:

version: '3.8'

services:
  vllm:
    image: vllm/vllm-openai:latest
    ports:
      - "8000:8000"
    command:
      - --model=meta-llama/Llama-2-7b-chat-hf
      - --otlp-traces-endpoint=http://jaeger:4317
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "4317:4317"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Run the stack with:

docker-compose up -d

You now have vLLM serving requests, Prometheus scraping metrics at http://localhost:9090, Jaeger collecting traces at http://localhost:16686, and Grafana ready for dashboards at http://localhost:3000.

Adding Custom Tracing to Your Application

If you are calling vLLM from your own application, you should propagate trace context so that spans from your application and vLLM appear in the same trace. This is called distributed tracing, and it works by passing trace context headers between services.

Instrumenting a FastAPI Client

Here is an example of a FastAPI application that calls vLLM and propagates trace context:

# app.py
from fastapi import FastAPI
import httpx
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Set up tracing
resource = Resource.create({"service.name": "my-llm-app"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()

@app.post("/chat")
async def chat(prompt: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:8000/v1/completions",
            json={
                "model": "meta-llama/Llama-2-7b-chat-hf",
                "prompt": prompt,
                "max_tokens": 100,
            },
        )
    return response.json()

The key here is the HTTPXClientInstrumentor, which automatically injects W3C trace context headers into outgoing HTTP requests. When vLLM receives these headers, it links its spans to the parent span from your application. In Jaeger, you will see a single trace that spans both services.

Adding Custom Spans

You can add custom spans to mark important business logic around your LLM calls:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/chat")
async def chat(prompt: str):
    with tracer.start_as_current_span("process_user_request") as span:
        span.set_attribute("user.prompt_length", len(prompt))
        span.set_attribute("user.prompt", prompt[:200])  # truncate for safety
        
        with tracer.start_as_current_span("validate_prompt"):
            if len(prompt) > 10000:
                span.set_status(trace.Status(trace.StatusCode.ERROR))
                return {"error": "Prompt too long"}
        
        with tracer.start_as_current_span("call_vllm"):
            async with httpx.AsyncClient() as client:
                response = await client.post(
                    "http://localhost:8000/v1/completions",
                    json={
                        "model": "meta-llama/Llama-2-7b-chat-hf",
                        "prompt": prompt,
                        "max_tokens": 100,
                    },
                )
        
        with tracer.start_as_current_span("post_process"):
            result = response.json()
            span.set_attribute("response.token_count", 
                             len(result["choices"][0]["text"].split()))
        
        return result

Structured Logging

Traces and metrics are powerful, but logs remain essential for debugging. vLLM uses Python's standard logging module, and you can configure it to output structured JSON logs that are easier to parse and search in tools like Elasticsearch or Loki.

# logging_config.py
import logging
import json
import sys
from datetime import datetime, timezone

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "module": record.module,
            "line": record.lineno,
        }
        if record.exc_info:
            log_entry["exception"] = self.formatException(record.exc_info)
        return json.dumps(log_entry)

def setup_logging():
    handler = logging.StreamHandler(sys.stdout)
    handler.setFormatter(JSONFormatter())
    
    root_logger = logging.getLogger()
    root_logger.setLevel(logging.INFO)
    root_logger.addHandler(handler)
    
    # Also configure vLLM's loggers
    for name in ["vllm", "vllm.engine", "vllm.core", "vllm.transformers_utils"]:
        logger = logging.getLogger(name)
        logger.setLevel(logging.INFO)
        logger.handlers = [handler]

Import and call setup_logging() before starting vLLM programmatically, or configure it in your entrypoint script. The JSON output makes it trivial to ingest logs into a centralized logging system.

Monitoring Streaming Responses

Streaming responses present a unique observability challenge because a single request produces multiple chunks over time. vLLM handles this internally, but you should instrument your client code to track streaming metrics.

import time
import httpx
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def stream_completion(prompt: str):
    with tracer.start_as_current_span("stream_completion") as span:
        first_token_time = None
        token_count = 0
        start_time = time.time()
        
        async with httpx.AsyncClient() as client:
            async with client.stream(
                "POST",
                "http://localhost:8000/v1/completions",
                json={
                    "model": "meta-llama/Llama-2-7b-chat-hf",
                    "prompt": prompt,
                    "max_tokens": 200,
                    "stream": True,
                },
            ) as response:
                async for line in response.aiter_lines():
                    if line.startswith("data: ") and line != "data: [DONE]":
                        if first_token_time is None:
                            first_token_time = time.time()
                            span.add_event("first_token_received", {
                                "ttft_ms": (first_token_time - start_time) * 1000
                            })
                        token_count += 1
        
        total_time = time.time() - start_time
        span.set_attribute("total_tokens", token_count)
        span.set_attribute("total_time_s", total_time)
        span.set_attribute("tokens_per_second", token_count / total_time if total_time > 0 else 0)
        if first_token_time:
            span.set_attribute("time_to_first_token_ms", 
                             (first_token_time - start_time) * 1000)

This code records time-to-first-token as a span event and captures throughput as span attributes. These values will be visible in Jaeger alongside vLLM's internal spans.

Alerting Strategies

Observability is not complete without alerting. You should define alerts that notify you before users experience problems. Here are key alerting rules for vLLM using Prometheus alertmanager:

# alerts.yml
groups:
  - name: vllm
    rules:
      - alert: HighTimeToFirstToken
        expr: histogram_quantile(0.95, sum(rate(vllm_time_to_first_token_seconds_bucket[5m])) by (le)) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "vLLM p95 time-to-first-token is above 5 seconds"

      - alert: HighQueueDepth
        expr: vllm_num_requests_waiting > 50
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "vLLM has {{ $value }} requests waiting in queue"

      - alert: GPUCacheNearFull
        expr: vllm_gpu_cache_usage_perc > 0.95
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "vLLM GPU cache usage is above 95%"

      - alert: HighPreemptionRate
        expr: rate(vllm_num_preemption[5m]) > 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "vLLM is preempting requests at a high rate"

      - alert: HighErrorRate
        expr: rate(vllm_request_success_total{finished_reason="error}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "vLLM error rate is above 5%"

Best Practices

1. Use Sampling Wisely

Tracing every single request can overwhelm your tracing backend and add overhead. Use head-based or tail-based sampling to collect a representative subset. For production, a sampling rate of 1-10% is often sufficient. However, always trace errors and slow requests at 100%.

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased

# Sample 10% of traces
sampler = ParentBased(TraceIdRatioBased(0.1))
provider = TracerProvider(resource=resource, sampler=sampler)

2. Correlate Traces, Metrics, and Logs

Always include the trace ID in your log entries. This lets you jump from a metric alert to a trace and then to the exact log lines for that request. The OpenTelemetry logging bridge makes this straightforward:

import logging
from opentelemetry import trace

logger = logging.getLogger(__name__)

def log_with_trace(message, **kwargs):
    span = trace.get_current_span()
    trace_id = format(span.get_span_context().trace_id, "032x")
    span_id = format(span.get_span_context().span_id, "016x")
    logger.info(message, extra={
        "trace_id": trace_id,
        "span_id": span_id,
        **kwargs
    })

3. Monitor GPU Health Separately

vLLM's metrics tell you about request processing, but they do not cover GPU-level health. Use DCGM exporter or nvidia-smi metrics to monitor GPU temperature, power usage, memory utilization, and clock speeds alongside vLLM metrics.

4. Set Meaningful SLOs

Define service level objectives based on user experience, not raw numbers. For example:

5. Tag Everything with Context

Add attributes to spans and labels to metrics that provide context. Model name, GPU type, instance ID, and request type (chat vs. completion) are all valuable for filtering and grouping in dashboards.

span.set_attribute("model.name", "llama-2-7b")
span.set_attribute("model.quantization", "none")
span.set_attribute("request.type", "chat")
span.set_attribute("request.max_tokens", 100)
span.set_attribute("gpu.type", "A100")

6. Watch for Preemption Events

Preemption is vLLM's mechanism for handling KV cache pressure by swapping or recomputing requests. High preemption rates indicate that your model cannot handle the current load with the available GPU memory. If you see sustained preemption, either reduce --max-num-seqs, increase GPU memory, or scale horizontally.

7. Benchmark Before and After Changes

Use your observability stack to establish baselines before making configuration changes. Compare time-to-first-token, throughput, and queue depth before and after. This data-driven approach prevents regressions from sneaking in unnoticed.

Programmatic vLLM with Tracing

If you are using vLLM as a library rather than as a server, you can still enable tracing. The LLM class accepts tracing configuration through engine arguments:

from vllm import LLM, SamplingParams
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Set up OpenTelemetry before creating the LLM
resource = Resource.create({"service.name": "vllm-embedded"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)

# Create the LLM — vLLM will automatically use the global tracer provider
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")

# Generate with tracing
sampling_params = SamplingParams(temperature=0.8, max_tokens=100)
prompts = [
    "The future of AI is",
    "Explain quantum computing in simple terms",
]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(output.outputs[0].text)

Each call to llm.generate produces a trace with spans for scheduling, prefill, and decode phases. This is invaluable when vLLM is embedded in a larger pipeline such as a RAG system or an agent framework.

Conclusion

Observability is not an afterthought — it is a foundational requirement for running vLLM in production. By leveraging vLLM's built-in Prometheus metrics endpoint, enabling OpenTelemetry tracing with the --otlp-traces-endpoint flag, and integrating with backends like Jaeger and Grafana, you gain complete visibility into every stage of the inference pipeline. The combination of metrics for aggregate health, traces for individual request debugging, and structured logs for detailed inspection creates a powerful toolkit for maintaining reliable LLM services. Start with the basics — enable the metrics endpoint and point Prometheus at it — then layer in tracing and custom instrumentation as your needs grow. With proper observability in place, you can confidently scale your vLLM deployment, diagnose issues in minutes instead of hours, and deliver a consistently fast experience to your users.

— Ad —

Google AdSense will appear here after approval

← Back to all articles