Jaeger Distributed Tracing: Complete Implementation Guide

The operator creates collector, query, and agent deployments automatically. It also supports auto-injection of tracing sidecars for annotated pods.

Instrumenting Your Applications with OpenTelemetry

Modern Jaeger deployments use OpenTelemetry (OTel) for instrumentation. OTel provides language-agnostic APIs and SDKs, with a Jaeger exporter that converts OTel spans to Jaeger's format. This approach future-proofs your instrumentation — you can switch backend from Jaeger to any OTel-compatible system without code changes.

Node.js / TypeScript Example

Let's instrument a simple Express application that calls a downstream service:

# Install dependencies
npm install @opentelemetry/sdk-node \
  @opentelemetry/instrumentation-http \
  @opentelemetry/instrumentation-express \
  @opentelemetry/exporter-otlp-grpc \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

Create tracing.js — the OpenTelemetry setup file:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-otlp-grpc');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');

// Define the resource identifying this service
const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'api-gateway',
  [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
  [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: 'development',
});

// Configure the OTLP exporter pointing to Jaeger's OTLP receiver
const traceExporter = new OTLPTraceExporter({
  url: 'http://localhost:4317', // Jaeger collector gRPC OTLP port
  // For HTTP-based OTLP, use http://localhost:4318/v1/traces
});

// Create the SDK
const sdk = new NodeSDK({
  resource,
  spanProcessor: new BatchSpanProcessor(traceExporter, {
    maxQueueSize: 1000,
    maxExportBatchSize: 100,
    scheduledDelayMillis: 5000,
    exportTimeoutMillis: 30000,
  }),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

sdk.start();

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown().then(() => console.log('Tracing terminated'));
});
process.on('SIGINT', () => {
  sdk.shutdown().then(() => console.log('Tracing terminated'));
});

Now create your Express server in app.js:

// IMPORTANT: Require tracing.js BEFORE any other imports
require('./tracing');

const express = require('express');
const axios = require('axios');
const { trace } = require('@opentelemetry/api');

const app = express();
const PORT = 3000;

// Get a tracer instance for this module
const tracer = trace.getTracer('api-gateway-tracer');

app.get('/users/:id', async (req, res) => {
  // Create a custom span for the entire handler
  return tracer.startActiveSpan('get-user-handler', async (span) => {
    try {
      const userId = req.params.id;
      
      // Add attributes to the span for searchability
      span.setAttribute('user.id', userId);
      span.setAttribute('http.method', req.method);
      span.setAttribute('http.route', '/users/:id');

      // Simulate a downstream call with context propagation
      const response = await axios.get(`http://localhost:3001/profiles/${userId}`, {
        headers: {
          // Axios instrumentation automatically propagates trace context
          // but we can do it manually if needed
        },
      });

      span.setAttribute('http.status_code', 200);
      span.end();
      res.json({ user: response.data });
    } catch (error) {
      span.setAttribute('error', true);
      span.setAttribute('error.message', error.message);
      span.recordException(error);
      span.end();
      res.status(500).json({ error: 'Failed to fetch user' });
    }
  });
});

// Health check endpoint
app.get('/health', (req, res) => {
  res.json({ status: 'ok' });
});

app.listen(PORT, () => {
  console.log(`API Gateway listening on port ${PORT}`);
});

For the downstream service profile-service, create a similar setup:

require('./tracing'); // Same tracing setup with SERVICE_NAME='profile-service'

const express = require('express');
const app = express();
const PORT = 3001;

app.get('/profiles/:id', (req, res) => {
  const profile = {
    id: req.params.id,
    name: 'Jane Doe',
    email: 'jane@example.com',
    created_at: new Date().toISOString(),
  };
  res.json(profile);
});

app.listen(PORT, () => {
  console.log(`Profile Service listening on port ${PORT}`);
});

Run both services and make a request:

curl http://localhost:3000/users/42

Open Jaeger UI at http://localhost:16686. Select api-gateway from the service dropdown, click "Find Traces," and you'll see your trace. The Gantt chart will show the get-user-handler span with the HTTP GET to /profiles/:id nested underneath it.

Python Example with FastAPI

Python instrumentation follows the same pattern. Install the required packages:

pip install opentelemetry-sdk \
  opentelemetry-exporter-otlp \
  opentelemetry-instrumentation-fastapi \
  opentelemetry-instrumentation-requests \
  fastapi uvicorn requests

Create tracing.py:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

def init_tracing(service_name: str = "python-service"):
    # Create resource attributes
    resource = Resource.create({
        ResourceAttributes.SERVICE_NAME: service_name,
        ResourceAttributes.SERVICE_VERSION: "1.0.0",
        ResourceAttributes.DEPLOYMENT_ENVIRONMENT: "development",
    })

    # Set up the tracer provider
    provider = TracerProvider(resource=resource)
    
    # Configure OTLP exporter to Jaeger
    otlp_exporter = OTLPSpanExporter(
        endpoint="http://localhost:4317",  # Jaeger OTLP gRPC
        insecure=True,
    )
    
    # Add batch processor
    processor = BatchSpanProcessor(otlp_exporter)
    provider.add_span_processor(processor)
    
    # Set global tracer provider
    trace.set_tracer_provider(provider)
    
    # Instrument libraries
    FastAPIInstrumentor().instrument()
    RequestsInstrumentor().instrument()
    
    return provider

# Call this at startup
init_tracing()

Create main.py:

# Import tracing setup FIRST
import tracing
tracing.init_tracing("order-service")

from fastapi import FastAPI
import requests
from opentelemetry import trace

app = FastAPI()
tracer = trace.get_tracer(__name__)

@app.get("/orders/{order_id}")
async def get_order(order_id: str):
    with tracer.start_as_current_span("fetch-order-details") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("custom.tag", "important-value")
        
        # Call downstream inventory service
        # The requests instrumentation automatically propagates context
        try:
            inventory_response = requests.get(
                f"http://localhost:8001/inventory/{order_id}",
                timeout=5,
            )
            inventory_data = inventory_response.json()
        except Exception as e:
            span.record_exception(e)
            span.set_attribute("error", True)
            raise
        
        return {
            "order_id": order_id,
            "items": inventory_data.get("items", []),
            "status": "processing",
        }

@app.get("/health")
async def health():
    return {"status": "ok"}

Run with:

uvicorn main:app --host 0.0.0.0 --port 8000

Go Example with gRPC

For Go applications, use the OpenTelemetry Go SDK:

package main

import (
    "context"
    "log"
    "net/http"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func initTracerProvider(serviceName string) (*sdktrace.TracerProvider, error) {
    ctx := context.Background()

    // Create OTLP exporter pointing to Jaeger
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("localhost:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    // Create resource identifying this service
    resource, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("1.0.0"),
            attribute.String("environment", "development"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Create tracer provider
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource),
        sdktrace.WithSampler(sdktrace.AlwaysSample()),
    )

    otel.SetTracerProvider(tp)
    return tp, nil
}

func main() {
    tp, err := initTracerProvider("go-payment-service")
    if err != nil {
        log.Fatal(err)
    }
    defer tp.Shutdown(context.Background())

    tracer := otel.Tracer("payment-handler")

    // Wrap HTTP handler with OpenTelemetry instrumentation
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx, span := tracer.Start(r.Context(), "process-payment")
        defer span.End()

        span.SetAttributes(
            attribute.String("payment.method", "credit_card"),
            attribute.Float64("payment.amount", 99.99),
            attribute.String("payment.currency", "USD"),
        )

        // Simulate processing
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(`{"status": "completed", "transaction_id": "txn_12345"}`))
    })

    // Wrap with otelhttp for automatic HTTP tracing
    wrappedHandler := otelhttp.NewHandler(handler, "payment-endpoint")
    http.Handle("/payments", wrappedHandler)

    log.Println("Payment service listening on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Install dependencies and run:

go mod init payment-service
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go run main.go

Java / Spring Boot Example

For Spring Boot applications, use the OpenTelemetry Java agent for zero-code instrumentation:

# Download the OpenTelemetry Java agent JAR
wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.32.0/opentelemetry-javaagent.jar

# Run your Spring Boot app with the agent
java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.traces.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
  -Dotel.resource.attributes="service.name=spring-boot-app,environment=dev" \
  -Dotel.propagators=tracecontext,baggage \
  -jar target/my-app.jar

This auto-instruments Spring MVC, Spring WebFlux, JDBC, Kafka, gRPC, and dozens of other libraries — no code changes required. You can also add manual instrumentation for custom business logic:

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.GlobalOpenTelemetry;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class OrderController {

    private final Tracer tracer = GlobalOpenTelemetry.getTracer("order-controller");

    @GetMapping("/orders/{id}")
    public String getOrder(String id) {
        Span span = tracer.spanBuilder("validate-order").startSpan();
        try {
            span.setAttribute("order.id", id);
            // Business logic here
            return "Order details for " + id;
        } finally {
            span.end();
        }
    }
}

Context Propagation: The Key to Connected Traces

For traces to span multiple services, trace context must propagate across process boundaries. OpenTelemetry handles this automatically for most HTTP and gRPC clients via instrumentation libraries. The mechanism uses W3C Trace Context headers:

• traceparent — carries the trace ID and parent span ID
• tracestate — vendor-specific trace data

When making outbound HTTP calls, the instrumentation library injects these headers. When receiving inbound requests, it extracts them. This creates a single trace ID that links all spans together. If you're writing a custom protocol or using a message queue, you'll need to handle propagation manually:

// Manual context propagation example (Node.js)
const { propagation, trace } = require('@opentelemetry/api');

// Outbound: inject context into carrier object
const carrier = {};
propagation.inject(trace.setSpan(context.active(), span), carrier);
// carrier now has traceparent and tracestate keys

// Send carrier along with your message
await sendMessage({ body: payload, ...carrier });

// Inbound: extract context from received message
const extractedContext = propagation.extract(context.active(), messageHeaders);
// Use extractedContext to create child spans

Sampling Strategies

In high-throughput systems, tracing every request is impractical. Jaeger supports multiple sampling strategies:

• Always Sample — for development; traces every request
• Never Sample — effectively disables tracing
• Probabilistic — sample a percentage (e.g., 0.1% = 1 in 1000)
• Rate Limiting — sample up to N traces per second
• Adaptive / Tail-based — sample based on latency or error occurrence

Configure sampling in OpenTelemetry:

// Probabilistic sampling at 10%
const { ParentBasedSampler, TraceIdRatioBasedSampler } = require('@opentelemetry/sdk-trace-base');

const sampler = new ParentBasedSampler({
  rootSampler: new TraceIdRatioBasedSampler(0.1), // 10% of root spans
});

const sdk = new NodeSDK({
  sampler,
  // ... rest of configuration
});

For production, a common strategy is head-based probabilistic sampling at 1-10%, combined with tail-based sampling where the collector retains traces that exceed latency thresholds or contain errors, even if they weren't initially sampled. Jaeger's collector supports tail-based sampling via a sampling processor.

Advanced Features and Custom Instrumentation

Baggage: Passing Data Across Services

Baggage allows you to propagate arbitrary key-value pairs across service boundaries alongside trace context. Use it for passing tenant IDs, feature flags, or experiment identifiers:

const { propagation, baggage } = require('@opentelemetry/api');

// Create baggage with tenant context
const bag = baggage.setBaggage(
  baggage.createBaggage({ tenantId: 'customer-42' }),
  { userId: 'user-789' }
);

// Propagate with context
const ctx = propagation.setBaggage(context.active(), bag);
propagation.inject(ctx, carrier);

Downstream services can extract baggage and use it for routing decisions or custom tagging without modifying service interfaces.

Custom Span Attributes and Events

Enrich spans with meaningful data to improve searchability in the Jaeger UI:

span.setAttribute('db.system', 'postgresql');
span.setAttribute('db.name', 'users_db');
span.setAttribute('db.operation', 'SELECT');
span.setAttribute('db.statement', 'SELECT * FROM users WHERE id = ?');
span.setAttribute('db.user', 'readonly_user');
span.setAttribute('peer.service', 'user-database');
span.setAttribute('net.peer.name', 'db-host.example.com');

// Add timestamped events for important milestones
span.addEvent('cache-hit', { 'cache.key': 'user:42', 'cache.ttl': 300 });
span.addEvent('query-started', { 'query.plan': 'index-scan' });

Linking Spans Across Traces

Sometimes a single operation relates to multiple traces (e.g., a batch job processing items from different upstream requests). Use span links to connect them:

// Create a span with links to related traces
const batchSpan = tracer.startSpan('batch-process', {
  links: [
    { context: { traceId: 'abc123', spanId: 'span-001' } },
    { context: { traceId: 'def456', spanId: 'span-002' } },
  ],
});

Best Practices for Jaeger Distributed Tracing

1. Keep Span Granularity Balanced

Spans should represent meaningful units of work. Too coarse (one span per service) loses detail; too fine (one span per function call) creates noise. A good rule: create spans for cross-service calls, database operations, message queue interactions, and significant computational boundaries. Avoid spans for trivial getters or log statements.

2. Use Semantic Conventions

OpenTelemetry defines semantic conventions for common operations. Follow them — they enable interoperability across tools. For HTTP calls, use http.method, http.status_code, http.url. For databases, use db.system, db.operation, db.name. Consistent naming makes traces queryable across your entire fleet.

3. Handle Errors Explicitly

Always record exceptions in spans:

span.recordException(error);
span.setAttribute('error', true);
span.setAttribute('error.message', error.message);
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });

This enables Jaeger to highlight failed spans in red, making error traces instantly visible.

4. Deploy Agents as Sidecars or DaemonSets

In Kubernetes, run the Jaeger agent as a DaemonSet on each node, or as a sidecar per pod. The sidecar pattern ensures network locality and avoids UDP packet loss. For OpenTelemetry SDKs that send directly to the collector (bypassing the agent), ensure the collector endpoint is highly available.

5. Plan Your Storage Retention

Trace data grows quickly. Define a retention policy based on your debugging needs: typically 7-30 days for full traces, with sampling reducing volume. Use Elasticsearch index lifecycle management or Cassandra TTL to automate cleanup.

6. Secure Your Jaeger Deployment

The Jaeger UI and collector endpoints are unprotected by default. In production, place them behind a reverse proxy with authentication, or use mTLS between components. Jaeger supports TLS configuration for collector and query service gRPC endpoints.

7. Test Trace Context Propagation

Write integration tests that verify trace context flows correctly:

// Jest test for trace propagation
it('propagates trace context to downstream service', async () => {
  const response = await request(app)
    .get('/users/42')
    .set('traceparent', '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01');
  
  expect(response.status).toBe(200);
  // Verify downstream service received the traceparent header
  expect(downstreamMock.lastRequestHeaders['traceparent']).toBeDefined();
});

8. Monitor Jaeger Itself

Monitor the collector's memory usage, dropped spans count, and queue depth. The collector exposes metrics at port 14271 in Prometheus format. Set up alerts for span drop rates exceeding thresholds.

Troubleshooting Common Issues

"No traces in Jaeger UI" — Verify the OTLP exporter endpoint is correct. Check that the collector's OTLP receiver is enabled (COLLECTOR_OTLP_ENABLED=true). Use console exporter temporarily to verify spans are being generated.

"Incomplete traces / missing spans" — Ensure trace context propagation headers are not being stripped by intermediaries (load balancers, API gateways). Check that your HTTP