Tempo Tracing Backend: Complete Implementation Guide

Tempo Tracing Backend: Complete Implementation Guide

Distributed tracing has become an essential observability pillar for modern microservices architectures. Grafana Tempo is an open-source, high-scale distributed tracing backend that stores traces efficiently and integrates seamlessly with the broader observability ecosystem. This guide walks you through everything from understanding Tempo's architecture to deploying it in production and instrumenting your applications.

What is Grafana Tempo?

Grafana Tempo is an open-source distributed tracing backend created by Grafana Labs. Unlike traditional tracing systems that require indexing every span and trace, Tempo takes a different approach: it stores traces in object storage without building indexes. Instead, Tempo relies on trace IDs for retrieval, making it dramatically more cost-effective and scalable than alternatives like Jaeger or Zipkin.

Tempo is compatible with multiple tracing protocols including OpenTelemetry, Jaeger, Zipkin, and OpenTracing. This means you can send traces from virtually any instrumented application without changing your existing instrumentation. Tempo pairs naturally with Grafana for visualization, Loki for log correlation, and Prometheus for metrics, forming a complete observability stack.

Why Tempo Matters

Traditional distributed tracing backends face a fundamental tradeoff: indexing every span provides flexible querying but creates enormous storage and operational costs. Tempo eliminates this tradeoff by removing indexes entirely. You query traces by their ID, and Tempo retrieves them from object storage on demand. This architectural decision has several important implications.

Cost Efficiency

Because Tempo stores traces in cheap object storage like Amazon S3, Google Cloud Storage, or MinIO, your per-trace storage cost drops dramatically. There is no expensive index database to maintain, no index cardinality explosions to worry about, and no need to sample aggressively just to control costs.

Scalability

Tempo scales horizontally with your trace volume. The ingestion path separates trace data into blocks that are flushed to object storage. Because there is no index to update, adding more ingestors simply increases throughput without coordination overhead.

Simplified Operations

Without an index database, there are fewer moving parts to operate. Tempo consists of a few core components, and the storage layer is managed by your object storage provider. This reduces operational burden significantly compared to running Elasticsearch or Cassandra clusters for trace storage.

Ecosystem Integration

Tempo integrates tightly with Grafana, Loki, and Prometheus. You can correlate traces with logs by trace ID, jump from a Prometheus metric alert to the relevant traces, and visualize spans in Grafana's trace view. This unified workflow is powerful for debugging distributed systems.

Tempo Architecture Overview

Understanding Tempo's architecture helps you deploy and operate it effectively. Tempo consists of several microservices that work together to ingest, store, and query traces.

Core Components

• Distributor: Receives traces from clients, validates them, and routes them to ingestors. The distributor handles incoming trace data from OpenTelemetry, Jaeger, and Zipkin protocols.
• Ingestor: Buffers incoming traces in memory and builds blocks. When a block reaches a configured size or age, the ingestor flushes it to object storage.
• Compactor: Combines smaller blocks into larger ones in the background, reducing the number of blocks and improving query performance.
• Querier: Handles trace queries by ID. It searches both the ingester's in-memory data and the blocks in object storage.
• Query Frontend: Splits and distributes queries, providing a scalable query layer that can handle concurrent requests.
• Metrics Generator: Optional component that generates metrics from span data, enabling span-based RED metrics without separate instrumentation.

Storage Model

Tempo stores traces in a columnar format called Parquet in object storage. Each block contains traces organized by trace ID. When you query a trace, Tempo identifies which blocks might contain it and retrieves only the relevant data. The Parquet format enables future advanced querying capabilities while maintaining efficient storage.

Installation and Setup

Let's walk through deploying Tempo. The most common approaches are Docker for local development, Docker Compose for testing, and Kubernetes for production.

Local Development with Docker

For quick local testing, you can run Tempo in a single Docker container with a minimal configuration. First, create a configuration file named tempo.yaml:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 48h

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal

Then run Tempo with Docker, mounting the configuration file and a local storage directory:

docker run -d \
  --name tempo \
  -p 3200:3200 \
  -p 4317:4317 \
  -p 4318:4318 \
  -v $(pwd)/tempo.yaml:/etc/tempo.yaml \
  -v $(pwd)/tempo-data:/var/tempo \
  grafana/tempo:latest \
  -config.file=/etc/tempo.yaml

This starts Tempo with OTLP ingestion on ports 4317 (gRPC) and 4318 (HTTP), and the Tempo query API on port 3200. The local storage backend writes blocks to a directory on your host machine.

Docker Compose with Grafana

For a more complete local setup, use Docker Compose to run Tempo alongside Grafana for visualization. Create a docker-compose.yaml file:

version: "3.8"

services:
  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - tempo

volumes:
  tempo-data:
  grafana-data:

Start the stack with docker-compose up -d. Then open Grafana at http://localhost:3000 and add Tempo as a data source with the URL http://tempo:3200.

Production Deployment on Kubernetes

For production deployments, the recommended approach is using the Tempo Helm chart or the Tempo distributed Helm chart. The distributed chart deploys each component separately for independent scaling. Here is a minimal values file for the distributed chart:

tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: my-tempo-traces
        endpoint: s3.us-east-1.amazonaws.com
        region: us-east-1

ingester:
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1
      memory: 2Gi

distributor:
  replicas: 2
  resources:
    requests:
      cpu: 250m
      memory: 512Mi

querier:
  replicas: 2

compactor:
  replicas: 1
  compaction:
    block_retention: 168h

metricsGenerator:
  enabled: true
  replicas: 2

Install the chart with Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install tempo grafana/tempo-distributed -f values.yaml

Configuration Deep Dive

Tempo's configuration file controls every aspect of its behavior. Let's examine the most important sections.

Receivers Configuration

Tempo can accept traces from multiple protocols simultaneously. The most common is OTLP from OpenTelemetry, but you can also enable Jaeger and Zipkin receivers:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    jaeger:
      protocols:
        thrift_http:
          endpoint: 0.0.0.0:14268
        grpc:
          endpoint: 0.0.0.0:14250
    zipkin:
      endpoint: 0.0.0.0:9411

Storage Configuration

For production, you should use object storage. Here is an S3 configuration example:

storage:
  trace:
    backend: s3
    s3:
      bucket: my-tempo-traces
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
    wal:
      path: /var/tempo/wal
    block:
      version: parquet

For MinIO or other S3-compatible storage, adjust the endpoint accordingly:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio:9000
      access_key: minioadmin
      secret_key: minioadmin
      insecure: true

Retention Configuration

Control how long traces are retained using the compactor configuration:

compactor:
  compaction:
    block_retention: 168h
    compaction_window: 1h
    max_block_bytes: 107374182400
    retention_concurrency: 10

The block_retention parameter determines how long traces are kept. Older blocks are deleted automatically during compaction.

Metrics Generator Configuration

The metrics generator derives RED metrics (Rate, Errors, Duration) from spans and exports them to Prometheus. This is extremely useful for creating dashboards and alerts based on trace data:

metrics_generator:
  ring:
    kvstore:
      store: memberlist
  processor:
    service_graphs:
      dimensions:
        - http.method
        - http.status_code
    span_metrics:
      dimensions:
        - http.method
        - http.status_code
        - http.route
  storage:
    remote_write:
      - endpoint: prometheus:9090/api/v1/write

Instrumenting Applications

Tempo stores traces, but you need to generate them from your applications. OpenTelemetry is the standard for instrumentation, and it works seamlessly with Tempo.

Python Application Instrumentation

Install the OpenTelemetry packages for Python:

pip install opentelemetry-api \
    opentelemetry-sdk \
    opentelemetry-exporter-otlp \
    opentelemetry-instrumentation-flask \
    opentelemetry-instrumentation-requests

Then instrument your Flask application:

from flask import Flask, jsonify
import requests
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

app = Flask(__name__)

# Configure the tracer provider
resource = Resource.create({
    "service.name": "order-service",
    "service.version": "1.0.0",
    "deployment.environment": "production"
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://tempo:4317", insecure=True)
    )
)
trace.set_tracer_provider(provider)

# Auto-instrument Flask and requests
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route("/orders/")
def get_order(order_id):
    tracer = trace.get_tracer(__name__)
    
    with tracer.start_as_current_span("fetch_order_details") as span:
        span.set_attribute("order.id", order_id)
        
        # Call another service
        response = requests.get(f"http://inventory-service/items/{order_id}")
        items = response.json()
        
        span.set_attribute("items.count", len(items))
        
    return jsonify({"order_id": order_id, "items": items})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Node.js Application Instrumentation

Install the OpenTelemetry packages for Node.js:

npm install @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/exporter-trace-otlp-grpc \
  @opentelemetry/auto-instrumentations-node

Create a tracing initialization file that runs before your application code:

// tracing.js
const opentelemetry = require('@opentelemetry/api');
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { resourceFromAttributes } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require('@opentelemetry/semantic-conventions');

const resource = resourceFromAttributes({
  [ATTR_SERVICE_NAME]: 'payment-service',
  [ATTR_SERVICE_VERSION]: '1.0.0',
});

const traceExporter = new OTLPTraceExporter({
  url: 'http://tempo:4317',
});

const sdk = new NodeSDK({
  resource,
  traceExporter,
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

Then start your application with the tracing module loaded first:

node -r ./tracing.js app.js

Go Application Instrumentation

Install the OpenTelemetry Go packages:

go get go.opentelemetry.io/otel \
    go.opentelemetry.io/otel/sdk \
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Create a setup function and instrument your HTTP server:

package main

import (
    "context"
    "log"
    "net/http"
    "os"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/otel/trace"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func initTracer(ctx context.Context) func() {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("tempo:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("shipping-service"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )
    if err != nil {
        log.Fatalf("failed to create resource: %v", err)
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})

    return func() {
        tp.Shutdown(ctx)
    }
}

func shipOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    tracer := otel.Tracer("shipping-service")
    
    ctx, span := tracer.Start(ctx, "process_shipment",
        trace.WithAttributes(attribute.String("shipment.method", "express")),
    )
    defer span.End()

    // Simulate work
    span.AddEvent("shipment_processed", trace.WithAttributes(
        attribute.String("status", "completed"),
    ))

    w.Write([]byte("Order shipped"))
}

func main() {
    ctx := context.Background()
    shutdown := initTracer(ctx)
    defer shutdown()

    handler := http.HandlerFunc(shipOrder)
    wrappedHandler := otelhttp.NewHandler(handler, "ship-order")

    log.Println("Server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", wrappedHandler))
}

Java Application Instrumentation

For Java applications, the OpenTelemetry Java agent provides automatic instrumentation without code changes. Download the agent and configure it as a JVM argument:

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=user-service \
  -Dotel.exporter.otlp.endpoint=http://tempo:4317 \
  -Dotel.exporter.otlp.protocol=grpc \
  -jar my-application.jar

For manual span creation in Java:

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.context.Scope;

public class UserService {
    private static final Tracer tracer = 
        GlobalOpenTelemetry.getTracer("user-service");

    public User getUser(String userId) {
        Span span = tracer.spanBuilder("get_user")
            .setAttribute("user.id", userId)
            .startSpan();
        
        try (Scope scope = span.makeCurrent()) {
            User user = database.findById(userId);
            span.setAttribute("user.found", user != null);
            return user;
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            throw e;
        } finally {
            span.end();
        }
    }
}

Querying Traces

Tempo provides multiple ways to query traces. The primary method is by trace ID, but Tempo also supports search capabilities when enabled.

Querying by Trace ID

You can query a trace directly via the Tempo API:

curl http://localhost:3200/api/traces/<trace_id>

The response is a JSON object containing the full trace with all spans:

{
  "batches": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {"stringValue": "order-service"}
          }
        ]
      },
      "instrumentationLibrarySpans": [
        {
          "spans": [
            {
              "traceId": "abcdef1234567890abcdef1234567890",
              "spanId": "1234567890abcdef",
              "name": "fetch_order_details",
              "kind": "SPAN_KIND_INTERNAL",
              "startTimeUnixNano": "1699999999000000000",
              "endTimeUnixNano": "1699999999005000000",
              "attributes": [
                {
                  "key": "order.id",
                  "value": {"stringValue": "order-123"}
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Trace Search

Tempo supports search queries when you enable the search capability. This requires using the Parquet block format and enabling search in the configuration:

storage:
  trace:
    block:
      version: parquet

querier:
  max_concurrent_queries: 5

query_frontend:
  search:
    duration_slo: 5s
    throughput_slo: 1

With search enabled, you can query traces using TraceQL, Tempo's query language:

# Find traces from a specific service with errors
{ resource.service.name = "payment-service" && status = error }

# Find traces with high latency
{ duration > 2s }

# Find traces with specific attributes
{ span.http.status_code = 500 }

# Combine conditions
{ resource.service.name = "order-service" && span.http.route = "/orders" && duration > 500ms }

You can execute TraceQL queries via the API:

curl -G http://localhost:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "payment-service" && status = error }'

Using the Tempo CLI

Tempo includes a CLI tool for querying traces. You can use it to inspect traces from the command line:

tempo-cli query api-traces \
  --endpoint=http://localhost:3200 \
  --trace-id=abcdef1234567890abcdef1234567890

Integrating with Grafana

Grafana provides the richest visualization experience for Tempo traces. After adding Tempo as a data source in Grafana, you can explore traces, view flame graphs, and correlate with logs and metrics.

Adding Tempo as a Data Source

In Grafana, navigate to Configuration > Data Sources > Add data source > Tempo. Configure the following settings:

URL: http://tempo:3200

# Enable search and TraceQL
Search: Enabled

# Configure node graph for service maps
Node Graph: Enabled

# Link to Loki for log correlation
Trace to Logs:
  Datasource: Loki
  Tags: service.name, span.name
  Map tag names: 
    service.name: service

# Link to Prometheus for metrics correlation
Trace to Metrics:
  Datasource: Prometheus

Creating Trace Dashboards

You can embed trace queries in Grafana dashboards. For example, create a panel that shows recent error traces:

# In a Grafana dashboard panel using Tempo data source
# Query: { status = error } 
# Visualization: Table
# Columns: service.name, span.name, duration

Exemplars Integration

If you have the metrics generator enabled, Tempo exports span metrics to Prometheus. You can configure Prometheus exemplars to link from metric charts directly to traces:

# In Prometheus configuration
remote_write:
  - url: http://tempo:9009/api/v1/push

# Exemplars link metrics to trace IDs
# In Grafana, configure the Prometheus data source:
# Exemplars trace ID label: trace_id
# Tempo data source for exemplar links: Tempo

Best Practices

Use Consistent Service Names and Attributes

Consistent naming is critical for effective tracing. Use the OpenTelemetry semantic conventions for standard attributes like service.name, http.method, http.route, and http.status_code. This consistency enables meaningful search queries and dashboards.

Propagate Trace Context Correctly

Ensure trace context propagation across service boundaries. OpenTelemetry uses W3C Trace Context headers by default. When making HTTP calls between services, the instrumentation libraries handle propagation automatically, but if you use custom transports, you must propagate context manually:

# Python example of manual context propagation
from opentelemetry.propagate import inject, extract

# Inject context into outgoing request headers
headers = {}
inject(headers)
response = requests.get(url, headers=headers)

# Extract context from incoming request on the receiving service
context = extract(request.headers)

Use Appropriate Sampling

While Tempo's storage is cheap, you may still want to sample at high volumes. Use head-based sampling for simplicity or tail-based sampling for more intelligent decisions. Tail-based sampling with the OpenTelemetry Collector allows you to sample based on trace characteristics:

# OpenTelemetry Collector tail sampling configuration
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow_traces
        type: latency
        latency:
          threshold_ms: 1000
      - name: sample_10_percent
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Monitor Tempo Itself

Tempo exposes Prometheus metrics on its metrics endpoint. Monitor key metrics like ingestion rate, block flush times, and query latency. Critical metrics to watch include:

# Ingestion rate
tempo_distributor_spans_received_total

# Ingestor memory usage
tempo_ingester_live_blocks

# Query performance
tempo_querier_query_duration_seconds

# Compaction
tempo_compactor_blocks_compacted_total

# Errors
tempo_request_duration_seconds{status_code=~"5.."}

Use the OpenTelemetry Collector as a Gateway

Instead of sending traces directly from applications to Tempo, route them through the OpenTelemetry Collector. This provides a buffering layer, enables preprocessing, and allows you to add tail sampling, attribute enrichment, and routing logic:

# OpenTelemetry Collector configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 20

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

Set Appropriate Resource Limits

In production, set resource limits carefully. Ingestors buffer traces in memory, so they need sufficient memory. The compactor needs CPU for block processing. Queriers need resources proportional to your query load. Start with the Helm chart defaults and adjust based on monitoring data.

Enable Compression

Tempo supports compression for data in transit and at rest. Enable gRPC compression for the OTLP exporter and ensure object storage uses appropriate compression:

# In application exporter configuration
OTLPSpanExporter(
    endpoint="http://tempo:4317",
    insecure=True,
    compression=Compression.Gzip
)

Secure Your Tempo Deployment

For production, enable TLS for all Tempo endpoints and configure authentication. Use Grafana's authentication proxy or an API gateway to protect the Tempo query API:

server:
  http_tls_config:
    cert_file: /etc/tempo/certs/server.crt
    key_file: /etc/tempo/certs/server.key
  grpc_tls_config:
    cert_file: /etc/tempo/certs/server.crt
    key_file: /etc/tempo/certs/server.key

Plan for Disaster Recovery

Since Tempo stores data in object storage, your disaster recovery strategy depends on your storage provider's replication and backup capabilities. Enable versioning and cross-region replication on your S3 buckets. Test recovery by restoring blocks to a new Tempo instance.

Conclusion

Grafana Tempo represents a paradigm shift in distributed tracing backends by eliminating indexes and leveraging cheap object storage. This architectural choice makes full-fidelity tracing economically viable at scale, allowing you to store 100% of your traces without sampling just to control costs. By combining Tempo with OpenTelemetry for instrumentation, Grafana for visualization, Loki for log correlation, and Prometheus for metrics, you build a unified observability platform that can handle the complexity of modern distributed systems. Start with a local Docker setup to understand the basics, then move to a Kubernetes deployment with object storage for production. Focus on consistent instrumentation, proper context propagation, and monitoring Tempo itself to ensure reliable trace data flows through your observability pipeline. With the practices and configurations outlined in this guide, you are well-equipped to implement Tempo as your tracing backend and gain deep visibility into your microservices architecture.