Observability and Tracing with llama.cpp: Complete Guide

Observability and Tracing with llama.cpp: Complete Guide

As LLM-powered applications move from prototypes to production, understanding what happens inside the inference pipeline becomes critical. llama.cpp is a popular C/C++ inference engine for running large language models locally and on edge devices. While it is celebrated for its performance and portability, observability and tracing are often treated as afterthoughts. This guide walks you through everything you need to know to make your llama.cpp deployments observable, debuggable, and production-ready.

What Is Observability in the Context of llama.cpp?

Observability is the ability to understand the internal state of a system based on its external outputs. For llama.cpp, this means having visibility into token generation latency, memory consumption, prompt processing time, context window usage, KV cache behavior, and model loading times. Tracing is a specific observability technique that records the causal journey of a single request through the system, allowing you to reconstruct what happened end-to-end.

Unlike managed services such as OpenAI or Anthropic, llama.cpp runs on your own hardware. This gives you full control but also full responsibility for monitoring. Without proper instrumentation, you are flying blind when a user reports that "the model is slow today."

Why Observability Matters for LLM Inference

• Latency diagnosis: Identify whether time is spent in prompt evaluation, token generation, or network transfer.
• Resource planning: Track VRAM and RAM usage to decide when to scale or switch to quantized models.
• Quality assurance: Correlate generation parameters with output quality to tune temperature, top-p, and repeat penalties.
• Cost attribution: In multi-tenant setups, trace which requests consume the most compute.
• Failure investigation: When a request fails or produces unexpected output, traces help you reconstruct the exact sequence of events.

Built-in Logging and Timing in llama.cpp

llama.cpp ships with a lightweight logging system that prints timing information to stderr by default. When you run the llama-cli or llama-server binaries, you will see output similar to this at the end of each generation:

llama_print_timings:        load time =     245.31 ms
llama_print_timings:      prompt eval time =     132.18 ms /    42 tokens (    3.15 ms per token,   317.71 tokens per second)
llama_print_timings:        eval time =    2841.92 ms /   128 runs   (   22.20 ms per token,    45.05 tokens per second)
llama_print_timings:       total time =    2974.10 ms

This built-in output is useful for quick checks, but it is not structured, not exportable, and not correlated with individual requests in a concurrent server. For production observability, you need to go further.

Enabling Verbose Logging

You can increase the verbosity of llama.cpp by setting the log level. When building from source, ensure that LLAMA_LOG is enabled. At runtime, you can control verbosity through environment variables and API calls:

# Set verbose logging via environment variable
export LLAMA_LOG=debug

# Or when running the server binary
./llama-server --model ./models/llama-3-8b.gguf --verbose

For programmatic control, the C API exposes llama_log_set(), which lets you register a custom callback to intercept all internal log messages:

#include "llama.h"
#include <stdio.h>

static void my_log_callback(enum ggml_log_level level, const char * text, void * user_data) {
    // Route logs to your observability backend
    FILE * f = (FILE *) user_data;
    fprintf(f, "[llama:%d] %s", level, text);
}

int main() {
    FILE * log_file = fopen("llama.log", "a");
    llama_log_set(my_log_callback, log_file);

    // Initialize llama as usual
    llama_backend_init();
    // ... rest of your code
    return 0;
}

Building a Custom Tracing Layer

The most effective way to add observability to llama.cpp is to wrap the inference calls in a tracing layer. This layer records timestamps, token counts, and parameters for each phase of inference. Below is a practical example using C++ that instruments a basic generation loop.

Defining a Trace Record Structure

#include <chrono>
#include <string>
#include <vector>
#include <iostream>
#include <fstream>
#include "llama.h"

struct TraceRecord {
    std::string request_id;
    std::string phase;          // "load", "prompt_eval", "token_gen", "total"
    int64_t duration_ms;
    int token_count;
    std::string model_name;
    int n_threads;
    int n_ctx;
    float temperature;
};

class Tracer {
private:
    std::vector<TraceRecord> records;
    std::string output_path;

public:
    Tracer(const std::string & path) : output_path(path) {}

    void record(const TraceRecord & r) {
        records.push_back(r);
    }

    template<typename Func>
    auto timed(const std::string & request_id,
               const std::string & phase,
               Func && fn) -> decltype(fn()) {
        auto start = std::chrono::high_resolution_clock::now();
        auto result = fn();
        auto end = std::chrono::high_resolution_clock::now();
        auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();
        records.push_back({request_id, phase, ms, 0, "", 0, 0, 0.0f});
        return result;
    }

    void flush() {
        std::ofstream out(output_path, std::ios::app);
        for (const auto & r : records) {
            out << r.request_id << ","
                << r.phase << ","
                << r.duration_ms << ","
                << r.token_count << ","
                << r.model_name << ","
                << r.n_threads << ","
                << r.n_ctx << ","
                << r.temperature << "\n";
        }
        records.clear();
        out.close();
    }
};

Instrumenting the Inference Loop

Now let us use the tracer to wrap a complete inference workflow. This example loads a model, processes a prompt, and generates tokens, recording timing for each phase:

int main() {
    Tracer tracer("traces.csv");

    // Generate a unique request ID
    std::string request_id = "req_001";

    // --- Phase 1: Model Loading ---
    llama_backend_init();
    llama_model_params model_params = llama_model_default_params();
    model_params.n_gpu_layers = 20;

    auto model = tracer.timed(request_id, "load", [&]() {
        return llama_load_model_from_file("models/llama-3-8b.gguf", model_params);
    });

    if (!model) {
        std::cerr << "Failed to load model\n";
        return 1;
    }

    // --- Phase 2: Context Creation ---
    llama_context_params ctx_params = llama_context_default_params();
    ctx_params.n_ctx = 2048;
    ctx_params.n_threads = 8;

    auto ctx = llama_new_context_with_model(model, ctx_params);

    // --- Phase 3: Prompt Evaluation ---
    std::string prompt = "Explain observability in three sentences.";
    std::vector<llama_token> tokens(llama_n_ctx(ctx));

    int n_prompt_tokens = tracer.timed(request_id, "prompt_eval", [&]() {
        return llama_tokenize(ctx, prompt.c_str(), prompt.size(),
                              tokens.data(), tokens.size(), true, true);
    });

    llama_batch batch = llama_batch_get_one(tokens.data(), n_prompt_tokens);
    llama_decode(ctx, batch);

    // --- Phase 4: Token Generation ---
    int n_generated = 0;
    int max_tokens = 128;

    auto gen_start = std::chrono::high_resolution_clock::now();

    for (int i = 0; i < max_tokens; i++) {
        float * logits = llama_get_logits(ctx);
        llama_token next = llama_sample_token_greedy(ctx, nullptr, logits, n_prompt_tokens + i);

        if (next == llama_token_eos(ctx)) break;

        n_generated++;
        batch = llama_batch_get_one(&next, 1);
        llama_decode(ctx, batch);
    }

    auto gen_end = std::chrono::high_resolution_clock::now();
    auto gen_ms = std::chrono::duration_cast<std::chrono::milliseconds>(gen_end - gen_start).count();

    tracer.record({request_id, "token_gen", gen_ms, n_generated,
                   "llama-3-8b", 8, 2048, 0.0f});

    // --- Cleanup ---
    llama_free(ctx);
    llama_free_model(model);
    llama_backend_free();

    tracer.flush();
    return 0;
}

The resulting traces.csv file will contain structured rows that you can import into any analytics tool, spreadsheet, or time-series database.

Integrating with OpenTelemetry

For production-grade distributed tracing, OpenTelemetry is the industry standard. You can instrument your llama.cpp server to emit OpenTelemetry spans that integrate with backends like Jaeger, Zipkin, Grafana Tempo, or Datadog.

Setting Up the C++ OpenTelemetry SDK

First, add the OpenTelemetry C++ SDK to your project. If you are using CMake:

# CMakeLists.txt
find_package(OpenTelemetryCpp REQUIRED)

target_link_libraries(your_target
    PRIVATE
        opentelemetry-cpp::trace
        opentelemetry-cpp::otlp_http_exporter
        opentelemetry-cpp::sdk
)

Creating Spans for Inference Phases

#include <opentelemetry/trace/tracer.h>
#include <opentelemetry/exporters/otlp/otlp_http_exporter.h>
#include <opentelemetry/sdk/trace/processor.h>
#include <opentelemetry/sdk/trace/simple_processor.h>
#include <opentelemetry/sdk/trace/tracer_provider.h>
#include "llama.h"

namespace trace = opentelemetry::trace;
namespace otlp = opentelemetry::exporter::otlp;
namespace sdktrace = opentelemetry::sdk::trace;

void init_tracing() {
    auto exporter = std::unique_ptr<otlp::OtlpHttpExporter>(
        new otlp::OtlpHttpExporter());
    auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
        new sdktrace::SimpleSpanProcessor(std::move(exporter)));
    auto provider = std::shared_ptr<trace::TracerProvider>(
        new sdktrace::TracerProvider(std::move(processor)));
    trace::Provider::SetTracerProvider(provider);
}

void run_inference_with_spans(llama_context * ctx,
                               const std::string & prompt,
                               int max_tokens) {
    auto tracer = trace::Provider::GetTracerProvider()->GetTracer("llama-cpp");

    // Root span for the entire request
    auto root_span = tracer->StartSpan("llama.inference");
    auto root_scope = tracer->WithActiveSpan(root_span);

    // Span for tokenization
    {
        auto span = tracer->StartSpan("llama.tokenize");
        auto scope = tracer->WithActiveSpan(span);
        std::vector<llama_token> tokens(llama_n_ctx(ctx));
        int n = llama_tokenize(ctx, prompt.c_str(), prompt.size(),
                               tokens.data(), tokens.size(), true, true);
        span->SetAttribute("prompt.token_count", n);
        span->End();
    }

    // Span for prompt evaluation
    {
        auto span = tracer->StartSpan("llama.prompt_eval");
        auto scope = tracer->WithActiveSpan(span);
        // ... prompt evaluation code ...
        span->SetAttribute("prompt_eval.tokens", 42);
        span->End();
    }

    // Span for generation
    {
        auto span = tracer->StartSpan("llama.generation");
        auto scope = tracer->WithActiveSpan(span);
        // ... generation loop ...
        span->SetAttribute("generation.max_tokens", max_tokens);
        span->SetAttribute("generation.actual_tokens", 96);
        span->End();
    }

    root_span->End();
}

Once exported, these spans will appear in your tracing backend as a hierarchical timeline, showing exactly how much time each phase consumed.

Observability for the llama.cpp HTTP Server

Many deployments use the built-in llama-server binary, which exposes an OpenAI-compatible HTTP API. To add observability without modifying C++ code, you can place a reverse proxy in front of the server that handles tracing.

Using a Python Sidecar for Tracing

# tracing_proxy.py
from fastapi import FastAPI, Request, Response
from fastapi.responses import StreamingResponse
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
import httpx
import time
import uuid

app = FastAPI()

# Initialize OpenTelemetry
trace.set_tracer_provider(TracerProvider())
span_processor = BatchSpanProcessor(OTLPSpanExporter(
    endpoint="http://localhost:4318/v1/traces"
))
trace.get_tracer_provider().add_span_processor(span_processor)
HTTPXClientInstrumentor().instrument()

tracer = trace.get_tracer("llama-proxy")
LLAMA_SERVER = "http://localhost:8080"

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    body = await request.json()
    request_id = str(uuid.uuid4())

    with tracer.start_as_current_span("llama.chat_completion") as span:
        span.set_attribute("request.id", request_id)
        span.set_attribute("request.model", body.get("model", "unknown"))
        span.set_attribute("request.max_tokens", body.get("max_tokens", 0))
        span.set_attribute("request.temperature", body.get("temperature", 1.0))

        prompt_len = sum(len(m.get("content", "")) for m in body.get("messages", []))
        span.set_attribute("request.prompt_chars", prompt_len)

        start = time.time()

        async with httpx.AsyncClient() as client:
            if body.get("stream", False):
                async def stream_response():
                    completion_tokens = 0
                    async with client.stream("POST", f"{LLAMA_SERVER}/v1/chat/completions",
                                             json=body) as resp:
                        async for line in resp.aiter_lines():
                            if line.startswith("data:") and "[DONE]" not in line:
                                completion_tokens += 1
                            yield line + "\n"
                    span.set_attribute("response.completion_tokens", completion_tokens)
                    span.set_attribute("response.duration_ms", (time.time() - start) * 1000)

                return StreamingResponse(stream_response(), media_type="text/event-stream")
            else:
                resp = await client.post(f"{LLAMA_SERVER}/v1/chat/completions", json=body)
                span.set_attribute("response.duration_ms", (time.time() - start) * 1000)
                span.set_attribute("response.status_code", resp.status_code)
                return Response(content=resp.content, media_type="application/json")

Run this proxy with uvicorn tracing_proxy:app --port 8000 and point your clients to port 8000 instead of 8080. Every request will now generate a trace with rich attributes.

Key Metrics to Track

Beyond traces, you should collect metrics that give you a system-wide view. Here are the most important metrics for llama.cpp deployments:

• tokens_per_second: Generation throughput, measured during the decode loop.
• prompt_eval_tokens_per_second: How fast the model processes the input prompt.
• time_to_first_token: Latency from request arrival to first generated token. Critical for user experience.
• kv_cache_usage_ratio: Fraction of the context window currently occupied. Helps detect context overflow.
• model_load_time_ms: Time to load the model into memory. Important for cold starts.
• memory_usage_mb: RSS or VRAM consumption. Track for memory leaks over long-running sessions.
• queue_depth: Number of requests waiting for inference. Indicates saturation.
• error_rate: Percentage of requests that fail. Segment by error type.

Exporting Metrics with Prometheus

If you are using the Python sidecar approach, you can easily expose Prometheus metrics:

# metrics.py
from prometheus_client import Counter, Histogram, Gauge, generate_latest

# Define metrics
REQUEST_COUNT = Counter(
    "llama_requests_total",
    "Total inference requests",
    ["model", "status"]
)

REQUEST_DURATION = Histogram(
    "llama_request_duration_seconds",
    "Request duration in seconds",
    ["model"],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
)

TOKENS_GENERATED = Counter(
    "llama_tokens_generated_total",
    "Total tokens generated",
    ["model"]
)

TIME_TO_FIRST_TOKEN = Histogram(
    "llama_time_to_first_token_seconds",
    "Time to first token in seconds",
    ["model"],
    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0]
)

ACTIVE_REQUESTS = Gauge(
    "llama_active_requests",
    "Currently active inference requests"
)

KV_CACHE_USAGE = Gauge(
    "llama_kv_cache_usage_ratio",
    "KV cache usage ratio (0-1)",
    ["model"]
)

Expose these metrics at a /metrics endpoint and configure Prometheus to scrape them. Pair with Grafana for dashboards.

Best Practices for llama.cpp Observability

1. Always Tag Traces with Request IDs

Generate a unique identifier for every inference request and propagate it through all logs, traces, and metrics. This allows you to correlate a user complaint with the exact trace that caused it.

2. Separate Prompt Evaluation from Generation in Traces

These two phases have very different performance characteristics. Prompt evaluation is often compute-bound and parallelizable, while token generation is memory-bandwidth-bound and sequential. Merging them into a single span hides the real bottleneck.

3. Log Generation Parameters as Span Attributes

Always record temperature, top-k, top-p, repeat penalty, and max tokens as trace attributes. When output quality issues arise, these parameters are the first thing you need to inspect.

4. Monitor KV Cache Health

The KV cache is the most common source of subtle bugs. If it fills up, the model will silently truncate context or produce degraded output. Track kv_cache_usage_ratio and alert when it exceeds 80%.

5. Use Sampling, Not Full Capture, in Production

Capturing every trace in a high-traffic system creates excessive overhead. Use OpenTelemetry's sampling capabilities to capture 100% of errors and slow requests, but only a fraction of normal requests:

# Python example: configure a parent-based sampler with 10% base rate
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

sampler = ParentBased(root=TraceIdRatioBased(0.1))
trace.set_tracer_provider(TracerProvider(sampler=sampler))

6. Version Your Models in Traces

Include the model filename, quantization type, and any fine-tune version in every trace. When you swap models, you need to be able to compare performance before and after the change.

7. Watch for Memory Fragmentation in Long-Running Processes

llama.cpp can experience memory fragmentation when repeatedly loading and unloading models. Track RSS memory over time and restart the process if it grows beyond a threshold. A common pattern is to use a process manager like systemd or supervisord with automatic restart policies.

8. Correlate GPU Metrics with Inference Traces

If you are using GPU acceleration via CUDA or Metal, collect GPU utilization, memory usage, and temperature alongside your inference traces. Tools like nvidia-smi can be polled and the results attached as span attributes or exported as Prometheus metrics.

Putting It All Together: A Complete Observability Stack

For a production llama.cpp deployment, the recommended observability stack consists of:

• llama-server running the model on the backend.
• Python or Go sidecar proxy handling tracing and metrics collection.
• OpenTelemetry Collector aggregating and forwarding traces.
• Jaeger or Grafana Tempo for trace visualization.
• Prometheus for metrics scraping and storage.
• Grafana for unified dashboards combining traces, metrics, and logs.
• Loki or Elasticsearch for structured log aggregation.

This stack gives you the three pillars of observability — logs, metrics, and traces — working together. When a user reports slow responses, you can look at the Grafana dashboard to see if tokens-per-second dropped, click through to the Jaeger trace to find which phase was slow, and check Loki logs for any errors emitted by llama.cpp at that time.

Conclusion

Observability is not a luxury for production LLM systems — it is a necessity. llama.cpp provides the raw performance and flexibility needed for efficient local inference, but it is up to you to build the observability layer around it. By leveraging the built-in logging callbacks, wrapping inference calls in a custom tracing layer, integrating with OpenTelemetry for distributed tracing, and exporting metrics to Prometheus, you can achieve full visibility into your inference pipeline. Start with the basics: capture timing for each inference phase, tag everything with request IDs, and monitor KV cache usage. As your deployment grows, layer in more sophisticated tooling until you have a complete picture of system health. The investment pays off the first time you can diagnose a performance regression in minutes instead of hours, simply by opening a trace and seeing exactly where the time went.