Building a Log Analysis Agent with llama.cpp: Complete Guide

Building a Log Analysis Agent with llama.cpp: Complete Guide

Log analysis is one of those operational tasks that scales poorly with human attention. As systems grow, the volume of logs explodes, and finding the meaningful signal in a sea of timestamps becomes a needle-in-a-haystack problem. Traditional approaches—grep, awk, ELK dashboards—work well for known patterns but struggle with novel errors, ambiguous stack traces, and cross-correlating events across services. This is where a local LLM-powered agent shines. By combining llama.cpp's efficient inference engine with a simple agent loop, you can build a tool that reads logs, reasons about them, and surfaces actionable insights—all running on your own hardware, with no data leaving your machine.

What Is a Log Analysis Agent?

A log analysis agent is a program that uses a large language model to interpret log files. Unlike a one-shot prompt where you paste logs into a chat window, an agent operates in a loop: it reads input, decides what to do, takes an action (such as grepping a file, counting error occurrences, or summarizing a time window), observes the result, and iterates until it has a useful answer. The "agent" part is the decision-making loop; the LLM is the reasoning engine behind it.

Using llama.cpp specifically gives you three advantages: it runs locally so sensitive logs never leave your infrastructure, it supports a wide range of quantized models (GGUF format) that fit on consumer GPUs or even CPUs, and it exposes a stable C API plus HTTP server that you can call from any language.

Why It Matters

• Privacy and compliance: Logs often contain PII, secrets, or internal architecture details. Sending them to a third-party API may violate policy or regulation.
• Cost predictability: No per-token billing. Once you have the hardware, inference is free.
• Latency: Local inference avoids network round-trips, which matters when an agent makes dozens of tool calls per analysis.
• Customization: You control the system prompt, the tools, and the stopping criteria. You can fine-tune behavior for your specific log formats.
• Offline operation: Air-gapped environments, on-call laptops on bad hotel Wi-Fi, edge deployments—all work fine.

Prerequisites and Setup

You will need a machine with at least 16 GB of RAM (32 GB preferred for larger models), a recent C++ compiler, CMake, and Python 3.10+. A CUDA-capable GPU is helpful but not required. For this tutorial we will use the Python bindings (llama-cpp-python) because they expose the full API while keeping the code readable.

First, build or install llama.cpp. The fastest path is the Python wheel:

pip install llama-cpp-python[server]

If you have an NVIDIA GPU and want hardware acceleration, install with the appropriate build flags:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python[server] --upgrade --force-reinstall --no-cache-dir

Next, download a model in GGUF format. For log analysis you want a model with strong reasoning and instruction-following ability. Good choices include Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, or Phi-3-medium-128k-instruct if you need a long context window for large log batches. Download a Q4_K_M quantization, which offers a good balance of quality and memory footprint:

mkdir -p models
curl -L -o models/qwen2.5-7b-instruct-q4_k_m.gguf \
  https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf


Architecture of the Agent

Our agent has four components:


  
Model wrapper: Loads the GGUF file and exposes a chat() method that returns text completions.
  
Tool registry: A dictionary of named functions the agent can call, each with a JSON schema describing its arguments.
  
Agent loop: Sends the conversation history to the model, parses any tool-call requests, executes them, appends results, and repeats until the model produces a final answer or hits a step limit.
  
Log tools: The actual functions: read_log_file, grep_logs, count_errors, summarize_window, and list_files.


Step 1: Loading the Model

Create a file called log_agent.py. We start with the model wrapper:

from llama_cpp import Llama
from typing import List, Dict, Any
import json
import re

MODEL_PATH = "models/qwen2.5-7b-instruct-q4_k_m.gguf"

llm = Llama(
    model_path=MODEL_PATH,
    n_ctx=16384,          # context window in tokens
    n_gpu_layers=-1,      # offload all layers to GPU if available
    verbose=False,
)

def chat(messages: List[Dict[str, str]], temperature: float = 0.2, max_tokens: int = 1024) -> str:
    """Send a chat-formatted message list and return the assistant text."""
    response = llm.create_chat_completion(
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens,
        stop=["</tool_call>"],
    )
    return response["choices"][0]["message"]["content"]


The n_ctx value matters: log analysis generates a lot of tokens. 16K is a reasonable starting point; if you load a long-context model you can push this higher. Keep temperature low because we want deterministic tool-calling behavior.

Step 2: Defining the Tools

Each tool is a plain Python function plus a schema describing its arguments. We use a simple XML-style protocol for tool calls so the tutorial works with any instruct model, not just those with native function-calling support.

import subprocess
from datetime import datetime
from collections import Counter

def list_files(directory: str) -> str:
    """List files in a directory."""
    import os
    try:
        files = os.listdir(directory)
        return json.dumps({"files": files})
    except Exception as e:
        return json.dumps({"error": str(e)})

def read_log_file(path: str, max_lines: int = 200) -> str:
    """Read the last N lines of a log file."""
    try:
        result = subprocess.run(
            ["tail", "-n", str(max_lines), path],
            capture_output=True, text=True, timeout=10
        )
        return result.stdout if result.returncode == 0 else result.stderr
    except Exception as e:
        return f"Error: {e}"

def grep_logs(path: str, pattern: str, max_matches: int = 50) -> str:
    """Search a log file for lines matching a regex pattern."""
    try:
        result = subprocess.run(
            ["grep", "-E", pattern, path],
            capture_output=True, text=True, timeout=15
        )
        lines = result.stdout.splitlines()
        return "\n".join(lines[:max_matches])
    except Exception as e:
        return f"Error: {e}"

def count_errors(path: str, level: str = "ERROR") -> str:
    """Count occurrences of a log level in a file."""
    try:
        result = subprocess.run(
            ["grep", "-c", level, path],
            capture_output=True, text=True, timeout=10
        )
        return json.dumps({"count": int(result.stdout.strip() or 0), "level": level})
    except Exception as e:
        return json.dumps({"error": str(e)})

def summarize_window(path: str, start_time: str, end_time: str) -> str:
    """Extract log lines between two ISO timestamps."""
    try:
        result = subprocess.run(
            ["awk", f"/{start_time}/,/{end_time}/", path],
            capture_output=True, text=True, timeout=15
        )
        lines = result.stdout.splitlines()
        return "\n".join(lines[:300])
    except Exception as e:
        return f"Error: {e}"

TOOLS = {
    "list_files": list_files,
    "read_log_file": read_log_file,
    "grep_logs": grep_logs,
    "count_errors": count_errors,
    "summarize_window": summarize_window,
}

TOOL_SCHEMAS = [
    {
        "name": "list_files",
        "description": "List files in a directory.",
        "params": {"directory": "string: path to directory"}
    },
    {
        "name": "read_log_file",
        "description": "Read the last N lines of a log file.",
        "params": {"path": "string: file path", "max_lines": "int: max lines (default 200)"}
    },
    {
        "name": "grep_logs",
        "description": "Search a log file for lines matching a regex pattern.",
        "params": {"path": "string", "pattern": "string: regex", "max_matches": "int (default 50)"}
    },
    {
        "name": "count_errors",
        "description": "Count occurrences of a log level in a file.",
        "params": {"path": "string", "level": "string: e.g. ERROR, WARN"}
    },
    {
        "name": "summarize_window",
        "description": "Extract log lines between two timestamps.",
        "params": {"path": "string", "start_time": "string", "end_time": "string"}
    },
]


Step 3: The System Prompt

The system prompt is the most important tuning lever. It must explain the agent's role, the available tools, the exact format for tool calls, and the rules for when to stop.

SYSTEM_PROMPT = """You are a log analysis agent. Your job is to investigate log files,
identify problems, and produce a clear written report for an on-call engineer.

You have access to the following tools:

{tools}

To call a tool, output exactly this format on its own line:

<tool_call>
{{"name": "tool_name", "args": {{"arg1": "value1"}}}}
</tool_call>

Rules:
1. Call only one tool at a time. Wait for the result before continuing.
2. After receiving a tool result, reason briefly about what it tells you.
3. When you have enough information, write your final report prefixed with "FINAL REPORT:".
4. Be specific: cite timestamps, error messages, and counts.
5. If a tool returns an error, try a different approach rather than repeating.
6. Do not invent log content. Only report what tools actually returned.
""".format(
    tools=json.dumps(TOOL_SCHEMAS, indent=2)
)


Step 4: The Agent Loop

The loop is the heart of the agent. It sends messages, looks for tool calls in the response, executes them, and feeds results back. We cap iterations to prevent infinite loops.

TOOL_CALL_RE = re.compile(r"<tool_call>\s*(\{.*?\})\s*</tool_call>", re.DOTALL)

def run_agent(query: str, max_steps: int = 12) -> str:
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": query},
    ]

    for step in range(max_steps):
        print(f"\n--- Step {step + 1} ---")
        assistant_text = chat(messages)
        print(f"Assistant: {assistant_text[:500]}")
        messages.append({"role": "assistant", "content": assistant_text})

        # Check for final report
        if "FINAL REPORT:" in assistant_text:
            return assistant_text.split("FINAL REPORT:", 1)[1].strip()

        # Check for tool calls
        match = TOOL_CALL_RE.search(assistant_text)
        if not match:
            # No tool call and no final report; prompt the model to conclude
            messages.append({
                "role": "user",
                "content": "Please either call a tool or provide your FINAL REPORT."
            })
            continue

        try:
            call = json.loads(match.group(1))
            tool_name = call["name"]
            tool_args = call.get("args", {})
        except (json.JSONDecodeError, KeyError) as e:
            messages.append({
                "role": "user",
                "content": f"Invalid tool call format: {e}. Please retry."
            })
            continue

        if tool_name not in TOOLS:
            messages.append({
                "role": "user",
                "content": f"Unknown tool: {tool_name}. Available: {list(TOOLS.keys())}"
            })
            continue

        print(f"Calling {tool_name} with {tool_args}")
        result = TOOLS[tool_name](**tool_args)
        print(f"Result (truncated): {result[:300]}")
        messages.append({
            "role": "user",
            "content": f"Tool result for {tool_name}:\n{result}"
        })

    return "Agent reached maximum steps without a final report."


Step 5: Running the Agent

Add an entry point and test it against a sample log directory:

if __name__ == "__main__":
    query = """Investigate the logs in /var/log/myapp/ from today.
    The on-call engineer reports elevated 500 errors starting around 14:00.
    Identify the root cause, list affected endpoints, and estimate blast radius."""

    report = run_agent(query)
    print("\n" + "=" * 60)
    print("AGENT REPORT")
    print("=" * 60)
    print(report)


Run it:

python log_agent.py


A typical run looks like this: the agent first calls list_files to see what logs exist, then count_errors on the main application log, then grep_logs for 500 around the reported time, then summarize_window to read the surrounding context, and finally produces a structured report. The whole loop might take 30 to 90 seconds on a 7B model depending on hardware.

Step 6: Streaming Output for Better UX

For interactive use, streaming tokens as they generate makes the agent feel much faster. llama-cpp-python supports this via a callback:

def chat_stream(messages, temperature=0.2, max_tokens=1024):
    chunks = []
    for chunk in llm.create_chat_completion(
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens,
        stream=True,
        stop=["</tool_call>"],
    ):
        delta = chunk["choices"][0]["delta"].get("content", "")
        chunks.append(delta)
        print(delta, end="", flush=True)
    print()
    return "".join(chunks)


Swap chat() for chat_stream() in the loop and you will see the agent's reasoning unfold in real time.

Best Practices

Choose the right model size. A 7B model handles structured log analysis well. For complex multi-step reasoning across many files, a 14B or 32B model may be worth the extra memory. Always benchmark on your actual log formats before committing.

Constrain the context window deliberately. Logs are token-dense. Reading a 10,000-line file directly into context will blow your budget. Instead, use tools to filter first—grep, count, and window—then feed only relevant slices to the model. This is why the agent pattern beats naive paste-the-whole-log prompting.

Use low temperature for tool calls. Set temperature=0.1 to 0.3 during investigation. Higher temperatures cause the model to hallucinate tool names or malformed JSON. If you want a more creative final summary, you can do a second pass at temperature=0.7 using only the collected evidence.

Implement retry and validation. Models occasionally emit malformed tool calls. The loop above handles this by asking the model to retry. For production, add JSON schema validation on arguments and rate-limit tool calls to prevent runaway loops.

Log everything the agent does. Persist the full message history to disk for each run. This is invaluable for debugging bad agent behavior and for building a dataset to fine-tune the system prompt or the model itself.

Sanitize sensitive data before analysis. Even though inference is local, logs may contain secrets that you do not want embedded in the agent's context or saved to disk. Consider a preprocessing step that redacts API keys, tokens, and PII before the agent reads them.

Cache repeated queries. If the agent frequently re-reads the same log file, cache the tool results keyed by file path and modification time. This avoids redundant tail and grep calls across agent invocations.

Set a hard step limit. The max_steps parameter is your safety net. A confused model can loop indefinitely between two tools. Twelve steps is usually enough for a focused investigation; raise it for complex multi-file forensics.

Test with synthetic logs. Create a fixture log file with known errors and verify the agent finds them. This gives you a regression test when you swap models or change prompts.

Extending the Agent

Once the basic loop works, you can add powerful tools:


  
query_prometheus—hit your metrics endpoint to correlate log errors with traffic spikes or latency.
  
trace_request—follow a request ID across multiple service logs.
  
diff_logs—compare today's error patterns against yesterday's baseline.
  
run_sql—query a local SQLite index of parsed logs for aggregation.
  
send_alert—post the final report to Slack or PagerDuty.


Each new tool is just a function plus a schema entry. The agent loop does not change. This extensibility is the core strength of the agent pattern over a monolithic prompt.

Performance Tuning

If inference feels slow, check these levers in order of impact:


  
GPU offload: Ensure n_gpu_layers=-1 and that the build actually linked against CUDA or Metal. Check llama_cpp logs at startup.
  
Batch size: Increase n_batch from the default 512 to 2048 for faster prompt processing on long contexts.
  
Quantization: Drop from Q4_K_M to Q3_K_M if memory-constrained, or move to Q5_K_M if you have headroom and want better reasoning.
  
Context length: Do not set n_ctx higher than you need. Attention cost scales quadratically with sequence length in many implementations.
  
Thread count: On CPU-only setups, set n_threads to your physical core count, not logical.


Conclusion

Building a log analysis agent with llama.cpp gives you a private, cost-effective, and highly customizable tool for operational debugging. The architecture is straightforward: a local GGUF model, a handful of shell-backed tools, a system prompt that enforces a tool-calling protocol, and a loop that ties them together. The real power emerges when you tailor the tools to your specific environment—adding metrics queries, trace following, and alerting transforms a generic chatbot into a specialized SRE assistant that lives inside your infrastructure. Start with the code above, point it at a real log directory, and iterate on the system prompt until its reports match what your on-call engineers actually need. The investment in setup pays back the first time the agent catches an incident pattern at 3 AM that would have taken you an hour to grep by hand.