Building a Multi-Agent Research Pipeline with llama.cpp: Complete Guide

Building a Multi-Agent Research Pipeline with llama.cpp: Complete Guide

Large language models are powerful, but a single prompt-response cycle rarely solves complex research tasks. When you need to gather information, synthesize findings, fact-check claims, and produce a polished report, a single agent often hallucinates, loses context, or produces shallow output. A multi-agent research pipeline solves this by decomposing the task into specialized roles, each handled by a dedicated LLM agent that focuses on one responsibility and passes structured output to the next stage.

This tutorial walks you through building a complete multi-agent research pipeline using llama.cpp, the lightweight C++ inference engine that runs quantized GGUF models locally. By the end, you will have a working pipeline with a Planner, Researcher, Analyzer, Fact-Checker, and Writer agent that collaborate to produce a research report — all running on your own hardware with no external API calls.

What Is a Multi-Agent Research Pipeline?

A multi-agent research pipeline is an orchestrated workflow where several LLM-powered agents, each with a distinct persona and responsibility, collaborate to complete a research objective. Instead of asking one model to do everything, you assign narrow tasks to specialized agents and chain their outputs together.

A typical pipeline includes these roles:

• Planner Agent — Decomposes the research question into sub-questions and a research plan.
• Researcher Agent — Explores each sub-question, retrieves relevant context, and drafts raw findings.
• Analyzer Agent — Synthesizes raw findings, identifies patterns, and extracts key insights.
• Fact-Checker Agent — Verifies claims, flags contradictions, and requests corrections.
• Writer Agent — Produces the final structured report from verified findings.

Each agent runs inference through llama.cpp, which loads a quantized model (such as Llama 3 8B Q4_K_M) into memory and generates text efficiently on CPU or GPU. Because everything runs locally, you get privacy, zero per-token cost, and full control over the inference parameters.

Why It Matters

Single-agent approaches fail on complex research for several reasons. Context windows get filled with irrelevant intermediate text, the model loses track of its role, and there is no internal verification. A multi-agent pipeline addresses these issues by isolating concerns, enabling parallel exploration of sub-questions, and introducing a verification step before output is finalized.

Running this on llama.cpp specifically gives you several advantages. You avoid vendor lock-in and API rate limits. You can run the entire pipeline offline, which matters for sensitive research domains. You can swap models freely — try a 7B model for speed or a 70B model for depth — without changing your pipeline code. And you can tune quantization levels to balance quality against memory usage.

Prerequisites and Setup

Installing llama-cpp-python

The easiest way to interact with llama.cpp from Python is through the llama-cpp-python package. Install it with hardware acceleration if available:

# CPU-only install
pip install llama-cpp-python

# With CUDA acceleration (Linux, NVIDIA GPU)
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

# With Metal acceleration (Apple Silicon)
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

You will also need a quantized GGUF model. Download one from Hugging Face, for example Meta-Llama-3-8B-Instruct.Q4_K_M.gguf. Place it in a models/ directory within your project.

Project Structure

research_pipeline/
├── models/
│   └── Meta-Llama-3-8B-Instruct.Q4_K_M.gguf
├── pipeline/
│   ├── __init__.py
│   ├── llm_engine.py
│   ├── agents.py
│   ├── orchestrator.py
│   └── memory.py
├── main.py
└── requirements.txt

Building the LLM Engine

The first component is a thin wrapper around llama-cpp-python that handles model loading, chat formatting, and generation. Centralizing this in one class lets every agent share the same loaded model instance, which saves memory and avoids reloading the model for each agent call.

# pipeline/llm_engine.py

from llama_cpp import Llama
from typing import Optional


class LLMEngine:
    """Shared inference engine wrapping llama.cpp."""

    def __init__(
        self,
        model_path: str,
        n_ctx: int = 8192,
        n_gpu_layers: int = -1,
        n_threads: Optional[int] = None,
    ):
        self.llm = Llama(
            model_path=model_path,
            n_ctx=n_ctx,
            n_gpu_layers=n_gpu_layers,
            n_threads=n_threads,
            verbose=False,
        )

    def chat(
        self,
        system_prompt: str,
        user_prompt: str,
        max_tokens: int = 1024,
        temperature: float = 0.7,
        top_p: float = 0.9,
        stop: Optional[list] = None,
    ) -> str:
        """Generate a response using chat formatting."""
        messages = [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ]
        response = self.llm.create_chat_completion(
            messages=messages,
            max_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            stop=stop or [],
        )
        return response["choices"][0]["message"]["content"].strip()

The n_ctx parameter sets the context window size. Research agents often need large context to hold retrieved material, so 8192 tokens is a reasonable starting point. The n_gpu_layers value of -1 offloads all layers to the GPU when available.

Designing the Agent Base Class

Every agent shares common behavior: it has a system prompt defining its role, it receives input, and it produces structured output. We define an abstract base class that encapsulates this pattern.

# pipeline/agents.py

from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Any
from pipeline.llm_engine import LLMEngine


@dataclass
class AgentResult:
    agent_name: str
    output: str
    metadata: dict = field(default_factory=dict)


class BaseAgent(ABC):
    """Base class for all pipeline agents."""

    name: str = "base_agent"
    system_prompt: str = "You are a helpful assistant."

    def __init__(self, engine: LLMEngine, temperature: float = 0.7):
        self.engine = engine
        self.temperature = temperature

    @abstractmethod
    def build_user_prompt(self, input_data: Any) -> str:
        """Construct the user prompt from input data."""
        pass

    def run(self, input_data: Any, max_tokens: int = 1024) -> AgentResult:
        user_prompt = self.build_user_prompt(input_data)
        output = self.engine.chat(
            system_prompt=self.system_prompt,
            user_prompt=user_prompt,
            max_tokens=max_tokens,
            temperature=self.temperature,
        )
        return AgentResult(
            agent_name=self.name,
            output=output,
            metadata={"temperature": self.temperature},
        )

Implementing the Specialized Agents

Now we implement each of the five agents. Each one overrides build_user_prompt and sets a tailored system prompt. Temperature settings vary by role: the Planner and Writer benefit from moderate creativity, while the Fact-Checker should be deterministic.

Planner Agent

class PlannerAgent(BaseAgent):
    name = "planner"
    system_prompt = (
        "You are a research planner. Given a research question, break it down "
        "into 3 to 6 specific sub-questions that collectively cover the topic. "
        "Output ONLY a numbered list of sub-questions, one per line. "
        "Do not include any other text or explanation."
    )

    def __init__(self, engine: LLMEngine):
        super().__init__(engine, temperature=0.5)

    def build_user_prompt(self, input_data: str) -> str:
        return f"Research question: {input_data}\n\nProvide the sub-questions:"

Researcher Agent

class ResearcherAgent(BaseAgent):
    name = "researcher"
    system_prompt = (
        "You are a research assistant. Given a specific sub-question, provide "
        "a thorough factual answer based on your training knowledge. Include "
        "relevant details, dates, names, and figures where applicable. "
        "If you are uncertain about a claim, explicitly state your uncertainty. "
        "Do not fabricate sources or citations."
    )

    def __init__(self, engine: LLMEngine):
        super().__init__(engine, temperature=0.3)

    def build_user_prompt(self, input_data: str) -> str:
        return f"Sub-question to research: {input_data}"

Analyzer Agent

class AnalyzerAgent(BaseAgent):
    name = "analyzer"
    system_prompt = (
        "You are a research analyst. You receive raw findings from multiple "
        "sub-questions. Synthesize them into a coherent summary. Identify "
        "key themes, contradictions, and gaps. Output your synthesis as "
        "structured text with clear section headers."
    )

    def __init__(self, engine: LLMEngine):
        super().__init__(engine, temperature=0.4)

    def build_user_prompt(self, input_data: str) -> str:
        return (
            "Below are raw research findings from multiple sub-questions.\n\n"
            f"{input_data}\n\n"
            "Synthesize these findings into a structured analysis."
        )

Fact-Checker Agent

class FactCheckerAgent(BaseAgent):
    name = "fact_checker"
    system_prompt = (
        "You are a fact-checker. Review the provided analysis for factual "
        "errors, logical inconsistencies, and unsupported claims. For each "
        "issue found, state the problematic claim and explain the concern. "
        "If no issues are found, output exactly: VERIFIED"
    )

    def __init__(self, engine: LLMEngine):
        super().__init__(engine, temperature=0.1)

    def build_user_prompt(self, input_data: str) -> str:
        return f"Analysis to fact-check:\n\n{input_data}"

    def is_verified(self, result: AgentResult) -> bool:
        return result.output.strip().upper() == "VERIFIED"

Writer Agent

class WriterAgent(BaseAgent):
    name = "writer"
    system_prompt = (
        "You are a technical writer. Produce a well-structured research "
        "report from the provided verified analysis. Use markdown headings, "
        "bullet points where appropriate, and a clear introduction and "
        "conclusion. Maintain an objective, professional tone."
    )

    def __init__(self, engine: LLMEngine):
        super().__init__(engine, temperature=0.6)

    def build_user_prompt(self, input_data: str) -> str:
        return (
            "Write the final research report based on this verified analysis:\n\n"
            f"{input_data}"
        )

Adding Shared Memory

Agents need to share intermediate results. A simple in-memory store keeps the pipeline state organized and makes it easy to inspect what each agent produced, which is invaluable for debugging.

# pipeline/memory.py

from dataclasses import dataclass, field
from typing import Any


@dataclass
class PipelineMemory:
    """Stores intermediate results between pipeline stages."""
    research_question: str = ""
    sub_questions: list = field(default_factory=list)
    findings: dict = field(default_factory=dict)
    analysis: str = ""
    fact_check_result: str = ""
    fact_check_passed: bool = False
    final_report: str = ""
    log: list = field(default_factory=list)

    def add_log(self, agent_name: str, output: str):
        self.log.append({"agent": agent_name, "output": output})

    def get_findings_text(self) -> str:
        lines = []
        for sq, finding in self.findings.items():
            lines.append(f"### Sub-question: {sq}\n{finding}\n")
        return "\n".join(lines)

Orchestrating the Pipeline

The orchestrator ties everything together. It instantiates the agents, sequences their execution, handles the fact-checking loop, and writes results to memory. The fact-checker can trigger a re-analysis if it finds issues, with a configurable maximum number of retries.

# pipeline/orchestrator.py

from pipeline.llm_engine import LLMEngine
from pipeline.agents import (
    PlannerAgent,
    ResearcherAgent,
    AnalyzerAgent,
    FactCheckerAgent,
    WriterAgent,
)
from pipeline.memory import PipelineMemory


class ResearchOrchestrator:
    def __init__(self, engine: LLMEngine, max_fact_check_retries: int = 2):
        self.planner = PlannerAgent(engine)
        self.researcher = ResearcherAgent(engine)
        self.analyzer = AnalyzerAgent(engine)
        self.fact_checker = FactCheckerAgent(engine)
        self.writer = WriterAgent(engine)
        self.max_retries = max_fact_check_retries

    def run(self, research_question: str) -> PipelineMemory:
        memory = PipelineMemory(research_question=research_question)
        print(f"[Pipeline] Starting research: {research_question}")

        # Step 1: Plan
        plan_result = self.planner.run(research_question, max_tokens=512)
        memory.sub_questions = [
            line.strip().lstrip("0123456789. ")
            for line in plan_result.output.split("\n")
            if line.strip()
        ]
        memory.add_log("planner", plan_result.output)
        print(f"[Pipeline] Planner generated {len(memory.sub_questions)} sub-questions")

        # Step 2: Research each sub-question
        for i, sq in enumerate(memory.sub_questions, 1):
            print(f"[Pipeline] Researching sub-question {i}/{len(memory.sub_questions)}")
            research_result = self.researcher.run(sq, max_tokens=1024)
            memory.findings[sq] = research_result.output
            memory.add_log("researcher", research_result.output)

        # Step 3 + 4: Analyze and fact-check (with retry loop)
        for attempt in range(self.max_retries + 1):
            print(f"[Pipeline] Analysis attempt {attempt + 1}")
            analysis_result = self.analyzer.run(
                memory.get_findings_text(), max_tokens=2048
            )
            memory.analysis = analysis_result.output
            memory.add_log("analyzer", analysis_result.output)

            print("[Pipeline] Fact-checking analysis...")
            check_result = self.fact_checker.run(memory.analysis, max_tokens=1024)
            memory.fact_check_result = check_result.output
            memory.add_log("fact_checker", check_result.output)

            if self.fact_checker.is_verified(check_result):
                memory.fact_check_passed = True
                print("[Pipeline] Fact-check passed")
                break
            else:
                print(f"[Pipeline] Fact-check found issues (attempt {attempt + 1})")
                # Append the fact-check feedback to findings so the analyzer
                # can address the concerns on the next iteration.
                memory.findings["__fact_check_feedback__"] = check_result.output

        # Step 5: Write the final report
        print("[Pipeline] Writing final report...")
        report_result = self.writer.run(memory.analysis, max_tokens=2048)
        memory.final_report = report_result.output
        memory.add_log("writer", report_result.output)
        print("[Pipeline] Done.")

        return memory

Running the Pipeline

With all components in place, the entry point loads the model, creates the engine, and runs the orchestrator on a research question.

# main.py

from pipeline.llm_engine import LLMEngine
from pipeline.orchestrator import ResearchOrchestrator

MODEL_PATH = "models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"


def main():
    engine = LLMEngine(
        model_path=MODEL_PATH,
        n_ctx=8192,
        n_gpu_layers=-1,
    )

    orchestrator = ResearchOrchestrator(engine, max_fact_check_retries=2)

    question = (
        "What are the key technical and economic trade-offs between "
        "nuclear fission and solar power for grid-scale electricity generation?"
    )

    memory = orchestrator.run(question)

    print("\n" + "=" * 60)
    print("FINAL RESEARCH REPORT")
    print("=" * 60)
    print(memory.final_report)

    print("\n" + "=" * 60)
    print("PIPELINE TRACE")
    print("=" * 60)
    for entry in memory.log:
        print(f"\n--- {entry['agent']} ---")
        print(entry["output"][:300] + "..." if len(entry["output"]) > 300 else entry["output"])


if __name__ == "__main__":
    main()

Run it with:

python main.py

Best Practices

• Keep system prompts narrow. Each agent should have a single, clearly defined responsibility. Avoid giving an agent multiple tasks, as this degrades output quality and makes debugging harder.
• Tune temperature per agent. Use low temperature (0.1–0.3) for fact-checking and research where accuracy matters. Use moderate temperature (0.5–0.7) for planning and writing where some creativity helps.
• Use structured output formats. Ask agents to output numbered lists, JSON, or markdown sections. This makes parsing between stages reliable. For stricter formatting, consider adding a JSON schema validator.
• Cap retry loops. The fact-checker loop should always have a maximum retry count to prevent infinite cycles where the analyzer and fact-checker disagree indefinitely.
• Log every stage. Store each agent's raw output in memory. When the final report has a problem, you can trace exactly which stage introduced the error.
• Choose the right model size. A 7B–8B quantized model works well for planning and drafting. For deeper reasoning in the analyzer or fact-checker, consider a larger model or a second model loaded for those specific stages.
• Manage context carefully. Concatenating all findings can exceed the context window on long research tasks. Summarize or truncate earlier findings before passing them to downstream agents if needed.
• Add retrieval for real research. The Researcher agent in this tutorial relies on model knowledge. For production use, integrate a retrieval step — such as a local vector database or web search — so the agent can ground its findings in current sources.

Conclusion

Building a multi-agent research pipeline with llama.cpp gives you a private, cost-effective, and highly controllable system for complex research tasks. By decomposing the work into specialized agents — each with a focused system prompt, tuned generation parameters, and a clear position in the workflow — you get higher-quality output than any single prompt could produce. The fact-checking loop adds a self-correction mechanism that catches errors before they reach the final report. From here, you can extend the pipeline by adding a retrieval-augmented Researcher agent, parallelizing sub-question research with concurrent inference calls, or swapping in different model sizes for different stages. The architecture is modular by design, so each agent can be improved or replaced independently without rewriting the entire pipeline.