← Back to DevBytes
Developer Tutorial August 17, 2026 10 min read

Building a Knowledge Base Chatbot with llama.cpp: Complete Guide

Building a Knowledge Base Chatbot with llama.cpp: Complete Guide

Large language models are powerful, but they are frozen in time at the moment of their training. When users ask about proprietary documents, internal wikis, or recent events, even the largest models hallucinate or simply refuse to answer. The solution is retrieval-augmented generation (RAG): a pattern where relevant context is fetched from a knowledge base and injected into the model's prompt before it generates a response. In this guide, we will build a fully functional knowledge base chatbot using llama.cpp, the lightweight C++ inference engine that lets you run quantized LLMs on commodity hardware.

What Is llama.cpp and Why Use It for a Knowledge Base Chatbot?

llama.cpp is an open-source C/C++ library originally created to run Meta's LLaMA models on consumer CPUs. It has since grown into a general-purpose inference engine supporting GGUF-formatted models from many families: Llama, Mistral, Phi, Qwen, Gemma, and more. It exposes a simple HTTP server with an OpenAI-compatible API, making it easy to plug into existing tooling.

For a knowledge base chatbot, llama.cpp offers several advantages:

Architecture Overview

Our chatbot will follow a classic RAG architecture with four components:

We will use Python for orchestration because it has excellent libraries for embeddings and vector storage, while llama.cpp handles the heavy lifting of generation. The same pattern can be implemented in any language that can speak HTTP.

Prerequisites and Installation

Before writing code, install the required components. You will need Python 3.10 or newer, a C++ compiler, and cmake.

First, clone and build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j

Next, download a chat model in GGUF format. A good starting point is a quantized Llama 3 8B model, which balances quality and speed. Place the .gguf file in a known path, for example ./models/llama-3-8b-instruct-q4_k_m.gguf.

Install the Python dependencies:

pip install sentence-transformers numpy

We will use sentence-transformers for embeddings and a simple NumPy-based vector store to keep the tutorial dependency-free. For production, you would swap this for Chroma, FAISS, or Qdrant.

Step 1: Preparing the Knowledge Base

Create a directory called knowledge_base and add some text files. For this tutorial, imagine a small company FAQ. Here is an example file at knowledge_base/leave_policy.txt:

Annual Leave Policy
====================
Employees are entitled to 25 days of paid annual leave per calendar year.
Unused leave can be carried over up to 5 days into the next year.
Leave requests must be submitted at least 14 days in advance through the HR portal.
Sick leave does not count against annual leave and requires a doctor's note after 3 consecutive days.

Add a few more files covering topics like remote work, expenses, and onboarding. The more documents you add, the more useful the chatbot becomes.

Step 2: Chunking and Embedding Documents

LLMs have limited context windows, so we cannot feed entire documents into the prompt. Instead, we split documents into smaller chunks, embed each chunk, and store the embeddings for later retrieval. A chunk size of around 300 to 500 characters works well for short FAQ-style documents.

Create a file called build_index.py:

import os
import json
import numpy as np
from sentence_transformers import SentenceTransformer

KB_DIR = "knowledge_base"
CHUNK_SIZE = 400
CHUNK_OVERLAP = 50
EMBED_MODEL = "all-MiniLM-L6-v2"

def chunk_text(text, size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks

def main():
    model = SentenceTransformer(EMBED_MODEL)
    documents = []

    for filename in os.listdir(KB_DIR):
        path = os.path.join(KB_DIR, filename)
        if not os.path.isfile(path):
            continue
        with open(path, "r", encoding="utf-8") as f:
            text = f.read()
        for i, chunk in enumerate(chunk_text(text)):
            documents.append({
                "id": f"{filename}_{i}",
                "source": filename,
                "text": chunk
            })

    texts = [d["text"] for d in documents]
    embeddings = model.encode(texts, normalize_embeddings=True)
    embeddings = np.array(embeddings, dtype=np.float32)

    np.save("embeddings.npy", embeddings)
    with open("documents.json", "w", encoding="utf-8") as f:
        json.dump(documents, f, ensure_ascii=False, indent=2)

    print(f"Indexed {len(documents)} chunks from {len(os.listdir(KB_DIR))} files.")

if __name__ == "__main__":
    main()

Run the script to build the index:

python build_index.py

This produces two files: embeddings.npy containing the vector matrix and documents.json mapping each vector back to its source text. The all-MiniLM-L6-v2 model produces 384-dimensional embeddings, which is small and fast while still performing well on semantic similarity tasks.

Step 3: Starting the llama.cpp Server

With the index built, start the llama.cpp server. The server binary is located in build/bin/ after compilation. Launch it with your chosen model:

./build/bin/llama-server \
  -m ./models/llama-3-8b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 8192 \
  -t 4 \
  -ngl 0

Key flags explained:

Once running, the server exposes an OpenAI-compatible endpoint at http://localhost:8080/v1/chat/completions. You can test it with curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}]}'

Step 4: Building the Retriever

The retriever takes a user question, embeds it using the same model, and finds the most similar document chunks via cosine similarity. Because we normalized embeddings during indexing, cosine similarity reduces to a simple dot product.

Create retriever.py:

import json
import numpy as np
from sentence_transformers import SentenceTransformer

class Retriever:
    def __init__(self, embeddings_path="embeddings.npy", docs_path="documents.json"):
        self.embeddings = np.load(embeddings_path)
        with open(docs_path, "r", encoding="utf-8") as f:
            self.documents = json.load(f)
        self.model = SentenceTransformer("all-MiniLM-L6-v2")

    def search(self, query, top_k=3):
        query_vec = self.model.encode([query], normalize_embeddings=True)
        scores = self.embeddings @ query_vec.T
        scores = scores.flatten()
        top_indices = np.argsort(scores)[::-1][:top_k]
        results = []
        for idx in top_indices:
            results.append({
                "text": self.documents[idx]["text"],
                "source": self.documents[idx]["source"],
                "score": float(scores[idx])
            })
        return results

if __name__ == "__main__":
    r = Retriever()
    for hit in r.search("How many days of annual leave do I get?"):
        print(f"[{hit['score']:.3f}] {hit['source']}: {hit['text'][:80]}")

Run it to verify retrieval works:

python retriever.py

You should see the leave policy chunk ranked highest, confirming that semantic search is functioning correctly.

Step 5: Assembling the Chatbot

Now we combine the retriever with the llama.cpp server. The chatbot follows a simple loop: receive a question, retrieve relevant chunks, construct a prompt with the context, send it to the model, and stream the answer back to the user.

Create chatbot.py:

import json
import requests
from retriever import Retriever

LLAMA_URL = "http://localhost:8080/v1/chat/completions"
MODEL_NAME = "local-llama"
TOP_K = 3

SYSTEM_PROMPT = """You are a helpful assistant that answers questions based strictly on the provided context.
If the answer is not contained in the context, say you do not know. Do not make up information.
Always cite the source file at the end of your answer."""

def build_context(hits):
    context_parts = []
    for i, hit in enumerate(hits, 1):
        context_parts.append(f"[{i}] (Source: {hit['source']})\n{hit['text']}")
    return "\n\n".join(context_parts)

def ask_question(retriever, question, history=None):
    hits = retriever.search(question, top_k=TOP_K)
    context = build_context(hits)

    user_content = f"Context:\n{context}\n\nQuestion: {question}"

    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    if history:
        messages.extend(history)
    messages.append({"role": "user", "content": user_content})

    payload = {
        "model": MODEL_NAME,
        "messages": messages,
        "temperature": 0.3,
        "max_tokens": 512,
        "stream": False
    }

    response = requests.post(LLAMA_URL, json=payload)
    response.raise_for_status()
    answer = response.json()["choices"][0]["message"]["content"]

    sources = [hit["source"] for hit in hits]
    return answer, sources, messages

def main():
    retriever = Retriever()
    history = []
    print("Knowledge Base Chatbot (type 'quit' to exit)\n")

    while True:
        question = input("You: ").strip()
        if question.lower() in ("quit", "exit"):
            break
        if not question:
            continue

        answer, sources, messages = ask_question(retriever, question, history)
        print(f"\nBot: {answer}")
        print(f"Sources: {', '.join(sources)}\n")

        history = messages
        history.append({"role": "assistant", "content": answer})
        history = history[-10:]

if __name__ == "__main__":
    main()

Run the chatbot:

python chatbot.py

Try asking questions like "How many days of annual leave do I get?" or "What is the policy on remote work?" The chatbot will retrieve the relevant chunks, pass them as context to llama.cpp, and return a grounded answer with source citations.

Step 6: Adding Streaming Responses

For a better user experience, enable streaming so the answer appears token by token. Update the ask_question function to use server-sent events:

def ask_question_stream(retriever, question, history=None):
    hits = retriever.search(question, top_k=TOP_K)
    context = build_context(hits)

    user_content = f"Context:\n{context}\n\nQuestion: {question}"
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    if history:
        messages.extend(history)
    messages.append({"role": "user", "content": user_content})

    payload = {
        "model": MODEL_NAME,
        "messages": messages,
        "temperature": 0.3,
        "max_tokens": 512,
        "stream": True
    }

    full_response = ""
    with requests.post(LLAMA_URL, json=payload, stream=True) as response:
        response.raise_for_status()
        for line in response.iter_lines():
            if not line:
                continue
            line = line.decode("utf-8")
            if line.startswith("data: "):
                data = line[6:]
                if data == "[DONE]":
                    break
                chunk = json.loads(data)
                delta = chunk["choices"][0]["delta"].get("content", "")
                full_response += delta
                print(delta, end="", flush=True)
    print()
    return full_response, [hit["source"] for hit in hits], messages

Replace the call in main() with ask_question_stream to see the streamed output in your terminal.

Best Practices

Building a working chatbot is only the beginning. The following practices will help you move from a prototype to a reliable system.

Production Considerations

When moving beyond a local prototype, consider these upgrades. Replace the NumPy vector store with a dedicated database like Chroma, Qdrant, or pgvector for persistence and fast approximate nearest neighbor search. Containerize the llama.cpp server using Docker so deployments are reproducible. Add authentication in front of the server endpoint to prevent unauthorized access. Implement rate limiting and request queuing, since LLM inference is expensive and a single slow request can block others. Finally, instrument the chatbot with logging that captures the question, retrieved chunks, model response, and latency so you can continuously monitor quality.

Conclusion

Building a knowledge base chatbot with llama.cpp gives you a fully local, privacy-preserving, and cost-effective alternative to hosted LLM APIs. By combining semantic retrieval with a quantized open-source model, you can ground answers in your own documents while keeping complete control over the infrastructure. The architecture presented here, chunking, embedding, retrieval, and generation, is the foundation of virtually every RAG system, and each component can be upgraded independently as your needs grow. Start with the simple NumPy store and a single GGUF model, then scale up to a vector database, GPU acceleration, and hybrid search when your corpus and user base demand it. The most important step is to build a small evaluation set early, so every change you make can be measured against real questions and real answers.

— Ad —

Google AdSense will appear here after approval

← Back to all articles