Building a Knowledge Base Chatbot with AutoGen: Complete Guide

Building a Knowledge Base Chatbot with AutoGen: Complete Guide

Conversational AI has moved far beyond simple FAQ bots. Modern chatbots need to reason over large document collections, cite sources, and collaborate with specialized agents to answer complex questions. Microsoft's AutoGen framework makes this possible by letting you orchestrate multiple LLM-powered agents that talk to each other, retrieve information, and produce grounded answers. In this tutorial, you'll build a production-style knowledge base chatbot that retrieves relevant documents and answers user questions with citations.

What Is AutoGen?

AutoGen is an open-source multi-agent conversation framework developed by Microsoft Research. Instead of treating an LLM as a single black box, AutoGen lets you define multiple agents—each with a role, system prompt, and set of tools—and have them exchange messages in a structured conversation. Agents can be backed by OpenAI, Azure OpenAI, Anthropic, or local models via LiteLLM.

A typical AutoGen setup includes:

• Conversable agents — agents that can send and receive messages.
• An assistant agent — the LLM-powered reasoner.
• A user proxy agent — represents the human and can execute code or call tools.
• Group chats — coordinate many agents with a manager.

Why Use AutoGen for a Knowledge Base Chatbot?

A naive RAG (Retrieval-Augmented Generation) pipeline retrieves chunks and stuffs them into a prompt. That works for simple lookups but breaks down when users ask multi-step questions, need clarification, or require synthesis across documents. AutoGen solves this by:

• Separating retrieval from reasoning into distinct agents.
• Allowing agents to ask follow-up questions before answering.
• Supporting tool calls so agents can query a vector store dynamically.
• Enabling self-critique — one agent drafts, another verifies.
• Providing built-in termination and human-in-the-loop controls.

Prerequisites and Installation

You'll need Python 3.10+, an OpenAI API key (or compatible endpoint), and a few libraries. Create a virtual environment and install the dependencies:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

pip install "autogen-agentchat==0.2.40" \
            "autogen-docdata" \
            "chromadb==0.5.5" \
            "sentence-transformers==3.0.1" \
            "python-dotenv==1.0.1" \
            "pypdf==4.3.1"

Create a .env file in your project root:

OPENAI_API_KEY=sk-your-key-here

Step 1: Prepare the Knowledge Base

First, build a small vector store from PDF documents. We'll use ChromaDB as the vector database and a local sentence-transformers embedding model so you don't need a separate embedding API.

# build_knowledge_base.py
import os
import chromadb
from chromadb.utils import embedding_functions
from pypdf import PdfReader

DOCS_DIR = "docs"
DB_DIR = "chroma_db"

def load_pdf_texts(directory):
    documents = []
    for filename in os.listdir(directory):
        if not filename.lower().endswith(".pdf"):
            continue
        reader = PdfReader(os.path.join(directory, filename))
        text = "\n".join(page.extract_text() or "" for page in reader.pages)
        documents.append({"id": filename, "text": text})
    return documents

def chunk_text(text, chunk_size=800, overlap=100):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

def build():
    os.makedirs(DB_DIR, exist_ok=True)
    client = chromadb.PersistentClient(path=DB_DIR)
    embed_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
        model_name="all-MiniLM-L6-v2"
    )
    collection = client.get_or_create_collection(
        name="knowledge_base",
        embedding_function=embed_fn,
    )

    docs = load_pdf_texts(DOCS_DIR)
    all_chunks, all_ids, all_metadata = [], [], []
    for doc in docs:
        for i, chunk in enumerate(chunk_text(doc["text"])):
            all_chunks.append(chunk)
            all_ids.append(f"{doc['id']}_chunk_{i}")
            all_metadata.append({"source": doc["id"], "chunk": i})

    if all_chunks:
        collection.add(
            documents=all_chunks,
            ids=all_ids,
            metadatas=all_metadata,
        )
    print(f"Indexed {len(all_chunks)} chunks from {len(docs)} documents.")

if __name__ == "__main__":
    build()

Drop a few PDF files into a docs/ folder and run python build_knowledge_base.py. You now have a persistent vector store ready for retrieval.

Step 2: Define the Retrieval Tool

AutoGen agents call Python functions as tools. We'll expose a search_knowledge_base function that queries ChromaDB and returns formatted results with source metadata.

# tools.py
import chromadb
from chromadb.utils import embedding_functions

DB_DIR = "chroma_db"

_client = chromadb.PersistentClient(path=DB_DIR)
_embed_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="all-MiniLM-L6-v2"
)
_collection = _client.get_or_create_collection(
    name="knowledge_base",
    embedding_function=_embed_fn,
)

def search_knowledge_base(query: str, top_k: int = 4) -> str:
    """Search the internal knowledge base for relevant passages.

    Args:
        query: A natural language search query.
        top_k: Number of passages to return.

    Returns:
        A formatted string of retrieved passages with source citations.
    """
    results = _collection.query(query_texts=[query], n_results=top_k)
    documents = results["documents"][0]
    metadatas = results["metadatas"][0]

    if not documents:
        return "No relevant documents found."

    output_lines = []
    for i, (doc, meta) in enumerate(zip(documents, metadatas), start=1):
        source = meta.get("source", "unknown")
        output_lines.append(f"[{i}] (source: {source})\n{doc}")
    return "\n\n".join(output_lines)

Step 3: Configure AutoGen Agents

Now we wire up the agents. We'll create two agents: a researcher that uses the retrieval tool, and a responder that synthesizes the final answer. A user proxy agent represents the human and triggers the conversation.

# chatbot.py
import os
import autogen
from dotenv import load_dotenv
from tools import search_knowledge_base

load_dotenv()

config_list = [
    {
        "model": "gpt-4o-mini",
        "api_key": os.getenv("OPENAI_API_KEY"),
    }
]

llm_config = {
    "config_list": config_list,
    "temperature": 0.2,
    "timeout": 60,
}

# Researcher agent: retrieves documents
researcher = autogen.AssistantAgent(
    name="Researcher",
    system_message=(
        "You are a retrieval specialist. When asked a question, call the "
        "search_knowledge_base tool with a focused query. Return the raw "
        "retrieved passages verbatim, including source citations. Do not "
        "invent information. If no results are found, say so explicitly."
    ),
    llm_config=llm_config,
)

# Register the tool with the researcher
autogen.register_function(
    search_knowledge_base,
    caller=researcher,
    name="search_knowledge_base",
    description="Search the internal knowledge base for relevant passages.",
)

# Responder agent: synthesizes the final answer
responder = autogen.AssistantAgent(
    name="Responder",
    system_message=(
        "You are a helpful assistant. You receive retrieved passages from "
        "the Researcher. Synthesize a clear, concise answer for the user. "
        "Always cite sources in the form [source: filename]. If the passages "
        "do not contain enough information, say you don't know. Never "
        "hallucinate facts not present in the retrieved context."
    ),
    llm_config=llm_config,
)

# User proxy: represents the human
user_proxy = autogen.UserProxyAgent(
    name="User",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=5,
    is_termination_msg=lambda msg: msg.get("content", "")
        and "TERMINATE" in msg["content"],
    code_execution_config=False,
)

Step 4: Orchestrate the Conversation

With agents defined, we use a GroupChat with a manager to coordinate the flow: the user asks a question, the Researcher retrieves, the Responder answers, and the conversation terminates.

# chatbot.py (continued)
groupchat = autogen.GroupChat(
    agents=[user_proxy, researcher, responder],
    messages=[],
    max_round=8,
    speaker_selection_method="auto",
)

manager = autogen.GroupChatManager(
    groupchat=groupchat,
    llm_config=llm_config,
)

def ask(question: str):
    print(f"\n=== User Question ===\n{question}\n")
    user_proxy.initiate_chat(
        manager,
        message=question,
        clear_history=True,
    )

if __name__ == "__main__":
    ask("What is our company's remote work policy regarding equipment?")
    ask("How many vacation days do new employees receive?")

Run the chatbot with python chatbot.py. You'll see the Researcher call the retrieval tool, pass results to the Responder, and the Responder produce a grounded answer with citations.

Step 5: Add a Streaming CLI Loop

For an interactive experience, replace the __main__ block with a REPL that reads questions from stdin:

if __name__ == "__main__":
    print("Knowledge Base Chatbot. Type 'exit' to quit.")
    while True:
        question = input("\nYou: ").strip()
        if question.lower() in {"exit", "quit"}:
            break
        if not question:
            continue
        ask(question)

Best Practices

• Keep chunks focused. 500–800 character chunks with overlap preserve context without diluting relevance.
• Use low temperature for retrieval. Set temperature to 0–0.3 for the Researcher to avoid creative rewrites of queries.
• Force citations. Instruct the Responder to cite sources explicitly and to say "I don't know" when evidence is missing.
• Limit rounds. Set max_round on the group chat to prevent infinite agent loops.
• Log tool calls. Wrap search_knowledge_base with logging to audit which queries are issued and what is returned.
• Re-embed on updates. When documents change, rebuild the collection or use ChromaDB's upsert to keep embeddings fresh.
• Cache embeddings. Sentence-transformers loads the model once; keep the collection persistent to avoid recomputation.
• Test for hallucination. Build a small eval set of question-answer pairs and check that responses are grounded in retrieved chunks.
• Consider a verifier agent. Add a third agent that checks the Responder's answer against retrieved passages before returning it to the user.

Extending the Chatbot

Once the core works, you can extend it in several directions. Add a summarizer agent that condenses long retrieved passages before they reach the Responder, reducing token usage. Add a query rewriter agent that reformulates ambiguous user questions into multiple search queries for better recall. Swap ChromaDB for a managed vector database like Pinecone or Weaviate if you need horizontal scale. Replace the OpenAI backend with a local model via LiteLLM or Ollama for air-gapped deployments. Finally, expose the chatbot through FastAPI so it can serve a web frontend.

AutoGen's strength is composability: each capability becomes an agent or a tool, and the framework handles message passing, tool dispatch, and termination. By separating retrieval from reasoning, you get a chatbot that is more accurate, easier to debug, and simpler to extend than a monolithic RAG prompt. With the structure above, you have a solid foundation for a knowledge base chatbot that can grow with your documentation and your users' needs.