Introduction to LlamaIndex Knowledge Base Chatbots
A knowledge base chatbot is an AI assistant that answers user questions based on a specific set of documents rather than relying solely on its pre-trained knowledge. This approach, often called Retrieval-Augmented Generation (RAG), dramatically improves accuracy and reduces hallucinations because the model grounds its responses in your actual data.
LlamaIndex is a leading data framework designed specifically for building LLM applications with custom data. It provides the tools to ingest, structure, and query your documents efficiently, making it the perfect foundation for a knowledge base chatbot. In this guide, you will build a fully functional chatbot from scratch, covering document loading, indexing, retrieval, and conversational memory.
Why LlamaIndex Matters for Knowledge Base Chatbots
Building a chatbot that "knows" your documents involves several complex steps: parsing different file formats, splitting text into manageable chunks, generating embeddings, storing them in a vector database, and retrieving the most relevant pieces when a user asks a question. LlamaIndex abstracts much of this complexity into a clean, modular API.
Key Advantages
- Data Connectors: LlamaIndex supports over 160 data sources including PDFs, Notion, GitHub, SQL databases, and web pages.
- Indexing Structures: It offers multiple index types such as vector indexes, list indexes, and tree indexes, each optimized for different query patterns.
- Advanced Retrieval: Built-in support for hybrid search, re-ranking, and sub-question query engines.
- Conversational Memory: First-class support for chat memory buffers so your bot remembers context across turns.
- Framework Agnostic: Works with OpenAI, Anthropic, Hugging Face, and local models through a unified interface.
Prerequisites and Setup
Before you start coding, make sure you have Python 3.9 or higher installed. You will also need an OpenAI API key, though you can swap in other providers later. Create a new project directory and set up your environment.
# Create project directory
mkdir kb-chatbot
cd kb-chatbot
# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install required packages
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai
pip install python-dotenv
# Create environment file
echo "OPENAI_API_KEY=your-api-key-here" > .env
Create a folder named data inside your project directory and place a few text or PDF files there. These documents will form your knowledge base.
Step 1: Loading Documents
The first step in any LlamaIndex pipeline is loading your data. LlamaIndex uses "readers" to convert raw files into Document objects, each containing text and metadata.
import os
from dotenv import load_dotenv
from llama_index.core import SimpleDirectoryReader
# Load environment variables
load_dotenv()
# Load documents from the data directory
reader = SimpleDirectoryReader(input_dir="./data", recursive=True)
documents = reader.load_data()
print(f"Loaded {len(documents)} documents")
for doc in documents[:3]:
print(f"- {doc.metadata.get('file_name', 'unknown')}: {len(doc.text)} chars")
The SimpleDirectoryReader automatically detects file types and uses the appropriate parser. It handles TXT, PDF, DOCX, CSV, and Markdown files out of the box. The recursive=True flag tells it to search subdirectories as well.
Step 2: Building the Index
Once documents are loaded, you need to build an index. The most common type is the VectorStoreIndex, which converts document chunks into embedding vectors and stores them for similarity search. When a query comes in, the index finds the chunks whose embeddings are closest to the query embedding.
from llama_index.core import VectorStoreIndex, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
# Configure global settings
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
# Build the index from documents
index = VectorStoreIndex.from_documents(documents)
print("Index built successfully!")
Setting temperature=0.1 keeps responses focused and factual, which is ideal for a knowledge base chatbot. The embedding model converts text into 1536-dimensional vectors that capture semantic meaning, enabling the system to find relevant information even when the user's wording differs from the source text.
Step 3: Creating a Basic Query Engine
Before adding conversational capabilities, let us test a simple query engine. This retrieves relevant chunks and synthesizes an answer.
# Create a query engine
query_engine = index.as_query_engine(similarity_top_k=3)
# Test with a question
response = query_engine.query("What is this knowledge base about?")
print(response)
The similarity_top_k=3 parameter tells the engine to retrieve the top 3 most similar chunks. You can increase this for more comprehensive answers or decrease it for faster, more focused responses.
Step 4: Adding Conversational Memory
A query engine answers isolated questions, but a chatbot needs to remember the conversation. LlamaIndex provides a ChatMemoryBuffer that stores message history and feeds it back to the LLM on each turn. You convert the index into a chat engine instead of a query engine.
from llama_index.core.memory import ChatMemoryBuffer
# Create a memory buffer
memory = ChatMemoryBuffer.from_defaults(token_limit=3900)
# Create a chat engine with memory
chat_engine = index.as_chat_engine(
chat_mode="context",
memory=memory,
system_prompt=(
"You are a knowledgeable assistant that answers questions "
"strictly based on the provided knowledge base. If the answer "
"is not found in the documents, say you don't know. Always "
"cite which document the information comes from."
),
similarity_top_k=3,
)
The chat_mode="context" strategy retrieves relevant documents on each turn and includes them as context, while also maintaining conversation history. The token_limit prevents memory from growing unbounded, which would eventually exceed the model's context window.
Step 5: Building the Interactive Chat Loop
Now you can wrap everything into an interactive command-line chat loop that maintains context across multiple turns.
def run_chatbot():
print("=" * 60)
print(" Knowledge Base Chatbot (type 'quit' to exit)")
print("=" * 60)
print()
while True:
user_input = input("You: ").strip()
if user_input.lower() in ["quit", "exit", "q"]:
print("Goodbye!")
break
if not user_input:
continue
print()
# Stream the response for a better user experience
response = chat_engine.stream_chat(user_input)
print("Bot: ", end="", flush=True)
for token in response.response_gen:
print(token, end="", flush=True)
print("\n")
if __name__ == "__main__":
run_chatbot()
Using stream_chat instead of chat streams tokens as they are generated, creating a more responsive and engaging experience. The chat engine automatically handles memory management, retrieving relevant context and maintaining conversation flow.
Step 6: Persisting the Index
Rebuilding the index from scratch every time you start the application is slow and wastes API calls. LlamaIndex lets you persist the index to disk and reload it instantly.
from llama_index.core import StorageContext, load_index_from_storage
import os
PERSIST_DIR = "./storage"
def build_or_load_index(documents=None):
if os.path.exists(PERSIST_DIR):
# Load existing index
storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
index = load_index_from_storage(storage_context)
print("Loaded index from storage.")
else:
# Build and save new index
index = VectorStoreIndex.from_documents(documents)
index.storage_context.persist(persist_dir=PERSIST_DIR)
print("Built and saved new index.")
return index
# Usage
if os.path.exists(PERSIST_DIR):
index = build_or_load_index()
else:
index = build_or_load_index(documents)
This pattern checks whether a persisted index exists. If it does, it loads instantly. Otherwise, it builds from documents and saves for future use. You only need to rebuild when your source documents change.
Step 7: Adding Document Citations
For a production knowledge base chatbot, users need to verify where information comes from. You can access the source nodes from each response and display them.
def chat_with_citations(user_input):
response = chat_engine.chat(user_input)
print(f"\nBot: {response}\n")
# Display source documents
if hasattr(response, "source_nodes") and response.source_nodes:
print("Sources:")
for i, node in enumerate(response.source_nodes, 1):
file_name = node.node.metadata.get("file_name", "unknown")
score = node.score if node.score else "N/A"
print(f" [{i}] {file_name} (relevance: {score})")
# Show a snippet of the source text
snippet = node.node.text[:200].replace("\n", " ")
print(f" \"{snippet}...\"")
print()
# Example usage
chat_with_citations("What are the key features mentioned in the documents?")
This transparency builds trust. Users can see exactly which documents the chatbot referenced and how relevant each one was to their question.
Step 8: Building a Web API with FastAPI
To make your chatbot accessible beyond the command line, you can wrap it in a REST API using FastAPI. This allows you to connect it to a web frontend, mobile app, or Slack integration.
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
app = FastAPI(title="Knowledge Base Chatbot API")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
# Initialize index and chat engine at startup
index = build_or_load_index(documents)
chat_engine = index.as_chat_engine(
chat_mode="context",
memory=ChatMemoryBuffer.from_defaults(token_limit=3900),
system_prompt="You are a helpful assistant answering from the knowledge base.",
similarity_top_k=3,
)
class ChatRequest(BaseModel):
message: str
class ChatResponse(BaseModel):
reply: str
sources: list
@app.post("/chat", response_model=ChatResponse)
def chat_endpoint(request: ChatRequest):
response = chat_engine.chat(request.message)
sources = []
if hasattr(response, "source_nodes"):
for node in response.source_nodes:
sources.append({
"file": node.node.metadata.get("file_name", "unknown"),
"text": node.node.text[:300],
})
return ChatResponse(reply=str(response), sources=sources)
@app.get("/health")
def health():
return {"status": "healthy"}
Run the server with uvicorn api:app --reload and test it with a simple curl command:
curl -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"message": "What is this knowledge base about?"}'
Best Practices
Optimize Chunk Size and Overlap
The default chunk size is 1024 tokens. For dense technical documents, smaller chunks (512 tokens) with more overlap (50-100 tokens) often produce better retrieval. For narrative documents, larger chunks preserve context better.
from llama_index.core import Settings
Settings.chunk_size = 512
Settings.chunk_overlap = 50
Use Hybrid Search for Better Retrieval
Vector similarity search is powerful but can miss exact keyword matches. Combining vector search with keyword search (BM25) often yields the best results.
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import SubQuestionQueryEngine
# You can also use retriever routers for hybrid approaches
vector_retriever = index.as_retriever(similarity_top_k=5)
Implement Guardrails
Always include a system prompt that instructs the model to say "I don't know" when information is not in the knowledge base. This prevents hallucinations and maintains trust. You can also add output parsers to validate responses.
Monitor and Log Queries
Log every query, retrieved sources, and response. This helps you identify gaps in your knowledge base and improve document coverage over time.
import logging
logging.basicConfig(
filename="chatbot.log",
level=logging.INFO,
format="%(asctime)s - %(message)s"
)
def logged_chat(user_input):
response = chat_engine.chat(user_input)
sources = [n.node.metadata.get("file_name") for n in response.source_nodes]
logging.info(f"Q: {user_input} | Sources: {sources}")
return response
Refresh Your Index Regularly
When documents change, rebuild the index. For frequently updated knowledge bases, consider setting up a scheduled job that checks for file modifications and rebuilds automatically.
Conclusion
Building a knowledge base chatbot with LlamaIndex gives you a powerful, flexible foundation for creating AI assistants that are grounded in your own data. By following the steps in this guide, you now have a complete system that loads documents, builds a searchable vector index, maintains conversational memory, provides source citations, and exposes a REST API for integration. The modular nature of LlamaIndex means you can extend this foundation in many directions: swap in a local open-source model for privacy, connect to enterprise data sources like Confluence or SharePoint, add re-ranking for improved retrieval accuracy, or deploy to cloud platforms for production scale. Start with the basic setup, test it thoroughly with real user questions, and iterate on chunk size, retrieval parameters, and system prompts until your chatbot delivers reliable, well-sourced answers that your users can trust.