Introduction to MCP for Knowledge Base Chatbots
The Model Context Protocol (MCP) is an open standard introduced by Anthropic that standardizes how AI applications connect to external data sources and tools. Think of it as the "USB-C for AI" — a universal plug that lets any LLM client talk to any data provider without bespoke integrations for every combination.
For knowledge base chatbots, MCP is transformative. Instead of hard-coding retrieval logic, vector store connections, and document parsers into your chatbot, you expose them as an MCP server. Any MCP-compatible client — Claude Desktop, Cursor, a custom LangChain agent, or your own web app — can then query your knowledge base through a consistent interface.
In this guide, we'll build a complete knowledge base chatbot: an MCP server that exposes a document store, and a client that connects to it and answers user questions using an LLM.
Why MCP Matters for Knowledge Bases
Traditional RAG (Retrieval-Augmented Generation) chatbots suffer from tight coupling. The retrieval pipeline, embedding model, vector database, and prompt assembly are all glued together in one application. If you want to switch from Pinecone to Qdrant, or expose the same knowledge base to a different chatbot, you rewrite significant chunks of code.
MCP solves this by separating concerns into three roles:
- MCP Host — the application the user interacts with (e.g., your chatbot UI).
- MCP Client — lives inside the host, manages connections to servers.
- MCP Server — exposes resources, tools, and prompts from a data source.
A knowledge base MCP server can expose documents as resources (readable content) and a semantic search function as a tool (callable action). The LLM decides when to read a resource or call a tool, giving it genuine agency over its own retrieval.
Prerequisites and Project Setup
You'll need Python 3.10+, an Anthropic API key (or OpenAI — the patterns transfer), and basic familiarity with async Python. We'll use the official MCP Python SDK.
Create a project directory and install dependencies:
mkdir kb-mcp-chatbot && cd kb-mcp-chatbot
python -m venv .venv
source .venv/bin/activate # on Windows: .venv\Scripts\activate
pip install mcp anthropic chromadb sentence-transformers python-dotenv
Create a .env file:
ANTHROPIC_API_KEY=sk-ant-...
Your project structure will look like this:
kb-mcp-chatbot/
├── .env
├── kb_server.py # The MCP server exposing the knowledge base
├── chat_client.py # The chatbot client that uses the server
├── ingest.py # Helper to load documents into the KB
└── documents/ # Source text files
└── sample.txt
Building the Knowledge Base MCP Server
The MCP server is the heart of the system. It owns the document store and exposes two capabilities: a search_kb tool for semantic retrieval, and a kb://documents resource listing available documents. We'll use ChromaDB as a local vector store and sentence-transformers for embeddings — no external API required.
Setting Up the Vector Store
First, create ingest.py to populate the knowledge base from text files:
import os
import chromadb
from sentence_transformers import SentenceTransformer
DOCS_DIR = "documents"
DB_PATH = "./chroma_db"
def ingest_documents():
client = chromadb.PersistentClient(path=DB_PATH)
collection = client.get_or_create_collection("knowledge_base")
embedder = SentenceTransformer("all-MiniLM-L6-v2")
files = [f for f in os.listdir(DOCS_DIR) if f.endswith(".txt")]
for fname in files:
with open(os.path.join(DOCS_DIR, fname), "r", encoding="utf-8") as fh:
text = fh.read()
# Split into chunks of ~500 characters for better retrieval
chunks = [text[i:i+500] for i in range(0, len(text), 500)]
embeddings = embedder.encode(chunks).tolist()
ids = [f"{fname}_{i}" for i in range(len(chunks))]
collection.upsert(
ids=ids,
documents=chunks,
embeddings=embeddings,
metadatas=[{"source": fname} for _ in chunks]
)
print(f"Ingested {len(chunks)} chunks from {fname}")
if __name__ == "__main__":
ingest_documents()
print("Ingestion complete.")
Drop a few .txt files into the documents/ directory and run python ingest.py. You now have a populated vector store.
Implementing the MCP Server
Now create kb_server.py. This file defines the server, registers a tool and a resource, and runs over standard I/O — the default transport for MCP servers launched as subprocesses.
import chromadb
from sentence_transformers import SentenceTransformer
from mcp.server.fastmcp import FastMCP
DB_PATH = "./chroma_db"
mcp = FastMCP("knowledge-base")
# Initialize once at module load
_client = chromadb.PersistentClient(path=DB_PATH)
_collection = _client.get_or_create_collection("knowledge_base")
_embedder = SentenceTransformer("all-MiniLM-L6-v2")
@mcp.tool()
def search_kb(query: str, top_k: int = 3) -> str:
"""Search the knowledge base for passages relevant to the query.
Args:
query: The natural-language question or topic to search for.
top_k: Maximum number of passages to return.
Returns:
A formatted string of the most relevant passages with their source files.
"""
query_embedding = _embedder.encode([query]).tolist()
results = _collection.query(
query_embeddings=query_embedding,
n_results=top_k,
include=["documents", "metadatas", "distances"]
)
if not results["documents"] or not results["documents"][0]:
return "No relevant documents found."
output_lines = []
for doc, meta, dist in zip(
results["documents"][0],
results["metadatas"][0],
results["distances"][0]
):
output_lines.append(
f"[Source: {meta.get('source', 'unknown')} | similarity: {1 - dist:.2f}]\n{doc}\n"
)
return "\n---\n".join(output_lines)
@mcp.resource("kb://documents")
def list_documents() -> str:
"""List all unique source documents available in the knowledge base."""
all_meta = _collection.get(include=["metadatas"])
sources = sorted({m["source"] for m in all_meta["metadatas"]})
return "Available documents:\n" + "\n".join(f"- {s}" for s in sources)
if __name__ == "__main__":
mcp.run(transport="stdio")
A few things to notice: the FastMCP class turns ordinary Python functions into MCP tools and resources via decorators. The docstrings become the descriptions the LLM sees, so write them carefully — they are the model's only guide for when and how to use each capability.
Building the Chatbot Client
The client connects to the server, sends user messages to the LLM, and lets the model invoke the search_kb tool when it needs information. We'll use the Anthropic SDK directly with MCP's client primitives.
Create chat_client.py:
import asyncio
import sys
import os
from dotenv import load_dotenv
from anthropic import Anthropic
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
load_dotenv()
anthropic = Anthropic()
async def chat_loop(session: ClientSession):
# Discover what the server offers
tools_result = await session.list_tools()
mcp_tools = tools_result.tools
# Convert MCP tool schemas to Anthropic tool format
anthropic_tools = [
{
"name": tool.name,
"description": tool.description,
"input_schema": tool.inputSchema,
}
for tool in mcp_tools
]
print("Knowledge Base Chatbot ready. Type 'quit' to exit.\n")
messages = []
while True:
user_input = input("You: ").strip()
if user_input.lower() in {"quit", "exit"}:
break
if not user_input:
continue
messages.append({"role": "user", "content": user_input})
# Agentic loop: keep going until the model stops calling tools
while True:
response = anthropic.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=(
"You are a helpful assistant that answers questions using a "
"knowledge base. When you need factual information, call the "
"search_kb tool. Always ground your answers in retrieved "
"passages and cite the source file."
),
tools=anthropic_tools,
messages=messages,
)
if response.stop_reason == "tool_use":
# Append the assistant's tool-call message
messages.append({"role": "assistant", "content": response.content})
# Execute every tool call the model requested
for block in response.content:
if block.type == "tool_use":
result = await session.call_tool(
block.name, block.input
)
tool_response_text = (
result.content[0].text
if result.content
else "No results."
)
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": block.id,
"content": tool_response_text,
}
],
})
# Loop back so the model can read the tool results and respond
continue
# No more tool calls — extract the final text answer
final_text = "".join(
block.text for block in response.content if block.type == "text"
)
messages.append({"role": "assistant", "content": final_text})
print(f"\nAssistant: {final_text}\n")
break
async def main():
server_params = StdioServerParameters(
command=sys.executable,
args=["kb_server.py"],
env={**os.environ},
)
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
await chat_loop(session)
if __name__ == "__main__":
asyncio.run(main())
Run it with python chat_client.py. The client spawns the server as a subprocess, negotiates capabilities, and enters a REPL. When you ask a question, the model decides whether to call search_kb, receives the retrieved passages, and synthesizes an answer.
How the Pieces Fit Together
The flow for a single user question is:
- The client sends the conversation (including tool definitions) to the LLM.
- The LLM either answers directly or returns a
tool_useblock requestingsearch_kb. - The client forwards the tool call over the MCP stdio transport to the server.
- The server embeds the query, searches ChromaDB, and returns formatted passages.
- The client feeds the tool result back to the LLM as a
tool_resultmessage. - The LLM generates a grounded answer, which is displayed to the user.
This agentic loop can repeat multiple times if the model needs to refine its search — for example, first searching broadly, then re-querying with a narrower term based on what it found.
Best Practices
Write Excellent Tool Descriptions
The LLM chooses tools based solely on the name and description. Vague descriptions lead to missed calls or hallucinated behavior. Be explicit about what the tool does, what inputs it expects, and what it returns. Include failure modes ("returns 'No relevant documents found' if nothing matches") so the model can recover gracefully.
Chunk Documents Thoughtfully
Retrieval quality depends on chunk size. Too small and you lose context; too large and similarity scores become noisy. A common starting point is 300–600 characters with overlap. For structured documents (Markdown, code), chunk by heading or function boundary instead of fixed character counts.
Expose Metadata, Not Just Text
Returning source filenames and similarity scores (as we do in search_kb) lets the model cite sources and judge confidence. Consider also exposing document titles, timestamps, or section headings as metadata the model can reason about.
Keep the Server Stateless Where Possible
MCP servers should ideally be stateless between tool calls. The vector store is loaded once at startup, but no per-session state should accumulate. This makes the server safe to restart and easy to scale horizontally with multiple instances.
Handle Errors Gracefully
Wrap tool bodies in try/except and return human-readable error strings rather than raising exceptions. An unhandled exception in a tool call will surface as an opaque error to the LLM, which often responds with confusion. A clear message like "Database connection failed; please retry" lets the model decide whether to try again or inform the user.
Use Resources for Browsable Content
Tools are for actions; resources are for data the client can browse. Expose a kb://documents resource (as we did) so clients can show users what's available, and consider per-document resources like kb://document/{filename} for full-text retrieval when the model needs complete context rather than chunks.
Test the Server Independently
MCP servers can be tested without an LLM using the MCP Inspector (npx @modelcontextprotocol/inspector python kb_server.py). This lets you call tools and read resources directly, validating your retrieval logic before wiring it into an expensive model loop.
Extending the System
Once the foundation works, common extensions include:
- Multiple tools — add
search_by_sourceto filter by file, orsummarize_documentfor long-form synthesis. - Hybrid search — combine vector similarity with keyword matching (BM25) for better recall on exact terms.
- Streaming transport — switch from stdio to SSE or WebSocket so the server can run as a remote service shared by many clients.
- Authentication — for remote servers, add token-based auth so only authorized clients can query sensitive knowledge bases.
- Citation rendering — parse source metadata from tool results in the client and render clickable citations in the UI.
Conclusion
MCP brings a clean separation of concerns to knowledge base chatbots: the server owns retrieval, the client owns conversation, and the LLM orchestrates between them. By exposing your document store as an MCP server with well-described tools and resources, you get a retrieval system that any MCP-compatible client can use — today's Claude Desktop, tomorrow's IDE plugin, or your own custom agent — without rewriting a line of retrieval code. The architecture we built here, with ChromaDB for storage and an agentic Anthropic loop for orchestration, is a production-ready starting point that scales from a personal notes assistant to an enterprise knowledge portal simply by swapping the vector store backend and adding authentication. The real power of MCP is that once your knowledge base speaks the protocol, it becomes a first-class citizen of the entire emerging ecosystem of AI tools.