Building a RAG-Enabled Agent with LangChain
Retrieval-Augmented Generation (RAG) has become one of the most practical patterns for building production-grade LLM applications. When you combine RAG with an agentic loop — where the model can reason, decide, and call tools autonomously — you get a system that doesn't just answer questions from a static knowledge base, but actively retrieves, synthesizes, and acts on information. In this tutorial, we'll build a complete RAG-enabled agent using LangChain, from document ingestion all the way to a working conversational interface.
What Is a RAG-Enabled Agent?
A RAG-enabled agent is the fusion of two powerful ideas. First, Retrieval-Augmented Generation grounds an LLM's responses in external documents by retrieving relevant chunks from a vector store and injecting them into the prompt context. Second, an agent is an LLM equipped with tools and a reasoning loop (often using ReAct-style prompting) that lets it decide which actions to take based on user input.
The key difference between a plain RAG pipeline and a RAG agent is autonomy. A plain RAG system retrieves documents on every query and stuffs them into the prompt. A RAG agent, by contrast, can choose whether to retrieve, what query to use for retrieval, and how to combine multiple retrieval calls with other tools or reasoning steps. This makes it far more robust for complex, multi-step questions.
Why It Matters
- Reduced hallucinations: Grounding responses in retrieved documents dramatically cuts down on fabricated answers.
- Up-to-date knowledge: You can refresh the vector store without retraining the model, keeping the agent current.
- Source attribution: Agents can cite which documents informed their answers, building user trust.
- Multi-step reasoning: The agent can decompose complex questions, retrieve multiple times, and synthesize results.
- Extensibility: The same agent framework lets you add web search, SQL queries, API calls, and more as additional tools.
Prerequisites and Setup
Before we start coding, install the required packages and set up your environment. We'll use OpenAI for embeddings and chat models, and Chroma as our local vector store.
pip install langchain langchain-openai langchain-chromadb langchain-community
pip install chromadb python-dotenv pypdf
Create a .env file with your API key:
OPENAI_API_KEY=sk-your-key-here
Now let's set up the basic imports and load environment variables:
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool
from langchain.agents import create_tool_calling_agent, AgentExecutor
load_dotenv()
Step 1: Document Ingestion and Indexing
The foundation of any RAG system is a well-organized knowledge base. We'll load PDF documents, split them into manageable chunks, embed those chunks, and store them in a vector database.
def build_vector_store(docs_dir: str, persist_dir: str = "./chroma_db"):
"""Load documents, split them, and create a persistent vector store."""
# Load all PDFs from the directory
loader = DirectoryLoader(
docs_dir,
glob="**/*.pdf",
loader_cls=PyPDFLoader,
show_progress=True
)
documents = loader.load()
print(f"Loaded {len(documents)} document pages")
# Split documents into chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")
# Create embeddings and persist to disk
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory=persist_dir
)
print(f"Vector store created at {persist_dir}")
return vector_store
The chunk size and overlap parameters are critical. A chunk size of 1000 characters with 200 characters of overlap is a reasonable starting point, but you should tune these based on your document structure. Smaller chunks give more precise retrieval but may lose context; larger chunks preserve context but dilute relevance signals.
Step 2: Creating the Retrieval Tool
For the agent to use retrieval, we need to expose it as a tool. LangChain's @tool decorator makes this straightforward. The tool's docstring is essential — it tells the LLM when and how to use the tool.
# Initialize or load the vector store
PERSIST_DIR = "./chroma_db"
if os.path.exists(PERSIST_DIR):
vector_store = Chroma(
persist_directory=PERSIST_DIR,
embedding_function=OpenAIEmbeddings(model="text-embedding-3-small")
)
else:
vector_store = build_vector_store("./docs", PERSIST_DIR)
@tool
def search_knowledge_base(query: str) -> str:
"""Search the internal knowledge base for relevant information.
Use this tool when the user asks questions about company policies,
product documentation, technical specifications, or any information
that might be stored in the document repository. Always use this
tool before answering factual questions.
Args:
query: A clear, specific search query related to the user's question.
Returns:
Relevant document excerpts that may contain the answer.
"""
results = vector_store.similarity_search_with_relevance_scores(
query, k=4
)
if not results:
return "No relevant documents found in the knowledge base."
formatted = []
for i, (doc, score) in enumerate(results, 1):
source = doc.metadata.get("source", "unknown")
page = doc.metadata.get("page", "unknown")
formatted.append(
f"[Document {i}] (Relevance: {score:.2f}, "
f"Source: {source}, Page: {page})\n{doc.page_content}"
)
return "\n\n---\n\n".join(formatted)
Notice that we include relevance scores and source metadata in the tool's output. This gives the agent the information it needs to cite sources and judge whether the retrieved content is actually relevant.
Step 3: Adding Additional Tools
The power of an agent over a plain RAG pipeline is the ability to combine multiple tools. Let's add a simple calculator tool and a current-date tool to demonstrate this.
from datetime import datetime
@tool
def calculator(expression: str) -> str:
"""Perform a mathematical calculation.
Use this tool when you need to compute numerical values,
such as totals, percentages, or any arithmetic operations.
Args:
expression: A mathematical expression as a string, e.g. "15 * 23 + 100"
Returns:
The result of the calculation.
"""
try:
# Safe evaluation of mathematical expressions
allowed = set("0123456789+-*/.() ")
if not all(c in allowed for c in expression):
return "Error: Expression contains invalid characters."
result = eval(expression)
return f"Result: {result}"
except Exception as e:
return f"Error calculating: {str(e)}"
@tool
def get_current_date() -> str:
"""Get the current date and time.
Use this tool when the user asks about dates, deadlines,
or when you need to reference the current time.
Returns:
The current date and time as a formatted string.
"""
return datetime.now().strftime("%Y-%m-%d %H:%M:%S")
Step 4: Building the Agent
Now we bring everything together. We'll create a system prompt that instructs the agent on its role, then use LangChain's create_tool_calling_agent to wire up the LLM with our tools.
# Initialize the LLM
llm = ChatOpenAI(
model="gpt-4o",
temperature=0.1,
streaming=True
)
# Define the system prompt
system_prompt = """You are a knowledgeable assistant with access to an internal
document knowledge base and several tools. Your job is to help users find
accurate information and answer their questions.
Guidelines:
1. ALWAYS use the search_knowledge_base tool when the user asks a factual
question that might be answered by the documents. Do not rely on your
training data for information that should be in the knowledge base.
2. If the search results don't contain enough information, say so honestly
rather than guessing.
3. When you use information from the knowledge base, cite the source
document and page number.
4. Use the calculator tool for any mathematical computations.
5. Use the get_current_date tool when date information is needed.
6. If a question requires multiple searches, perform them sequentially
and synthesize the results.
7. Be concise but thorough in your responses.
Remember: it is better to search and find nothing than to fabricate an answer.
"""
# Create the prompt template
prompt = ChatPromptTemplate.from_messages([
("system", system_prompt),
("placeholder", "{chat_history}"),
("human", "{input}"),
("placeholder", "{agent_scratchpad}"),
])
# List of tools available to the agent
tools = [search_knowledge_base, calculator, get_current_date]
# Create the agent
agent = create_tool_calling_agent(llm, tools, prompt)
# Create the executor with memory
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
agent_executor = AgentExecutor(
agent=agent,
tools=tools,
verbose=True,
max_iterations=10,
handle_parsing_errors=True,
return_intermediate_steps=False
)
# Wrap with message history for conversational memory
message_history = InMemoryChatMessageHistory()
agent_with_history = RunnableWithMessageHistory(
agent_executor,
lambda session_id: message_history,
input_messages_key="input",
history_messages_key="chat_history",
)
Step 5: Running the Agent
Let's test the agent with a few queries to see how it behaves in practice.
def ask(question: str, session_id: str = "default"):
"""Send a question to the agent and print the response."""
print(f"\n{'='*60}")
print(f"USER: {question}")
print(f"{'='*60}")
response = agent_with_history.invoke(
{"input": question},
config={"configurable": {"session_id": session_id}}
)
print(f"\nAGENT: {response['output']}")
print(f"{'='*60}\n")
return response
# Example queries
ask("What is our company's remote work policy?")
ask("If an employee works 22 days remotely and 8 days in office, what percentage of time is remote?")
ask("Can you summarize the key points from the onboarding document?")
Step 6: Adding a Conversational Interface
For a more interactive experience, here's a simple REPL loop you can use for testing:
def run_interactive_session():
"""Run an interactive chat session with the agent."""
print("RAG Agent Ready. Type 'quit' to exit.\n")
session_id = "interactive"
while True:
user_input = input("You: ").strip()
if user_input.lower() in ("quit", "exit", "q"):
print("Goodbye!")
break
if not user_input:
continue
try:
response = agent_with_history.invoke(
{"input": user_input},
config={"configurable": {"session_id": session_id}}
)
print(f"\nAgent: {response['output']}\n")
except Exception as e:
print(f"\nError: {str(e)}\n")
if __name__ == "__main__":
run_interactive_session()
Best Practices
Building a RAG agent that works reliably in production requires attention to several details beyond the basic implementation above. Here are the most important practices to follow.
Optimize Your Chunking Strategy
Chunking is the single most impactful parameter in a RAG system. Avoid naive character-based splitting when possible. Consider semantic chunking, splitting by document structure (headers, sections), or using parent-child strategies where you retrieve small chunks but return their larger parent documents for context. Always test retrieval quality with a set of representative queries.
Use Hybrid Search
Vector similarity search alone can miss exact keyword matches. Combine it with BM25 or keyword search for better recall. Chroma and most vector stores support hybrid search modes. This is especially important for queries involving proper nouns, IDs, or technical terms.
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
# Create both retrievers
vector_retriever = vector_store.as_retriever(search_kwargs={"k": 5})
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 5
# Combine them
ensemble_retriever = EnsembleRetriever(
retrievers=[vector_retriever, bm25_retriever],
weights=[0.5, 0.5]
)
Implement Relevance Filtering
Not all retrieved documents are useful. Add a relevance threshold to filter out low-quality matches before they reach the LLM. This reduces noise and saves token costs.
def filtered_search(query: str, threshold: float = 0.5, k: int = 5):
"""Search with a minimum relevance score threshold."""
results = vector_store.similarity_search_with_relevance_scores(query, k=k)
filtered = [(doc, score) for doc, score in results if score >= threshold]
if not filtered:
return "No sufficiently relevant documents found."
return filtered
Add Query Transformation
Users often ask questions in ways that don't match document language well. Use an LLM to rewrite or expand queries before retrieval. This is especially powerful in conversational settings where the user's question may reference prior context.
query_rewrite_prompt = ChatPromptTemplate.from_messages([
("system", "Rewrite the following question into an effective search query "
"for a document retrieval system. Make it specific and keyword-rich. "
"Return only the rewritten query, nothing else."),
("human", "{question}")
])
rewrite_chain = query_rewrite_prompt | llm
def rewrite_query(question: str) -> str:
return rewrite_chain.invoke({"question": question}).content.strip()
Monitor and Evaluate
Build an evaluation set of question-answer pairs and regularly test your agent against it. Track metrics like retrieval recall, answer faithfulness, and response latency. LangSmith integrates directly with LangChain for tracing and evaluation, giving you visibility into which tools the agent uses and why.
Handle Edge Cases Gracefully
Set max_iterations on the agent executor to prevent infinite loops. Use handle_parsing_errors=True so the agent can recover from malformed tool calls. Always validate tool inputs, and consider rate limiting to prevent abuse.
Use Persistent Storage
For production, replace InMemoryChatMessageHistory with a persistent backend like Redis or PostgreSQL. This ensures conversation continuity across sessions and server restarts.
from langchain_redis.chat_message_history import RedisChatMessageHistory
def get_history(session_id: str) -> RedisChatMessageHistory:
return RedisChatMessageHistory(
session_id=session_id,
url="redis://localhost:6379"
)
agent_with_history = RunnableWithMessageHistory(
agent_executor,
get_history,
input_messages_key="input",
history_messages_key="chat_history",
)
Conclusion
Building a RAG-enabled agent with LangChain gives you a system that combines the grounding power of retrieval with the flexibility of autonomous tool use. By exposing your vector store as a tool alongside other utilities, the agent can decide when to search, how to search, and how to combine retrieved information with computation and reasoning. The key to a successful deployment lies in careful chunking, hybrid search strategies, relevance filtering, and continuous evaluation. Start with the basic setup we've built here, then iteratively improve each component based on real user queries. As your knowledge base grows and your agent handles more complex questions, the investment in retrieval quality and agent design will pay off in more accurate, trustworthy, and useful responses.