LangChain Retrieval Chains: RAG Done Right

What is a Retrieval Chain?

A Retrieval Chain is LangChain's composable architecture for implementing Retrieval-Augmented Generation (RAG). At its core, it connects a retriever that fetches relevant documents from a knowledge base to a language model that synthesizes those documents into a coherent, grounded answer. Instead of relying solely on the LLM's parametric memory, the chain injects retrieved context into the prompt, drastically reducing hallucinations and grounding responses in actual data.

LangChain abstracts this pattern into a declarative pipeline. You define a retriever (backed by a vector store, traditional search index, or hybrid system), choose a chain type that dictates how documents are formatted and fed to the model, and optionally attach memory for multi-turn conversations. The result is a single callable object—chain.invoke({"question": "..."})—that handles the entire retrieval-to-generation flow.

Why Retrieval Chains Matter

Foundation models are trained on static snapshots of the internet. They don't know your internal documentation, last week's product launch, or the latest regulatory filing. Fine-tuning can help, but it's expensive, slow to update, and doesn't guarantee factual accuracy. Retrieval Chains solve this by keeping knowledge external and fresh. You index documents once, and every subsequent query benefits from that living index. Key advantages include:

Source attribution — every answer can cite exactly which chunks informed it, making outputs auditable.
Zero retraining — add new documents and they're immediately available to the retriever.
Cost efficiency — you avoid stuffing the entire corpus into the context window, paying only for the retrieved chunks plus a small reasoning step.
Separation of concerns — retrieval logic, prompt engineering, and model choice evolve independently.

Core Components

Before writing code, it's essential to understand the building blocks LangChain expects in a retrieval pipeline:

Document Loaders

Document loaders ingest raw content from diverse sources—PDFs, web pages, databases, CSV files, markdown repositories—and convert them into LangChain's Document objects, which carry page_content and optional metadata.

from langchain_community.document_loaders import WebBaseLoader, PyPDFLoader

# Load a web page
loader = WebBaseLoader("https://docs.langchain.com")
docs = loader.load()

# Load a PDF
pdf_loader = PyPDFLoader("annual_report.pdf")
pdf_docs = pdf_loader.load()

print(f"Loaded {len(docs)} documents")
# Each document has: doc.page_content, doc.metadata

Text Splitters

Raw documents are often too large for embedding models or LLM context windows. Text splitters break them into semantically meaningful chunks while preserving overlap to prevent information loss at boundaries.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=150,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")

Embeddings & Vector Stores

Each chunk is embedded into a dense vector using an embedding model, then indexed in a vector store for fast approximate nearest-neighbor search. LangChain supports dozens of providers.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
print(f"Indexed {vectorstore._collection.count()} vectors")

Retrievers

A retriever is the interface that, given a query, returns relevant documents. The simplest is a vector-store backed retriever, but LangChain offers many flavors—keyword, hybrid, ensemble, contextual compression, and more.

retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}
)

# Test it
docs = retriever.invoke("What is the refund policy?")
for d in docs:
    print(d.page_content[:200], "...")

Building Your First Retrieval Chain

With the retriever ready, you can create a full RAG pipeline in one line using create_retrieval_chain and a pre-built question-answering prompt. The chain handles: (1) retrieving documents, (2) formatting them into a context block, (3) combining context with the user query in a prompt, and (4) calling the LLM.

from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4o", temperature=0)

# The "stuff" chain concatenates all retrieved docs into one prompt
combine_docs_chain = create_stuff_documents_chain(
    llm=llm,
    prompt=ChatPromptTemplate.from_messages([
        ("system", "You are a helpful assistant. Use the following pieces of retrieved context to answer the question. If you don't know, say you don't know.\n\nContext:\n{context}"),
        ("human", "{input}")
    ])
)

# Wrap with retrieval
rag_chain = create_retrieval_chain(
    retriever=retriever,
    combine_docs_chain=combine_docs_chain
)

# Query
result = rag_chain.invoke({"input": "What's the company's refund policy?"})
print("Answer:", result["answer"])
print("Source documents:", result["context"])

This pattern gives you a clean separation: result["input"] holds the original query, result["context"] holds the retrieved documents, and result["answer"] holds the generated response. You can inspect, log, or display each independently.

Advanced Retrieval Strategies

Stuff (Default)

The "stuff" method naively concatenates all retrieved documents into a single context block. It's fast and works beautifully when the total retrieved content fits comfortably within the model's context window. For most RAG use cases with k=4 and chunk sizes of 800 tokens, this is the sweet spot.

Map-Reduce

When you need to retrieve many documents (e.g., k=20) and they overflow the context window, map-reduce shines. It processes each document independently in a "map" step—extracting key information—then aggregates those summaries in a "reduce" step that produces the final answer.

from langchain.chains.combine_documents import create_map_reduce_documents_chain
from langchain_core.prompts import PromptTemplate

map_template = """Summarize the key points from this document relevant to the question: {question}

Document:
{context}

Key points:"""

reduce_template = """You are answering a question based on summaries of multiple documents.
Combine these summaries into a comprehensive answer.

Question: {question}

Document summaries:
{context}

Answer:"""

map_reduce_chain = create_map_reduce_documents_chain(
    llm=llm,
    map_prompt=PromptTemplate.from_template(map_template),
    reduce_prompt=PromptTemplate.from_template(reduce_template),
    document_variable_name="context"
)

rag_chain_map_reduce = create_retrieval_chain(
    retriever=retriever,
    combine_docs_chain=map_reduce_chain
)

result = rag_chain_map_reduce.invoke({
    "input": "What are the key strategic initiatives across all business units?",
    "question": "What are the key strategic initiatives across all business units?"
})
print(result["answer"])

Refine

The refine strategy iterates through documents sequentially, starting with an initial answer from the first document and refining it with each subsequent document. It's more nuanced than map-reduce but slower and more token-intensive due to the sequential LLM calls.

from langchain.chains.combine_documents import create_refine_documents_chain

refine_chain = create_refine_documents_chain(
    llm=llm,
    initial_prompt=PromptTemplate.from_template(
        "Answer the question based on the document below.\n\nQuestion: {question}\n\nDocument: {context}\n\nAnswer:"
    ),
    refine_prompt=PromptTemplate.from_template(
        "Refine the existing answer with new information from the document below.\n\nExisting answer: {existing_answer}\n\nNew document: {context}\n\nRefined answer:"
    ),
    document_variable_name="context"
)

rag_chain_refine = create_retrieval_chain(
    retriever=retriever,
    combine_docs_chain=refine_chain
)
result = rag_chain_refine.invoke({
    "input": "Explain the evolution of the company's sustainability efforts.",
    "question": "Explain the evolution of the company's sustainability efforts."
})
print(result["answer"])

Conversational Retrieval Chains with Memory

For multi-turn conversations where the user asks follow-up questions, you need to weave chat history into the retrieval and generation steps. LangChain provides create_history_aware_retriever which reformulates the user's query using conversation history before retrieval, and chains that inject memory into the prompt.

from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import MessagesPlaceholder
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

# Prompt to reformulate query given chat history
contextualize_q_prompt = ChatPromptTemplate.from_messages([
    ("system", "Given the chat history and the latest user question, formulate a standalone question suitable for vector retrieval. Do NOT answer the question, just rephrase it if needed."),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}")
])

history_aware_retriever = create_history_aware_retriever(
    llm=llm,
    retriever=retriever,
    contextualize_q_prompt=contextualize_q_prompt
)

# QA prompt with history
qa_prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant. Use the retrieved context and conversation history to answer.\n\nContext:\n{context}"),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}")
])

combine_docs = create_stuff_documents_chain(llm=llm, prompt=qa_prompt)
rag_chain_conversational = create_retrieval_chain(
    retriever=history_aware_retriever,
    combine_docs_chain=combine_docs
)

# In-memory chat history store
store = {}

def get_session_history(session_id: str):
    if session_id not in store:
        store[session_id] = ChatMessageHistory()
    return store[session_id]

chain_with_memory = RunnableWithMessageHistory(
    rag_chain_conversational,
    get_session_history,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="answer"
)

# First turn
response1 = chain_with_memory.invoke(
    {"input": "What's the refund policy?"},
    config={"configurable": {"session_id": "user-123"}}
)
print(response1["answer"])

# Follow-up (history-aware retriever reformulates before retrieval)
response2 = chain_with_memory.invoke(
    {"input": "Does it apply to digital products too?"},
    config={"configurable": {"session_id": "user-123"}}
)
print(response2["answer"])

Here's what happens under the hood on the second call: the contextualize_q_prompt sees the full chat history and rewrites "Does it apply to digital products too?" into something like "Does the refund policy apply to digital products?" before sending it to the retriever. This dramatically improves retrieval quality in conversational settings.

Best Practices for Production RAG

1. Chunk Strategy Is Everything

Experiment heavily with chunk_size and chunk_overlap. Too small and you lose semantic coherence; too large and you dilute retrieval precision. For dense technical docs, 500–800 tokens works well. For narrative content, 1000–1500 may be better. Always maintain 10–20% overlap to prevent splitting a critical sentence across two chunks.

2. Metadata Filtering

Tag documents with metadata (source, date, category, language) and apply filters at retrieval time. This prevents irrelevant chunks from polluting the context.

retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 4,
        "filter": {"category": "finance", "year": 2024}
    }
)

3. Contextual Compression

Use a ContextualCompressionRetriever with an LLMChainFilter to post-process retrieved documents. It drops irrelevant chunks and extracts only the sentences that actually answer the query, reducing prompt bloat.

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainFilter

compressor = LLMChainFilter.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=retriever
)
# Now only truly relevant content reaches the LLM

4. Evaluate Retrieval Quality First

Before tuning the generation prompt, validate your retriever in isolation. Check recall@k, precision@k, and mean reciprocal rank on a curated test set. A poorly performing retriever will produce poor answers regardless of prompt engineering.

5. Guard Against Prompt Injection in Retrieved Documents

If your knowledge base contains user-generated content, assume bad actors have embedded LLM instructions in it. Use system prompts that explicitly isolate context from instruction, and consider running retrieved documents through a sanitization step that strips suspicious patterns.

6. Streaming for UX

For long answers, stream tokens back to the user. LangChain's Runnable interface supports .stream() out of the box.

for chunk in rag_chain.stream({"input": "Explain our entire product roadmap."}):
    if "answer" in chunk:
        print(chunk["answer"], end="", flush=True)

7. Cache Embeddings and Responses

Embedding computation is expensive. Persist your vector store to disk and use LangChain's caching layer to avoid redundant LLM calls for identical queries. A simple InMemoryCache or Redis-backed cache saves both latency and cost.

from langchain_core.caches import InMemoryCache
from langchain_core.globals import set_llm_cache

set_llm_cache(InMemoryCache())
# Subsequent identical queries hit the cache

Conclusion

LangChain Retrieval Chains transform the abstract promise of RAG into a concrete, composable pipeline that you can inspect, debug, and scale. Starting from a simple vector-store-backed retriever and a stuff documents chain, you can evolve to history-aware conversational retrieval, map-reduce for large document sets, and contextual compression for noisy corpora. The key insight is that retrieval and generation are independent levers—tune each separately, evaluate rigorously, and let the chain abstraction handle the orchestration. Whether you're building a customer-support bot, an internal knowledge assistant, or a compliance research tool, mastering retrieval chains gives you a repeatable pattern for grounding LLM outputs in trustworthy, up-to-date information.