Parent-Document Retrieval: Preserving Context in RAG

Retrieval-Augmented Generation (RAG) has become the standard pattern for grounding large language models in private or domain-specific data. However, a subtle but persistent problem haunts most naive RAG implementations: the chunk-size tradeoff. Small chunks retrieve precisely but lack context. Large chunks preserve context but dilute the signal that embedding models need to match queries accurately. Parent-Document Retrieval is the architectural answer to this dilemma.

What Is Parent-Document Retrieval?

Parent-Document Retrieval (PDR) is a two-tier retrieval strategy where you search using small, semantically focused chunks but return the larger parent document (or a substantial section of it) that contains those chunks. The idea is to decouple the unit of retrieval from the unit of generation context.

Concretely, a long document is split into parent chunks — say, 2000 tokens each. Each parent chunk is then further subdivided into child chunks of 200 tokens. The child chunks are embedded and indexed. At query time, the system embeds the query, finds the most similar child chunks, maps each back to its parent, deduplicates, and passes the parent chunks to the LLM as context.

Why It Matters

Better retrieval precision. Embedding models perform best on short, topically coherent passages. A 200-token child chunk about "OAuth 2.0 token refresh" will match a user query far better than a 2000-token parent chunk that also covers unrelated topics.
Better generation quality. LLMs need surrounding context to interpret any passage correctly. A child chunk that says "the value must be set to true" is meaningless without the parent that explains what "the value" refers to.
Reduced hallucination. When the LLM sees the full parent passage, it is less likely to invent missing context that wasn't in the small chunk.
Flexible granularity. You can tune child size for retrieval and parent size for generation independently, giving you two knobs instead of one.

How to Use It

Below is a complete, runnable example using LangChain. We will build a parent-document retriever over a set of markdown documents, store child embeddings in an in-memory vector store, and demonstrate a query that retrieves precise child chunks but returns rich parent context.

# pip install langchain langchain-openai langchain-community chromadb

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore

# 1. Load source documents
loader = TextLoader("api_docs.md")
docs = loader.load()

# 2. Parent splitter: large chunks that preserve context
parent_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000,
    chunk_overlap=200,
)

# 3. Child splitter: small chunks optimized for embedding match
child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=300,
    chunk_overlap=50,
)

# 4. Vector store for child chunks (search index)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(
    collection_name="child_chunks",
    embedding_function=embeddings,
)

# 5. Key-value store mapping child_id -> parent document
parent_store = InMemoryStore()

# 6. Build the retriever
retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=parent_store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

# 7. Index documents: this splits into parents, then children,
#    embeds children, and stores parents in the docstore.
retriever.add_documents(docs)

# 8. Query: retrieves child chunks, returns parent documents
query = "How do I refresh an expired access token?"
results = retriever.invoke(query)

for i, doc in enumerate(results):
    print(f"--- Parent Document {i} ---")
    print(f"Length: {len(doc.page_content)} chars")
    print(doc.page_content[:500])
    print()

Notice that even though the vector store only contains 300-character child chunks, the results list contains full 2000-character parent documents. The retriever handles the mapping transparently.

Wiring It Into a RAG Chain

The retriever plugs directly into a standard LangChain RAG chain. Because the returned documents are full parent chunks, the LLM receives richer context than it would with naive chunk retrieval.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template("""
You are a technical assistant. Answer the question using only
the provided context. If the context is insufficient, say so.

Context:
{context}

Question: {question}
""")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

answer = rag_chain.invoke("How do I refresh an expired access token?")
print(answer)

Using a Persistent Parent Store

The InMemoryStore is fine for prototyping but loses data on restart. For production, use a persistent backing store such as Redis, a local filesystem store, or a SQL database. LangChain ships with several options:

from langchain.storage import LocalFileStore

# Persist parent documents to disk
parent_store = LocalFileStore("./parent_docs")

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=parent_store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

The vector store (Chroma) persists child embeddings to its own on-disk directory, while the parent store keeps the full parent text. On restart, both layers reload automatically, and the retriever reconstructs the mapping.

Best Practices

Choose parent size based on LLM context budget. If you return k parents, ensure k * parent_size fits comfortably within your model's context window alongside the prompt and expected answer.
Choose child size based on embedding model behavior. Most modern embedding models work well between 200 and 500 tokens. Run retrieval evaluations on your own data before committing.
Add overlap on both levels. Overlap in the parent splitter prevents losing context at parent boundaries; overlap in the child splitter prevents losing semantic continuity at child boundaries.
Deduplicate parents. Multiple child chunks from the same parent will frequently all match a query. Parent-Document Retrievers handle this automatically, but if you build a custom version, deduplication is essential to avoid feeding the same passage twice.
Preserve metadata. Carry source file, page number, and section title through both splitting stages. This metadata is invaluable for citations and debugging.
Consider a third tier for very large documents. For books or long manuals, you can add a "grandparent" layer: search grandchildren, return parents, but also include the grandparent heading for navigation context.
Evaluate with retrieval metrics. Measure recall@k and context relevance on a held-out question set. PDR typically improves answer faithfulness even when raw retrieval recall looks similar to naive chunking.

When Not to Use It

Parent-Document Retrieval adds complexity: a second store, a mapping layer, and two splitters to tune. For small, homogeneous corpora where each document is already short and topically focused, naive single-pass chunking may perform just as well with less operational overhead. PDR shines most when documents are long, heterogeneous, and contain sections whose meaning depends on surrounding context — exactly the situation in most real-world knowledge bases.

Parent-Document Retrieval resolves the fundamental tension between retrieval precision and generation context by treating them as separate concerns. By embedding small, focused child chunks and returning their larger parent passages to the LLM, you get the best of both worlds: accurate semantic search and rich, interpretable context. For any RAG system operating over substantial documents, it is one of the highest-leverage architectural choices you can make.