Evaluating Chunk Size Impact on RAG Performance

Retrieval-Augmented Generation (RAG) has become the de facto architecture for building LLM-powered applications over private or domain-specific data. Yet, one of the most overlooked levers in a RAG pipeline is the humble chunk size — the number of tokens (or characters) used to split source documents before embedding and indexing. This single parameter influences retrieval precision, context completeness, latency, cost, and ultimately the quality of the generated answer. In this tutorial, we'll explore what chunk size is, why it matters, how to systematically evaluate its impact, and the best practices you should adopt.

What Is Chunk Size in RAG?

In a RAG system, source documents are rarely short enough to embed as a single unit. Most embedding models accept a maximum input length (often 512 or 8192 tokens), and even when they accept more, retrieval works better when the indexed units are topically focused. Chunk size refers to the length of each text segment produced by a document splitter before it is embedded and stored in a vector database.

A typical chunking pipeline looks like this:

Document -> Splitter (chunk_size, chunk_overlap) -> Chunks -> Embedding Model -> Vector Store

Two parameters control the splitter's behavior:

chunk_size: the target length of each chunk (in characters or tokens).
chunk_overlap: the number of characters/tokens shared between adjacent chunks to preserve context across boundaries.

Why Chunk Size Matters

Chunk size sits at the intersection of two competing forces: retrieval precision and context completeness. Understanding this trade-off is essential.

Small chunks (e.g., 128–256 tokens): Each chunk is highly focused, so the embedding captures a narrow semantic meaning. This improves precision when the user's query maps cleanly to a specific passage. However, small chunks often lose surrounding context, which can leave the LLM without enough information to synthesize a complete answer.

Large chunks (e.g., 1024+ tokens): Each chunk carries more context, reducing the risk of missing information. But the embedding now averages over multiple topics, diluting semantic specificity. This can hurt recall for narrow queries and also increases token consumption in the generation step, raising cost and latency.

The impact extends beyond answer quality:

Cost: Larger chunks mean more tokens sent to the LLM at generation time.
Latency: More tokens in the prompt increase time-to-first-token.
Storage: More chunks (smaller size) increase vector store footprint and query cost.
Context window limits: Retrieving many large chunks can exceed the model's context window.

How to Evaluate Chunk Size Impact

The right way to evaluate chunk size is empirically. You need a labeled evaluation dataset, a consistent RAG pipeline, and metrics that capture both retrieval and generation quality. Let's build a practical evaluation harness.

Step 1: Prepare an Evaluation Dataset

Create a small set of question-answer pairs drawn from your corpus. Each entry should include the question, the expected answer, and optionally the source document reference.

eval_data = [
    {
        "question": "What is the maximum context window of GPT-4 Turbo?",
        "answer": "GPT-4 Turbo has a 128,000 token context window."
    },
    {
        "question": "How does cosine similarity differ from dot product?",
        "answer": "Cosine similarity measures the angle between vectors, ignoring magnitude, while dot product depends on both direction and magnitude."
    },
    {
        "question": "What is the purpose of chunk overlap in RAG?",
        "answer": "Chunk overlap preserves context across chunk boundaries so that information split between chunks is not lost."
    }
]

Step 2: Build a Configurable RAG Pipeline

Use LangChain to build a pipeline where chunk size is a parameter. This lets you sweep across values without rewriting code.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough

def build_rag_pipeline(documents, chunk_size, chunk_overlap=50):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", ". ", " ", ""]
    )
    chunks = splitter.split_documents(documents)

    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = FAISS.from_documents(chunks, embeddings)
    retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

    prompt = ChatPromptTemplate.from_template("""
    Answer the question based only on the following context:

    {context}

    Question: {question}
    """)

    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

    def format_docs(docs):
        return "\n\n".join(d.page_content for d in docs)

    rag_chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
    )
    return rag_chain, retriever

Step 3: Define Evaluation Metrics

You need metrics for both retrieval and generation. For retrieval, measure whether the correct source appears in the top-k results. For generation, compare the produced answer against the expected answer using semantic similarity or an LLM-as-a-judge approach.

from langchain_openai import OpenAIEmbeddings
import numpy as np

embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")

def cosine_similarity(a, b):
    a = np.array(a)
    b = np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def answer_similarity(expected, generated):
    e = embeddings_model.embed_query(expected)
    g = embeddings_model.embed_query(generated)
    return cosine_similarity(e, g)

def llm_judge(question, expected, generated):
    judge_prompt = f"""
    Question: {question}
    Expected answer: {expected}
    Generated answer: {generated}

    Score the generated answer from 0 to 1 based on correctness and completeness.
    Return only the number.
    """
    judge = ChatOpenAI(model="gpt-4o", temperature=0)
    score = judge.invoke(judge_prompt).content.strip()
    try:
        return float(score)
    except ValueError:
        return 0.0

Step 4: Run the Sweep

Now iterate over a range of chunk sizes, rebuild the pipeline for each, and record metrics across the evaluation dataset.

chunk_sizes = [128, 256, 512, 1024, 2048]
results = []

for size in chunk_sizes:
    rag_chain, retriever = build_rag_pipeline(documents, chunk_size=size)

    sim_scores = []
    judge_scores = []
    token_counts = []

    for item in eval_data:
        question = item["question"]
        expected = item["answer"]

        # Retrieve context to measure token usage
        retrieved_docs = retriever.invoke(question)
        context_tokens = sum(len(d.page_content) // 4 for d in retrieved_docs)
        token_counts.append(context_tokens)

        # Generate answer
        response = rag_chain.invoke(question)
        generated = response.content if hasattr(response, "content") else str(response)

        # Score
        sim_scores.append(answer_similarity(expected, generated))
        judge_scores.append(llm_judge(question, expected, generated))

    results.append({
        "chunk_size": size,
        "avg_similarity": np.mean(sim_scores),
        "avg_judge_score": np.mean(judge_scores),
        "avg_context_tokens": np.mean(token_counts)
    })

for r in results:
    print(f"Chunk size: {r['chunk_size']:>5} | "
          f"Sim: {r['avg_similarity']:.3f} | "
          f"Judge: {r['avg_judge_score']:.3f} | "
          f"Tokens: {r['avg_context_tokens']:.0f}")

Step 5: Analyze the Results

A typical output might look like this:

Chunk size:   128 | Sim: 0.71 | Judge: 0.65 | Tokens:  480
Chunk size:   256 | Sim: 0.83 | Judge: 0.79 | Tokens:  920
Chunk size:   512 | Sim: 0.88 | Judge: 0.86 | Tokens: 1820
Chunk size:  1024 | Sim: 0.84 | Judge: 0.82 | Tokens: 3640
Chunk size:  2048 | Sim: 0.76 | Judge: 0.74 | Tokens: 7280

In this example, 512 tokens produces the best balance. Smaller chunks lose context; larger chunks dilute embeddings and waste tokens. Your results will differ depending on your corpus, so always run this evaluation on your own data.

Best Practices

Start with 512 tokens and a 10–15% overlap. This is a reasonable default for most prose-heavy corpora. Tune from there.
Match chunk size to document structure. For code, split on function boundaries. For markdown, split on headers. For legal documents, split on clauses. Structural splitting often outperforms fixed-size splitting.
Use overlap to prevent boundary loss. A 50–100 token overlap catches sentences that straddle chunk boundaries. Too much overlap, however, creates redundant embeddings and wastes storage.
Evaluate on representative data. Your evaluation set should reflect real user questions, not just easy lookups. Include multi-hop and ambiguous questions.
Consider semantic chunking. Instead of fixed sizes, use embedding-based or LLM-based splitters that break at natural semantic boundaries. These can outperform fixed-size splitters but add complexity and cost.
Track cost alongside quality. A chunk size that improves answer quality by 2% but doubles token cost may not be worth it. Always report cost in your evaluation.
Re-evaluate when you change components. Switching embedding models, retrievers, or LLMs can shift the optimal chunk size. Re-run the sweep whenever you change a core component.
Log retrieved chunks during evaluation. Inspecting which chunks were retrieved for each question helps diagnose whether poor answers stem from retrieval failure or generation failure.

Conclusion

Chunk size is one of the highest-impact, lowest-effort parameters you can tune in a RAG system. By building a configurable pipeline, defining clear metrics, and sweeping across chunk sizes on your own evaluation data, you can empirically identify the sweet spot that balances retrieval precision, context completeness, cost, and latency. There is no universally optimal chunk size — it depends on your corpus, your embedding model, and your users' questions — but the evaluation methodology outlined here will help you find the right value for your specific application and give you a repeatable process to fall back on whenever your system changes.