Chunking Strategies for RAG: Fixed Size vs Semantic Chunking

Chunking Strategies for RAG: Fixed Size vs Semantic Chunking

Retrieval-Augmented Generation (RAG) has become the standard pattern for building LLM-powered applications that ground their answers in your private data. But before an embedding model ever sees your documents, you have to solve a deceptively simple problem: how do you split a long document into smaller pieces? This process is called chunking, and the strategy you choose has an outsized impact on retrieval quality, token cost, and answer accuracy.

In this tutorial, we'll compare the two most common chunking strategies — fixed-size chunking and semantic chunking — and show you how to implement both in Python. By the end, you'll understand when to reach for each approach and how to combine them for production-grade RAG pipelines.

Why Chunking Matters

LLMs have a finite context window, and embedding models typically perform best on short, focused passages. If you embed an entire 50-page PDF as a single vector, the embedding becomes a blurry average of many topics — retrieval will struggle to match it against specific user queries. On the other hand, if you split text into fragments that are too small or cut mid-sentence, you lose the context needed to answer questions correctly.

Good chunking balances three competing goals:

• Retrieval precision: Each chunk should be topically coherent so embeddings capture a clear meaning.
• Context sufficiency: Each chunk should contain enough information to answer a question without requiring the model to stitch fragments together.
• Cost efficiency: Fewer, well-sized chunks mean fewer embeddings to store and fewer tokens to process.

The wrong chunking strategy can silently degrade your RAG system. A chunk that splits a definition away from its term, or a conclusion away from its supporting evidence, will produce embeddings that retrieve poorly even with a great model.

Fixed-Size Chunking

Fixed-size chunking is the simplest and most widely used strategy. You split text into chunks of a predetermined number of tokens (or characters), optionally with an overlap between consecutive chunks to preserve context across boundaries.

The typical parameters are:

• chunk_size — the number of tokens (or characters) per chunk.
• chunk_overlap — the number of tokens shared between adjacent chunks.

Here's a minimal implementation using the langchain text splitter:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = """
Retrieval-Augmented Generation combines a retriever with a language model.
The retriever finds relevant passages from a knowledge base. The language
model then uses those passages to generate an answer. This approach reduces
hallucinations because the model can cite its sources.
"""

splitter = RecursiveCharacterTextSplitter(
    chunk_size=100,
    chunk_overlap=20,
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_text(text)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i} ---")
    print(chunk)
    print()

The RecursiveCharacterTextSplitter is a refinement of naive fixed-size splitting. It tries to break on paragraph boundaries first, then sentence boundaries, then word boundaries, only falling back to character-level splits when necessary. This produces chunks that are close to your target size while respecting natural text boundaries as much as possible.

You can also implement fixed-size token chunking directly with tiktoken if you want full control:

import tiktoken

def fixed_size_chunk(text, chunk_size=200, overlap=20, model="text-embedding-3-small"):
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        start = end - overlap
    return chunks

text = "Your long document text goes here..."
chunks = fixed_size_chunk(text, chunk_size=200, overlap=20)

print(f"Total chunks: {len(chunks)}")
for i, c in enumerate(chunks):
    print(f"Chunk {i}: {len(c)} chars")

Pros and Cons of Fixed-Size Chunking

• Pros: Simple to implement, deterministic, fast, predictable cost, easy to tune with just two parameters.
• Cons: Can split mid-sentence or mid-thought, ignores semantic boundaries, may produce chunks that mix unrelated topics or fragment related ones.

Fixed-size chunking works well for homogeneous text like transcripts, logs, or uniformly structured documents. It struggles with heterogeneous content like technical documentation, where a single 200-token chunk might span two completely unrelated sections.

Semantic Chunking

Semantic chunking splits text based on meaning rather than size. The core idea: group sentences that are topically related and split when the topic shifts. This produces chunks of variable length, but each chunk is semantically coherent.

The most common implementation, introduced by Greg Kamradt, works as follows:

• Split the document into individual sentences.
• Generate an embedding for each sentence.
• Compute the cosine similarity between consecutive sentence embeddings.
• Identify breakpoints where similarity drops below a threshold (e.g., a percentile-based cutoff).
• Group sentences between breakpoints into chunks.

Here's a complete implementation using OpenAI embeddings and numpy:

import numpy as np
from openai import OpenAI
import re

client = OpenAI()

def split_sentences(text):
    # Split on sentence-ending punctuation followed by whitespace
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    return [s for s in sentences if len(s) > 0]

def get_embeddings(texts, model="text-embedding-3-small"):
    response = client.embeddings.create(input=texts, model=model)
    return [item.embedding for item in response.data]

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def semantic_chunk(text, percentile=95, buffer_size=1):
    sentences = split_sentences(text)
    if len(sentences) <= 1:
        return sentences

    embeddings = get_embeddings(sentences)

    # Compute similarity between consecutive sentences
    similarities = []
    for i in range(len(embeddings) - 1):
        sim = cosine_similarity(embeddings[i], embeddings[i + 1])
        similarities.append(sim)

    # Find the threshold for breakpoints
    threshold = np.percentile(similarities, percentile)

    # Identify split points
    split_indices = []
    for i, sim in enumerate(similarities):
        if sim < threshold:
            split_indices.append(i)

    # Group sentences into chunks with a buffer for context
    chunks = []
    start = 0
    for idx in split_indices:
        end = idx + 1
        chunk_sentences = sentences[start:end + buffer_size]
        chunks.append(" ".join(chunk_sentences))
        start = end

    # Add the final chunk
    if start < len(sentences):
        chunks.append(" ".join(sentences[start:]))

    return chunks

text = """
Machine learning is a subset of artificial intelligence.
It focuses on algorithms that learn from data.
Deep learning is a specialized branch of machine learning.
Neural networks with many layers are used in deep learning.
Cooking Italian pasta requires fresh ingredients.
Tomato sauce and basil are essential for marinara.
You should boil pasta in salted water for best results.
"""

chunks = semantic_chunk(text, percentile=50)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i} ---")
    print(chunk)
    print()

Notice how the semantic chunker naturally separates the machine-learning sentences from the cooking sentences, even though no explicit topic labels were provided. The similarity between "deep learning" sentences and "Italian pasta" sentences drops sharply, triggering a split.

Pros and Cons of Semantic Chunking

• Pros: Produces topically coherent chunks, respects natural topic boundaries, often improves retrieval precision.
• Cons: More expensive (requires embedding every sentence), non-deterministic chunk sizes, slower, more complex to tune and debug.

Comparing the Two Strategies

Here's a side-by-side benchmark script that applies both strategies to the same document and reports chunk count and average chunk length:

from langchain_text_splitters import RecursiveCharacterTextSplitter

def compare_strategies(text):
    # Fixed-size
    fixed_splitter = RecursiveCharacterTextSplitter(
        chunk_size=300, chunk_overlap=30
    )
    fixed_chunks = fixed_splitter.split_text(text)

    # Semantic
    semantic_chunks = semantic_chunk(text, percentile=80)

    print("=== Fixed-Size Chunking ===")
    print(f"Chunks: {len(fixed_chunks)}")
    print(f"Avg length: {sum(len(c) for c in fixed_chunks) / len(fixed_chunks):.0f} chars")
    print()
    print("=== Semantic Chunking ===")
    print(f"Chunks: {len(semantic_chunks)}")
    print(f"Avg length: {sum(len(c) for c in semantic_chunks) / len(semantic_chunks):.0f} chars")

compare_strategies(your_document_text)

Typically, you'll observe that fixed-size produces uniform chunks while semantic produces variable-length chunks that align with topic shifts. The right choice depends on your content and retrieval goals.

Best Practices

• Start simple. Begin with fixed-size chunking (200–500 tokens, 10–20% overlap) as your baseline. Only move to semantic chunking if you observe retrieval quality issues caused by topic fragmentation.
• Add metadata. Attach source document, page number, section heading, and chunk index as metadata. This lets you filter during retrieval and cite sources in answers.
• Use parent-child chunking. Embed small chunks for precise retrieval, but return the larger parent chunk (or surrounding context) to the LLM. This combines the precision of small chunks with the context of large ones.
• Respect document structure. For Markdown or HTML, split on headers (#, <h2>) before applying size-based splits. LangChain's MarkdownHeaderTextSplitter and HTMLSectionSplitter handle this.
• Tune overlap carefully. Too little overlap loses context at boundaries; too much wastes tokens and creates redundant embeddings. 10–20% of chunk size is a good starting range.
• Evaluate empirically. Build a small evaluation set of questions and expected answers. Measure retrieval recall and answer faithfulness for each chunking strategy before committing.
• Consider hybrid approaches. Use semantic chunking for heterogeneous documents (wikis, manuals) and fixed-size for homogeneous ones (transcripts, logs). You can even apply different strategies per document type in the same pipeline.

Conclusion

Chunking is the foundation of every RAG system, and the strategy you choose ripples through retrieval quality, cost, and answer accuracy. Fixed-size chunking is fast, predictable, and good enough for many use cases — especially when paired with recursive character splitting and sensible overlap. Semantic chunking trades computational cost for coherence, producing chunks that respect natural topic boundaries and often improve retrieval precision on heterogeneous documents. The best approach is rarely dogmatic: start with a simple fixed-size baseline, measure retrieval quality on a real evaluation set, and adopt semantic or hybrid strategies only where the data demands it. By treating chunking as a first-class design decision rather than an afterthought, you set your RAG pipeline up for reliable, grounded, and cost-effective answers.