Introduction to Chunking in RAG
Retrieval-Augmented Generation (RAG) pipelines rely on one critical preprocessing step: chunking. Before your documents can be indexed, embedded, and later retrieved to answer user queries, they must be split into smaller, manageable pieces. These pieces — chunks — directly impact the quality of the generated answers, the relevance of retrieved context, and the overall efficiency of your RAG system. Choosing the right chunking strategy means balancing size, overlap, and structural awareness. This tutorial walks you through the theory and provides ready-to-use code examples so you can implement robust chunking in your own projects.
What is Chunking?
Chunking is the process of dividing a large document or corpus into smaller segments (chunks). Each chunk becomes a standalone piece of text that is embedded as a single vector and stored in a vector database. When a user query arrives, the system compares the query embedding against these chunk embeddings to retrieve the most relevant chunks, which are then fed to the language model as context.
Without chunking, an entire book or lengthy article would be represented by a single embedding vector. That vector would be too coarse to capture detailed semantic nuances, making precise retrieval nearly impossible. Moreover, language models have token limits, and you cannot stuff an entire document into the prompt. Chunking solves both problems: it enables fine-grained retrieval and respects LLM context windows.
Why Chunking Matters
Chunking is not a one-size-fits-all operation. The way you split your documents affects:
- Retrieval accuracy – chunks that are too large may contain multiple topics, diluting the embedding signal. Chunks that are too small may lack necessary context to answer the query.
- Embedding quality – most embedding models are trained on passages of a certain length (e.g. 256–512 tokens). Sending chunks significantly longer than that may degrade semantic representation.
- LLM context utilization – retrieved chunks will be concatenated into the prompt. You want each chunk to be dense with relevant information, not padded with irrelevant filler.
- Latency and cost – more chunks mean more embeddings to generate and store, and potentially more retrieval overhead.
In short, the chunking strategy is a design decision that directly impacts the end-to-end performance of your RAG system.
Core Chunking Parameters
Three primary parameters define a chunking strategy: size, overlap, and structure.
Chunk Size
Chunk size is the maximum number of characters or tokens each chunk will contain. It determines how much information a single vector encodes. Common sizes range from 128 tokens (for very granular, factoid retrieval) to 1024 tokens (for longer, self-contained passages). The optimal size depends on your documents, the embedding model, and the expected queries. A chunk size that matches the embedding model’s training length often yields the best semantic matching. For example, if you use text-embedding-ada-002 (which handles up to 8192 tokens but works best around 256–512 tokens per chunk), setting chunk size to 256–512 tokens is a solid starting point.
Overlap
Overlap is the amount of text shared between consecutive chunks. For instance, with a chunk size of 400 tokens and an overlap of 50 tokens, the first chunk covers tokens 1–400, the second chunk covers 351–750, the third 701–1100, and so on. Overlap prevents cutting a sentence or a thought in the middle, ensuring that context that might be split across a boundary is available in both chunks. This improves retrieval continuity and reduces the risk of missing information that straddles a chunk edge.
Structure-Aware Splitting
Structure-aware chunking respects the natural boundaries of a document — paragraphs, headings, sections, or even semantic clusters. Instead of blindly splitting after a fixed number of characters, you split on meaningful separators like newlines, markdown headers, or sentence endings. This preserves the logical coherence of each chunk and often leads to better retrieval and generation quality.
How to Choose the Right Chunk Size
Choosing the right chunk size is an empirical process. Start by considering the embedding model's maximum input length and its recommended chunk size. Then experiment with different sizes on a representative query set and evaluate retrieval precision and answer quality. Below are common approaches to implement different chunk sizes.
Fixed-size Character Chunking
The simplest method: split the text after every N characters. This is fast but often breaks words and sentences. It works for unstructured text where semantic boundaries are fuzzy, but you'll almost always want to combine it with overlap and a sensible separator.
# Basic fixed-size character chunking
def fixed_size_chunk(text: str, chunk_size: int, overlap: int = 0):
chunks = []
start = 0
while start < len(text):
end = min(start + chunk_size, len(text))
chunks.append(text[start:end])
start += chunk_size - overlap
return chunks
# Example
text = "Your long document here..." * 100
chunks = fixed_size_chunk(text, chunk_size=500, overlap=50)
print(f"Produced {len(chunks)} chunks")
Sentence-aware Chunking with spaCy
By splitting on sentence boundaries, you keep entire sentences intact. This preserves grammatical coherence and avoids cutting a sentence mid-way. You can then combine sentences until you reach a token limit.
import spacy
from typing import List
nlp = spacy.load("en_core_web_sm")
def sentence_chunk(text: str, max_tokens: int, overlap_sentences: int = 1) -> List[str]:
doc = nlp(text)
sentences = [sent.text for sent in doc.sents]
chunks = []
current_chunk = []
current_token_count = 0
for i, sentence in enumerate(sentences):
sentence_tokens = len(sentence.split())
if current_token_count + sentence_tokens > max_tokens and current_chunk:
chunks.append(" ".join(current_chunk))
# Keep overlap sentences from the end of the previous chunk
current_chunk = current_chunk[-overlap_sentences:] if overlap_sentences > 0 else []
current_token_count = sum(len(s.split()) for s in current_chunk)
current_chunk.append(sentence)
current_token_count += sentence_tokens
if current_chunk:
chunks.append(" ".join(current_chunk))
return chunks
# Example usage
document = "Your multi-sentence text. More sentences here..."
chunks = sentence_chunk(document, max_tokens=256, overlap_sentences=2)
Recursive Character Splitting (LangChain)
The most widely adopted method in RAG pipelines is recursive splitting using a hierarchy of separators. It tries to split on paragraph breaks first, then newlines, then sentences, and finally characters, until each chunk falls within the desired size. This preserves structure as much as possible. LangChain provides RecursiveCharacterTextSplitter for this purpose.
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Initialize the splitter with chunk size, overlap, and separators
splitter = RecursiveCharacterTextSplitter(
chunk_size=400, # target characters per chunk
chunk_overlap=50, # overlap in characters
separators=["\n\n", "\n", ". ", " ", ""], # ordered priority
length_function=len,
is_separator_regex=False,
)
# Split a document
document = "Your long text... with paragraphs.\n\nSecond paragraph..."
chunks = splitter.split_text(document)
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1} ({len(chunk)} chars): {chunk[:60]}...")
Implementing Overlap
Overlap is crucial when chunks are formed by fixed-size or recursive splitting, because the cut point can fall in the middle of a relevant phrase. Overlap ensures that any information that would otherwise be split is contained in both adjacent chunks. A typical overlap is 10–20% of the chunk size. Too little overlap may still lose context; too much wastes storage and retrieval capacity.
The code examples above already incorporate overlap. The RecursiveCharacterTextSplitter handles overlap internally. In custom implementations, you simply subtract overlap from the step size when advancing the window, as shown in the fixed-size example. When using sentence chunking, overlap is controlled by keeping the last overlap_sentences sentences from the previous chunk.
Structure-Preserving Chunking Strategies
Beyond size and overlap, preserving the document's inherent structure yields chunks that are more coherent and self-contained. Here are practical strategies.
Markdown Header Splitting
For documents with clear markdown headings (e.g., technical manuals, READMEs), you can split by headers so that each section becomes a chunk. This keeps related content together and often aligns perfectly with the user's information need.
import re
def split_by_markdown_headers(text: str, max_section_length: int = 1000) -> List[str]:
# Split on any markdown header (e.g., ##, ###)
sections = re.split(r'\n(?=#{1,6}\s)', text)
chunks = []
current = ""
for section in sections:
if len(current) + len(section) > max_section_length and current:
chunks.append(current.strip())
current = section
else:
current += "\n" + section
if current:
chunks.append(current.strip())
return chunks
# Example
markdown_doc = """
## Introduction
This is intro text...
## Usage
Instructions here...
"""
chunks = split_by_markdown_headers(markdown_doc, max_section_length=800)
Semantic Chunking (Experimental)
Semantic chunking uses embeddings to find natural breakpoints where the topic shifts. You can compute sentence embeddings, then split when the similarity between consecutive sentences drops below a threshold. This creates chunks that are topically coherent.
import numpy as np
from typing import List
# Assume you have an embedding function embed(text) -> np.array
def semantic_chunk(sentences: List[str], embed_func, similarity_threshold: float = 0.5) -> List[str]:
embeddings = [embed_func(s) for s in sentences]
chunks = []
current_chunk = [sentences[0]]
for i in range(1, len(sentences)):
sim = np.dot(embeddings[i-1], embeddings[i]) / (np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i]))
if sim < similarity_threshold:
chunks.append(" ".join(current_chunk))
current_chunk = []
current_chunk.append(sentences[i])
if current_chunk:
chunks.append(" ".join(current_chunk))
return chunks
# Pseudocode; requires an embedding model like sentence-transformers
# sentences = nltk.sent_tokenize(document)
# chunks = semantic_chunk(sentences, model.encode, threshold=0.6)
Best Practices for Chunking in RAG
When designing your chunking pipeline, keep these recommendations in mind:
- Start with the embedding model's sweet spot. Check the model's documentation for recommended chunk sizes. For
text-embedding-3-small, 256–512 tokens works well; fortext-embedding-ada-002, 256–1024 tokens. - Evaluate with real queries. Run a set of representative queries and check retrieval recall. If chunks are too small, you might miss context; too large, you'll retrieve irrelevant content. Iterate on size and overlap.
- Use a modest overlap. 10–20% of chunk size is a safe range. For 500-character chunks, 50–100 characters overlap. Overlap prevents fragmentation but too much duplicates information unnecessarily.
- Preserve document structure. Whenever possible, split on natural boundaries like paragraphs, sections, or sentences. Recursive character splitting with well-chosen separators is a strong default.
- Handle special elements. Tables, lists, and code blocks often break when split blindly. Consider extracting them into separate chunks with metadata or using structure-aware splitters that treat them as indivisible blocks.
- Monitor token counts, not just characters. Embedding models and LLMs count tokens. Use a tokenizer (e.g.,
tiktoken) to measure chunk size in tokens to avoid exceeding limits. - Combine chunking with metadata. Attach section titles, page numbers, or document names as metadata. This enriches the context and helps the LLM understand provenance.
Putting It All Together: A Complete Chunking Pipeline
Below is an end-to-end example using LangChain's RecursiveCharacterTextSplitter with token-based chunk sizing, overlap, and metadata attachment. It reads a document, splits it, and prepares chunks for embedding.
from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken
# Tokenizer for accurate token counting
tokenizer = tiktoken.get_encoding("cl100x_base") # Used by many models
def token_length(text: str) -> int:
return len(tokenizer.encode(text))
# Configure splitter with token-based chunk size
splitter = RecursiveCharacterTextSplitter(
chunk_size=400, # target tokens per chunk
chunk_overlap=50, # token overlap
separators=["\n\n", "\n", ". ", " ", ""],
length_function=token_length, # use token count, not characters
is_separator_regex=False,
)
# Sample document
document = """
# Product Overview
The XYZ Widget is a revolutionary device designed to simplify daily tasks.
It features an intuitive interface, robust durability, and seamless connectivity.
## Key Benefits
- Increased productivity by up to 30%
- Energy-efficient operation
- 24/7 customer support
## Technical Specifications
| Feature | Details |
|---------------|------------------|
| Weight | 2.3 kg |
| Battery Life | 12 hours |
| Connectivity | Bluetooth 5.0, Wi-Fi 6 |
"""
chunks = splitter.create_documents([document])
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1} (tokens: {token_length(chunk.page_content)}):\n{chunk.page_content}\n")
This pipeline ensures chunks are sized within the model's preferred token range, respects paragraph and sentence boundaries, and includes overlap to maintain context continuity. You can then embed chunks with your embedding model and store them in a vector database.
Conclusion
Chunking is a foundational step in any RAG system that directly influences retrieval quality, response accuracy, and system efficiency. By carefully tuning chunk size, overlap, and respecting document structure, you can dramatically improve your pipeline's performance. Start with a proven approach like recursive character splitting with token-aware sizes, then iterate based on real-world query evaluations. The code examples in this tutorial give you the building blocks to implement robust chunking strategies and adapt them to your specific domain. Remember: a well-chunked document base is the backbone of a reliable RAG application.