Hybrid Search for RAG: Combining Keywords and Vectors
Retrieval-Augmented Generation (RAG) systems are only as good as the retrieval step that feeds them. If your retriever pulls back irrelevant documents, even the most powerful language model will produce weak answers. Hybrid search — blending traditional keyword-based retrieval with modern vector similarity search — has emerged as a powerful strategy to significantly boost retrieval quality. This tutorial walks you through what hybrid search is, why it matters, how to implement it from scratch, and the best practices that make it production-ready.
What Is Hybrid Search?
Hybrid search is the practice of running two or more retrieval methods in parallel on the same query, then merging their results into a single ranked list. The most common pairing combines:
- Sparse / lexical retrieval — Classic keyword matching using inverted indices (BM25, TF-IDF). Excels at exact term matching, handles rare terms well, and is fast and deterministic.
- Dense / semantic retrieval — Vector similarity search using embeddings (text-embedding-3-small, Cohere Embed, etc.). Captures semantic meaning, synonyms, and conceptual relationships that keyword search misses.
By marrying the two, you get the best of both worlds: the precision of exact keyword matching and the semantic understanding of neural embeddings. This is especially critical in RAG pipelines where queries can be ambiguous, contain domain-specific jargon, or require factual precision that embeddings alone may blur.
Why Hybrid Search Matters for RAG
Pure vector search has well-documented failure modes. It can retrieve superficially similar but factually wrong documents, struggle with rare entities or acronyms, and underperform on queries that contain very specific filter criteria. Pure keyword search, on the other hand, misses semantically relevant documents that use different vocabulary. In a RAG context, these failures translate directly into hallucinated or incomplete LLM responses.
Consider a user query: "What was the revenue impact of Project Nightingale in Q4?"
- A vector search might return documents about "bird conservation programs" because "Nightingale" is semantically associated with birds, missing the internal project codename entirely.
- A keyword search would lock onto "Project Nightingale" and "Q4 revenue" precisely, but might miss a document titled "Financial Summary for Avian Initiative" that contains the answer.
Hybrid search resolves this tension. It ensures that both the exact-match documents and the semantically related documents surface, dramatically improving recall and giving the LLM a richer context to work with.
Core Approaches to Combining Results
Once you have two ranked lists — one from BM25 and one from vector similarity — you need to fuse them. There are three primary strategies:
1. Reciprocal Rank Fusion (RRF)
RRF is the most popular approach due to its simplicity and robustness. It ignores raw scores entirely and works solely with document ranks, making it immune to score calibration differences between the two retrievers.
def reciprocal_rank_fusion(rankings, k=60):
"""
rankings: list of dicts mapping doc_id -> rank (0-based or 1-based)
Returns fused scores dict: doc_id -> RRF score
"""
fused_scores = {}
for ranking in rankings:
for doc_id, rank in ranking.items():
# RRF formula: 1 / (k + rank)
# Using rank starting from 1 for consistency
rrf_score = 1.0 / (k + rank)
fused_scores[doc_id] = fused_scores.get(doc_id, 0) + rrf_score
return fused_scores
# Example usage
bm25_ranking = {"doc_1": 1, "doc_3": 2, "doc_2": 3, "doc_5": 4, "doc_4": 5}
vector_ranking = {"doc_2": 1, "doc_4": 2, "doc_1": 3, "doc_6": 4, "doc_3": 5}
rankings = [bm25_ranking, vector_ranking]
fused = reciprocal_rank_fusion(rankings, k=60)
# Sort by descending RRF score
final_ranking = sorted(fused.items(), key=lambda x: x[1], reverse=True)
# Result: [('doc_1', 0.0327), ('doc_2', 0.0326), ('doc_3', 0.0319), ...]
print(final_ranking)
The constant k (typically 60) dampens the impact of high ranks. A document ranked 1st in one list gets a score of 1/(60+1) ≈ 0.0164, while a document ranked 100th gets 1/(60+100) ≈ 0.00625. The difference shrinks as ranks grow, which prevents a single top-ranked result from dominating. RRF requires no score normalization and works across any number of retrieval sources.
2. Score Normalization and Weighted Sum
If you need fine-grained control, normalize each retriever's scores to a common scale (e.g., min-max or softmax) and then take a weighted linear combination.
import numpy as np
def min_max_normalize(scores_dict):
"""Normalize scores to [0, 1] range using min-max scaling."""
values = list(scores_dict.values())
min_val = min(values)
max_val = max(values)
if max_val == min_val:
return {k: 1.0 for k in scores_dict}
return {
k: (v - min_val) / (max_val - min_val)
for k, v in scores_dict.items()
}
def weighted_fusion(bm25_scores, vector_scores, alpha=0.3):
"""
alpha: weight for BM25 (0.3 means 30% BM25, 70% vector)
Returns fused scores dict
"""
norm_bm25 = min_max_normalize(bm25_scores)
norm_vector = min_max_normalize(vector_scores)
all_doc_ids = set(norm_bm25.keys()) | set(norm_vector.keys())
fused = {}
for doc_id in all_doc_ids:
bm25_score = norm_bm25.get(doc_id, 0.0)
vec_score = norm_vector.get(doc_id, 0.0)
fused[doc_id] = alpha * bm25_score + (1 - alpha) * vec_score
return fused
# Example with raw scores
bm25_raw = {"doc_1": 4.5, "doc_3": 3.2, "doc_2": 2.1, "doc_5": 1.8, "doc_4": 0.9}
vector_raw = {"doc_2": 0.95, "doc_4": 0.88, "doc_1": 0.82, "doc_6": 0.79, "doc_3": 0.65}
fused_weighted = weighted_fusion(bm25_raw, vector_raw, alpha=0.3)
final = sorted(fused_weighted.items(), key=lambda x: x[1], reverse=True)
print(final)
This approach gives you explicit control over the importance of each retriever. However, it requires careful calibration — the raw score distributions from BM25 and cosine similarity are fundamentally different, and a poor normalization strategy can skew results.
3. Re-Ranking with a Cross-Encoder
For maximum quality, you can use a cross-encoder model to re-rank the pooled candidates from both retrievers. This is the most computationally expensive option but often yields the best results.
from sentence_transformers import CrossEncoder
# Initialize cross-encoder (runs on GPU if available)
cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
def cross_encoder_rerank(query, documents, top_k=10):
"""
query: str
documents: list of (doc_id, doc_text) tuples
Returns top_k documents sorted by cross-encoder relevance
"""
pairs = [(query, doc_text) for doc_id, doc_text in documents]
scores = cross_encoder.predict(pairs)
# Pair scores with doc_ids and sort
scored_docs = list(zip([d[0] for d in documents], scores))
scored_docs.sort(key=lambda x: x[1], reverse=True)
return scored_docs[:top_k]
# Example: pool candidates from both retrievers
# First, fetch top-20 from BM25 and top-20 from vector search
bm25_candidates = [
("doc_1", "Project Nightingale Q4 2024 revenue report shows 12% growth..."),
("doc_3", "Quarterly financial summary for avian conservation initiative..."),
# ... more candidates
]
vector_candidates = [
("doc_2", "Nightingale project financials indicate strong Q4 performance..."),
("doc_4", "Q4 earnings call transcript mentions Project Nightingale revenue..."),
# ... more candidates
]
# Deduplicate and combine
all_candidates = {}
for doc_id, text in bm25_candidates + vector_candidates:
all_candidates[doc_id] = text
# Re-rank with cross-encoder
query = "What was the revenue impact of Project Nightingale in Q4?"
reranked = cross_encoder_rerank(
query,
list(all_candidates.items()),
top_k=10
)
print(reranked)
Cross-encoder re-ranking is ideal when you can afford the extra latency (typically 50–200ms per query) and want the highest precision. A common production pattern is: retrieve top-50 from BM25 and top-50 from vector search, deduplicate, then re-rank the combined pool with a cross-encoder down to the final top-10 that gets passed to the LLM.
Complete Hybrid RAG Pipeline
Let's build a full hybrid retrieval pipeline that integrates BM25, vector search, RRF fusion, and feeds results into a RAG chain. We'll use rank-bm25 for lexical search and a local embedding model for dense retrieval.
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import json
class HybridRetriever:
"""
A hybrid retriever combining BM25 (sparse) and embedding-based (dense) search
with Reciprocal Rank Fusion.
"""
def __init__(self, documents, embedding_model_name='all-MiniLM-L6-v2'):
"""
documents: list of dicts with 'id' and 'text' keys
"""
self.documents = documents
self.doc_texts = [doc['text'] for doc in documents]
self.doc_ids = [doc['id'] for doc in documents]
self.id_to_text = {doc['id']: doc['text'] for doc in documents}
# Initialize BM25
tokenized_corpus = [self._tokenize(text) for text in self.doc_texts]
self.bm25 = BM25Okapi(tokenized_corpus)
# Initialize embedding model and pre-compute document embeddings
self.embedder = SentenceTransformer(embedding_model_name)
self.doc_embeddings = self.embedder.encode(
self.doc_texts,
show_progress_bar=True,
convert_to_numpy=True
)
# Normalize for fast cosine similarity
self.doc_embeddings = self.doc_embeddings / np.linalg.norm(
self.doc_embeddings, axis=1, keepdims=True
)
def _tokenize(self, text):
"""Simple whitespace tokenizer for BM25."""
return text.lower().split()
def bm25_search(self, query, top_k=50):
"""Return dict of doc_id -> BM25 score for top_k results."""
tokenized_query = self._tokenize(query)
scores = self.bm25.get_scores(tokenized_query)
# Get top_k indices
top_indices = np.argsort(scores)[::-1][:top_k]
results = {}
for idx in top_indices:
if scores[idx] > 0:
results[self.doc_ids[idx]] = scores[idx]
return results
def vector_search(self, query, top_k=50):
"""Return dict of doc_id -> cosine similarity for top_k results."""
query_embedding = self.embedder.encode([query], convert_to_numpy=True)
query_embedding = query_embedding / np.linalg.norm(query_embedding)
similarities = cosine_similarity(query_embedding, self.doc_embeddings)[0]
top_indices = np.argsort(similarities)[::-1][:top_k]
results = {}
for idx in top_indices:
results[self.doc_ids[idx]] = float(similarities[idx])
return results
def hybrid_search(self, query, top_k=10, rrf_k=60):
"""
Execute hybrid search: BM25 + vector, fuse with RRF, return top_k.
"""
# Get rankings from both retrievers
bm25_scores = self.bm25_search(query, top_k=50)
vector_scores = self.vector_search(query, top_k=50)
# Convert scores to rankings (1-based, best = rank 1)
bm25_ranking = {}
for rank, (doc_id, _) in enumerate(
sorted(bm25_scores.items(), key=lambda x: x[1], reverse=True), start=1
):
bm25_ranking[doc_id] = rank
vector_ranking = {}
for rank, (doc_id, _) in enumerate(
sorted(vector_scores.items(), key=lambda x: x[1], reverse=True), start=1
):
vector_ranking[doc_id] = rank
# Reciprocal Rank Fusion
fused_scores = {}
for ranking in [bm25_ranking, vector_ranking]:
for doc_id, rank in ranking.items():
fused_scores[doc_id] = fused_scores.get(doc_id, 0) + 1.0 / (rrf_k + rank)
# Sort and return top_k with text
sorted_docs = sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)
results = []
for doc_id, score in sorted_docs[:top_k]:
results.append({
'id': doc_id,
'text': self.id_to_text[doc_id],
'rrf_score': round(score, 6),
'bm25_rank': bm25_ranking.get(doc_id, None),
'vector_rank': vector_ranking.get(doc_id, None)
})
return results
# =============================================
# Example: Build and query the hybrid retriever
# =============================================
# Sample document corpus (in production, this would be your knowledge base)
corpus = [
{"id": "doc_001", "text": "Project Nightingale is an internal initiative to modernize healthcare data pipelines. The project began in Q2 2023 and has shown 15% cost reduction."},
{"id": "doc_002", "text": "Q4 2024 financial results: Revenue increased 12% year-over-year, driven by strong performance in the enterprise segment."},
{"id": "doc_003", "text": "The Nightingale conservation program protects endangered bird species across North America. Funding increased in 2024."},
{"id": "doc_004", "text": "Project Nightingale revenue contribution reached $4.2M in Q4 2024, exceeding targets by 8% according to the quarterly business review."},
{"id": "doc_005", "text": "Enterprise API usage grew 40% in Q4, contributing to overall revenue growth of $12M for the fiscal year."},
{"id": "doc_006", "text": "Bird species population surveys indicate a 3% decline in nightingale numbers across European habitats in 2024."},
{"id": "doc_007", "text": "The healthcare data modernization project (codenamed Nightingale) delivered $4.2M in incremental revenue during the fourth quarter of 2024."},
{"id": "doc_008", "text": "Team standup notes: Discussed Nightingale migration timeline and Q4 budget allocation for cloud infrastructure."},
{"id": "doc_009", "text": "Annual shareholder letter highlights strong Q4 performance across all business units including the Nightingale healthcare initiative."},
{"id": "doc_010", "text": "Lucerne Festival Orchestra performed Nightingale-themed compositions to critical acclaim in spring 2024."},
]
# Initialize retriever
retriever = HybridRetriever(corpus)
# Execute a hybrid search
query = "What was the revenue impact of Project Nightingale in Q4?"
results = retriever.hybrid_search(query, top_k=5)
print("Hybrid Search Results:")
print("=" * 60)
for i, result in enumerate(results, 1):
print(f"\nRank {i}: {result['id']} (RRF: {result['rrf_score']})")
print(f" BM25 rank: {result['bm25_rank']}, Vector rank: {result['vector_rank']}")
print(f" Text: {result['text'][:120]}...")
# Now you can feed these results into your RAG LLM call
# context = "\n\n".join([r['text'] for r in results])
# prompt = f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
This pipeline produces ranked results where documents containing exact matches for "Project Nightingale" and "Q4 revenue" are boosted by BM25, while semantically related documents about financial performance also surface through vector similarity. Notice how doc_004 and doc_007 — which contain the precise answer — rank highly because they satisfy both retrieval methods.
Integrating with LangChain and LlamaIndex
Most production RAG systems use frameworks like LangChain or LlamaIndex. Here's how to plug hybrid search into each.
LangChain Hybrid Retriever
from langchain.retrievers import BM25Retriever, EnsembleRetriever
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.schema import Document
# Prepare documents as LangChain Document objects
docs = [
Document(page_content=text, metadata={"id": doc_id})
for doc_id, text in your_documents # your document list
]
# BM25 retriever
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 10 # retrieve top 10
# Vector retriever
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})
# Ensemble retriever with RRF (default weights [0.5, 0.5])
ensemble_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, vector_retriever],
weights=[0.4, 0.6], # give vector search slightly more weight
c=60 # RRF constant
)
# Use in a RAG chain
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
qa_chain = RetrievalQA.from_chain_type(
llm=OpenAI(),
chain_type="stuff",
retriever=ensemble_retriever
)
result = qa_chain.run("What was the revenue impact of Project Nightingale in Q4?")
print(result)
LlamaIndex Hybrid Search
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import HybridRetriever
from llama_index.embeddings.openai import OpenAIEmbedding
# Build vector index
storage_context = StorageContext.from_defaults()
vector_index = VectorStoreIndex.from_documents(
documents,
storage_context=storage_context,
embed_model=OpenAIEmbedding()
)
# Build BM25 retriever (requires tokenized nodes)
bm25_retriever = BM25Retriever.from_defaults(
nodes=vector_index.docstore.docs.values(),
similarity_top_k=10
)
# Vector retriever
vector_retriever = vector_index.as_retriever(similarity_top_k=10)
# Hybrid retriever
hybrid_retriever = HybridRetriever(
vector_retriever,
bm25_retriever,
vector_weight=0.6,
bm25_weight=0.4
)
# Query
nodes = hybrid_retriever.retrieve("What was the revenue impact of Project Nightingale in Q4?")
for node in nodes:
print(f"Score: {node.score:.4f} | {node.text[:100]}...")
Best Practices for Production Hybrid Search
1. Tune the Fusion Weights on Your Data
The optimal balance between BM25 and vector search depends entirely on your domain. For technical documentation with lots of unique identifiers and acronyms, lean heavier on BM25 (alpha 0.5–0.7). For conversational FAQ or support knowledge bases, vector search may deserve more weight (alpha 0.2–0.3). Run an evaluation set with ground-truth relevance judgments and sweep the alpha parameter to find the sweet spot.
2. Use Separate Top-K Retrieval Before Fusion
A common mistake is fusing only the top-5 or top-10 from each retriever. Instead, retrieve a wider pool (top-50 to top-200) from each method, fuse, and then take the final top-K. This ensures documents that rank poorly in one method but highly in the other still have a chance to surface.
3. Deduplicate Carefully
When the same document appears in both retrievers' results, ensure your fusion logic handles it correctly — RRF naturally sums scores for duplicates, which is desirable. With weighted sum fusion, take the maximum or average of the two normalized scores to avoid double-counting.
4. Add Metadata Filters as a Pre-Retrieval Step
Hybrid search works best when combined with structured metadata filtering. Before running BM25 and vector search, apply filters like date ranges, document types, or access control lists. This shrinks the candidate pool and ensures both retrievers operate on relevant subsets.
# Example: pre-filtering with metadata
def filtered_hybrid_search(query, filters, top_k=10):
"""
filters: dict like {'date_range': ('2024-10-01', '2024-12-31'), 'category': 'financial'}
"""
# Apply filters to document corpus first
filtered_docs = [
doc for doc in full_corpus
if doc['date'] >= filters.get('date_range', ('2000-01-01',))[0]
and doc['date'] <= filters.get('date_range', ('2099-12-31',))[1]
and doc.get('category') == filters.get('category', doc.get('category'))
]
# Now run hybrid search on filtered_docs only
retriever = HybridRetriever(filtered_docs)
return retriever.hybrid_search(query, top_k=top_k)
5. Monitor and Log Both Retrievers' Performance
In production, log the individual top-K from BM25 and vector search alongside the final fused results. This gives you visibility into which retriever is contributing the most relevant documents and helps diagnose regressions. If BM25 consistently contributes nothing to the final top-10, your fusion weights may need adjustment — or your BM25 index may need rebuilding.
6. Consider Elasticsearch or Vespa for Unified Hybrid Search
If you're operating at scale, consider platforms that natively support hybrid search within a single engine. Elasticsearch 8.x+ offers built-in vector search alongside BM25 with native RRF fusion. Vespa provides similar capabilities with its rank-profile configuration. This avoids the operational complexity of maintaining separate BM25 and vector indexes.
# Example Elasticsearch hybrid search query (conceptual)
# POST /your-index/_search
{
"query": {
"hybrid": {
"queries": [
{"match": {"text": "Project Nightingale revenue Q4"}},
{"text_expansion": {"vector_field": "embedding", "model_id": ".your-model"}}
],
"rrf": {"window_size": 50, "k": 60}
}
},
"size": 10
}
7. Evaluate with Both Recall-Oriented and Precision-Oriented Metrics
Hybrid search typically improves recall@K, but you should also measure MRR (Mean Reciprocal Rank) and NDCG to ensure the fused ranking places the most relevant document at the top. A hybrid system might improve recall but degrade MRR if fusion pulls up marginally relevant documents above highly relevant ones. Run both metric suites on your evaluation set before deploying.
When to Use Each Fusion Strategy
- RRF — Default choice. Works out of the box, no score calibration needed, handles any number of retrieval sources. Use when you want reliability without tuning.
- Weighted Sum with Normalization — Use when you have strong domain intuition about the relative importance of keyword vs. semantic search and you've calibrated normalization on a representative query set.
- Cross-Encoder Re-Ranking — Use when retrieval quality is paramount and the additional 100–300ms latency is acceptable. Ideal for question-answering over complex knowledge bases where the LLM's context window is precious and every document counts.
Conclusion
Hybrid search is not a luxury — it's quickly becoming table stakes for production RAG systems. By combining the precision of keyword retrieval with the semantic breadth of vector search, you dramatically reduce the chance that critical information slips through the cracks. Whether you implement it from scratch with the code patterns above, or leverage platforms like Elasticsearch, LangChain, or LlamaIndex, the core principles remain the same: retrieve broadly from both methods, fuse thoughtfully, and evaluate rigorously. The result is a RAG pipeline that your LLM — and your users — can genuinely rely on.