← Back to DevBytes
Developer Tutorial August 18, 2026 7 min read

BM25 vs Vector Search: When to Use Which in RAG

BM25 vs Vector Search: When to Use Which in RAG

Retrieval-Augmented Generation (RAG) has become the standard architecture for grounding large language models in private or domain-specific data. But the quality of a RAG system is only as good as its retrieval layer. If the retriever surfaces the wrong chunks, the generator cannot produce a correct answer — no matter how capable the LLM. Two retrieval methods dominate the landscape: BM25, a classical lexical algorithm, and vector search, a modern embedding-based approach. Understanding when to use each — and how to combine them — is one of the most impactful decisions you can make when building a RAG pipeline.

What Is BM25?

BM25 (Best Matching 25) is a probabilistic ranking function that extends the classic TF-IDF approach. It scores documents based on how often query terms appear in a document, while normalizing for document length and term frequency saturation. Unlike vector embeddings, BM25 operates purely on exact token matching. If the user searches for "GPT-4o", BM25 will strongly favor documents that contain that literal string.

The BM25 scoring formula balances three signals: term frequency (how often a term appears in a document), inverse document frequency (how rare the term is across the corpus), and document length normalization (to prevent longer documents from being unfairly favored). This makes BM25 remarkably robust for keyword-heavy queries, product names, error codes, identifiers, and any scenario where exact terminology matters.

What Is Vector Search?

Vector search works by embedding both documents and queries into a high-dimensional semantic space using a model such as OpenAI's text-embedding-3-small, Cohere's embed-v3, or open-source alternatives like BGE and E5. Retrieval is performed by computing the similarity — typically cosine similarity — between the query vector and document vectors. The closest vectors in semantic space are returned as results.

The strength of vector search lies in its ability to match on meaning rather than wording. A query like "how do I reset my password" can retrieve a document titled "credential recovery procedure" even though no words overlap. This semantic matching is invaluable for natural language questions, paraphrased queries, and conceptual lookups.

Why the Choice Matters in RAG

The retrieval method you choose directly determines which chunks reach the LLM context window. Each method has characteristic failure modes:

In real-world RAG systems, blindly choosing one method over the other leads to predictable blind spots. The most effective pipelines understand the tradeoffs and often combine both.

How to Use BM25 in a RAG Pipeline

BM25 is lightweight, requires no GPU, and can be implemented with libraries like rank_bm25 in Python or via full-text search engines like Elasticsearch, OpenSearch, and Postgres' built-in ts_vector capabilities. Below is a minimal implementation using rank_bm25.

from rank_bm25 import BM25Okapi

documents = [
    "The GPT-4o model supports 128k context window.",
    "Claude 3.5 Sonnet excels at coding tasks.",
    "To reset your password, click the recovery link.",
    "GPT-4o pricing is $5 per million input tokens.",
    "Credential recovery requires email verification."
]

# Tokenize documents (simple whitespace split for demo)
tokenized_docs = [doc.lower().split() for doc in documents]
bm25 = BM25Okapi(tokenized_docs)

query = "GPT-4o pricing"
tokenized_query = query.lower().split()
scores = bm25.get_scores(tokenized_query)

ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked:
    print(f"{score:.4f}  {doc}")

Notice how BM25 cleanly surfaces the two documents mentioning "GPT-4o" and "pricing". A vector model might have ranked the Claude document higher due to semantic similarity around "models" and "tokens". For queries involving specific product names, BM25 is hard to beat.

How to Use Vector Search in a RAG Pipeline

Vector search requires an embedding model and a vector store. For production systems, dedicated vector databases like Pinecone, Weaviate, Qdrant, or Milvus are common. For smaller projects, pgvector or FAISS work well. Here is a self-contained example using sentence-transformers and FAISS.

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

documents = [
    "The GPT-4o model supports 128k context window.",
    "Claude 3.5 Sonnet excels at coding tasks.",
    "To reset your password, click the recovery link.",
    "GPT-4o pricing is $5 per million input tokens.",
    "Credential recovery requires email verification."
]

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
embeddings = model.encode(documents, normalize_embeddings=True)

dimension = embeddings.shape[1]
index = faiss.IndexFlatIP(dimension)  # inner product = cosine on normalized vectors
index.add(embeddings.astype(np.float32))

query = "how do I get back into my account"
query_vec = model.encode([query], normalize_embeddings=True).astype(np.float32)

distances, indices = index.search(query_vec, k=3)
for rank, idx in enumerate(indices[0]):
    print(f"{distances[0][rank]:.4f}  {documents[idx]}")

Here the query "how do I get back into my account" has zero word overlap with "Credential recovery requires email verification", yet vector search retrieves it correctly because the embedding model captures the semantic intent. This is exactly the scenario where BM25 would fail.

Hybrid Search: The Best of Both Worlds

Modern RAG systems increasingly use hybrid retrieval, which combines BM25 and vector scores into a single ranked list. The most common fusion algorithm is Reciprocal Rank Fusion (RRF), which combines rankings rather than raw scores — making it robust to the different score distributions of each method.

def reciprocal_rank_fusion(bm25_ranked, vector_ranked, k=60):
    """
    bm25_ranked and vector_ranked are lists of document indices
    ordered from most to least relevant.
    """
    rrf_scores = {}
    for rank, doc_id in enumerate(bm25_ranked):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(vector_ranked):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(rrf_scores, key=rrf_scores.get, reverse=True)

# Example usage
bm25_ranked = [0, 3, 1, 2, 4]   # from BM25 search
vector_ranked = [2, 4, 0, 1, 3] # from vector search

fused = reciprocal_rank_fusion(bm25_ranked, vector_ranked)
print("Fused ranking:", fused)

RRF is simple, parameter-light, and works well in practice. Many vector databases now offer hybrid search out of the box — Weaviate, Qdrant, and Pinecone all support combining keyword and vector retrieval natively, often with configurable weighting.

When to Use Which: A Decision Guide

Use BM25 When

Use Vector Search When

Use Hybrid Search When

Best Practices for RAG Retrieval

Beyond choosing a retrieval method, several engineering practices significantly improve RAG retrieval quality:

Conclusion

BM25 and vector search are not competing technologies — they are complementary tools that excel in different retrieval scenarios. BM25 remains unmatched for exact keyword matching, identifiers, and transparent ranking, while vector search enables semantic understanding that bridges vocabulary gaps. For most production RAG systems, the right answer is hybrid retrieval combining both methods, ideally followed by a cross-encoder re-ranker and informed by rigorous evaluation. By understanding the strengths and failure modes of each approach, you can build retrieval pipelines that surface the right context consistently, giving your LLM the grounding it needs to produce accurate, trustworthy answers.

— Ad —

Google AdSense will appear here after approval

← Back to all articles