Maximizing Recall in Vector Search with Quantization

Vector search has become the backbone of modern retrieval systems, powering semantic search, recommendation engines, and retrieval-augmented generation (RAG) pipelines. As datasets grow into the billions of vectors, the memory and compute cost of storing and searching high-dimensional embeddings becomes a serious bottleneck. Quantization offers a way to compress vectors dramatically while maintaining acceptable — and often near-lossless — recall. However, naive quantization can destroy recall if applied carelessly. This tutorial walks through what quantization is, why it matters for recall, how to apply it correctly, and the best practices that separate production-grade systems from toy demos.

What Is Vector Quantization?

Quantization is the process of mapping high-precision floating-point vectors into a lower-precision representation. A typical embedding is stored as a 768- or 1536-dimensional array of 32-bit floats. Quantization reduces each value to fewer bits — 8-bit integers (scalar quantization), compact codes derived from learned codebooks (product quantization), or even binary representations (binary quantization). The trade-off is always the same: smaller memory footprint and faster distance computations in exchange for some loss of precision in the similarity scores.

>>> NEW: The Self-Hosted Agent Factory — Production Kit

Run a 3-agent production line (orchestrator + workers + adversarial QC) on a $6/mo VPS. 120-page book + the complete working Docker stack: task ledger, model router, 15-min watchdog, QC rubric.

Launch price $29 (first 100 copies, then $59) →

The three dominant flavors are:

Scalar Quantization (SQ): Each float is independently mapped to an 8-bit (or 4-bit/16-bit) integer by linearly scaling the value range. Simple, fast, and typically preserves 95–99% of recall.
Product Quantization (PQ): The vector is split into sub-vectors, each quantized independently against a learned codebook (usually via k-means). Much higher compression (often 32x–64x) but more recall loss.
Binary Quantization (BQ): Each dimension is reduced to a single bit based on its sign. Extreme compression (32x for float32) and enables ultra-fast Hamming distance computation, but requires careful handling to preserve recall.

Why Quantization Matters for Recall

At first glance, quantization and recall seem to be in tension — you are literally throwing away information. The key insight is that recall is measured relative to an exact brute-force search over the full-precision vectors. If quantization introduces too much noise, the approximate nearest neighbor (ANN) results diverge from the ground truth, and recall@k drops. But quantization also enables you to hold far more vectors in RAM, use faster distance kernels, and scan larger candidate sets in the same time budget — all of which can increase effective recall when tuned correctly.

The challenge is that quantization error is not uniform. It is larger for vectors that are close to the decision boundary between two quantization cells, and it accumulates across dimensions. Product quantization in particular suffers from the "curse of dimensionality" because each sub-vector carries less information as the dimension count grows. The art of maximizing recall is therefore about choosing the right quantization method, training it well, and pairing it with a re-scoring step that recovers lost precision.

How to Use Quantization Effectively

The most reliable pattern in production systems is a two-stage retrieval pipeline: use quantized vectors for fast candidate generation, then re-rank the top candidates using full-precision vectors. This combines the speed and memory savings of quantization with the accuracy of full-precision scoring. Below we walk through a concrete implementation using FAISS, the most widely used open-source vector search library.

Setting Up the Environment

pip install faiss-cpu numpy datasets

We will use a synthetic dataset for reproducibility, but the same code works with real embeddings from OpenAI, Cohere, or open-source models like all-MiniLM-L6-v2.

Building a Baseline with Full-Precision Search

Before applying quantization, establish a recall baseline. This gives you a ground truth to measure against.

import numpy as np
import faiss

np.random.seed(42)

dim = 768
n_docs = 200_000
n_queries = 1_000
k = 10

# Simulate normalized embeddings (as produced by most modern encoders)
xb = np.random.randn(n_docs, dim).astype(np.float32)
faiss.normalize_L2(xb)
xq = np.random.randn(n_queries, dim).astype(np.float32)
faiss.normalize_L2(xq)

# Exact brute-force baseline
index_exact = faiss.IndexFlatIP(dim)
index_exact.add(xb)

_, gt = index_exact.search(xq, k)
print("Ground truth computed for", n_queries, "queries")

The gt array now holds the true top-10 neighbors for each query. Every quantized index we build will be evaluated against this ground truth.

Applying Scalar Quantization

Scalar quantization to 8-bit is the safest first step. It typically reduces memory by 4x with negligible recall loss.

# Build an IVF index with scalar quantization (8-bit)
nlist = 4096  # number of clusters
quantizer = faiss.IndexFlatIP(dim)

index_sq = faiss.IndexIVFScalarQuantizer(
    quantizer, dim, nlist,
    faiss.ScalarQuantizer.QT_8bit,
    faiss.METRIC_INNER_PRODUCT
)

# Train on a representative sample
train_size = min(50_000, n_docs)
index_sq.train(xb[:train_size])
index_sq.add(xb)
index_sq.nprobe = 32  # number of clusters to probe at query time

_, sq_results = index_sq.search(xq, k)

# Compute recall@10
def recall_at_k(gt, results, k):
    total = 0
    for i in range(gt.shape[0]):
        gt_set = set(gt[i][:k])
        res_set = set(results[i][:k])
        total += len(gt_set & res_set) / k
    return total / gt.shape[0]

print(f"Scalar Quantization recall@{k}: {recall_at_k(gt, sq_results, k):.4f}")

You should see recall above 0.95 in most cases. If it is lower, increase nprobe or the number of training clusters.

Applying Product Quantization

Product quantization achieves much higher compression. The key parameters are m (number of sub-quantizers) and nbits (bits per sub-quantizer). A common configuration for 768-dimensional vectors is m=64 and nbits=8, which compresses each vector from 3072 bytes to 64 bytes — a 48x reduction.

m = 64       # number of sub-vectors
nbits = 8    # bits per code

quantizer_pq = faiss.IndexFlatIP(dim)
index_pq = faiss.IndexIVFPQ(quantizer_pq, dim, nlist, m, nbits)
index_pq.train(xb[:train_size])
index_pq.add(xb)
index_pq.nprobe = 64

_, pq_results = index_pq.search(xq, k)
print(f"PQ recall@{k}: {recall_at_k(gt, pq_results, k):.4f}")

PQ recall will typically be lower than scalar quantization — often in the 0.80–0.92 range depending on the data. This is where re-scoring becomes essential.

Re-Scoring to Recover Recall

FAISS supports a built-in re-ranking mechanism through the RFlat or Refine wrapper. The idea is simple: retrieve a larger candidate set using the fast quantized index, then re-score those candidates with full-precision vectors.

# Retrieve more candidates with PQ, then re-rank with full precision
nprobe = 64
k_reorder = 100  # over-fetch before re-ranking

index_pq.nprobe = nprobe
_, pq_candidates = index_pq.search(xq, k_reorder)

# Re-rank candidates using the exact index
reranked = np.zeros((n_queries, k), dtype=np.int64)
for i in range(n_queries):
    candidate_ids = pq_candidates[i]
    candidate_vecs = xb[candidate_ids]
    scores = candidate_vecs @ xq[i]
    top_local = np.argsort(-scores)[:k]
    reranked[i] = candidate_ids[top_local]

print(f"PQ + Re-rank recall@{k}: {recall_at_k(gt, reranked, k):.4f}")

With re-ranking, PQ-based retrieval often recovers 98%+ of the recall of exact search while still benefiting from the massive memory savings during the candidate generation phase. The full-precision vectors only need to be loaded for a small number of candidates, which can be done from disk or a memory-mapped file if RAM is constrained.

Binary Quantization with Re-Scoring

Binary quantization is the most aggressive option. Each dimension becomes a single bit, reducing a 768-dimensional float32 vector from 3072 bytes to 96 bytes. Hamming distance on binary codes is extremely fast because it uses bitwise XOR and popcount instructions. The trick to preserving recall is to over-fetch aggressively and re-rank.

# Binary quantization using IndexBinaryIVF
index_bq = faiss.IndexBinaryIVF(faiss.IndexBinaryFlat(dim), nlist)
index_bq.train(xb[:train_size].astype(np.float32))
index_bq.add(xb.astype(np.float32))
index_bq.nprobe = 128

k_fetch = 200
_, bq_candidates = index_bq.search(xq.astype(np.float32), k_fetch)

# Re-rank with full precision
reranked_bq = np.zeros((n_queries, k), dtype=np.int64)
for i in range(n_queries):
    candidate_ids = bq_candidates[i]
    candidate_vecs = xb[candidate_ids]
    scores = candidate_vecs @ xq[i]
    top_local = np.argsort(-scores)[:k]
    reranked_bq[i] = candidate_ids[top_local]

print(f"BQ + Re-rank recall@{k}: {recall_at_k(gt, reranked_bq, k):.4f}")

Binary quantization with aggressive over-fetching and re-ranking is the approach used by large-scale systems at companies like Spotify and Pinterest, where the primary index must fit in a constrained memory budget but recall must remain high.

Best Practices for Maximizing Recall

Always normalize embeddings before quantization. Inner product on normalized vectors is equivalent to cosine similarity, and normalization stabilizes the dynamic range that scalar and binary quantization depend on.
Train quantizers on representative data. PQ codebooks learned from a small or biased sample will produce poor quantization cells. Use at least 50,000–100,000 diverse vectors for training.
Tune nprobe and over-fetch together. Recall is a function of both how many clusters you probe and how many candidates you re-rank. Profile both parameters and find the knee in the latency-recall curve.
Use re-ranking whenever possible. The single most effective technique for recovering recall lost to quantization is to re-score the top candidates with full-precision vectors. The cost is minimal compared to the recall gain.
Choose the quantization method based on your constraint. If memory is the bottleneck, use PQ or BQ with re-ranking. If latency is the bottleneck and memory is acceptable, scalar quantization gives the best recall-to-speed ratio.
Monitor recall in production. Embedding distributions drift over time as models are updated. Periodically retrain quantizers and re-measure recall against a held-out ground truth set.
Consider OPQ for difficult distributions. Optimized Product Quantization (OPQ) applies a rotation to the vectors before quantization, which can significantly improve recall when the embedding dimensions are highly correlated. FAISS supports it via faiss.IndexIVFPQ with the do_opq flag or by wrapping with faiss.IndexPreTransform.
Shard large indexes. For billion-scale datasets, combine quantization with sharding across multiple machines. Each shard uses quantized search with re-ranking, and results are merged at the gateway layer.

Putting It All Together

The following snippet shows a complete, production-oriented pipeline that combines IVF, product quantization, OPQ rotation, and re-ranking into a single index.

import faiss

dim = 768
nlist = 4096
m = 64
nbits = 8

# OPQ rotation + IVF + PQ
opq_matrix = faiss.OPQMatrix(dim, m)
quantizer = faiss.IndexFlatIP(dim)
ivf_pq = faiss.IndexIVFPQ(quantizer, dim, nlist, m, nbits)

# Wrap with OPQ as a pre-transform
index = faiss.IndexPreTransform(opq_matrix, ivf_pq)

# Train
index.train(xb[:100_000])
index.add(xb)

# Search with re-ranking built in using IndexRefineFlat
refine_index = faiss.IndexRefineFlat(index)
refine_index.k_factor = 10  # over-fetch factor for re-ranking

refine_index.search(xq, k)

The IndexRefineFlat wrapper automatically retrieves k * k_factor candidates from the underlying quantized index and re-ranks them using stored full-precision vectors, all in a single call. This is the recommended approach for most production deployments.

Conclusion

Quantization is not a recall-destroying compromise — it is a precision engineering problem with well-understood solutions. By choosing the right quantization method for your memory and latency constraints, training on representative data, tuning probe and over-fetch parameters, and consistently applying full-precision re-ranking, you can achieve recall within 1–2% of exact brute-force search while reducing memory usage by an order of magnitude or more. The two-stage pattern of fast quantized retrieval followed by precise re-scoring is the foundation of virtually every large-scale vector search system in production today, and with the techniques covered in this tutorial, you have everything you need to build one yourself.