Distributed Vector Search with Vespa: Setup Guide

Distributed Vector Search with Vespa: Setup Guide

Vector search has become a cornerstone of modern AI applications, powering semantic retrieval, recommendation systems, and retrieval-augmented generation (RAG) pipelines. While many solutions handle vector search on a single node, production workloads often demand horizontal scalability across multiple machines. Vespa, an open-source platform originally developed at Yahoo, is purpose-built for distributed search and ranking at scale. This tutorial walks you through setting up a distributed vector search system with Vespa, from installation to querying.

What Is Vespa?

Vespa is an engine for storing, searching, and ranking large datasets at high throughput and low latency. Unlike standalone vector databases, Vespa combines traditional text search, vector search, and machine-learned ranking in a single unified system. It supports approximate nearest neighbor (ANN) search using HNSW (Hierarchical Navigable Small World) graphs, and it scales horizontally by distributing data and computation across a cluster of nodes.

A Vespa deployment typically consists of several containerized services: the configuration server, which manages schema and application packages; the container cluster, which handles query processing and feeding; and the content cluster, which stores data and executes searches. Each of these can be scaled independently.

Why Distributed Vector Search Matters

• Scale beyond a single machine: Datasets with hundreds of millions or billions of vectors cannot fit in the memory of one node. Vespa shards data across content nodes automatically.
• Throughput for production traffic: Distributing queries across nodes reduces per-node load and enables higher queries-per-second.
• Fault tolerance: With replication, Vespa continues serving results even when individual nodes fail.
• Hybrid retrieval: Vespa lets you combine vector similarity with filters, text matching, and custom ranking functions in a single query.
• Cost efficiency: You can scale compute and storage independently, matching infrastructure to workload characteristics.

Prerequisites

Before starting, ensure you have the following installed on your machine:

• Docker (version 20.10 or later)
• Docker Compose (version 2.0 or later)
• Python 3.9 or later
• The Vespa CLI, which you can install via Homebrew on macOS (brew install vespa-cli) or by downloading the binary from the Vespa releases page
• At least 8 GB of free RAM allocated to Docker

Step 1: Create the Application Package

Vespa applications are defined by an application package, a directory structure containing schemas, services configuration, and ranking expressions. Start by creating the project directory.

mkdir vespa-vector-search
cd vespa-vector-search
mkdir -p app/schemas app/services

The services.xml file defines the topology of your Vespa cluster. For a distributed setup, we configure a container cluster for query handling and a content cluster with multiple nodes for storage and search.

<?xml version="1.0" encoding="utf-8" ?>
<services version="1.0">

  <container id="default" version="1.0">
    <search/>
    <document-api/>
    <nodes>
      <node hostalias="node1"/>
    </nodes>
  </container>

  <content id="vector_content" version="1.0">
    <redundancy>2</redundancy>
    <documents>
      <document type="doc" mode="index"/>
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0"/>
      <node hostalias="node2" distribution-key="1"/>
      <node hostalias="node3" distribution-key="2"/>
    </nodes>
    <engine>
      <proton>
        <searchable-copies>1</searchable-copies>
      </proton>
    </engine>
  </content>

</services>

The redundancy value of 2 means each document is stored on two nodes, providing fault tolerance. The three content nodes will shard the data across the cluster.

Step 2: Define the Schema

The schema defines the document structure, indexing options, and ranking profiles. Create app/schemas/doc.sd.

schema doc {

  document doc {

    field id type string {
      indexing: attribute | summary
    }

    field title type string {
      indexing: index | summary
      index: enable-bm25
    }

    field text type string {
      indexing: index | summary
      index: enable-bm25
    }

    field embedding type tensor<float>(x[384]) {
      indexing: attribute | index
      attribute {
        distance-metric: angular
      }
      index {
        hnsw {
          max-links-per-node: 16
          neighbors-to-explore-at-insert: 50
        }
      }
    }
  }

  rank-profile default {
    inputs {
      query(query_embedding) tensor<float>(x[384])
    }
    first-phase {
      expression: closeness(field, embedding)
    }
  }

  rank-profile hybrid inherits default {
    first-phase {
      expression: 0.7 * closeness(field, embedding) + 0.3 * bm25(text)
    }
  }
}

This schema defines a 384-dimensional embedding field, which matches the output size of common sentence embedding models like all-MiniLM-L6-v2. The HNSW parameters control the trade-off between search accuracy and indexing speed. The default rank profile uses pure vector similarity, while hybrid combines vector closeness with BM25 text scoring.

Step 3: Configure Docker Compose for a Multi-Node Cluster

To simulate a distributed deployment locally, use Docker Compose with multiple Vespa containers. Create docker-compose.yml in the project root.

version: "3.9"

services:

  vespa-configserver:
    image: vespaengine/vespa:latest
    hostname: vespa-configserver
    container_name: vespa-configserver
    ports:
      - "19071:19071"
    volumes:
      - vespa-config:/opt/vespa/var
    command: configserver

  node1:
    image: vespaengine/vespa:latest
    hostname: node1
    container_name: node1
    depends_on:
      - vespa-configserver
    volumes:
      - vespa-data1:/opt/vespa/var
    command: >
      sh -c "vespa-start-configserver-proxy --configserver vespa-configserver &
             vespa-start-container &
             vespa-start-content"

  node2:
    image: vespaengine/vespa:latest
    hostname: node2
    container_name: node2
    depends_on:
      - vespa-configserver
    volumes:
      - vespa-data2:/opt/vespa/var
    command: >
      sh -c "vespa-start-configserver-proxy --configserver vespa-configserver &
             vespa-start-container &
             vespa-start-content"

  node3:
    image: vespaengine/vespa:latest
    hostname: node3
    container_name: node3
    depends_on:
      - vespa-configserver
    volumes:
      - vespa-data3:/opt/vespa/var
    command: >
      sh -c "vespa-start-configserver-proxy --configserver vespa-configserver &
             vespa-start-container &
             vespa-start-content"

volumes:
  vespa-config:
  vespa-data1:
  vespa-data2:
  vespa-data3:

Start the cluster with the following command:

docker compose up -d

Wait for the services to become healthy. You can check the status using the Vespa CLI:

vespa status --target http://localhost:19071

Step 4: Deploy the Application Package

With the cluster running, deploy your application package using the Vespa CLI. From the project root, run:

vespa deploy --target http://localhost:19071 app

If the deployment succeeds, you will see a confirmation message indicating the application is now active across all nodes. The configuration server propagates the schema and service definitions to each node in the cluster.

Step 5: Generate Embeddings and Feed Documents

To populate the index, you need to generate embeddings for your documents and feed them to Vespa. The following Python script uses the sentence-transformers library to produce embeddings and the vespa Python SDK to feed documents.

pip install sentence-transformers pyvespa

Create a file named feed_documents.py:

from sentence_transformers import SentenceTransformer
from vespa.application import Vespa

# Load a lightweight sentence embedding model
model = SentenceTransformer("all-MiniLM-L6-v2")

# Connect to the Vespa container endpoint
app = Vespa(url="http://localhost:8080")

# Sample documents
documents = [
    {"id": "1", "title": "Vector Databases", "text": "Vector databases store and query high-dimensional embeddings for semantic search."},
    {"id": "2", "title": "Approximate Nearest Neighbors", "text": "ANN algorithms trade exactness for speed using graph-based indexes like HNSW."},
    {"id": "3", "title": "Distributed Systems", "text": "Distributed systems partition data across nodes to achieve horizontal scalability."},
    {"id": "4", "title": "Information Retrieval", "text": "Modern IR combines lexical matching with dense vector similarity for better results."},
    {"id": "5", "title": "Scalability Patterns", "text": "Sharding and replication are fundamental patterns for scaling data-intensive applications."},
]

# Generate embeddings and feed
for doc in documents:
    embedding = model.encode(doc["text"]).tolist()
    vespa_doc = {
        "id": doc["id"],
        "fields": {
            "id": doc["id"],
            "title": doc["title"],
            "text": doc["text"],
            "embedding": embedding,
        },
    }
    response = app.feed_data_point(
        schema="doc",
        data_id=doc["id"],
        fields=vespa_doc["fields"],
    )
    print(f"Fed document {doc['id']}: status {response.status_code}")

Run the script:

python feed_documents.py

Each document is distributed across the content nodes according to the redundancy setting. Vespa automatically builds the HNSW index as documents arrive.

Step 6: Query the Vector Index

Now you can issue vector search queries. The following script encodes a natural language query and sends it to Vespa, requesting the nearest neighbors.

from sentence_transformers import SentenceTransformer
from vespa.application import Vespa

model = SentenceTransformer("all-MiniLM-L6-v2")
app = Vespa(url="http://localhost:8080")

query_text = "How do you scale search across multiple machines?"
query_embedding = model.encode(query_text).tolist()

response = app.query(
    body={
        "yql": 'select * from doc where ({targetHits: 3})nearestNeighbor(embedding, query_embedding)',
        "input.query(query_embedding)": query_embedding,
        "ranking.profile": "default",
        "presentation.timing": True,
    }
)

hits = response.hits
for hit in hits:
    fields = hit["fields"]
    score = hit["relevance"]
    print(f"Score: {score:.4f} | Title: {fields['title']}")

The nearestNeighbor operator triggers the HNSW search on each content node. Vespa fans out the query across all shards, merges the results, and returns the top hits. The targetHits parameter controls how many candidates each node returns before merging.

Step 7: Hybrid Search with Filtering

One of Vespa's strengths is combining vector search with structured filters and text ranking. The following query restricts the vector search to documents whose title contains the word "distributed" and applies the hybrid ranking profile.

response = app.query(
    body={
        "yql": 'select * from doc where title contains "distributed" and ({targetHits: 10})nearestNeighbor(embedding, query_embedding)',
        "input.query(query_embedding)": query_embedding,
        "ranking.profile": "hybrid",
    }
)

Vespa pushes the filter down into each shard before executing the vector search, which avoids scanning irrelevant partitions and keeps latency low even at scale.

Best Practices

• Tune HNSW parameters carefully. Increasing max-links-per-node improves recall but increases memory usage and indexing time. Start with the defaults and benchmark against your dataset.
• Choose the right distance metric. Use angular for normalized embeddings (cosine similarity), euclidean for L2 distance, and innerproduct for maximum inner product search. The metric must match how your model was trained.
• Use phased ranking. Use vector similarity in first-phase for fast candidate selection, then apply a more expensive model in second-phase on a smaller candidate set.
• Monitor shard balance. Uneven data distribution can create hotspots. Vespa's distribution-key mechanism helps balance load, but you should monitor per-node metrics in production.
• Batch your feed operations. Feeding documents one at a time is inefficient. Use the Vespa feeding API with batches of 100 to 1000 documents for better throughput.
• Set appropriate redundancy. A redundancy of 2 provides basic fault tolerance. For higher availability, use 3 and ensure nodes are on different physical hosts or availability zones.
• Warm up the index. HNSW performance improves after the graph is fully built. Run warm-up queries before routing production traffic to a freshly fed index.
• Use tensor fields for multi-vector documents. If a document has multiple embeddings (for example, chunk-level embeddings), use a tensor of shape tensor<float>(chunk{}, x[384]) and the nearestNeighbor operator with a mixed tensor query.

Scaling to Production

For a real production deployment, you would move beyond Docker Compose to a container orchestration platform. Vespa provides a Helm chart for Kubernetes, which handles node provisioning, rolling upgrades, and automatic recovery. You should also configure persistent volumes for each content node, set up monitoring with Prometheus and Grafana using Vespa's built-in metrics endpoints, and tune the JVM heap sizes based on your dataset size and query load.

When growing the cluster, you can add content nodes without downtime. Vespa redistributes data automatically to balance the new topology. Plan capacity by estimating the number of vectors, the dimensionality, and the expected queries per second, then size your nodes with enough headroom for HNSW graph overhead, which typically adds 20 to 40 percent on top of the raw vector data.

Conclusion

Vespa provides a robust foundation for distributed vector search, combining HNSW-based approximate nearest neighbor search with horizontal scaling, replication, and hybrid ranking. By defining a schema with an embedding field, configuring a multi-node content cluster, and using the Python SDK for feeding and querying, you can build a production-grade semantic search system that scales with your data. The key to success lies in tuning index parameters, choosing the right distance metric, and leveraging Vespa's phased ranking to balance latency and result quality. With this setup as a starting point, you can extend the system with custom rank models, multi-vector documents, and real-time updates to meet the demands of modern AI applications.