Designing a Search Engine with Microservices Architecture

Understanding Microservices Architecture for Search Engines

A search engine built on microservices breaks the traditional monolithic search stack into independently deployable, single-purpose services. Each service owns a well-defined domain—crawling, document processing, indexing, query serving, ranking, or analytics—and communicates over lightweight protocols such as REST, gRPC, or asynchronous messaging. This approach turns the search platform into a resilient, scalable ecosystem where teams can develop, test, and release components in parallel without disrupting the entire system.

Why Microservices for Search Engines Matter

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Search workloads are inherently uneven. Indexing might spike during a data import, while query traffic peaks during business hours. A monolithic search engine forces you to scale the entire application even if only one function is under pressure. Microservices solve this by allowing independent scaling of each service. They also improve fault isolation: a bug in the crawler won’t take down the query API. Moreover, different services can use the best-fit technology—Python for NLP preprocessing, Go for high-performance crawling, and Elasticsearch for full-text search—creating a polyglot architecture that optimizes every layer.

Core Microservices in a Search Engine

A production search engine typically decomposes into the following services:

Web Crawler Service – Fetches pages, respects robots.txt, extracts raw content and links, and pushes them into the ingestion pipeline.
Document Processing Service – Performs text extraction, language detection, tokenization, and entity recognition before indexing.
Indexing Service – Consumes processed documents, builds inverted indexes (or forwards them to Elasticsearch/Solr), and manages shard allocation.
Query Service – Receives user queries, parses them, retrieves candidate documents, applies scoring/ranking, and returns results.
Ranking & Scoring Service – Computes relevance scores using BM25, learning-to-rank models, or custom business logic. Often embedded within the query service or called as a sidecar.
Analytics & Logging Service – Collects query logs, click-through data, and system metrics for monitoring and relevance tuning.

Designing the Architecture

Communication patterns dictate the system’s behavior. For synchronous request-response paths like user queries, a REST or gRPC API works well behind an API gateway. For asynchronous ingestion, use a durable message broker (Kafka, RabbitMQ, or Redis Streams) to decouple crawlers from processors and indexers. This allows back-pressure handling and ensures no data is lost during spikes.

Example High-Level Design

Below is a typical layout expressed as a Docker Compose stack, illustrating service boundaries and communication channels.

# docker-compose.yml
version: '3.8'
services:
  elasticsearch:
    image: elasticsearch:8.10.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  crawler:
    build: ./crawler
    environment:
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis

  indexer:
    build: ./indexer
    environment:
      - REDIS_URL=redis://redis:6379
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    depends_on:
      - redis
      - elasticsearch

  query-api:
    build: ./query-api
    ports:
      - "8080:8080"
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  gateway:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - query-api

The crawler publishes raw content to a Redis list. The indexer pops tasks, processes them, and indexes into Elasticsearch. The query API searches Elasticsearch and returns JSON responses. An NGINX gateway routes external traffic to the query service, handling authentication and rate limiting.

Practical Implementation Example

Let’s build a minimal search engine for articles. We’ll implement three services in Python: a crawler that simulates fetching URLs, an indexer that stores documents in Elasticsearch, and a query API that serves search requests.

Web Crawler Service

This service simulates crawling by iterating over a list of URLs, fetching content (using a mock or real HTTP call), and pushing a JSON document onto a Redis queue.

# crawler/crawler.py
import time
import redis
import requests
from bs4 import BeautifulSoup
import os

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
QUEUE_KEY = "crawl_queue"

def fetch_page(url):
    # Simulate or actually fetch; here we use a mock for demonstration
    try:
        resp = requests.get(url, timeout=5)
        soup = BeautifulSoup(resp.text, 'html.parser')
        text = soup.get_text(separator=' ', strip=True)
        return {"url": url, "content": text}
    except Exception as e:
        return {"url": url, "error": str(e)}

def main():
    r = redis.Redis.from_url(REDIS_URL)
    urls = [
        "https://example.com/article/1",
        "https://example.com/article/2",
        "https://example.com/article/3"
    ]
    for url in urls:
        doc = fetch_page(url)
        r.lpush(QUEUE_KEY, str(doc))
        print(f"Pushed {url}")
        time.sleep(2)  # politeness delay

if __name__ == "__main__":
    main()

Document Processing & Indexing Service

The indexer continuously pulls messages from the Redis queue, performs basic text cleanup (lowercasing, stripping punctuation), and indexes the document into Elasticsearch.

# indexer/indexer.py
import time
import json
import redis
import elasticsearch
import os
import re

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
ES_URL = os.getenv("ELASTICSEARCH_URL", "http://localhost:9200")
QUEUE_KEY = "crawl_queue"
INDEX_NAME = "articles"

def clean_text(text):
    text = text.lower()
    text = re.sub(r'[^a-z0-9\s]', '', text)
    return text

def process_doc(raw_str):
    doc = json.loads(raw_str)
    if "error" in doc:
        return None
    doc["content"] = clean_text(doc["content"])
    return doc

def main():
    r = redis.Redis.from_url(REDIS_URL)
    es = elasticsearch.Elasticsearch(ES_URL)
    
    # Create index if not exists
    if not es.indices.exists(index=INDEX_NAME):
        es.indices.create(index=INDEX_NAME, body={
            "mappings": {
                "properties": {
                    "url": {"type": "keyword"},
                    "content": {"type": "text"}
                }
            }
        })
    
    while True:
        # Blocking pop from left (FIFO)
        result = r.blpop(QUEUE_KEY, timeout=10)
        if result is None:
            print("No messages, waiting...")
            time.sleep(5)
            continue
        _, raw = result
        doc = process_doc(raw.decode())
        if doc:
            es.index(index=INDEX_NAME, body=doc)
            print(f"Indexed {doc['url']}")

if __name__ == "__main__":
    main()

Query Service

The query service exposes a REST endpoint that accepts a user query, searches Elasticsearch, and returns ranked results. We’ll use Flask and the official Elasticsearch client.

# query-api/app.py
from flask import Flask, request, jsonify
import elasticsearch
import os

app = Flask(__name__)
ES_URL = os.getenv("ELASTICSEARCH_URL", "http://localhost:9200")
INDEX_NAME = "articles"

es = elasticsearch.Elasticsearch(ES_URL)

@app.route('/search', methods=['GET'])
def search():
    q = request.args.get('q', '')
    size = request.args.get('size', 10, type=int)
    if not q:
        return jsonify({"error": "Missing query parameter 'q'"}), 400
    
    body = {
        "query": {
            "match": {
                "content": q
            }
        },
        "size": size
    }
    res = es.search(index=INDEX_NAME, body=body)
    hits = []
    for hit in res['hits']['hits']:
        hits.append({
            "url": hit['_source']['url'],
            "score": hit['_score'],
            "snippet": hit['_source']['content'][:200]
        })
    return jsonify({"results": hits, "total": res['hits']['total']['value']})

@app.route('/health', methods=['GET'])
def health():
    return jsonify({"status": "ok"})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

API Gateway Configuration

A minimal NGINX configuration routes search traffic to the query service and adds caching headers.

# nginx.conf
events {}
http {
    server {
        listen 80;
        location /search {
            proxy_pass http://query-api:8080/search;
            proxy_set_header Host $host;
            add_header Cache-Control "no-store";
        }
        location /health {
            proxy_pass http://query-api:8080/health;
        }
    }
}

With these services, you can start the stack using docker-compose up. The crawler pushes mock articles, the indexer processes them, and the query API serves searches at /search?q=example. This pattern scales horizontally: run multiple indexer instances to handle heavy ingestion, or add more query API replicas behind a load balancer during traffic surges.

Best Practices for Search Engine Microservices

Idempotent Indexing – Design index operations so that reprocessing the same document yields the same index state. Use deterministic document IDs (e.g., URL hash) to avoid duplicates.
Back-Pressure Handling – Use bounded queues and monitor consumer lag. When the indexer falls behind, slow down the crawler or throttle via queue length alerts.
Circuit Breakers – Protect the query service from cascading failures if Elasticsearch becomes slow. Implement fallback responses or cached results.
Observability – Emit structured logs, expose Prometheus metrics (request latency, indexing rate, error counts), and use distributed tracing (Jaeger/Zipkin) across service boundaries.
Data Consistency – Embrace eventual consistency for search indexes. The query API may serve slightly stale data during reindexing, which is acceptable for most search use cases.
Versioned Index Schemas – When updating mappings, create a new index version (e.g., articles_v2) and reindex without downtime using aliases.
Graceful Degradation – If the ranking service fails, fall back to simple BM25 scoring. The search should still work, just with reduced relevance quality.

Conclusion

Designing a search engine with microservices transforms a rigid monolith into a flexible, scalable system that can evolve with your data and traffic patterns. By decomposing crawling, indexing, and query serving into independent services, you gain the ability to scale components precisely where needed, isolate failures, and iterate faster. The practical example above demonstrates a simple yet complete pipeline using Python, Redis, Elasticsearch, and Docker—ready to be extended with advanced NLP, machine-learned ranking, and robust observability. Start small, adopt asynchronous boundaries for ingestion, keep query paths synchronous and fast, and follow the best practices to build a search engine that grows gracefully with your users and content.