Introduction to Semantic Routing for Multi-Agent Systems
Modern AI applications rarely rely on a single model or agent. Instead, they orchestrate multiple specialized agents—each with distinct capabilities, tools, and knowledge domains. The challenge becomes: how do you decide which agent should handle a given request? This is where semantic routing comes in.
Semantic routing is the process of dispatching user queries or tasks to the most appropriate agent in a multi-agent system based on the meaning of the input rather than rigid keyword matching or hard-coded rules. By leveraging embeddings and similarity search, semantic routers can generalize across phrasings, synonyms, and paraphrases, making multi-agent systems more robust and maintainable.
What Is Semantic Routing?
A semantic router is essentially an intent classifier powered by vector similarity. Instead of writing brittle if/else branches or training a dedicated classifier for every new intent, you define a set of routes, each associated with example utterances. At runtime, the incoming query is embedded and compared against the route embeddings. The closest match wins, and the request is forwarded to the corresponding agent.
Conceptually, the pipeline looks like this:
- Route definitions — each route has a name, a set of example phrases, and a target agent.
- Embedding step — both the examples and the incoming query are converted into dense vectors.
- Similarity scoring — cosine similarity (or another metric) ranks the routes.
- Dispatch — the top-scoring route's agent receives the request.
This approach scales gracefully: adding a new agent only requires adding a new route with a handful of examples, no retraining required.
Why Semantic Routing Matters
Traditional routing strategies—rule-based keyword matching, regex patterns, or LLM-as-router prompts—each have significant drawbacks in production multi-agent systems.
Limitations of Rule-Based Routing
Keyword matching fails when users phrase requests unexpectedly. A route triggered by the word "refund" misses "I want my money back" or "cancel my purchase and credit me." Maintaining exhaustive synonym lists becomes a maintenance nightmare.
Limitations of LLM-as-Router
Using a large language model to decide routing is flexible but expensive and slow. Every request incurs a token cost and latency penalty before any actual work begins. For high-throughput systems, this overhead is unacceptable.
Advantages of Semantic Routing
- Speed — embedding similarity is a sub-millisecond operation with modern vector indexes.
- Cost efficiency — no LLM call is needed for routing decisions.
- Flexibility — handles paraphrases, synonyms, and novel phrasings naturally.
- Extensibility — new routes are added by supplying examples, not by retraining.
- Transparency — similarity scores provide a confidence signal you can threshold.
Building a Semantic Router from Scratch
Let's build a minimal but functional semantic router in Python. We'll use sentence-transformers for embeddings and numpy for similarity computation. The design will be agent-agnostic so you can plug in any callable.
Defining the Router
import numpy as np
from sentence_transformers import SentenceTransformer
class SemanticRouter:
def __init__(self, embedding_model_name="all-MiniLM-L6-v2"):
self.encoder = SentenceTransformer(embedding_model_name)
self.routes = [] # list of dicts: {name, examples, agent}
self.route_vectors = None # np.ndarray of shape (n_routes, n_examples, dim)
def add_route(self, name, examples, agent):
"""Register a new route with example utterances and a target agent."""
self.routes.append({
"name": name,
"examples": examples,
"agent": agent,
})
def _build_index(self):
"""Embed all examples and store them for similarity search."""
all_examples = []
route_indices = [] # maps each example index to a route index
for route_idx, route in enumerate(self.routes):
for example in route["examples"]:
all_examples.append(example)
route_indices.append(route_idx)
embeddings = self.encoder.encode(
all_examples, normalize_embeddings=True, convert_to_numpy=True
)
self.embeddings = embeddings
self.route_indices = np.array(route_indices)
def route(self, query, threshold=0.35):
"""
Return (route_name, agent, score) for the best matching route.
Returns (None, None, score) if below threshold.
"""
if self.route_vectors is None and not hasattr(self, "embeddings"):
self._build_index()
query_vec = self.encoder.encode(
[query], normalize_embeddings=True, convert_to_numpy=True
)[0]
# Cosine similarity (vectors are normalized)
scores = self.embeddings @ query_vec
best_idx = int(np.argmax(scores))
best_score = float(scores[best_idx])
best_route_idx = self.route_indices[best_idx]
if best_score < threshold:
return None, None, best_score
route = self.routes[best_route_idx]
return route["name"], route["agent"], best_score
Defining Agents and Routes
Now let's create a few simple agents and register routes for them. Each agent is just a callable that takes a user query string and returns a response string.
# --- Define simple agents ---
def billing_agent(query):
return f"[Billing Agent] Handling: {query}"
def support_agent(query):
return f"[Support Agent] Handling: {query}"
def sales_agent(query):
return f"[Sales Agent] Handling: {query}"
def general_agent(query):
return f"[General Agent] Handling: {query}"
# --- Configure the router ---
router = SemanticRouter()
router.add_route(
name="billing",
examples=[
"I want a refund for my order",
"How do I update my payment method?",
"My invoice looks incorrect",
"Charge on my card that I don't recognize",
"Can I get a receipt for last month?",
"I need to dispute a transaction",
],
agent=billing_agent,
)
router.add_route(
name="support",
examples=[
"The app keeps crashing when I upload a file",
"I can't log into my account",
"How do I reset my password?",
"Something is broken on the dashboard",
"I'm getting an error code 500",
"The page won't load properly",
],
agent=support_agent,
)
router.add_route(
name="sales",
examples=[
"I'd like a demo of your enterprise plan",
"What are your pricing tiers?",
"Can I talk to someone about a custom contract?",
"Do you offer discounts for startups?",
"I want to upgrade my subscription",
"Tell me about your team features",
],
agent=sales_agent,
)
router.add_route(
name="general",
examples=[
"What does your company do?",
"Where are you located?",
"How can I contact you?",
"Tell me about your privacy policy",
],
agent=general_agent,
)
# Build the embedding index
router._build_index()
Testing the Router
test_queries = [
"I need my money back for order 4821",
"The login page is completely broken",
"We're a 200-person company, can we get a custom quote?",
"What's your refund policy?",
"I see a weird charge on my statement",
]
for q in test_queries:
name, agent, score = router.route(q)
if agent:
print(f"Query: {q}")
print(f" -> Route: {name} (score={score:.3f})")
print(f" -> Response: {agent(q)}")
else:
print(f"Query: {q}")
print(f" -> No route matched (best score={score:.3f})")
print()
Expected output will show that paraphrased queries like "I need my money back" correctly route to the billing agent, and "The login page is completely broken" routes to support—even though neither query shares exact words with the example utterances.
Integrating with a Multi-Agent Orchestrator
A semantic router is most powerful when embedded inside an orchestration layer that handles fallbacks, logging, and agent handoffs. Below is a minimal orchestrator that wraps the router with these concerns.
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("orchestrator")
class MultiAgentOrchestrator:
def __init__(self, router, fallback_agent=None):
self.router = router
self.fallback_agent = fallback_agent or general_agent
self.conversation_log = []
def handle(self, user_query, user_id="anonymous"):
logger.info(f"[{user_id}] Incoming: {user_query}")
route_name, agent, score = self.router.route(user_query)
if agent is None:
logger.warning(
f"[{user_id}] No route matched (score={score:.3f}), using fallback"
)
response = self.fallback_agent(user_query)
route_name = "fallback"
else:
logger.info(f"[{user_id}] Routed to '{route_name}' (score={score:.3f})")
response = agent(user_query)
self.conversation_log.append({
"user_id": user_id,
"query": user_query,
"route": route_name,
"score": score,
"response": response,
})
return response
def get_log(self):
return self.conversation_log
# Usage
orchestrator = MultiAgentOrchestrator(router, fallback_agent=general_agent)
response = orchestrator.handle("Hey, I was double charged last Tuesday")
print(response)
Using the Semantic Router Library
While building your own router is instructive, the open-source semantic-router library by Aurelio AI provides a production-ready implementation with additional features like hybrid search (keyword + semantic), dynamic routes, and async support.
pip install semantic-routerBasic Usage
from semantic_router import Route
from semantic_router.layer import RouteLayer
from semantic_router.encoders import HuggingFaceEncoder
# Define routes
billing = Route(
name="billing",
utterances=[
"I want a refund",
"How do I update my payment method?",
"My invoice is wrong",
"Dispute a charge on my card",
],
)
support = Route(
name="support",
utterances=[
"The app keeps crashing",
"I can't log in",
"How do I reset my password?",
"I'm getting a 500 error",
],
)
sales = Route(
name="sales",
utterances=[
"I'd like a demo",
"What are your pricing tiers?",
"Can we get a custom contract?",
"Do you offer startup discounts?",
],
)
# Use a local encoder (no API key needed)
encoder = HuggingFaceEncoder(name="sentence-transformers/all-MiniLM-L6-v2")
# Build the route layer
route_layer = RouteLayer(encoder=encoder, routes=[billing, support, sales])
# Route a query
result = route_layer("I need to get my money back for a purchase")
print(result.name) # Output: billing
Hybrid Routing
The library supports hybrid routing that combines dense semantic similarity with sparse keyword matching (BM25). This is especially useful when domain-specific terminology matters—exact product names or error codes that embeddings might underweight.
from semantic_router.encoders import CohereEncoder
from semantic_router.layer import RouteLayer
# Hybrid requires an encoder that supports both dense and sparse vectors
encoder = CohereEncoder(
name="embed-english-v3.0",
cohere_api_key="your-api-key",
)
route_layer = RouteLayer(encoder=encoder, routes=[billing, support, sales])
result = route_layer("Error code ERR_4029 on upload")
print(result.name) # Output: support
Best Practices
1. Curate High-Quality Example Utterances
The router is only as good as its examples. Aim for 5–20 diverse utterances per route. Include variations in tone, length, and vocabulary. Avoid near-duplicate examples that skew the similarity space.
2. Set Appropriate Confidence Thresholds
Always configure a minimum similarity threshold below which the system falls back to a general agent or asks for clarification. A threshold that's too low causes misrouting; one that's too high causes unnecessary fallbacks. Calibrate using a held-out evaluation set.
3. Monitor and Log Routing Decisions
Log every routing decision with its score. Over time, analyze low-confidence routings and misroutes to identify routes that need better examples or routes that overlap too much.
4. Handle Route Overlap Deliberately
If two agents handle related domains (e.g., "billing" and "subscriptions"), their example utterances may produce similar embeddings. Consider merging closely related routes, or use a secondary LLM-based disambiguation step only when the top two scores are within a small margin.
5. Re-embed When You Update Routes
Any time you add or modify routes, rebuild the embedding index. If you persist embeddings in a vector database, ensure your pipeline invalidates and regenerates them atomically to avoid serving stale routes.
6. Choose the Right Embedding Model
General-purpose models like all-MiniLM-L6-v2 are fast and work well for common intents. For domain-specific applications (legal, medical, technical support), consider fine-tuned or domain-adapted embedding models that better capture specialized vocabulary.
7. Evaluate Routinely
Maintain a golden test set of labeled queries. Run it against the router after every route change and track metrics like accuracy, precision per route, and fallback rate. This catches regressions before they reach production.
Conclusion
Semantic routing provides a fast, scalable, and maintainable way to dispatch requests across multiple specialized agents. By replacing brittle rules and expensive LLM-based dispatchers with embedding similarity, you get a system that generalizes across language variation while keeping latency and cost low. Whether you build a custom router or adopt a library like semantic-router, the key to success lies in curating high-quality example utterances, calibrating confidence thresholds, and continuously monitoring routing performance. As your multi-agent system grows, a well-tuned semantic router becomes the connective tissue that keeps the entire architecture coherent and responsive.