How to Build a Multi-Model Serving Platform

How to Build a Multi-Model Serving Platform

As machine learning adoption grows inside organizations, teams inevitably face a familiar problem: dozens of trained models, each with different frameworks, dependencies, and hardware needs, all needing to be exposed as reliable APIs. Serving one model is easy. Serving fifty models efficiently, without drowning in infrastructure costs, is an entirely different challenge. That is where a multi-model serving platform comes in.

In this tutorial, you will learn what a multi-model serving platform is, why it matters, how to architect one, and how to build a working example using Python, FastAPI, and a simple model registry. By the end, you will have a clear blueprint for serving many models behind a single, scalable API gateway.

What Is a Multi-Model Serving Platform?

A multi-model serving platform is an infrastructure layer that hosts multiple machine learning models simultaneously and exposes them through a unified interface, typically HTTP or gRPC endpoints. Instead of deploying one container per model, the platform loads several models into shared or isolated runtimes, routes incoming requests to the correct model, and manages lifecycle operations such as loading, unloading, versioning, and scaling.

Key characteristics of a mature platform include:

• Dynamic model loading: Models can be registered and loaded at runtime without redeploying the entire service.
• Request routing: Incoming requests are dispatched to the correct model based on path, header, or payload metadata.
• Resource sharing: GPU and CPU memory are shared across models where possible, with on-demand loading to reduce idle cost.
• Versioning: Multiple versions of the same model can coexist, enabling canary releases and A/B testing.
• Observability: Metrics, logs, and traces are collected per model for monitoring latency, throughput, and error rates.

Why It Matters

The naive approach of one container per model works for a handful of models, but it breaks down quickly. Each container carries its own runtime overhead, cold-start cost, and operational burden. When you have hundreds of models, many of which receive traffic only occasionally, the waste becomes significant.

A multi-model serving platform addresses several real problems:

• Cost efficiency: Infrequently used models share compute instead of occupying dedicated resources.
• Operational simplicity: A single deployment surface means fewer pipelines to maintain.
• Faster iteration: Data scientists can register new models without waiting for DevOps to provision infrastructure.
• Consistent APIs: All models share a uniform request and response contract, simplifying client integration.
• Better GPU utilization: Multiple models can time-share a single accelerator, dramatically improving ROI on expensive hardware.

Architecture Overview

Before writing code, it helps to understand the components of a multi-model serving platform. A typical architecture consists of the following layers:

• Model Registry: A storage backend (object store, database, or filesystem) that holds model artifacts and metadata.
• Model Manager: The component responsible for loading, unloading, and tracking which models are currently in memory.
• Inference Server: The runtime that executes predictions. This may be a custom Python process or a specialized engine like Triton, BentoML, or KServe.
• API Gateway: The front door that accepts requests, routes them to the correct model, and returns responses.
• Scaler: A controller that adjusts the number of replicas or GPU allocations based on traffic patterns.

In the next sections, we will build a simplified but functional version of this architecture.

Building the Platform: Step by Step

We will build our platform using Python and FastAPI. The example will support multiple scikit-learn models loaded dynamically from disk. The same pattern extends to PyTorch, TensorFlow, or ONNX models with minor adjustments.

Step 1: Project Setup

Create a new project directory and install the required dependencies:

mkdir multi-model-server && cd multi-model-server
python -m venv venv
source venv/bin/activate
pip install fastapi uvicorn scikit-learn joblib pydantic

Your project structure will look like this:

multi-model-server/
├── main.py
├── model_manager.py
├── registry.py
└── models/
    ├── iris_v1.joblib
    └── wine_v1.joblib

Step 2: Creating the Model Registry

The registry tracks metadata about each available model. In production, you would use a database or object store, but for this tutorial we use a simple in-memory dictionary backed by a local directory.

# registry.py
import os
import json
from pathlib import Path

MODEL_DIR = Path(__file__).parent / "models"

class ModelRegistry:
    def __init__(self, model_dir: Path = MODEL_DIR):
        self.model_dir = model_dir
        self.model_dir.mkdir(parents=True, exist_ok=True)
        self.metadata_file = self.model_dir / "registry.json"
        self.metadata = self._load_metadata()

    def _load_metadata(self):
        if self.metadata_file.exists():
            with open(self.metadata_file, "r") as f:
                return json.load(f)
        return {}

    def _save_metadata(self):
        with open(self.metadata_file, "w") as f:
            json.dump(self.metadata, f, indent=2)

    def register(self, name: str, version: str, filename: str, framework: str = "sklearn"):
        key = f"{name}:{version}"
        self.metadata[key] = {
            "name": name,
            "version": version,
            "filename": filename,
            "framework": framework,
            "path": str(self.model_dir / filename),
        }
        self._save_metadata()
        return self.metadata[key]

    def get(self, name: str, version: str):
        return self.metadata.get(f"{name}:{version}")

    def list_models(self):
        return list(self.metadata.values())

    def remove(self, name: str, version: str):
        key = f"{name}:{version}"
        if key in self.metadata:
            del self.metadata[key]
            self._save_metadata()
            return True
        return False

Step 3: Building the Model Manager

The model manager handles loading and unloading models into memory. It keeps a cache of loaded models and supports lazy loading, meaning a model is only loaded when first requested.

# model_manager.py
import joblib
import threading
from typing import Any, Dict, Optional
from registry import ModelRegistry

class ModelManager:
    def __init__(self, registry: ModelRegistry, max_loaded: int = 10):
        self.registry = registry
        self.max_loaded = max_loaded
        self.loaded: Dict[str, Any] = {}
        self.lock = threading.Lock()

    def _key(self, name: str, version: str) -> str:
        return f"{name}:{version}"

    def load(self, name: str, version: str) -> Any:
        key = self._key(name, version)
        with self.lock:
            if key in self.loaded:
                return self.loaded[key]

            meta = self.registry.get(name, version)
            if meta is None:
                raise ValueError(f"Model {key} not found in registry")

            if len(self.loaded) >= self.max_loaded:
                # Evict the oldest loaded model (simple LRU-ish strategy)
                oldest_key = next(iter(self.loaded))
                del self.loaded[oldest_key]

            model = joblib.load(meta["path"])
            self.loaded[key] = model
            return model

    def unload(self, name: str, version: str) -> bool:
        key = self._key(name, version)
        with self.lock:
            if key in self.loaded:
                del self.loaded[key]
                return True
            return False

    def get_loaded_models(self):
        with self.lock:
            return list(self.loaded.keys())

    def predict(self, name: str, version: str, input_data):
        model = self.load(name, version)
        return model.predict(input_data)

Notice the use of a threading lock. Even though FastAPI runs asynchronously, model inference is often CPU-bound and may run in thread pools, so protecting shared state is important.

Step 4: Creating the API Gateway

Now we build the FastAPI application that exposes endpoints for listing models, registering new ones, and running predictions.

# main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List
from registry import ModelRegistry
from model_manager import ModelManager

app = FastAPI(title="Multi-Model Serving Platform")
registry = ModelRegistry()
manager = ModelManager(registry, max_loaded=10)

class RegisterRequest(BaseModel):
    name: str
    version: str
    filename: str
    framework: str = "sklearn"

class PredictRequest(BaseModel):
    inputs: List[List[float]]

@app.get("/models")
def list_models():
    return {"models": registry.list_models(), "loaded": manager.get_loaded_models()}

@app.post("/models/register")
def register_model(req: RegisterRequest):
    meta = registry.register(req.name, req.version, req.filename, req.framework)
    return {"status": "registered", "model": meta}

@app.delete("/models/{name}/{version}")
def remove_model(name: str, version: str):
    manager.unload(name, version)
    removed = registry.remove(name, version)
    if not removed:
        raise HTTPException(status_code=404, detail="Model not found")
    return {"status": "removed"}

@app.post("/models/{name}/{version}/predict")
def predict(name: str, version: str, req: PredictRequest):
    try:
        predictions = manager.predict(name, version, req.inputs)
        return {"model": f"{name}:{version}", "predictions": predictions.tolist()}
    except ValueError as e:
        raise HTTPException(status_code=404, detail=str(e))
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Inference error: {str(e)}")

@app.delete("/models/{name}/{version}/unload")
def unload_model(name: str, version: str):
    success = manager.unload(name, version)
    if not success:
        raise HTTPException(status_code=404, detail="Model was not loaded")
    return {"status": "unloaded"}

Step 5: Training and Registering Sample Models

To test the platform, train two small scikit-learn models and save them to the models directory.

# train_samples.py
from sklearn.datasets import load_iris, load_wine
from sklearn.ensemble import RandomForestClassifier
import joblib
from pathlib import Path

model_dir = Path("models")
model_dir.mkdir(exist_ok=True)

# Train iris model
iris = load_iris()
iris_clf = RandomForestClassifier(n_estimators=50, random_state=42)
iris_clf.fit(iris.data, iris.target)
joblib.dump(iris_clf, model_dir / "iris_v1.joblib")

# Train wine model
wine = load_wine()
wine_clf = RandomForestClassifier(n_estimators=50, random_state=42)
wine_clf.fit(wine.data, wine.target)
joblib.dump(wine_clf, model_dir / "wine_v1.joblib")

print("Models trained and saved.")

Run the training script:

python train_samples.py

Step 6: Running the Server and Testing

Start the FastAPI server:

uvicorn main:app --host 0.0.0.0 --port 8000 --reload

Register the two models:

curl -X POST http://localhost:8000/models/register \
  -H "Content-Type: application/json" \
  -d '{"name": "iris", "version": "v1", "filename": "iris_v1.joblib"}'

curl -X POST http://localhost:8000/models/register \
  -H "Content-Type: application/json" \
  -d '{"name": "wine", "version": "v1", "filename": "wine_v1.joblib"}'

List available models:

curl http://localhost:8000/models

Run a prediction against the iris model:

curl -X POST http://localhost:8000/models/iris/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"inputs": [[5.1, 3.5, 1.4, 0.2], [6.7, 3.0, 5.0, 1.7]]}'

You should receive a JSON response with predicted class labels for both input samples. The model is loaded lazily on first request and remains in memory for subsequent calls.

Best Practices

Building a toy platform is straightforward, but making it production-ready requires careful attention to several areas.

1. Use Specialized Inference Engines for Scale

For production workloads, especially with deep learning models, consider using dedicated serving engines rather than a custom Python loop. NVIDIA Triton Inference Server, BentoML, TorchServe, and KServe all provide mature multi-model serving capabilities with GPU sharing, dynamic batching, and model versioning built in.

2. Implement Dynamic Batching

Individual inference requests are inefficient for GPU workloads. Dynamic batching collects multiple requests arriving within a short time window and processes them together, dramatically improving throughput. Most production engines support this out of the box.

3. Add Model Health Checks

Each loaded model should have a health check endpoint or internal ping. A model can fail to load due to corrupted weights, missing dependencies, or version mismatches. The platform should report model health separately from service health.

4. Enforce Resource Limits

Without limits, a single large model can consume all available memory. Set per-model memory budgets, enforce maximum loaded model counts, and implement eviction policies such as LRU or time-based unloading for idle models.

5. Version Everything

Always track model versions explicitly. This enables rollback, canary deployments, and A/B testing. Store version metadata alongside the artifact, including training data references, hyperparameters, and evaluation metrics.

6. Secure the API

Model endpoints often handle sensitive data. Add authentication, rate limiting, and input validation. Consider using API keys or OAuth tokens, and log all prediction requests for audit purposes.

7. Monitor Per-Model Metrics

Track latency, throughput, error rate, and memory usage for each model individually. Tools like Prometheus and Grafana work well for this. Alert on anomalies such as sudden latency spikes, which may indicate model drift or resource contention.

8. Plan for Cold Starts

Lazy loading reduces memory usage but introduces latency on first request. For latency-sensitive models, preload them at startup or keep them pinned in memory. For infrequent models, accept the cold start but communicate expected latency to clients.

Extending the Platform

The example in this tutorial covers the fundamentals. To take it further, consider adding the following capabilities:

• gRPC endpoints: For high-throughput, low-latency use cases, gRPC is more efficient than HTTP/JSON.
• Model warm-up: Run dummy inference at load time to initialize lazy frameworks and avoid first-request penalties.
• Autoscaling: Integrate with Kubernetes Horizontal Pod Autoscaler or KServe for automatic scaling based on custom metrics.
• Multi-framework support: Add loaders for PyTorch, TensorFlow, ONNX, and Hugging Face transformers behind a common interface.
• Canary routing: Split traffic between model versions using weighted routing rules.
• Persistent storage: Replace the local filesystem registry with S3, GCS, or a database-backed registry for distributed deployments.

Conclusion

A multi-model serving platform is the foundation that allows organizations to move beyond one-off model deployments and treat machine learning serving as a shared, efficient infrastructure. By combining a model registry, a model manager with lazy loading and eviction, and a unified API gateway, you can serve many models from a single service while keeping costs and operational complexity under control. The example built in this tutorial demonstrates the core patterns, and the best practices outlined here will guide you toward a production-ready system. As your needs grow, layering in specialized inference engines, autoscaling, and observability will transform this foundation into a robust platform capable of supporting hundreds of models at scale.