How to Handle Model Upgrades with Zero Downtime

How to Handle Model Upgrades with Zero Downtime

Machine learning models are not static artifacts. They evolve over time as you retrain on new data, tune hyperparameters, or swap architectures entirely. But deploying a new model version into production carries risk: a bad rollout can degrade predictions, break downstream consumers, or take your service offline entirely. Zero-downtime model upgrades ensure that users never experience an interruption while you transition from one model version to the next.

What Is a Zero-Downtime Model Upgrade?

A zero-downtime model upgrade is a deployment strategy that swaps an old model (version N) for a new one (version N+1) without interrupting inference availability. Throughout the upgrade, the service continues to respond to prediction requests. If the new model misbehaves, you can roll back instantly without users noticing.

This is distinct from a simple redeploy, which typically involves stopping the inference server, loading the new weights, and restarting. That approach introduces a window of unavailability — unacceptable for production-facing systems.

Why It Matters

• Availability: Prediction services often back real-time features like fraud detection, recommendations, or search ranking. Any downtime directly impacts user experience and revenue.
• Safety: New models can silently produce worse predictions. Zero-downtime strategies like canary releases let you catch regressions before they affect all traffic.
• Reversibility: If a new model underperforms, you need to revert in seconds, not minutes. Keeping the previous version hot makes this possible.
• Operational confidence: Teams ship more often when upgrades are safe and reversible, accelerating iteration.

Core Strategies

There are several complementary techniques. In practice, you combine them.

• Blue-green deployment: Run two identical environments. Switch traffic from blue (old) to green (new) atomically.
• Canary release: Route a small percentage of traffic to the new model, monitor metrics, and gradually increase.
• Shadow mode: The new model receives real traffic and produces predictions, but its outputs are logged, not served. You compare against the old model offline.
• Multi-model serving: Load multiple model versions in the same process and route per request.

How to Use It: A Practical Example

Below we walk through a canary-style upgrade using a lightweight model server in Python. The server keeps both the old and new model loaded, routes traffic based on a configurable percentage, and exposes a health endpoint for orchestration.

1. Define the Model Server

# server.py
import os
import random
import time
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import joblib

app = FastAPI()

# Configuration via environment variables
OLD_MODEL_PATH = os.getenv("OLD_MODEL_PATH", "/models/v1/model.joblib")
NEW_MODEL_PATH = os.getenv("NEW_MODEL_PATH", "/models/v2/model.joblib")
CANARY_PERCENT = float(os.getenv("CANARY_PERCENT", "0"))  # 0-100

# Load both models at startup so switching is instant
print("Loading models...")
old_model = joblib.load(OLD_MODEL_PATH)
new_model = joblib.load(NEW_MODEL_PATH)
print("Models loaded.")

class PredictRequest(BaseModel):
    features: list[float]

class PredictResponse(BaseModel):
    prediction: float
    model_version: str

def select_model():
    """Return (model, version_label) based on canary percentage."""
    if CANARY_PERCENT <= 0:
        return old_model, "v1"
    if CANARY_PERCENT >= 100:
        return new_model, "v2"
    if random.random() * 100 < CANARY_PERCENT:
        return new_model, "v2"
    return old_model, "v1"

@app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
    model, version = select_model()
    try:
        pred = float(model.predict([req.features])[0])
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
    return PredictResponse(prediction=pred, model_version=version)

@app.get("/health")
def health():
    return {"status": "ok", "canary_percent": CANARY_PERCENT}

@app.get("/config/canary")
def get_canary():
    return {"canary_percent": CANARY_PERCENT}

@app.post("/config/canary")
def set_canary(percent: float):
    global CANARY_PERCENT
    CANARY_PERCENT = max(0.0, min(100.0, percent))
    return {"canary_percent": CANARY_PERCENT}

2. Run the Server

export OLD_MODEL_PATH=/models/v1/model.joblib
export NEW_MODEL_PATH=/models/v2/model.joblib
export CANARY_PERCENT=0

uvicorn server:app --host 0.0.0.0 --port 8000

At this point, 100% of traffic is served by the old model. The new model is loaded but idle.

3. Begin the Canary Rollout

# Route 5% of traffic to the new model
curl -X POST http://localhost:8000/config/canary \
     -H "Content-Type: application/json" \
     -d '{"percent": 5}'

# Send a test prediction
curl -X POST http://localhost:8000/predict \
     -H "Content-Type: application/json" \
     -d '{"features": [1.2, 3.4, 0.8]}'

Each response includes model_version, so you can attribute predictions to v1 or v2 in your logs and metrics.

4. Monitor and Ramp Up

# After validating metrics at 5%, increase gradually
for pct in 10 25 50 75 100; do
  curl -X POST http://localhost:8000/config/canary \
       -H "Content-Type: application/json" \
       -d "{\"percent\": $pct}"
  echo "Set canary to $pct%, waiting 10 minutes..."
  sleep 600
done

Between each step, observe key metrics: latency, error rate, business KPIs (click-through rate, fraud catch rate, etc.). If anything degrades, immediately roll back.

5. Roll Back If Needed

# Instant rollback — all traffic back to v1
curl -X POST http://localhost:8000/config/canary \
     -H "Content-Type: application/json" \
     -d '{"percent": 0}'

Because the old model is still loaded in memory, rollback is instantaneous. No restart, no reload.

Best Practices

• Always run shadow mode first. Before sending any real traffic to a new model, run it in shadow mode and compare its predictions against the current model on live inputs. This catches data schema mismatches and gross errors.
• Tag every prediction with its model version. Without version attribution, you cannot diagnose regressions or compute per-version metrics.
• Define explicit success criteria before ramping. Decide in advance which metrics must hold and what thresholds trigger a rollback. Do not improvise during the rollout.
• Automate the ramp. Manual canary steps are error-prone. Use a script or orchestrator that waits, checks metrics, and either advances or rolls back automatically.
• Keep the previous model hot. The ability to roll back in milliseconds depends on the old model remaining loaded in memory. Do not unload it until the new model has served 100% of traffic for a sustained period.
• Watch for feature drift during rollout. A model trained on stale data may behave differently on live traffic than on your validation set. Monitor input distributions, not just outputs.
• Use a feature store for consistency. If v1 and v2 expect different feature pipelines, coordinate the feature schema upgrade alongside the model upgrade to avoid silent mismatches.
• Plan for memory. Holding two models in memory doubles GPU/CPU memory usage during the rollout window. Size your infrastructure accordingly, or use model hot-swapping with a brief single-model fallback.
• Version your model artifacts immutably. Store each version at a unique path (e.g., /models/v1/, /models/v2/). Never overwrite a deployed artifact in place.

Conclusion

Zero-downtime model upgrades are a discipline, not a single tool. By keeping multiple model versions loaded, routing traffic incrementally, tagging every prediction with its version, and automating both the ramp-up and the rollback, you turn model deployment from a risky event into a routine operation. The investment pays off every time you ship an improvement without a 3 a.m. incident — and every time you catch a bad model before your users do.