Error Recovery Patterns with vLLM: Complete Guide

Introduction to Error Recovery Patterns with vLLM

vLLM is a high-throughput, memory-efficient inference engine for Large Language Models (LLMs). While it excels at serving models at scale, production deployments inevitably encounter failures: GPU out-of-memory errors, network timeouts, malformed prompts, model loading issues, and transient infrastructure hiccups. Without robust error recovery patterns, a single failure can cascade into service-wide outages, lost requests, and poor user experiences.

This guide walks through the most important error recovery patterns you can apply when building applications on top of vLLM. We cover what each pattern does, when to use it, and provide copy-paste-ready code examples that you can adapt to your own serving stack.

What Are Error Recovery Patterns?

Error recovery patterns are reusable strategies for detecting, containing, and recovering from failures in a distributed or resource-constrained system. In the context of vLLM, they address failures that occur at several layers:

Request layer: malformed prompts, token-limit overruns, invalid sampling parameters.
Engine layer: CUDA out-of-memory, KV cache exhaustion, scheduler stalls.
Transport layer: HTTP timeouts, gRPC stream resets, broken connections.
Infrastructure layer: GPU drops, node restarts, model weight corruption.

A good recovery pattern combines three ingredients: detection (knowing something went wrong), containment (preventing the failure from spreading), and recovery (returning the system to a healthy state, ideally without user-visible impact).

Why Error Recovery Matters for vLLM

vLLM is designed for high concurrency. A single vLLM instance may process dozens of concurrent requests using paged attention and continuous batching. This architecture is efficient but introduces failure modes that traditional request-response servers do not face:

A single oversized prompt can trigger OOM and abort the entire batch.
Preemption of sequences under memory pressure can cause unpredictable latency spikes.
Long-running streaming responses can be interrupted mid-generation.
Model loading failures on startup can leave a replica permanently unhealthy.

Without recovery patterns, these issues translate into 500 errors, dropped streams, and retry storms that make the problem worse. With them, you can degrade gracefully, retry intelligently, and keep serving traffic even when components fail.

Common vLLM Error Types

Before implementing recovery, you need to recognize the errors vLLM surfaces. The most common ones include:

ValueError — invalid sampling parameters, unsupported tokenizers, or prompt issues.
torch.cuda.OutOfMemoryError — GPU memory exhausted during prefill or decoding.
ConnectionError / asyncio.TimeoutError — client-to-server transport failures.
RuntimeError — CUDA kernel failures, weight loading issues, or scheduler deadlocks.
AssertionError — internal invariant violations, often indicating version mismatches.

The recovery strategy should match the error class. Transient errors (timeouts, OOM under load) are good candidates for retry. Permanent errors (invalid parameters, missing weights) require fast-fail and user feedback.

Pattern 1: Retry with Exponential Backoff and Jitter

The simplest and most widely applicable pattern is retrying failed requests with exponential backoff plus jitter. Backoff prevents overwhelming a recovering server, while jitter spreads retries across time to avoid thundering-herd effects.

Implementation

import asyncio
import random
from openai import AsyncOpenAI, APIConnectionError, APITimeoutError, InternalServerError

client = AsyncOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

RETRYABLE_ERRORS = (
    APIConnectionError,
    APITimeoutError,
    InternalServerError,
)

async def generate_with_retry(
    prompt: str,
    model: str = "meta-llama/Llama-3-8B-Instruct",
    max_retries: int = 5,
    base_delay: float = 0.5,
    max_delay: float = 30.0,
):
    attempt = 0
    while True:
        try:
            response = await client.completions.create(
                model=model,
                prompt=prompt,
                max_tokens=256,
                temperature=0.7,
            )
            return response.choices[0].text
        except RETRYABLE_ERRORS as exc:
            attempt += 1
            if attempt > max_retries:
                raise
            # Exponential backoff with full jitter
            delay = min(max_delay, base_delay * (2 ** attempt))
            jittered = random.uniform(0, delay)
            print(f"Attempt {attempt} failed: {exc}. Retrying in {jittered:.2f}s")
            await asyncio.sleep(jittered)
        except Exception:
            # Non-retryable: fail fast
            raise

The key design choice is the RETRYABLE_ERRORS tuple. Only retry errors that have a reasonable chance of succeeding on a second attempt. Retrying a ValueError caused by an invalid temperature value wastes resources and delays user feedback.

Pattern 2: Circuit Breaker

When a vLLM instance is repeatedly failing — for example, due to a corrupted model load or a GPU fault — retrying only makes things worse. A circuit breaker monitors failure rates and temporarily stops sending traffic to a failing backend, giving it time to recover.

Implementation

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=30, success_threshold=2):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.success_threshold = success_threshold
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = 0

    def record_success(self):
        if self.state == CircuitState.HALF_OPEN:
            self.success_count += 1
            if self.success_count >= self.success_threshold:
                self.state = CircuitState.CLOSED
                self.failure_count = 0
                self.success_count = 0
        elif self.state == CircuitState.CLOSED:
            self.failure_count = 0

    def record_failure(self):
        self.last_failure_time = time.monotonic()
        if self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN
            self.success_count = 0
        elif self.state == CircuitState.CLOSED:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN

    def allow_request(self):
        if self.state == CircuitState.CLOSED:
            return True
        if self.state == CircuitState.OPEN:
            if time.monotonic() - self.last_failure_time >= self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
                self.success_count = 0
                return True
            return False
        # HALF_OPEN: allow a probe request
        return True

# Usage with multiple vLLM backends
backends = [
    {"url": "http://vllm-1:8000/v1", "breaker": CircuitBreaker()},
    {"url": "http://vllm-2:8000/v1", "breaker": CircuitBreaker()},
]

def pick_healthy_backend():
    for b in backends:
        if b["breaker"].allow_request():
            return b
    return None

With this pattern, a failing vLLM replica is automatically quarantined. Once the recovery timeout elapses, the circuit enters HALF_OPEN and allows a single probe request. If the probe succeeds a few times, the circuit closes and traffic resumes normally.

Pattern 3: Fallback Models and Graceful Degradation

Sometimes the best recovery is not retrying the same model but falling back to a smaller, cheaper, or more robust one. For example, if a 70B model OOMs under load, you can serve the request from a 7B model with slightly lower quality but guaranteed availability.

Implementation

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("vllm-fallback")

FALLBACK_CHAIN = [
    {"model": "meta-llama/Llama-3-70B-Instruct", "max_tokens": 1024},
    {"model": "meta-llama/Llama-3-8B-Instruct",  "max_tokens": 1024},
    {"model": "meta-llama/Llama-3.2-1B-Instruct", "max_tokens": 512},
]

async def generate_with_fallback(prompt: str):
    last_error = None
    for tier in FALLBACK_CHAIN:
        try:
            response = await client.completions.create(
                model=tier["model"],
                prompt=prompt,
                max_tokens=tier["max_tokens"],
            )
            if tier is not FALLBACK_CHAIN[0]:
                logger.warning(f"Served request using fallback model: {tier['model']}")
            return {
                "text": response.choices[0].text,
                "model_used": tier["model"],
                "degraded": tier is not FALLBACK_CHAIN[0],
            }
        except Exception as exc:
            logger.warning(f"Model {tier['model']} failed: {exc}")
            last_error = exc
    raise RuntimeError(f"All fallback models failed: {last_error}")

This pattern pairs well with the circuit breaker: each model tier can have its own breaker, so a failing 70B instance is skipped automatically without waiting for a timeout on every request.

Pattern 4: Request Validation and Pre-flight Checks

Many vLLM errors are preventable. Validating requests before they reach the engine eliminates entire classes of failures and frees the engine to focus on genuine transient issues.

Implementation

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B-Instruct")
MAX_MODEL_LEN = 8192

class RequestValidationError(Exception):
    pass

def validate_request(prompt: str, temperature: float, max_tokens: int):
    if not prompt or not prompt.strip():
        raise RequestValidationError("Prompt must not be empty")

    token_count = len(tokenizer.encode(prompt))
    if token_count + max_tokens > MAX_MODEL_LEN:
        raise RequestValidationError(
            f"Request too long: {token_count} prompt tokens + "
            f"{max_tokens} max_tokens exceeds limit of {MAX_MODEL_LEN}"
        )

    if not (0.0 <= temperature <= 2.0):
        raise RequestValidationError("temperature must be between 0.0 and 2.0")

    if max_tokens <= 0 or max_tokens > 4096:
        raise RequestValidationError("max_tokens must be between 1 and 4096")

    return {"token_count": token_count}

# Wrap in your handler
async def safe_generate(prompt: str, temperature: float, max_tokens: int):
    try:
        meta = validate_request(prompt, temperature, max_tokens)
    except RequestValidationError as e:
        return {"error": str(e), "status": "invalid_request"}, 400

    try:
        text = await generate_with_retry(prompt, temperature=temperature, max_tokens=max_tokens)
        return {"text": text, "prompt_tokens": meta["token_count"], "status": "ok"}, 200
    except Exception as e:
        return {"error": "internal_error", "status": "failed"}, 503

Pre-flight validation turns permanent errors into immediate 400 responses, which are cheap to produce and informative for the client. This keeps the vLLM engine queue clear of work that was doomed to fail.

Pattern 5: Streaming Recovery and Checkpointing

Streaming responses are especially vulnerable to mid-generation failures. If a connection drops after 500 of 1000 tokens, naive retry strategies either restart from scratch (wasteful) or return a truncated response (incorrect). A checkpointing pattern stores partial output so retries can resume or return the best available result.

Implementation

import asyncio
from dataclasses import dataclass, field

@dataclass
class StreamCheckpoint:
    prompt: str
    accumulated_text: str = ""
    finished: bool = False
    attempt: int = 0
    max_attempts: int = 3

async def stream_with_checkpoint(prompt: str, model: str = "meta-llama/Llama-3-8B-Instruct"):
    ckpt = StreamCheckpoint(prompt=prompt)

    while not ckpt.finished and ckpt.attempt < ckpt.max_attempts:
        ckpt.attempt += 1
        try:
            stream = await client.completions.create(
                model=model,
                prompt=prompt,
                max_tokens=512,
                stream=True,
            )
            async for event in stream:
                delta = event.choices[0].text
                ckpt.accumulated_text += delta
                yield delta
            ckpt.finished = True
        except (APIConnectionError, APITimeoutError) as exc:
            if ckpt.attempt >= ckpt.max_attempts:
                # Return what we have so far rather than failing entirely
                yield f"\n[stream interrupted after {len(ckpt.accumulated_text)} chars]"
                return
            await asyncio.sleep(1.0 * ckpt.attempt)
            # On retry, yield a marker so the client knows we restarted
            yield "\n[recovering...]\n"

This pattern is particularly useful for chat applications where a partial answer is still valuable. The client receives a transparent marker indicating recovery, and the final output contains as much useful content as possible.

Pattern 6: Bulkhead Isolation

A bulkhead pattern isolates different classes of workloads so that a failure in one does not starve another. For vLLM, this often means running separate engine instances for different priority levels or model sizes, each with its own resource pool.

Implementation

import asyncio
from asyncio import Semaphore

# Separate semaphores for different priority tiers
HIGH_PRIORITY_POOL = Semaphore(10)   # reserved for interactive chat
LOW_PRIORITY_POOL = Semaphore(2)     # background batch jobs

async def generate_priority(prompt: str, priority: str = "low"):
    pool = HIGH_PRIORITY_POOL if priority == "high" else LOW_PRIORITY_POOL
    async with pool:
        try:
            return await generate_with_retry(prompt)
        except Exception:
            if priority == "high":
                # High-priority requests can overflow into the low-priority pool
                async with LOW_PRIORITY_POOL:
                    return await generate_with_retry(prompt)
            raise

With bulkheads, a flood of low-priority batch jobs cannot exhaust the connections needed by interactive users. The overflow rule for high-priority requests ensures that critical traffic still gets served even when its dedicated pool is saturated.

Pattern 7: Health Checks and Readiness Probes

Load balancers need accurate signals about whether a vLLM instance is ready to accept traffic. A shallow TCP check is not enough — the engine may be listening but still loading weights or in a degraded state. A deep health check should exercise the model end-to-end.

Implementation

from fastapi import FastAPI, Response
import time

app = FastAPI()
ENGINE_READY = False
LAST_HEALTHY = 0
HEALTH_STALE_SECONDS = 60

@app.on_event("startup")
async def warmup():
    global ENGINE_READY, LAST_HEALTHY
    try:
        # Tiny inference to confirm the engine is fully loaded
        await generate_with_retry("ping", max_retries=1)
        ENGINE_READY = True
        LAST_HEALTHY = time.monotonic()
    except Exception:
        ENGINE_READY = False

@app.get("/health")
async def health(response: Response):
    global LAST_HEALTHY
    if not ENGINE_READY:
        response.status_code = 503
        return {"status": "not_ready"}

    try:
        await client.completions.create(
            model="meta-llama/Llama-3-8B-Instruct",
            prompt="hi",
            max_tokens=1,
        )
        LAST_HEALTHY = time.monotonic()
        return {"status": "healthy"}
    except Exception as e:
        response.status_code = 503
        return {"status": "unhealthy", "detail": str(e)}

@app.get("/ready")
async def ready(response: Response):
    stale = time.monotonic() - LAST_HEALTHY > HEALTH_STALE_SECONDS
    if not ENGINE_READY or stale:
        response.status_code = 503
        return {"ready": False}
    return {"ready": True}

Use /ready for the load balancer's routing decisions and /health for deeper diagnostics. The staleness check ensures that an instance which has not successfully served a health probe recently is treated as not ready, even if the process is still running.

Pattern 8: Graceful Shutdown and Drain

When deploying a new version of a vLLM server, you need to drain in-flight requests without aborting them. A graceful shutdown pattern stops accepting new requests, finishes active generations, and then exits cleanly.

Implementation

import signal
import asyncio

DRAINING = False
ACTIVE_REQUESTS = asyncio.Event()
ACTIVE_REQUESTS.set()  # set means "no active requests"

request_counter = 0

async def handle_generate(prompt: str):
    global request_counter
    if DRAINING:
        raise RuntimeError("Server is draining, try another replica")
    request_counter += 1
    if request_counter == 1:
        ACTIVE_REQUESTS.clear()
    try:
        return await generate_with_retry(prompt)
    finally:
        request_counter -= 1
        if request_counter == 0:
            ACTIVE_REQUESTS.set()

async def graceful_shutdown():
    global DRAINING
    DRAINING = True
    print("Draining: no new requests accepted")
    try:
        await asyncio.wait_for(ACTIVE_REQUESTS.wait(), timeout=120)
        print("All in-flight requests completed")
    except asyncio.TimeoutError:
        print("Drain timeout reached, forcing shutdown")
    # Close engine, release GPU memory
    # engine.shutdown()  # if using vLLM's Python API directly

def register_shutdown():
    loop = asyncio.get_event_loop()
    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, lambda: asyncio.create_task(graceful_shutdown()))

Combined with a load balancer that respects the /ready probe, this pattern ensures zero dropped requests during rolling deploys.

Best Practices

Classify errors before retrying. Distinguish transient from permanent failures. Retrying a bad request burns GPU time and delays user feedback.
Always use jitter. Synchronized retries after a recovery create load spikes that can trigger secondary failures.
Set generous timeouts for prefill. Long prompts have high prefill latency. A 5-second timeout will falsely fail legitimate requests.
Monitor KV cache pressure. vLLM exposes metrics like vllm:num_preemptions and vllm:gpu_cache_usage_perc. Alert on sustained high cache usage before it becomes OOM.
Keep fallback models warm. A fallback model that needs cold-start loading is not a real fallback. Pre-load smaller models on standby replicas.
Idempotency keys for retries. If your application has side effects (writing to a database, charging a user), tag each logical request with an idempotency key so retries do not double-charge.
Log recovery events separately. Track how often retries, fallbacks, and circuit opens occur. A rising trend signals capacity problems before users notice.
Test failure paths in staging. Inject OOM, kill replicas, and sever network links in a controlled environment. Recovery code that has never run in anger will fail when you need it most.
Right-size gpu_memory_utilization. Setting it too high leaves no headroom for variable-length batches and increases OOM risk. Values around 0.85–0.90 are a good starting point for production.
Use max_num_seqs and max_model_len deliberately. These caps prevent a single batch from growing unbounded and protect latency SLOs under load spikes.

Putting It All Together

A production-grade vLLM serving stack typically combines several of these patterns. A request flows through validation, then a circuit breaker, then retry-with-backoff, with fallback models on standby and bulkheads protecting priority tiers. Health checks feed the load balancer, and graceful shutdown handles deploys. The following snippet shows a simplified end-to-end handler:

async def production_generate(prompt: str, priority: str = "low"):
    # 1. Validate
    try:
        validate_request(prompt, temperature=0.7, max_tokens=512)
    except RequestValidationError as e:
        return {"error": str(e)}, 400

    # 2. Pick a healthy backend via circuit breaker
    backend = pick_healthy_backend()
    if backend is None:
        return {"error": "all backends unavailable"}, 503

    # 3. Acquire bulkhead slot
    pool = HIGH_PRIORITY_POOL if priority == "high" else LOW_PRIORITY_POOL
    async with pool:
        try:
            # 4. Retry with fallback
            result = await generate_with_fallback(prompt)
            backend["breaker"].record_success()
            return result, 200
        except Exception as e:
            backend["breaker"].record_failure()
            return {"error": "generation failed", "detail": str(e)}, 503

Conclusion

Error recovery is not an afterthought — it is a core part of serving LLMs reliably with vLLM. The patterns in this guide, from retry-with-backoff to circuit breakers, fallback models, bulkheads, and graceful shutdown, form a layered defense that keeps your service responsive even when individual components fail. Start by classifying the errors you actually see in production, then apply the smallest set of patterns that addresses them. Measure the results, tune thresholds, and revisit your recovery logic as your traffic and model catalog grow. A well-instrumented recovery layer is the difference between a demo that works on a good day and a service that customers can depend on every day.