Debugging Out of Memory (OOM) Errors in vLLM

Debugging Out of Memory (OOM) Errors in vLLM

vLLM is a high-throughput, memory-efficient inference engine for Large Language Models (LLMs) that leverages PagedAttention to manage the KV cache dynamically. Despite its efficiency, users frequently encounter Out of Memory (OOM) errors, especially when running large models on consumer GPUs or when pushing batch sizes to their limits. This tutorial walks you through understanding, diagnosing, and resolving OOM errors in vLLM with practical, hands-on examples.

What Is an OOM Error in vLLM?

An Out of Memory error occurs when vLLM attempts to allocate more GPU memory than is physically available on the device. This typically surfaces as a CUDA out of memory exception raised by PyTorch, or as an internal vLLM error indicating that KV cache blocks could not be allocated. OOM can happen at three distinct stages:

• Model loading: The model weights themselves exceed available GPU memory.
• KV cache allocation: vLLM tries to pre-allocate KV cache space after loading the model and runs out of room.
• Runtime: During inference, dynamic batch growth or very long sequences exhaust the pre-allocated KV cache.

Each stage requires a different debugging strategy, so identifying where the failure occurs is the first step.

Why OOM Debugging Matters

OOM errors are among the most common blockers for teams deploying LLMs in production. Left unresolved, they cause service crashes, failed requests, and unpredictable latency spikes when the system falls back to eviction or preemption. Understanding how vLLM manages memory allows you to:

• Maximize throughput by tuning the KV cache to fit your hardware.
• Stabilize serving under variable load and sequence lengths.
• Run larger models on limited hardware through quantization and offloading.
• Diagnose whether a crash is a configuration issue or a genuine hardware limitation.

Understanding vLLM's Memory Model

Before debugging, it helps to understand how vLLM partitions GPU memory. When the engine starts, it divides total GPU VRAM into three regions:

• Model weights: Loaded first, sized by the model parameters and precision (FP16, BF16, INT8, etc.).
• Activation memory: Temporary tensors used during forward passes.
• KV cache: The remaining space, allocated as fixed-size blocks via PagedAttention.

By default, vLLM reserves 90% of free GPU memory for the KV cache (controlled by gpu_memory_utilization). If the model weights alone consume too much, or if activations spike during long-context inference, the KV cache region shrinks and OOM becomes likely.

Step 1: Reproduce and Capture the Error

Start by reproducing the OOM in a controlled environment. Run vLLM with verbose logging enabled so you can see exactly where allocation fails.

import os
os.environ["VLLM_LOGGING_LEVEL"] = "DEBUG"

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-2-13b-hf",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    enforce_eager=False,
)

prompts = ["Hello, my name is"] * 64
sampling = SamplingParams(temperature=0.7, max_tokens=512)

try:
    outputs = llm.generate(prompts, sampling)
except RuntimeError as e:
    print(f"Caught error: {e}")

If the error occurs during model loading, you will see it before any prompt is processed. If it occurs during generate(), the issue is likely KV cache exhaustion under load.

Step 2: Inspect GPU Memory Usage

Use nvidia-smi and PyTorch's memory utilities to understand the baseline memory consumption before and after model loading.

import torch
from vllm import LLM

print(f"Total GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.2f} GB")
print(f"Free before load:  {torch.cuda.mem_get_info()[0] / 1e9:.2f} GB")

llm = LLM(model="meta-llama/Llama-2-13b-hf", gpu_memory_utilization=0.9)

print(f"Free after load:   {torch.cuda.mem_get_info()[0] / 1e9:.2f} GB")

This tells you how much VRAM the weights consume and how much headroom remains for the KV cache. A 13B model in FP16 requires roughly 26 GB just for weights, which already strains a 24 GB GPU like the RTX 4090.

Step 3: Reduce gpu_memory_utilization

The quickest fix for KV cache OOM is to lower the memory utilization target. This tells vLLM to reserve less VRAM for the cache, leaving more headroom for activations and fragmentation.

llm = LLM(
    model="meta-llama/Llama-2-13b-hf",
    gpu_memory_utilization=0.7,   # was 0.9
    max_model_len=4096,
)

Trade-off: a smaller KV cache means fewer concurrent sequences and lower throughput. This is a safe first step for stability, but not a long-term throughput solution.

Step 4: Reduce max_model_len

The max_model_len parameter caps the maximum sequence length vLLM will accept. Lowering it reduces the worst-case KV cache requirement per sequence, allowing more sequences to fit in the same cache space.

llm = LLM(
    model="meta-llama/Llama-2-13b-hf",
    gpu_memory_utilization=0.85,
    max_model_len=2048,   # was 4096
)

This is particularly effective when your actual workloads use short prompts and completions. There is no benefit to reserving cache for 4096-token sequences if your requests average 512 tokens.

Step 5: Enable Quantization

If the model weights themselves cause OOM, quantization is the most effective remedy. vLLM supports several quantization schemes including AWQ, GPTQ, and bitsandbytes.

# Using a pre-quantized AWQ model
llm = LLM(
    model="TheBloke/Llama-2-13B-AWQ",
    quantization="awq",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
)

# Using bitsandbytes 4-bit on the fly
llm = LLM(
    model="meta-llama/Llama-2-13b-hf",
    quantization="bitsandbytes",
    load_format="bitsandbytes",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
)

A 13B model in 4-bit precision consumes roughly 7 GB instead of 26 GB, freeing substantial VRAM for the KV cache and enabling larger batch sizes.

Step 6: Limit Concurrent Sequences

When OOM occurs at runtime rather than startup, the engine is accepting more concurrent requests than the KV cache can hold. Use max_num_seqs to cap concurrency.

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    max_num_seqs=16,   # limit concurrent sequences
)

In a serving deployment with the OpenAI-compatible API server, pass the same flag:

vllm serve meta-llama/Llama-2-7b-hf \
    --gpu-memory-utilization 0.9 \
    --max-model-len 4096 \
    --max-num-seqs 16

Step 7: Enable Preemption for Graceful Degradation

vLLM supports preemption, which allows the scheduler to swap out or recompute sequences when the KV cache is full, rather than crashing. This is controlled via the scheduler policy.

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    max_num_seqs=64,
    # Preemption is enabled by default; explicitly set for clarity
    scheduler_policy="fcfs",  # or "priority"
)

With preemption active, vLLM will drop or recompute the least recently generated tokens of a sequence to free blocks, preventing hard OOM crashes at the cost of some recomputation overhead.

Step 8: Use Tensor Parallelism for Multi-GPU

When a single GPU cannot hold the model, distribute it across multiple GPUs using tensor parallelism. This splits both weights and KV cache across devices.

llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,      # split across 4 GPUs
    gpu_memory_utilization=0.9,
    max_model_len=4096,
)

For the CLI server:

vllm serve meta-llama/Llama-2-70b-hf \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 4096

Ensure that all GPUs have identical memory capacity and that NCCL communication is properly configured.

Step 9: Diagnose with Profiling Tools

vLLM includes a profiling mode that logs detailed memory allocation information. Enable it to see exactly how much memory is consumed by weights, activations, and the KV cache.

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.9,
    max_model_len=4096,
    enforce_eager=True,          # disable CUDA graphs for accurate profiling
    disable_log_stats=False,     # enable memory stats logging
)

You can also use PyTorch's built-in memory profiler for deeper inspection:

import torch
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-hf", enforce_eager=True)

with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CUDA],
    profile_memory=True,
) as prof:
    llm.generate(["Hello world"], SamplingParams(max_tokens=32))

print(prof.key_averages().table(sort_by="self_cuda_memory_usage", row_limit=20))

This table reveals which operations consume the most GPU memory and can uncover unexpected spikes in activation memory during attention computation.

Best Practices

• Start conservative, then tune up: Begin with gpu_memory_utilization=0.7 and max_num_seqs=8, confirm stability, then increase gradually until you find the OOM boundary.
• Match max_model_len to your workload: Do not reserve cache for context lengths you never use. Profile your real request distribution and set the cap accordingly.
• Prefer quantized models on small GPUs: AWQ and GPTQ models offer near-lossless quality at half or quarter the memory footprint, making them ideal for 8 GB to 24 GB GPUs.
• Use enforce_eager when debugging: CUDA graph capture allocates additional memory. Disabling it with enforce_eager=True simplifies memory accounting during troubleshooting.
• Monitor in production: Expose vLLM's metrics endpoint and alert on KV cache utilization. Sustained utilization above 95% indicates you are near the OOM threshold and should scale out or reduce max_num_seqs.
• Keep some VRAM headroom: Never set gpu_memory_utilization to 1.0. Fragmentation, CUDA context overhead, and activation spikes need buffer space. A value between 0.85 and 0.92 is typically optimal.
• Test with realistic batch shapes: OOM often only appears under production load. Reproduce with prompts that match your real sequence length distribution and concurrency levels.

Common Pitfalls and Fixes

Pitfall: Forgetting that gpu_memory_utilization is a fraction of total VRAM, not free VRAM. If other processes occupy GPU memory, vLLM may still try to allocate up to the configured fraction of total memory, causing OOM. Ensure the GPU is exclusive to vLLM or account for co-located processes.

Pitfall: Setting max_model_len higher than the model's configured maximum context. This wastes KV cache space and can cause confusing allocation failures. Always check the model's config.json for max_position_embeddings.

Pitfall: Mixing quantization formats. Loading an AWQ model without specifying quantization="awq" causes vLLM to treat weights as FP16, leading to immediate OOM. Always match the quantization argument to the checkpoint format.

Conclusion

Debugging OOM errors in vLLM is a systematic process of identifying which memory region is exhausted and applying the appropriate lever: reducing gpu_memory_utilization for headroom, lowering max_model_len for cache efficiency, enabling quantization for smaller weights, capping max_num_seqs for runtime stability, or distributing across GPUs with tensor parallelism. By combining vLLM's built-in profiling tools with a methodical approach to parameter tuning, you can achieve a stable, high-throughput deployment that makes full use of available hardware without crossing the OOM threshold. The key is to measure first, tune incrementally, and always leave a safety margin for the dynamic memory patterns inherent in autoregressive generation.