← Back to DevBytes
Developer Tutorial August 17, 2026 14 min read

Migrating from Ollama to vLLM: Complete Migration Guide

Migrating from Ollama to vLLM: Complete Migration Guide

As AI workloads scale from local experimentation to production-grade inference, many developers find themselves outgrowing Ollama's simplicity and needing the performance, throughput, and flexibility that vLLM provides. This guide walks you through every step of migrating from Ollama to vLLM, covering architecture differences, installation, model loading, API adaptation, and production best practices.

What Is This Migration About?

Ollama is a popular tool for running LLMs locally with a focus on simplicity. It bundles model weights, configuration, and a runtime into a single package called a "Modelfile" ecosystem. vLLM, on the other hand, is a high-throughput, memory-efficient inference engine designed for production deployments. It implements PagedAttention, continuous batching, and tensor parallelism to maximize GPU utilization.

Migrating from Ollama to vLLM means moving from a developer-friendly local runner to a production inference server. The migration involves changing how models are loaded, how the API is called, and how the server is deployed.

Why This Migration Matters

Understanding the Key Differences

Architecture Comparison

Ollama uses a llama.cpp-based backend with GGUF model format. It is optimized for CPU inference and single-user scenarios. vLLM uses PyTorch and Hugging Face Transformers with native GPU optimization. The fundamental difference is that Ollama treats inference as a local application, while vLLM treats it as a server workload.

Model Format Differences

Ollama uses GGUF format, which is a quantized format designed for llama.cpp. vLLM uses Hugging Face model formats including safetensors and pytorch_model.bin. This means you cannot directly use Ollama's GGUF files with vLLM — you need the original Hugging Face model or a vLLM-compatible quantized version.

API Differences

Ollama exposes a custom REST API at http://localhost:11434/api/generate and http://localhost:11434/api/chat. vLLM exposes an OpenAI-compatible API at http://localhost:8000/v1/chat/completions. The request and response schemas are different, requiring client code changes.

Prerequisites and Preparation

System Requirements

Before starting the migration, verify your system meets vLLM's requirements. Unlike Ollama, which can run on CPU, vLLM requires a CUDA-capable GPU. Here is what you need:

Inventory Your Current Ollama Setup

Before migrating, document what you are currently running. List all models, their sizes, custom Modelfiles, system prompts, and any client applications that call the Ollama API. Here is a script to inventory your Ollama installation:

# List all installed Ollama models
ollama list

# Show details of a specific model
ollama show llama3:8b

# Check Ollama version
ollama --version

# View running models
curl http://localhost:11434/api/ps

Save the output of these commands. You will need the model names and any custom parameters when configuring vLLM.

Installing vLLM

Installation via pip

The simplest way to install vLLM is through pip. Create a fresh virtual environment to avoid dependency conflicts:

# Create a virtual environment
python -m venv vllm-env
source vllm-env/bin/activate

# Upgrade pip
pip install --upgrade pip

# Install vLLM
pip install vllm

# Verify installation
python -c "import vllm; print(vllm.__version__)"

Installation via Docker

For production deployments, Docker is recommended. vLLM provides official Docker images with all dependencies pre-installed:

# Pull the latest vLLM Docker image
docker pull vllm/vllm-openai:latest

# Run a quick test
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3-8B-Instruct

Loading Models in vLLM

Loading from Hugging Face Hub

vLLM can download and load models directly from the Hugging Face Hub. This is the most common approach. The model name you pass is the Hugging Face repository identifier:

# Start vLLM server with a Hugging Face model
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

Mapping Ollama Models to Hugging Face Equivalents

Since Ollama uses GGUF and vLLM uses Hugging Face format, you need to find the equivalent Hugging Face model. Here is a mapping of common Ollama models to their Hugging Face counterparts:

# Common model mappings
# Ollama: llama3:8b          -> HF: meta-llama/Meta-Llama-3-8B-Instruct
# Ollama: llama3:70b         -> HF: meta-llama/Meta-Llama-3-70B-Instruct
# Ollama: mistral:7b         -> HF: mistralai/Mistral-7B-Instruct-v0.3
# Ollama: mixtral:8x7b       -> HF: mistralai/Mixtral-8x7B-Instruct-v0.1
# Ollama: qwen2:7b           -> HF: Qwen/Qwen2-7B-Instruct
# Ollama: phi3:14b           -> HF: microsoft/Phi-3-medium-14b-instruct
# Ollama: gemma2:9b          -> HF: google/gemma-2-9b-it
# Ollama: codellama:13b      -> HF: codellama/CodeLlama-13b-Instruct-hf

Loading Quantized Models

If you were using quantized models in Ollama for memory savings, vLLM supports several quantization formats. AWQ and GPTQ are the most common:

# Load an AWQ quantized model
vllm serve TheBloke/Mistral-7B-Instruct-v0.2-AWQ \
  --quantization awq \
  --port 8000

# Load a GPTQ quantized model
vllm serve TheBloke/Llama-3-8B-Instruct-GPTQ \
  --quantization gptq \
  --port 8000

# Load an FP8 quantized model (requires Hopper or Ada GPU)
vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 \
  --quantization fp8 \
  --port 8000

Loading Local Model Files

If you have model files stored locally (downloaded from Hugging Face), you can point vLLM to the local directory:

# Load a local model directory
vllm serve /path/to/local/model \
  --port 8000 \
  --max-model-len 8192

# The directory should contain:
# - config.json
# - model-00001-of-0000X.safetensors (or pytorch_model.bin)
# - tokenizer.json
# - tokenizer_config.json
# - special_tokens_map.json

Migrating Client Applications

Understanding the API Differences

The most significant code change during migration is adapting your API calls. Ollama uses its own API format, while vLLM uses the OpenAI-compatible format. Let's look at the differences side by side.

Ollama API Example (Before Migration)

Here is a typical Ollama API call using Python's requests library:

import requests

# Ollama generate endpoint
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3:8b",
        "prompt": "Explain quantum computing in simple terms.",
        "stream": False,
        "options": {
            "temperature": 0.7,
            "top_p": 0.9,
            "num_predict": 512
        }
    }
)

result = response.json()
print(result["response"])

And here is the Ollama chat endpoint, which is more commonly used for conversational applications:

import requests

# Ollama chat endpoint
response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "llama3:8b",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "stream": False,
        "options": {
            "temperature": 0.7,
            "top_p": 0.9
        }
    }
)

result = response.json()
print(result["message"]["content"])

vLLM API Example (After Migration)

Here is the equivalent code using vLLM's OpenAI-compatible API. Notice the differences in endpoint URL, request structure, and response parsing:

import requests

# vLLM OpenAI-compatible chat endpoint
response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "model": "meta-llama/Meta-Llama-3-8B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "stream": False,
        "temperature": 0.7,
        "top_p": 0.9,
        "max_tokens": 512
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])

Using the OpenAI Python SDK

Since vLLM is OpenAI-compatible, you can use the official OpenAI Python SDK by simply changing the base URL. This is often the cleanest approach:

from openai import OpenAI

# Point the OpenAI client to your vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy-key"  # vLLM doesn't require a real key by default
)

# Chat completion
response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ],
    temperature=0.7,
    max_tokens=512
)

print(response.choices[0].message.content)

Streaming Responses

Both Ollama and vLLM support streaming. Here is how to stream from vLLM using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy-key"
)

# Streaming chat completion
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Write a short poem about the ocean."}
    ],
    stream=True,
    max_tokens=256
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

print()  # Final newline

Migrating Embedding Calls

If you were using Ollama for embeddings, vLLM also supports embedding endpoints. Here is the migration:

# Ollama embeddings (before)
import requests

response = requests.post(
    "http://localhost:11434/api/embeddings",
    json={
        "model": "nomic-embed-text",
        "prompt": "The quick brown fox jumps over the lazy dog."
    }
)
embedding = response.json()["embedding"]

# vLLM embeddings (after)
response = requests.post(
    "http://localhost:8000/v1/embeddings",
    json={
        "model": "BAAI/bge-large-en-v1.5",
        "input": "The quick brown fox jumps over the lazy dog."
    }
)
embedding = response.json()["data"][0]["embedding"]

To serve an embedding model with vLLM, start the server with the embedding model flag:

vllm serve BAAI/bge-large-en-v1.5 \
  --port 8000 \
  --task embed

Handling Custom Modelfiles

Translating Modelfile Parameters

Ollama Modelfiles allow you to customize model behavior with system prompts, parameters, and templates. In vLLM, these are handled differently. Here is how to translate common Modelfile settings:

# Example Ollama Modelfile
# -------------------------
# FROM llama3:8b
# PARAMETER temperature 0.7
# PARAMETER top_p 0.9
# PARAMETER num_ctx 8192
# SYSTEM "You are a coding assistant that writes clean, documented code."
# TEMPLATE """{{ .System }}
# User: {{ .Prompt }}
# Assistant:"""

# Equivalent vLLM configuration
# --------------------------------
# Start vLLM with the base model and context length:
# vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
#   --max-model-len 8192 \
#   --port 8000
#
# Then pass system prompt and parameters in each API call:

import requests

response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "model": "meta-llama/Meta-Llama-3-8B-Instruct",
        "messages": [
            {
                "role": "system",
                "content": "You are a coding assistant that writes clean, documented code."
            },
            {
                "role": "user",
                "content": "Write a Python function to reverse a linked list."
            }
        ],
        "temperature": 0.7,
        "top_p": 0.9,
        "max_tokens": 1024
    }
)

print(response.json()["choices"][0]["message"]["content"])

Using vLLM's Chat Templates

vLLM uses the chat template defined in the model's tokenizer_config.json. Most instruct models on Hugging Face already have appropriate chat templates. If you need a custom template, you can override it at startup:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --chat-template '{"role":"system","content":"You are a helpful assistant."}
{% for message in messages %}
{% if message.role == "user" %}User: {{ message.content }}
{% elif message.role == "assistant" %}Assistant: {{ message.content }}
{% endif %}
{% endfor %}Assistant:'

Multi-GPU and Distributed Deployment

Tensor Parallelism

One of vLLM's key advantages over Ollama is native multi-GPU support through tensor parallelism. This allows you to run models that are too large for a single GPU:

# Run a 70B model across 4 GPUs
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

# Run Mixtral 8x22B across 8 GPUs
vllm serve mistralai/Mixtral-8x22B-Instruct-v0.1 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 32768

Pipeline Parallelism

For models spread across multiple nodes, vLLM supports pipeline parallelism in addition to tensor parallelism:

# Run across 2 nodes with 4 GPUs each
# On node 0 (head node):
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --port 8000 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

Production Deployment

Docker Compose Setup

For production, a Docker Compose file provides reproducible deployments. Here is a complete example:

# docker-compose.yml
version: '3.8'

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm-server
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    ports:
      - "8000:8000"
    volumes:
      - ./models:/root/.cache/huggingface
      - ./logs:/app/logs
    ipc: host
    command:
      - --model=meta-llama/Meta-Llama-3-8B-Instruct
      - --port=8000
      - --max-model-len=8192
      - --gpu-memory-utilization=0.9
      - --tensor-parallel-size=1
      - --uvicorn-log-level=info
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

Start the deployment with:

# Set your Hugging Face token (needed for gated models)
export HF_TOKEN=your_token_here

# Start the service
docker compose up -d

# Check if the server is healthy
curl http://localhost:8000/health

# Test a completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Adding Authentication

By default, vLLM does not require authentication. For production, you should add an API key. vLLM supports this natively:

# Start vLLM with API key authentication
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --api-key sk-your-secret-api-key-here

# Then use the key in your client
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-your-secret-api-key-here"
)

Performance Tuning

Key Configuration Parameters

vLLM exposes many parameters that affect performance. Understanding these is critical for getting the most out of your hardware:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 256 \
  --swap-space 4 \
  --enforce-eager \
  --dtype auto

Here is what each parameter does:

Benchmarking Your Setup

vLLM includes a benchmarking tool to measure throughput and latency. Use it to compare performance before and after tuning:

# Benchmark online serving throughput
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 &

# Run the benchmark
python -m vllm.entrypoints.openai.benchmark_serving \
  --backend vllm \
  --base-url http://localhost:8000 \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --num-prompts 1000 \
  --request-rate 10

Best Practices

Model Selection

Always use the instruct or chat versions of models when serving conversational applications. Base models without instruction tuning will produce poor results with chat templates. Verify the model card on Hugging Face to confirm it supports chat formatting.

Memory Management

Set --gpu-memory-utilization to 0.85-0.9 for production. Leaving 10-15% headroom prevents out-of-memory errors during traffic spikes. Monitor GPU memory usage with nvidia-smi and adjust accordingly.

Graceful Shutdown

When stopping the vLLM server, use SIGTERM for graceful shutdown. This ensures in-flight requests complete before the server stops:

# Graceful shutdown
docker compose down  # or
kill -TERM $(pgrep -f vllm)

# Wait for in-flight requests to complete
# The server will stop accepting new requests immediately
# but will finish processing existing ones

Monitoring and Logging

Enable structured logging for production observability. vLLM supports different log levels and formats:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --uvicorn-log-level info \
  --disable-log-requests

For production monitoring, expose Prometheus metrics by adding a metrics endpoint. vLLM natively exposes metrics at /metrics:

# Scrape vLLM metrics with Prometheus
curl http://localhost:8000/metrics

# Key metrics to monitor:
# - vllm:num_requests_running
# - vllm:num_requests_waiting
# - vllm:gpu_cache_usage_perc
# - vllm:time_to_first_token_seconds
# - vllm:e2e_request_latency_seconds

Handling Gated Models

Many popular models on Hugging Face are gated and require authentication. Set your Hugging Face token before starting vLLM:

# Set the token as an environment variable
export HUGGING_FACE_HUB_TOKEN=hf_your_token_here

# Or pass it directly
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --port 8000 \
  --hf-token hf_your_token_here

Migration Checklist

Use this checklist to ensure a smooth migration:

Common Migration Issues and Solutions

Model Not Found Errors

If vLLM cannot find a model, ensure you are using the correct Hugging Face repository name. Ollama model names like llama3:8b do not work in vLLM. You must use the full Hugging Face identifier like meta-llama/Meta-Llama-3-8B-Instruct.

Out of Memory Errors

If you encounter CUDA out of memory errors, try these solutions in order: reduce --max-model-len, reduce --gpu-memory-utilization, use a quantized model, or increase the number of GPUs with tensor parallelism.

Chat Template Errors

Some models may not have a chat template defined. If you see errors about missing chat templates, either use a model that includes one or provide a custom template with the --chat-template flag.

Different Output Quality

If outputs differ between Ollama and vLLM, this is expected. Different inference engines, quantization methods, and sampling implementations produce slightly different results. The differences should be minor for well-known models. If quality degrades significantly, verify you are using the correct model variant and that the chat template matches.

Conclusion

Migrating from Ollama to vLLM is a strategic decision that trades local simplicity for production-grade performance and scalability. The migration primarily involves three changes: switching from GGUF to Hugging Face model formats, adapting API calls from Ollama's custom format to the OpenAI-compatible format, and configuring vLLM's server parameters for your specific hardware. By following this guide, you can systematically migrate your models and client applications while taking advantage of vLLM's continuous batching, PagedAttention, and multi-GPU support. Start with a single model and client application, validate performance with benchmarks, and gradually migrate your entire stack. The performance gains — often 5-20x in throughput — make the migration effort worthwhile for any workload moving beyond local development into production serving.

— Ad —

Google AdSense will appear here after approval

← Back to all articles