Building a Docker Image for GPU-Accelerated LLM Serving

Serving Large Language Models (LLMs) in production requires more than just loading weights into memory — it demands optimized inference runtimes, careful dependency management, and access to GPU hardware. Docker has become the de facto standard for packaging these workloads, but GPU-accelerated containers introduce unique challenges around driver compatibility, CUDA versions, and runtime configuration. This tutorial walks through building a production-ready Docker image for serving an LLM with GPU acceleration, using vLLM as the inference engine.

What It Is

A GPU-accelerated LLM serving container is a Docker image that bundles the model-serving framework, its dependencies, CUDA libraries, and optionally the model weights themselves, all configured to run on NVIDIA GPU hardware. Unlike CPU-only containers, these images must be built on top of a CUDA-enabled base image and run with the NVIDIA Container Toolkit, which exposes host GPU devices to the container runtime.

The typical stack includes:

A CUDA base image (e.g., nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04)
Python and a virtual environment for isolation
An inference engine such as vLLM, TGI, or TensorRT-LLM
Optional model weights cached at build time or mounted at runtime
A serving entrypoint that exposes an OpenAI-compatible API

Why It Matters

Packaging LLM serving into a Docker image delivers several concrete benefits. First, it guarantees reproducibility — the same CUDA, PyTorch, and vLLM versions that worked in staging will run identically in production. Second, it simplifies deployment across environments, from a single workstation with one GPU to a Kubernetes cluster with multiple nodes. Third, it enables clean separation between infrastructure and application code, letting platform teams manage GPU node pools while ML teams iterate on model versions independently.

Without containerization, teams frequently hit "dependency hell" when moving from development to production: mismatched CUDA versions, missing cuDNN libraries, or Python package conflicts that silently degrade inference performance. A well-built Docker image eliminates these issues by freezing the entire runtime environment.

How to Use It

1. Prerequisites

Before building the image, ensure your host machine has the NVIDIA driver installed and the NVIDIA Container Toolkit configured. Verify with:

nvidia-smi
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If both commands print the GPU table, your environment is ready.

2. The Dockerfile

Below is a complete Dockerfile that builds a vLLM serving image. It uses a multi-stage approach to keep the final image lean while ensuring all GPU libraries are present.

# syntax=docker/dockerfile:1.6

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 AS base

ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    HF_HOME=/models

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.11 python3.11-venv python3-pip curl ca-certificates git \
    && rm -rf /var/lib/apt/lists/*

RUN python3.11 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

RUN pip install --upgrade pip wheel

# ---- Builder stage ----
FROM base AS builder

RUN pip install vllm==0.6.3 torch==2.4.0 transformers==4.45.0

# ---- Runtime stage ----
FROM base AS runtime

COPY --from=builder /opt/venv /opt/venv

RUN mkdir -p /models /app
WORKDIR /app

COPY serve.py /app/serve.py

EXPOSE 8000

ENTRYPOINT ["python", "/app/serve.py"]

3. The Serving Script

The serve.py script wraps vLLM's engine in a lightweight FastAPI application that exposes an OpenAI-compatible chat endpoint.

import os
from fastapi import FastAPI
from pydantic import BaseModel
from vllm import LLM, SamplingParams

MODEL_ID = os.environ.get("MODEL_ID", "meta-llama/Meta-Llama-3-8B-Instruct")
PORT = int(os.environ.get("PORT", "8000"))

app = FastAPI()
llm = LLM(model=MODEL_ID, tensor_parallel_size=1, gpu_memory_utilization=0.9)

class ChatRequest(BaseModel):
    prompt: str
    max_tokens: int = 512
    temperature: float = 0.7

@app.post("/v1/chat/completions")
def chat(req: ChatRequest):
    params = SamplingParams(
        max_tokens=req.max_tokens,
        temperature=req.temperature
    )
    outputs = llm.generate([req.prompt], params)
    return {"choices": [{"text": outputs[0].outputs[0].text}]}

4. Building and Running the Image

Build the image with a descriptive tag so you can track versions:

docker build -t llm-server:vllm-0.6.3-llama3-8b .

Run the container, passing the GPU and the Hugging Face token as environment variables:

docker run --gpus all \
  -p 8000:8000 \
  -e MODEL_ID=meta-llama/Meta-Llama-3-8B-Instruct \
  -e HF_TOKEN=hf_your_token_here \
  -v /mnt/models:/models \
  llm-server:vllm-0.6.3-llama3-8b

Test the endpoint with curl:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quantum computing in one sentence.", "max_tokens": 100}'

Best Practices

Pin every dependency version. CUDA, PyTorch, vLLM, and transformers must be mutually compatible. A floating version can silently pull a CUDA 12.6 build onto a 12.4 driver and fail at runtime.
Use multi-stage builds. The builder stage compiles and installs packages; the runtime stage copies only what is needed. This can shrink the final image by several gigabytes.
Mount weights instead of baking them in. Embedding a 16 GB model in an image slows builds and bloats registries. Mount a host volume or pull weights at container startup from Hugging Face Hub.
Set gpu_memory_utilization conservatively. A value of 0.9 leaves headroom for the CUDA context and avoids OOM errors when multiple containers share a GPU.
Use a non-root user in production. Create a dedicated user with useradd and switch to it with the USER directive to reduce the attack surface.
Enable health checks. Add a HEALTHCHECK instruction that curls the API root so orchestrators can restart unhealthy containers automatically.
Leverage layer caching. Place rarely changing instructions (apt installs, pip installs) before frequently changing ones (application code) so rebuilds stay fast.
Tag images semantically. Use tags like vllm-0.6.3-llama3-8b-cuda12.4 rather than latest to make rollbacks trivial.

Conclusion

Building a Docker image for GPU-accelerated LLM serving is a foundational skill for any team shipping generative AI to production. By starting from a CUDA base image, layering in a proven inference engine like vLLM, and following best practices around version pinning, multi-stage builds, and weight management, you create a portable, reproducible artifact that runs identically on a developer laptop and a multi-GPU cluster. The Dockerfile and serving script in this tutorial provide a solid starting point that you can adapt to different models, engines, and hardware configurations as your serving needs evolve.