Docker with NVIDIA GPU Acceleration: Best Practices and Common Pitfalls

Understanding Docker GPU Acceleration with NVIDIA

Docker containers, by default, run in isolated environments without direct access to host hardware. For GPU-accelerated workloads—deep learning training, scientific simulation, video transcoding—this isolation is a dealbreaker. NVIDIA's container runtime bridges this gap, allowing containers to leverage CUDA-capable GPUs while preserving the portability and reproducibility benefits of containerization. This tutorial walks you through the ecosystem, setup, best practices, and the pitfalls that trip up even experienced developers.

What Is NVIDIA Container Toolkit?

NVIDIA provides a suite of tools collectively known as the NVIDIA Container Toolkit. Its core components include:

nvidia-container-runtime — a modified OCI runtime that mounts NVIDIA drivers and device nodes into the container
nvidia-container-toolkit — a CLI and library that configures the container runtime
libnvidia-container — the low-level library handling driver/device mounting

Together, these allow a Docker container to see the host's GPU hardware, CUDA drivers, and NVIDIA libraries exactly as a bare-metal process would—without baking drivers into the image. This separation means your container images remain portable across different driver versions and GPU architectures.

Why GPU Acceleration in Containers Matters

Running GPU workloads in containers offers compelling advantages:

Reproducibility: Pin exact versions of CUDA, cuDNN, Python packages, and framework builds
Dependency isolation: Multiple projects with conflicting CUDA versions coexist on the same host
CI/CD for ML: Automated testing of model training pipelines on GPU-equipped runners
Resource scheduling: Orchestrators like Kubernetes can allocate specific GPUs to specific containers
Portability: Move workloads seamlessly between on-premises DGX servers and cloud GPU instances

Prerequisites and Installation

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Host Requirements

A Linux host with NVIDIA GPU(s) installed
NVIDIA driver version ≥ 418.81.07 (older versions may work but are unsupported)
Docker Engine ≥ 19.03 (which ships with native GPU support via --gpus flag)
The nvidia-container-toolkit package installed on the host

Step-by-Step Setup

First, verify your driver is loaded and CUDA is visible:

# Check driver version
nvidia-smi

# Verify CUDA toolkit presence
nvcc --version

Install the NVIDIA Container Toolkit. For Debian/Ubuntu systems:

# Add NVIDIA package repositories
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/stable/$distribution/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install and configure
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

For RHEL/CentOS/Fedora systems:

# Add repository
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/stable/$distribution/libnvidia-container.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

# Install via yum or dnf
sudo yum install -y nvidia-container-toolkit
sudo systemctl restart docker

Verify the runtime is registered with Docker:

# Check that nvidia runtime appears in Docker info
docker info | grep -i runtime

# Expected output should list something like:
#  Runtimes: nvidia runc

Windows and WSL2 Considerations

On Windows with WSL2, GPU acceleration works through the NVIDIA Windows driver and the WSL2 kernel driver layer. Install the NVIDIA Container Toolkit inside your WSL2 Linux distribution following the same Linux instructions above. Ensure you have the latest NVIDIA Windows driver that supports WSL2 GPU acceleration (driver version ≥ 510).

Running Your First GPU-Accelerated Container

The Modern --gpus Flag

Docker 19.03+ introduced the --gpus flag, which abstracts away the runtime selection. This is the recommended approach:

# Run a simple CUDA container with all GPUs visible
docker run --gpus all -it nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

# Allocate only specific GPUs (GPU 0 and GPU 2)
docker run --gpus '"device=0,2"' -it nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

Using Docker Compose with GPUs

For multi-container applications, Docker Compose supports GPU resource allocation (requires Compose v2.3+ format with Docker 19.03+):

version: '3.8'
services:
  trainer:
    image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - ./data:/workspace/data
    command: python train.py

The deploy.resources.reservations.devices stanza tells Compose to request NVIDIA GPUs. The count: 2 field requests two GPUs; use count: all or omit it for all available GPUs.

Best Practices for GPU Docker Workloads

1. Choose the Right Base Image

NVIDIA publishes a hierarchy of CUDA images. Understanding them saves gigabytes of disk space and avoids version conflicts:

# Image hierarchy (smallest to largest)
# base — CUDA compiler and runtime, no OS packages, minimal
# runtime — adds cuDNN, minimal OS
# devel — adds compile tools, headers, debugging utilities
# cudnn-runtime — CUDA + cuDNN runtime, good for inference
# cudnn-devel — CUDA + cuDNN development, for building custom ops

# For production inference, use:
FROM nvidia/cuda:12.1-cudnn8-runtime-ubuntu22.04

# For development and building custom kernels:
FROM nvidia/cuda:12.1-cudnn8-devel-ubuntu22.04

Tag conventions: {cuda-version}-{flavour}-{os}. Always pin a specific tag; never use latest in production—it moves across CUDA versions unpredictably.

2. Keep Driver Dependencies Outside the Container

One of the biggest architectural wins of NVIDIA Container Toolkit is that the kernel driver (nvidia.ko) and device nodes are mounted from the host at container start time. Do not install NVIDIA drivers inside your Dockerfile. Your Dockerfile should rely on the CUDA toolkit libraries already present in the NVIDIA base images. This separation lets you upgrade host drivers without rebuilding images.

3. Use Multi-Stage Builds for Production Images

Development images carry compilers, headers, and debugging symbols. For production, use multi-stage builds to copy only runtime artifacts:

# Stage 1: Build custom CUDA kernels or compile extensions
FROM nvidia/cuda:12.1-devel-ubuntu22.04 AS builder
WORKDIR /build
COPY custom_ops/ .
RUN make -j$(nproc)

# Stage 2: Minimal runtime
FROM nvidia/cuda:12.1-runtime-ubuntu22.04
COPY --from=builder /build/libcustom.so /usr/lib/
COPY inference_app/ /app/
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip && \
    rm -rf /var/lib/apt/lists/*
WORKDIR /app
CMD ["python3", "inference.py"]

4. Pin CUDA and cuDNN Versions Explicitly

Framework compatibility matrices are strict. PyTorch, TensorFlow, and JAX each target specific CUDA and cuDNN versions. Mismatched libraries cause runtime errors or silently fall back to CPU execution:

# Example: Pin PyTorch 2.1 with CUDA 12.1 and cuDNN 8
FROM nvidia/cuda:12.1-cudnn8-runtime-ubuntu22.04
RUN pip install torch==2.1.0 --index-url https://download.pytorch.org/whl/cu121

5. Set Environment Variables for GPU Visibility

The NVIDIA Container Toolkit respects these environment variables:

NVIDIA_VISIBLE_DEVICES — comma-separated GPU UUIDs or indices, or all, none, void
NVIDIA_DRIVER_CAPABILITIES — comma-separated list: compute, utility, graphics, video, display
NVIDIA_REQUIRE_* — enforce minimum driver/CUDA version constraints

# Expose only GPU 1 with compute and utility capabilities
docker run --rm \
  -e NVIDIA_VISIBLE_DEVICES=1 \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
  nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

The compute capability enables CUDA and OpenCL. The utility capability enables nvidia-smi and NVML-based monitoring. For EGL/OpenGL rendering, add graphics.

6. Monitor GPU Usage Inside Containers

The nvidia-smi tool works inside containers when utility capability is granted. For programmatic monitoring, use the NVML library bindings or the nvtop tool:

# Install nvtop inside container for real-time monitoring
apt-get install -y nvtop
nvtop

For production monitoring, consider exporting Prometheus metrics via NVIDIA's DCGM (Data Center GPU Manager):

# Run DCGM exporter alongside your workload
docker run -d --gpus all --name dcgm-exporter \
  -p 9400:9400 \
  nvidia/dcgm-exporter:latest

7. Leverage Docker Volumes for Dataset and Model Caching

GPU workloads often involve large datasets and model weights. Bind mounts or named volumes prevent re-downloading data on each container start:

# Named volume for HuggingFace model cache
docker run --gpus all \
  -v hf_cache:/root/.cache/huggingface \
  -v $(pwd)/data:/workspace/data:ro \
  my-training-image

For multi-node setups, consider network filesystems like NFS or dedicated storage orchestrators, but be aware of I/O bottlenecks when many GPUs read from a single storage server.

Common Pitfalls and How to Avoid Them

Pitfall 1: "CUDA Not Found" Despite Correct Setup

Symptom: Container exits with CUDA_ERROR_NO_DEVICE or torch.cuda.is_available() returns False.

Root Causes:

You used the wrong Docker runtime. Always use --gpus all or explicitly specify --runtime=nvidia on older Docker versions
The base image doesn't include CUDA runtime libraries. Verify with ldconfig -p | grep libcuda inside the container
You're running on a CPU-only image tag (e.g., pytorch/pytorch:latest without the cuda suffix)

# Correct invocation
docker run --gpus all -it pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime \
  python3 -c "import torch; print(torch.cuda.device_count())"

Pitfall 2: Driver Version Mismatch

The host driver must support the CUDA version used inside the container. NVIDIA drivers are backward-compatible: a driver that supports CUDA 12.x can run containers built with CUDA 11.x, but not vice versa. Check the minimum required driver for your CUDA version:

# Check driver compatibility
nvidia-smi --query-gpu=driver_version --format=csv,noheader

# Compare against NVIDIA's compatibility matrix:
# CUDA 12.1 requires driver >= 525.60.13
# CUDA 11.8 requires driver >= 450.80.02

If your driver is too old, containers will fail with CUDA_ERROR_SYSTEM_DRIVER_MISMATCH.

Pitfall 3: Missing GPU Capabilities for Specific Workloads

By default, Docker with --gpus grants compute and utility capabilities. For workloads requiring GPU Direct RDMA, video encoding/decoding, or graphics interop, you must explicitly add capabilities:

# For video encoding (NVENC)
docker run --gpus all,capabilities=video \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video \
  nvidia/cuda:12.1-runtime-ubuntu22.04 \
  ffmpeg -hwaccel cuda -i input.mp4 output.mp4

# For GPU Direct RDMA (high-speed interconnects)
docker run --gpus all,capabilities=compute,utility,rdma \
  my_rdma_app

Pitfall 4: All GPUs Visible to All Containers

Without explicit GPU allocation, every container sees every GPU. This causes contention, OOM errors, and unpredictable performance. Use NVIDIA_VISIBLE_DEVICES or the --gpus device filter to isolate GPUs:

# Container A gets GPU 0 and 1
docker run --gpus '"device=0,1"' --name trainer-1 my_image

# Container B gets GPU 2 and 3
docker run --gpus '"device=2,3"' --name trainer-2 my_image

In Kubernetes, use the NVIDIA device plugin which exposes GPUs as schedulable resources, ensuring proper isolation.

Pitfall 5: CUDA Out-of-Memory Due to Shared GPU Contexts

When multiple processes share a GPU, CUDA context creation and memory allocation are not container-aware. A process in container A can exhaust GPU memory, starving container B. Solutions:

Use MIG (Multi-Instance GPU) on supported hardware (A100, H100) to partition GPU memory and compute at the hardware level
Implement application-level memory limits using CUDA_MEMORY_LIMIT environment variables
Use NVIDIA's MPS (Multi-Process Service) for time-slicing compute resources

# Enable MIG on A100 (requires supported driver and GPU mode)
sudo nvidia-smi -i 0 -mig 1
sudo nvidia-smi mig -i 0 -cgi 9,9,9,9  # Create 4 equal instances

# Then assign containers to specific MIG instances
docker run --gpus '"device=0:0"' my_app  # First MIG instance on GPU 0

Pitfall 6: Building Images Without GPU Access

Docker builds (docker build) do not have access to host GPUs. If your Dockerfile runs CUDA code during the build phase (e.g., compiling PTX or running tests), it will fail. Instead:

Pre-compile GPU kernels in a separate step on a GPU-equipped machine
Use docker build only for non-GPU steps; perform GPU-dependent operations at container runtime via entrypoint scripts

# Bad: Dockerfile attempts GPU computation during build
RUN python -c "import torch; torch.cuda.is_available()"  # Will always return False

# Good: Defer GPU checks to entrypoint
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
# entrypoint.sh:
#   python -c "import torch; assert torch.cuda.is_available()"

Pitfall 7: Ignoring CPU-GPU Data Transfer Overhead

Containers don't reduce the latency of PCIe transfers. When datasets are mounted via network volumes or read from slow storage, GPU utilization suffers. Profile your I/O patterns:

# Use NVIDIA Nsight Systems to profile data transfer bottlenecks
nsys profile --stats=true -o profile_output docker run --gpus all my_image python train.py

Pre-stage datasets on local NVMe storage, use /dev/shm for small frequently-accessed tensors, and consider NVIDIA GPUDirect Storage for bypassing CPU memory.

Pitfall 8: Inconsistent Library Resolution Between Host and Container

The NVIDIA Container Toolkit mounts host libraries like libcuda.so, libnvidia-ml.so into the container at /usr/lib/x86_64-linux-gnu/ (or equivalent). If your container image ships its own versions of these libraries, they may conflict. Always use NVIDIA's official CUDA base images which are designed to be compatible with this mounting mechanism. Never manually install NVIDIA drivers or libcuda.so inside a container image.

Advanced: GPU-Enabled Multi-Container Orchestration

Docker Compose with Multiple GPU Services

version: '3.8'
services:
  preprocessor:
    image: my_preprocess:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - shared_data:/data

  trainer:
    image: my_trainer:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]
    volumes:
      - shared_data:/data
    depends_on:
      - preprocessor

volumes:
  shared_data:

Kubernetes with NVIDIA Device Plugin

For production-scale orchestration, deploy the NVIDIA device plugin on your Kubernetes cluster:

# Deploy device plugin daemonset
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/nvidia-device-plugin.yml

# Pod spec requesting GPUs
apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-pod
spec:
  containers:
  - name: trainer
    image: my_trainer:latest
    resources:
      limits:
        nvidia.com/gpu: 2
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: "all"

Security Considerations

GPU access introduces security nuances. A container with GPU access can potentially:

Read GPU memory from other processes (if MIG or proper isolation is not configured)
Access display outputs if graphics capability is granted
Consume all GPU resources affecting co-located workloads

Apply the principle of least privilege: grant only the capabilities needed (compute for CUDA, utility for monitoring). Avoid running GPU containers in privileged mode. When possible, use MIG to enforce hardware-level isolation for multi-tenant GPU environments.

Troubleshooting Quick Reference

Common issues and immediate diagnostic commands:

# Check if Docker recognizes the NVIDIA runtime
docker info | grep -A 5 Runtimes

# Verify GPU visibility inside container
docker run --gpus all nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

# Test CUDA library loading
docker run --gpus all nvidia/cuda:12.1-base-ubuntu22.04 \
  bash -c "ldconfig -p | grep -i cuda"

# Check environment variables passed to container
docker run --gpus all --rm nvidia/cuda:12.1-base-ubuntu22.04 printenv | grep NVIDIA

# Validate driver version compatibility
nvidia-smi --query-gpu=driver_version,name --format=csv

Conclusion

Docker with NVIDIA GPU acceleration transforms how teams develop, ship, and run compute-intensive workloads. The separation of drivers from container images, combined with explicit GPU allocation, gives you reproducibility without sacrificing performance. The key takeaways are straightforward: use the --gpus flag, pin your CUDA base images, never install drivers inside containers, and isolate GPUs explicitly to prevent resource contention. Pay attention to capability flags for specialized workloads like video encoding or RDMA, and leverage MIG for true hardware multi-tenancy when running production services. With these practices in place, your GPU-accelerated container workflows will be reliable, portable, and ready for scale—from a single developer workstation to a Kubernetes cluster with hundreds of GPUs.