← Back to DevBytes
Kubernetes August 19, 2026 14 min read

How to Manage GPU Drivers in Kubernetes AI Clusters

How to Manage GPU Drivers in Kubernetes AI Clusters

Running AI workloads on Kubernetes has become the standard for scaling machine learning training and inference across distributed infrastructure. However, unlike CPU-based workloads, GPU-accelerated containers require a careful orchestration of host-level kernel modules, device plugins, container runtime configuration, and scheduling extensions. Managing GPU drivers in Kubernetes AI clusters is the foundational layer that determines whether your pods can actually access GPU hardware. This tutorial walks through the complete lifecycle of GPU driver management, from installation to monitoring and upgrades.

What Is GPU Driver Management in Kubernetes?

GPU driver management in Kubernetes refers to the set of practices and tooling used to install, configure, maintain, and expose NVIDIA (or other vendor) GPU hardware to containerized workloads. It involves four primary components working together: the host kernel driver, the NVIDIA Container Toolkit, the Kubernetes device plugin, and optionally the GPU operator for declarative management.

The host kernel driver communicates directly with the GPU hardware. The NVIDIA Container Toolkit (formerly nvidia-docker) bridges the host driver with the container runtime, allowing containers to access GPU devices. The Kubernetes NVIDIA Device Plugin registers GPUs with the kubelet so the scheduler can allocate them. The GPU Operator encapsulates all of this into a declarative Kubernetes-native workflow.

Why It Matters

Without proper GPU driver management, your AI cluster will silently fail in ways that are difficult to debug. Pods may schedule on nodes with no functional GPU access, training jobs may fall back to CPU execution, or containers may crash on startup with cryptic library errors. In production AI clusters, driver management matters for several reasons:

Prerequisites and Architecture Overview

Before diving into implementation, ensure your environment meets the baseline requirements. You need a Kubernetes cluster (version 1.24 or later recommended), nodes with NVIDIA GPUs, a compatible Linux kernel, and containerd as the container runtime. The architecture follows a layered model where each component depends on the one below it.

The stack from bottom to top is: physical GPU hardware, host kernel driver, NVIDIA Container Toolkit, container runtime (containerd with nvidia runtime handler), Kubernetes device plugin, and finally your AI workloads. The GPU Operator can automate layers two through five, which is the recommended approach for most production deployments.

Manual Driver Installation Approach

For understanding what happens under the hood, it is valuable to walk through manual driver installation. This approach gives you full control but requires more maintenance effort. Start by installing the NVIDIA driver on each GPU node.

Installing the Host Driver

On an Ubuntu-based node, install the NVIDIA driver using the package manager. The example below installs driver version 535, which supports CUDA 12.1 and later.

# Add the NVIDIA graphics drivers PPA
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update

# Install the driver (version 535 supports CUDA 12.x)
sudo apt-get install -y nvidia-driver-535

# Reboot the node to load the kernel module
sudo reboot

# After reboot, verify the driver is loaded
nvidia-smi

The nvidia-smi command should display a table showing your GPU model, driver version, CUDA version, and current utilization. If this fails, the driver is not correctly installed or the kernel module is not loaded.

Installing the NVIDIA Container Toolkit

Next, install the NVIDIA Container Toolkit, which configures the container runtime to pass GPU devices into containers. This is the critical bridge between the host driver and your pods.

# Configure the repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure containerd to use the nvidia runtime handler
sudo nvidia-ctk runtime configure --runtime=containerd

# Restart containerd to apply changes
sudo systemctl restart containerd

After configuring containerd, you need to add the NVIDIA runtime class to your cluster so pods can request it. Create a RuntimeClass resource:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

Apply this with kubectl apply -f runtimeclass.yaml. Pods that need GPU access will reference this runtime class in their spec.

Deploying the NVIDIA Device Plugin

The device plugin is what makes Kubernetes aware of available GPUs. It runs as a DaemonSet on GPU nodes and registers GPU resources with the kubelet. Deploy it using the official manifest:

# Create a namespace for GPU-related resources
kubectl create namespace gpu-resources

# Deploy the NVIDIA device plugin DaemonSet
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.5/nvidia-device-plugin.yml

# Verify the plugin is running
kubectl get pods -n kube-system -l app=nvidia-device-plugin-daemonset

# Check that GPUs are now visible as allocatable resources
kubectl describe node <gpu-node-name> | grep -A 10 "Allocated resources"

You should see nvidia.com/gpu listed under both capacity and allocatable resources. This is the extended resource that pods request in their resource limits.

Using the GPU Operator (Recommended Approach)

While the manual approach works, it does not scale well across large clusters with frequent node additions. The NVIDIA GPU Operator automates the entire driver and toolkit lifecycle using Kubernetes controllers and DaemonSets. It can install drivers directly on nodes using a containerized driver, eliminating the need for manual host configuration.

Installing the GPU Operator with Helm

Install the GPU Operator using Helm. The operator will deploy DaemonSets for the driver, container toolkit, device plugin, DCGM exporter for monitoring, and additional components for MIG and node labeling.

# Add the NVIDIA Helm repository
helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update

# Create a namespace for the operator
kubectl create namespace gpu-operator

# Install the GPU Operator
helm install --wait gpu-operator \
  nvidia/gpu-operator \
  -n gpu-operator \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set dcgmExporter.enabled=true

# Monitor the operator deployment
kubectl get pods -n gpu-operator -w

Wait until all pods in the gpu-operator namespace reach the Running state. The operator will automatically detect GPU nodes and deploy the appropriate driver containers. You can verify the driver is loaded on a node by exec-ing into the driver container:

# Find the driver container pod on a specific node
DRIVER_POD=$(kubectl get pods -n gpu-operator \
  -l app=nvidia-driver-daemonset \
  --field-selector spec.nodeName=<gpu-node-name> \
  -o jsonpath='{.items[0].metadata.name}')

# Check the driver version from inside the container
kubectl exec -n gpu-operator $DRIVER_POD -- nvidia-smi --query-gpu=driver_version --format=csv,noheader

Configuring Driver Versions with the GPU Operator

One of the key advantages of the GPU Operator is the ability to manage driver versions declaratively. You can specify a particular driver version or let the operator select the latest compatible version. Create a values file to customize the driver configuration:

# gpu-operator-values.yaml
driver:
  enabled: true
  version: "535"
  image:
    repository: nvcr.io/nvidia/driver
    tag: 535.129.03-ubuntu22.04
  rdma:
    enabled: true
  manager:
    env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "true"
      - name: ENABLE_AUTO_DRAIN
        value: "true"

toolkit:
  enabled: true
  image:
    repository: nvcr.io/nvidia/k8s/container-toolkit
    tag: v1.15.0-ubuntu22.04

devicePlugin:
  enabled: true
  config:
    name: device-plugin-config
    default: default
    data:
      default: |-
        version: v1
        sharing:
          timeSlicing:
            resources:
              - name: nvidia.com/gpu
                replicas: 2

dcgmExporter:
  enabled: true
  serviceMonitor:
    enabled: true

Apply this configuration by upgrading the Helm release:

helm upgrade gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  -f gpu-operator-values.yaml

This configuration pins the driver to version 535.129.03, enables RDMA for high-performance networking, configures GPU pod eviction during driver upgrades, and sets up time-slicing with two replicas per GPU for improved utilization.

Running GPU Workloads

Once the driver stack is in place, you can schedule GPU workloads. Pods request GPUs using the nvidia.com/gpu extended resource. Here is a simple example that runs a CUDA validation workload:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
  namespace: default
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
    - name: cuda-vector-add
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu22.04
      command: ["/cuda-vector-add"]
      resources:
        limits:
          nvidia.com/gpu: 1
  nodeSelector:
    nvidia.com/gpu.present: "true"

Apply this and check the logs to confirm the GPU is being used:

kubectl apply -f gpu-test.yaml
kubectl logs gpu-test -f

You should see output indicating the vector addition completed successfully on the GPU. If you see CPU fallback messages or library errors, the driver or toolkit is not properly configured.

Training Workload Example

For a more realistic AI training workload, here is a Job manifest that runs a PyTorch training script with GPU support:

apiVersion: batch/v1
kind: Job
metadata:
  name: pytorch-gpu-training
spec:
  template:
    spec:
      runtimeClassName: nvidia
      restartPolicy: Never
      containers:
        - name: trainer
          image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
          command: ["python", "-c"]
          args:
            - |
              import torch
              print(f"CUDA available: {torch.cuda.is_available()}")
              print(f"Device count: {torch.cuda.device_count()}")
              print(f"Device name: {torch.cuda.get_device_name(0)}")
              x = torch.randn(1000, 1000).cuda()
              y = torch.randn(1000, 1000).cuda()
              z = torch.mm(x, y)
              print(f"Matrix multiplication on GPU completed. Result shape: {z.shape}")
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 8Gi
            requests:
              cpu: 2
              memory: 4Gi
      nodeSelector:
        nvidia.com/gpu.present: "true"
  backoffLimit: 2

GPU Sharing and Partitioning

In many AI clusters, full GPU allocation per pod is wasteful for inference workloads. The GPU Operator supports several sharing mechanisms that depend on driver capabilities.

Time-Slicing Configuration

Time-slicing allows multiple pods to share a single GPU by time-multiplexing execution. This is configured through the device plugin config, as shown earlier. Create a ConfigMap that the device plugin reads:

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  default: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
            devices: ["0", "1"]
  mig-config: |-
    version: v1
    sharing:
      mig:
        strategy: single
    mig:
      devices: all
      strategy: mixed

With this configuration, each physical GPU exposes four virtual GPU resources. Pods can request fractional GPUs:

resources:
  limits:
    nvidia.com/gpu: 1  # This now represents 1/4 of a physical GPU

MIG (Multi-Instance GPU) Configuration

For A100 and H100 GPUs, MIG provides hardware-level isolation between partitions. This is superior to time-slicing for multi-tenant environments. Enable MIG through the GPU Operator:

# Label nodes to enable MIG
kubectl label node <gpu-node-name> nvidia.com/mig.config=all-1g.5gb --overwrite

# The GPU Operator will apply the MIG configuration
kubectl get pods -n gpu-operator -l app=nvidia-mig-manager -w

# Verify MIG devices are available
kubectl exec -n gpu-operator <device-plugin-pod> -- nvidia-smi mig -lgi

After MIG is configured, the device plugin exposes MIG instances as separate GPU resources. Pods request them the same way as regular GPUs, but each MIG instance provides guaranteed memory and compute isolation.

Monitoring GPU Health and Utilization

The GPU Operator deploys DCGM (Data Center GPU Manager) exporter, which exposes Prometheus-compatible metrics. These metrics are essential for understanding cluster health and identifying underutilized or failing GPUs.

Setting Up Prometheus Integration

If you have the Prometheus Operator installed, enable the ServiceMonitor for DCGM exporter:

# Update the GPU Operator to enable ServiceMonitor
helm upgrade gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --set dcgmExporter.serviceMonitor.additionalLabels.release=prometheus

# Verify the DCGM exporter is running and exposing metrics
kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter

# Port-forward to check available metrics
kubectl port-forward -n gpu-operator svc/nvidia-dcgm-exporter 9400:9400

# In another terminal, query the metrics
curl -s http://localhost:9400/metrics | grep -E "DCGM_FI_DEV_GPU_UTIL|DCGM_FI_DEV_MEM_COPY_UTIL|DCGM_FI_DEV_GPU_TEMP"

Key metrics to monitor include DCGM_FI_DEV_GPU_UTIL for GPU compute utilization, DCGM_FI_DEV_MEM_COPY_UTIL for memory bandwidth utilization, DCGM_FI_DEV_GPU_TEMP for temperature, DCGM_FI_DEV_FB_USED for framebuffer memory usage, and DCGM_FI_DEV_POWER_USAGE for power consumption.

Creating Grafana Dashboards

NVIDIA provides pre-built Grafana dashboards for DCGM metrics. Import dashboard ID 12239 from Grafana's dashboard repository. Here is a basic Prometheus alerting rule for GPU health:

groups:
  - name: gpu-alerts
    rules:
      - alert: GPUHighTemperature
        expr: DCGM_FI_DEV_GPU_TEMP > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU temperature is high on {{ $labels.gpu }}"
          description: "GPU {{ $labels.gpu }} on node {{ $labels.node }} has temperature {{ $value }}°C"

      - alert: GPUHighUtilization
        expr: avg by (node) (DCGM_FI_DEV_GPU_UTIL) > 95
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "GPU utilization is consistently high on {{ $labels.node }}"

      - alert: GPUDriverError
        expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "XID error detected on GPU {{ $labels.gpu }}"
          description: "GPU {{ $labels.gpu }} on node {{ $labels.node }} reported an XID error"

Driver Upgrades and Maintenance

Upgrading GPU drivers is one of the most sensitive operations in a Kubernetes AI cluster. A driver upgrade requires unloading the kernel module, which means all GPU workloads on that node must be drained first. The GPU Operator can automate this process.

Coordinated Driver Upgrades

To upgrade the driver version, update the Helm values and let the operator handle the rollout. The operator will cordon and drain nodes, wait for GPU workloads to terminate, install the new driver, and then uncordon the node.

# Update the driver version in values
cat > driver-upgrade.yaml << 'EOF'
driver:
  enabled: true
  version: "550"
  image:
    repository: nvcr.io/nvidia/driver
    tag: 550.54.15-ubuntu22.04
  manager:
    env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "true"
      - name: ENABLE_AUTO_DRAIN
        value: "true"
      - name: DRAIN_NODE_TIMEOUT
        value: "600s"
EOF

# Apply the upgrade
helm upgrade gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  -f driver-upgrade.yaml

# Monitor the upgrade progress
kubectl get nodes -o wide -w
kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -w

The operator performs a rolling upgrade, processing one node at a time. The DRAIN_NODE_TIMEOUT setting controls how long the operator waits for workloads to terminate before forcefully evicting them. Set this based on your longest-running checkpoint-enabled training job.

Handling Stuck Driver Upgrades

Sometimes a driver upgrade gets stuck because a pod refuses to terminate. You can manually intervene:

# Check which pods are blocking the drain
kubectl get pods --all-namespaces --field-selector spec.nodeName=<gpu-node-name> \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'

# If a pod has a PodDisruptionBudget protecting it, temporarily reduce it
kubectl get pdb --all-namespaces

# Force delete a stuck pod if necessary (use with caution)
kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force

# Check driver container logs for errors
kubectl logs -n gpu-operator <driver-pod-name> -c nvidia-driver-ctr

Best Practices

Conclusion

Managing GPU drivers in Kubernetes AI clusters is a multi-layered challenge that spans host kernel modules, container runtime configuration, Kubernetes scheduling, and observability. The NVIDIA GPU Operator has become the de facto standard for handling this complexity, providing a declarative, Kubernetes-native approach to driver lifecycle management. By combining the GPU Operator with proper monitoring through DCGM, thoughtful GPU sharing strategies like MIG, and disciplined upgrade procedures, you can build a robust GPU infrastructure that supports demanding AI training and inference workloads at scale. The key is to treat GPU driver management as a first-class operational concern rather than an afterthought, investing in automation and observability early to avoid costly debugging sessions when production workloads fail to access GPU resources.

— Ad —

Google AdSense will appear here after approval

← Back to all articles