Managing GPU Memory Fragmentation in Kubernetes

Introduction to GPU Memory Fragmentation in Kubernetes

As organizations increasingly run machine learning (ML) and high-performance computing (HPC) workloads on Kubernetes, efficient GPU resource management has become a critical concern. GPUs are expensive, power-hungry accelerators, and wasting their compute capacity directly translates to wasted infrastructure budget. One of the most insidious causes of GPU underutilization is memory fragmentation — a condition where GPU memory becomes divided into small, non-contiguous blocks that cannot satisfy larger allocation requests, even when the total free memory would otherwise be sufficient.

In a Kubernetes cluster, this problem is amplified by the orchestration layer. The kube-scheduler makes bin-packing decisions based on coarse-grained resource requests, while the actual GPU memory consumption is dictated by the workload runtime (PyTorch, TensorFlow, CUDA kernels, etc.). The mismatch between scheduling logic and runtime behavior leads to stranded GPUs, failed pods, and poor cluster utilization. This tutorial explains what GPU memory fragmentation is, why it matters in Kubernetes specifically, and how to detect, mitigate, and prevent it using practical tooling and configuration.

What Is GPU Memory Fragmentation?

GPU memory fragmentation occurs when the available VRAM on a GPU becomes split into many small, non-contiguous free regions. Although the sum of free bytes may be large enough to satisfy an allocation request, no single contiguous block is large enough to fulfill it. This is conceptually identical to classic memory fragmentation in operating systems, but it manifests differently because GPU memory allocators (such as the CUDA driver and the caching allocators in PyTorch) have their own allocation strategies and constraints.

There are two forms of fragmentation to be aware of:

External fragmentation: Free memory is scattered between live allocations. A new large tensor or model cannot be placed because no contiguous hole is big enough.
Internal fragmentation: Allocated blocks are larger than requested, leaving unused space inside allocated regions that cannot be reclaimed.

In Kubernetes, fragmentation appears at two layers. First, at the node level, a single GPU may host multiple pods (via technologies like NVIDIA MPS or time-slicing) whose combined memory usage leaves unusable gaps. Second, at the cluster level, the scheduler may place pods on nodes in a way that leaves each node with a small amount of free GPU memory — none of which is individually large enough to schedule a new pod.

Why It Matters in Kubernetes

Kubernetes was originally designed for stateless CPU-bound microservices where memory fragmentation was a node-level concern handled by the Linux kernel. GPUs break this assumption in several ways:

Coarse scheduling granularity: The default NVIDIA device plugin exposes GPUs as integer extended resources (nvidia.com/gpu). Pods request whole GPUs, so partial GPU memory cannot be reused by other pods.
No memory-aware bin packing: The default scheduler does not understand VRAM capacity or current allocation. It only knows whether a GPU is "free" or "in use."
Long-lived workloads: Training jobs and inference servers run for hours or days, locking fragmented memory patterns in place.
Multi-tenant clusters: Different teams submit heterogeneous workloads with varying memory footprints, increasing the chance of incompatible allocation patterns.

The practical impact includes failed pod admissions with OutOfMemory or Insufficient nvidia.com/gpu errors, GPUs sitting idle at 10–20% utilization while new pods remain pending, and unpredictable performance degradation when the CUDA caching allocator falls back to slower allocation paths.

Detecting GPU Memory Fragmentation

Before mitigating fragmentation, you need observability. The combination of nvidia-smi, the NVIDIA DCGM exporter, and Prometheus gives you the data needed to identify fragmented nodes.

Node-Level Inspection with nvidia-smi

The fastest way to inspect a single node is to run nvidia-smi inside a privileged pod or directly on the host. The memory output shows total, used, and free VRAM, but it does not show fragmentation directly. To detect fragmentation, compare the reported free memory against the largest contiguous free block, which you can approximate using a small CUDA probe.

# Run a quick GPU memory probe on a node
kubectl debug node/gpu-node-01 -it --image=nvidia/cuda:12.2.0-base-ubuntu22.04 -- nvidia-smi

# Example output:
# +-----------------------------------------------------------------------------+
# | NVIDIA-SMI 535.104.05   Driver Version: 535.104.05   CUDA Version: 12.2     |
# |-------------------------------+----------------------+----------------------+
# | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC|
# |   0  A100-SXM4-40GB      Off  | 00000000:07:00.0 Off |                    0|
# | 30%  45C    P0    55W / 400W |  28012MiB / 40960MiB |     68%      Default|
# +-------------------------------+----------------------+----------------------+

To measure the largest contiguous free block, use a small Python probe with PyTorch's caching allocator, which exposes fragmentation statistics directly.

import torch

def report_fragmentation():
    if not torch.cuda.is_available():
        print("No CUDA device available")
        return
    for i in range(torch.cuda.device_count()):
        total = torch.cuda.get_device_properties(i).total_memory
        allocated = torch.cuda.memory_allocated(i)
        reserved = torch.cuda.memory_reserved(i)
        free = total - allocated
        frag = reserved - allocated  # reserved but unused by tensors
        print(f"GPU {i}: total={total/1e9:.2f}GB "
              f"allocated={allocated/1e9:.2f}GB "
              f"reserved={reserved/1e9:.2f}GB "
              f"fragmented_reserved={frag/1e9:.2f}GB "
              f"fragmentation_ratio={frag/total*100:.1f}%")

if __name__ == "__main__":
    report_fragmentation()

Cluster-Level Monitoring with DCGM Exporter

For ongoing observability, deploy the NVIDIA DCGM exporter as a DaemonSet. It exposes Prometheus metrics including DCGM_FI_DEV_FB_USED (frame buffer used) and DCGM_FI_DEV_FB_FREE (frame buffer free). You can build a fragmentation alert by tracking nodes where free GPU memory is non-zero but no pod can be scheduled.

# Example Prometheus query: nodes with stranded GPU memory
# Free GPU memory exists but no pod can use it because requests are whole-GPU
sum by (node) (DCGM_FI_DEV_FB_FREE) > 0
and on(node)
count by (node) (kube_pod_container_resource_requests{resource="nvidia.com/gpu"}) == 0

This query identifies nodes where GPUs have free VRAM but no pods are currently requesting GPU resources — a strong signal of stranded capacity that often results from fragmentation and poor bin packing.

Mitigation Strategies

Once you can observe fragmentation, the next step is mitigation. There is no single silver bullet; effective mitigation combines scheduling configuration, device sharing technologies, and workload-level tuning.

1. Enable GPU Memory Sharing with Time-Slicing

The NVIDIA GPU device plugin supports time-slicing, which allows multiple pods to share a single GPU. While time-slicing itself does not partition memory, it enables finer-grained scheduling so that smaller workloads can fill gaps left by larger ones. Configure it by creating a ConfigMap that defines sharing rules.

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-sharing-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
            devices: all

Then update the device plugin DaemonSet to reference this ConfigMap. Each GPU will now be exposed as four virtual GPUs, allowing the scheduler to place up to four pods on a single physical device. This reduces cluster-level fragmentation by enabling denser bin packing.

2. Use NVIDIA MPS for Memory Isolation

Multi-Process Service (MPS) provides a stronger form of sharing with better isolation than time-slicing. MPS allows multiple CUDA processes to share a single GPU context, reducing per-process overhead and improving memory utilization. Configure MPS in the device plugin ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-mps-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    sharing:
      mps:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
            devices: all

MPS is particularly effective for inference workloads with many small models, because it reduces the per-process context memory overhead that contributes to fragmentation.

3. Adopt the NVIDIA GPU Operator with MIG Support

For A100 and H100 GPUs, Multi-Instance GPU (MIG) partitions a single physical GPU into isolated instances with dedicated memory and compute. MIG eliminates fragmentation at the hardware level because each instance has a fixed, contiguous memory allocation that cannot be fragmented by other tenants.

# Enable MIG on a node by labeling it
kubectl label node gpu-node-01 nvidia.com/mig.config=all-1g.5gb --overwrite

# Verify MIG instances are created
kubectl exec -n gpu-operator nvidia-driver-daemonset-xxxxx -- nvidia-smi mig -lgi

MIG configurations like all-1g.5gb create seven 5GB instances per A100, while all-2g.10gb creates three 10GB instances. Choose a partitioning strategy that matches your workload mix. The trade-off is that MIG partitions are relatively static — changing them requires draining the node.

4. Use the Dynamic Resource Allocation (DRA) Framework

Kubernetes 1.26 introduced Dynamic Resource Allocation as an alpha feature, and it has matured in subsequent releases. DRA allows drivers to expose fine-grained, claim-based allocation of GPU resources, including partial GPU memory. The NVIDIA DRA driver can allocate specific fractions of a GPU to a pod, which directly addresses the coarse-granularity problem that causes cluster-level fragmentation.

apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
  name: gpu-claim-1
spec:
  devices:
    requests:
      - name: gpu
        deviceClassName: nvidia.com/gpu
        selectors:
          - cel:
              expression: "device.attributes['nvidia.com/gpu'].memory >= 16000000000"

DRA is still evolving, but it represents the future direction for GPU resource management in Kubernetes and is worth evaluating for greenfield clusters running Kubernetes 1.30 or later.

Workload-Level Best Practices

Cluster-level configuration is only half the battle. The way workloads allocate GPU memory at runtime has a direct impact on fragmentation. The following practices help reduce fragmentation inside the CUDA caching allocator.

Set PYTORCH_CUDA_ALLOC_CONF

PyTorch's caching allocator can be tuned to reduce fragmentation. The max_split_size_mb parameter prevents the allocator from splitting large blocks, which is a common cause of external fragmentation during training.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: training-job
spec:
  replicas: 1
  selector:
    matchLabels:
      app: training-job
  template:
    metadata:
      labels:
        app: training-job
    spec:
      containers:
        - name: trainer
          image: pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: PYTORCH_CUDA_ALLOC_CONF
              value: "max_split_size_mb:512,expandable_segments:True"
          command: ["python", "train.py"]

The expandable_segments:True option is particularly valuable in Kubernetes because it allows the allocator to grow memory segments dynamically rather than pre-allocating large contiguous blocks, which reduces fragmentation when multiple pods share a GPU.

Preallocate and Pin Memory

For inference servers, preallocate the maximum expected memory at startup and reuse buffers. This avoids the allocation and deallocation churn that fragments the caching allocator over time. In PyTorch, you can warm up the allocator with a dummy allocation:

import torch

def warmup_allocator(device=0, target_gb=8):
    """Preallocate memory to warm the caching allocator."""
    chunks = []
    bytes_per_chunk = 256 * 1024 * 1024  # 256MB
    target_bytes = int(target_gb * 1e9)
    allocated = 0
    while allocated < target_bytes:
        chunk = torch.empty(bytes_per_chunk // 4, dtype=torch.float32, device=f"cuda:{device}")
        chunks.append(chunk)
        allocated += bytes_per_chunk
    del chunks
    torch.cuda.empty_cache()
    print(f"Allocator warmed up with {target_gb}GB on GPU {device}")

warmup_allocator(device=0, target_gb=8)

Right-Size Pod Resource Requests

Many teams over-request GPU resources "just in case," which causes the scheduler to reserve entire GPUs for workloads that use only a fraction of their memory. Use metrics from DCGM exporter to right-size requests. If a workload consistently uses 6GB of VRAM, do not request a full 40GB A100 — use MIG or time-slicing to place it on a smaller partition.

Operational Best Practices

Beyond configuration and workload tuning, several operational practices help keep fragmentation under control over the long term.

Regularly drain and rebalance GPU nodes: Schedule periodic node drains during maintenance windows to reset fragmented allocator state. Use the cluster-autoscaler scale-down feature to remove underutilized nodes.
Use node affinity and taints to separate workload classes: Keep large training jobs and small inference jobs on separate node pools. Mixing them on the same GPU node is a primary cause of fragmentation.
Monitor fragmentation ratios over time: Alert when the PyTorch fragmentation ratio exceeds 20% on production inference nodes, indicating the allocator is struggling.
Pin CUDA and driver versions: Fragmentation behavior changes between CUDA versions. Pin versions across the cluster to ensure predictable allocator behavior.
Leverage the Kubernetes Descheduler: The descheduler can evict pods from fragmented nodes so they can be rescheduled more efficiently. Use the RemoveDuplicates and LowNodeUtilization strategies with GPU-aware thresholds.

Descheduler Configuration Example

apiVersion: v1
kind: ConfigMap
metadata:
  name: descheduler-policy
  namespace: kube-system
data:
  policy.yaml: |
    apiVersion: "descheduler/v1alpha1"
    kind: "DeschedulerPolicy"
    strategies:
      LowNodeUtilization:
        enabled: true
        params:
          nodeResourceUtilizationThresholds:
            thresholds:
              nvidia.com/gpu: 20
            targetThresholds:
              nvidia.com/gpu: 80
      RemoveDuplicates:
        enabled: true

This configuration evicts pods from nodes where GPU utilization is below 20%, allowing the scheduler to reconsolidate workloads and free up whole GPUs for larger pending pods.

Conclusion

GPU memory fragmentation in Kubernetes is a multi-layered problem that spans hardware partitioning, cluster scheduling, and runtime allocator behavior. No single tool eliminates it entirely, but a combination of MIG partitioning for hardware-level isolation, time-slicing or MPS for finer-grained sharing, DRA for claim-based allocation, and careful tuning of the PyTorch caching allocator can dramatically reduce stranded capacity. The key is to treat GPU memory as a first-class observability target: instrument it with DCGM exporter, alert on fragmentation ratios, and continuously right-size workload requests against actual consumption. By combining the strategies in this tutorial, platform teams can push GPU utilization from the typical 20–30% range to 70% or higher, turning fragmented, underused accelerators into efficiently packed, cost-effective infrastructure.