How to Containerize Ollama for Kubernetes Deployment

Introduction to Containerizing Ollama for Kubernetes

Ollama is an open-source tool that lets you run large language models (LLMs) such as Llama 3, Mistral, Phi-3, and Gemma locally on your own infrastructure. While Ollama is typically installed as a native binary on a single machine, production deployments often require the scalability, resilience, and orchestration capabilities of Kubernetes. Containerizing Ollama and deploying it on Kubernetes allows you to serve LLMs as a managed service with rolling updates, persistent model storage, horizontal scaling, and integration with the broader cloud-native ecosystem.

Why Containerize Ollama for Kubernetes?

Running Ollama directly on a host works for experimentation, but production workloads demand more. Containerization combined with Kubernetes orchestration delivers several key benefits:

• Reproducibility: A container image pins the Ollama version and runtime dependencies, ensuring identical behavior across development, staging, and production.
• Persistence: Downloaded models can be large (several gigabytes). Kubernetes PersistentVolumeClaims (PVCs) ensure models survive pod restarts and rescheduling.
• Scalability: Kubernetes Deployments let you run multiple Ollama replicas behind a Service, distributing inference load.
• GPU Management: Kubernetes with the NVIDIA Device Plugin can schedule pods on GPU-enabled nodes, critical for running larger models efficiently.
• Observability: Containerized Ollama integrates with Prometheus, Grafana, and standard Kubernetes logging stacks.
• Declarative Management: GitOps workflows with Argo CD or Flux can manage your Ollama deployment alongside other services.

Prerequisites

Before you begin, ensure you have the following:

• A running Kubernetes cluster (version 1.24 or later recommended)
• kubectl installed and configured to access your cluster
• docker or podman for building container images
• A container registry (Docker Hub, GitHub Container Registry, or a private registry)
• Optional: NVIDIA GPU nodes with the NVIDIA Device Plugin installed for GPU acceleration

Building the Ollama Container Image

Ollama publishes official container images on Docker Hub at ollama/ollama. In most cases, you can use the official image directly. However, building a custom image is useful when you want to pre-bake models into the image or add sidecar utilities. Below is a custom Dockerfile that extends the official image and pre-pulls a small model during the build.

Custom Dockerfile

# Dockerfile
FROM ollama/ollama:latest

# Set environment variables
ENV OLLAMA_HOST=0.0.0.0:11434
ENV OLLAMA_KEEP_ALIVE=24h

# Copy a startup script that pre-pulls a model
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

EXPOSE 11434

ENTRYPOINT ["/entrypoint.sh"]

Entrypoint Script

The entrypoint script starts the Ollama server in the background, waits for it to become healthy, pulls a default model, and then brings the server to the foreground so Kubernetes can manage the process lifecycle.

#!/bin/bash
# entrypoint.sh
set -e

# Start Ollama server in the background
ollama serve &
OLLAMA_PID=$!

# Wait for the server to be ready
echo "Waiting for Ollama server to start..."
until curl -sf http://localhost:11434/api/tags >/dev/null 2>&1; do
  sleep 2
done

echo "Ollama server is ready."

# Pre-pull a default model if not already present
DEFAULT_MODEL="${DEFAULT_MODEL:-llama3.2:1b}"
echo "Pulling default model: $DEFAULT_MODEL"
ollama pull "$DEFAULT_MODEL" || echo "Failed to pull $DEFAULT_MODEL, continuing..."

# Bring the background process to the foreground
wait $OLLAMA_PID

Building and Pushing the Image

# Build the image
docker build -t myregistry.example.com/ollama-custom:0.1.0 .

# Push to your registry
docker push myregistry.example.com/ollama-custom:0.1.0

If you prefer to use the official image without customization, you can skip this step entirely and reference ollama/ollama:latest directly in your Kubernetes manifests.

Creating Kubernetes Manifests

We will create a namespace, a PersistentVolumeClaim for model storage, a Deployment, and a Service. We will also include a GPU-enabled variant for clusters with NVIDIA hardware.

Namespace

# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ollama
  labels:
    app.kubernetes.io/name: ollama

PersistentVolumeClaim

Downloaded models are stored under /root/.ollama inside the container. Mounting a PVC at this path ensures models persist across pod restarts, avoiding repeated multi-gigabyte downloads.

# k8s/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-models
  namespace: ollama
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: standard

Deployment (CPU)

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
  labels:
    app.kubernetes.io/name: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ollama
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 11434
              name: http
              protocol: TCP
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0:11434"
            - name: OLLAMA_KEEP_ALIVE
              value: "24h"
            - name: OLLAMA_MAX_LOADED_MODELS
              value: "2"
            - name: OLLAMA_NUM_PARALLEL
              value: "2"
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"
            limits:
              cpu: "4"
              memory: "8Gi"
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
          readinessProbe:
            httpGet:
              path: /api/tags
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /api/tags
              port: http
            initialDelaySeconds: 30
            periodSeconds: 30
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models

Note the use of strategy: Recreate. Because the PVC uses ReadWriteOnce, only one pod can mount it at a time. The Recreate strategy ensures the old pod terminates before a new one starts, preventing volume mount conflicts during updates.

Deployment with GPU Support

If your cluster has NVIDIA GPU nodes and the NVIDIA Device Plugin installed, you can request GPU resources. The official Ollama image includes CUDA runtime support in its GPU-tagged variants.

# k8s/deployment-gpu.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama-gpu
  namespace: ollama
  labels:
    app.kubernetes.io/name: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ollama
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ollama
    spec:
      nodeSelector:
        accelerator: nvidia
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: ollama
          image: ollama/ollama:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 11434
              name: http
              protocol: TCP
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0:11434"
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 16Gi
            requests:
              cpu: "2"
              memory: 8Gi
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
          readinessProbe:
            httpGet:
              path: /api/tags
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models

Service

The Service exposes Ollama within the cluster so other workloads can call the API. For external access, you can add an Ingress or a LoadBalancer Service.

# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ollama
  labels:
    app.kubernetes.io/name: ollama
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: ollama
  ports:
    - name: http
      port: 11434
      targetPort: http
      protocol: TCP

Optional Ingress

# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ollama
  namespace: ollama
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
spec:
  ingressClassName: nginx
  rules:
    - host: ollama.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ollama
                port:
                  name: http

Deploying to Kubernetes

Apply all manifests in order:

# Create the namespace
kubectl apply -f k8s/namespace.yaml

# Create the persistent volume claim
kubectl apply -f k8s/pvc.yaml

# Deploy Ollama (choose CPU or GPU variant)
kubectl apply -f k8s/deployment.yaml
# or
kubectl apply -f k8s/deployment-gpu.yaml

# Create the service
kubectl apply -f k8s/service.yaml

# Optional: create the ingress
kubectl apply -f k8s/ingress.yaml

Verify the deployment:

# Check pod status
kubectl get pods -n ollama

# Check the PVC
kubectl get pvc -n ollama

# View logs
kubectl logs -f deployment/ollama -n ollama

# Port-forward for local testing
kubectl port-forward svc/ollama 11434:11434 -n ollama

Pulling and Testing Models

Once Ollama is running, you can pull models and test inference. Use kubectl exec to run commands inside the pod, or call the API from another pod in the cluster.

Pulling a Model

# Pull a model inside the running pod
kubectl exec -it deployment/ollama -n ollama -- ollama pull llama3.2:1b

# List installed models
kubectl exec -it deployment/ollama -n ollama -- ollama list

Testing the API

# From a pod in the cluster, or via port-forward
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:1b",
  "prompt": "Explain Kubernetes in one sentence.",
  "stream": false
}'

Using a Kubernetes Job to Pre-Pull Models

For automated deployments, you can use a Kubernetes Job to pull models after the Deployment is ready. This is useful in CI/CD pipelines.

# k8s/model-pull-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ollama-pull-models
  namespace: ollama
spec:
  ttlSecondsAfterFinished: 300
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: model-puller
          image: curlimages/curl:latest
          command:
            - /bin/sh
            - -c
            - |
              echo "Waiting for Ollama to be ready..."
              until curl -sf http://ollama:11434/api/tags; do
                sleep 3
              done
              echo "Pulling models..."
              curl -X POST http://ollama:11434/api/pull \
                -H "Content-Type: application/json" \
                -d '{"name": "llama3.2:1b"}'
              curl -X POST http://ollama:11434/api/pull \
                -H "Content-Type: application/json" \
                -d '{"name": "nomic-embed-text"}'
              echo "Done."

Best Practices

Resource Management

Always set CPU and memory requests and limits. LLM inference is memory-intensive; undersized limits cause OOMKilled pods. Monitor actual usage with kubectl top pods and adjust accordingly. For GPU workloads, ensure only one Ollama replica mounts a given GPU unless you explicitly configure multi-instance GPU (MIG) slicing.

Persistent Storage Strategy

Use ReadWriteOnce PVCs with the Recreate deployment strategy for single-replica setups. If you need multiple replicas, either use ReadWriteMany storage (such as NFS or a distributed filesystem) or give each replica its own PVC using a StatefulSet. StatefulSets are generally the better choice for multi-replica Ollama deployments because each pod gets a stable identity and its own volume.

StatefulSet for Multi-Replica Deployments

# k8s/statefulset.yaml (excerpt)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ollama
  namespace: ollama
spec:
  serviceName: ollama
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: ollama
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
              name: http
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
  volumeClaimTemplates:
    - metadata:
        name: models
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi

Security

• Run Ollama as a non-root user where possible by setting securityContext.runAsUser and ensuring the PVC is writable by that user.
• Use NetworkPolicies to restrict which namespaces can reach the Ollama Service.
• Never expose the Ollama API directly to the internet without authentication. Place it behind an API gateway or an authenticating reverse proxy.
• Scan your custom images for vulnerabilities using Trivy or Grype.

Health Checks

The /api/tags endpoint is a lightweight health check that returns the list of installed models. Use it for both readiness and liveness probes. Set a generous initialDelaySeconds because the Ollama server may take time to initialize, especially when loading models from disk on startup.

Model Lifecycle Management

Use the OLLAMA_KEEP_ALIVE environment variable to control how long models stay loaded in memory after the last request. For always-on services, set it to a high value (for example, 24h) to avoid cold-start latency. For cost-sensitive deployments, set it lower to free memory between bursts of traffic.

Observability

Ollama does not expose Prometheus metrics natively, but you can wrap it with a sidecar or use the API to build custom exporters. At minimum, collect logs via the cluster logging stack and monitor pod resource usage. Alert on high memory usage, pod restarts, and failed readiness probes.

Image Tagging

Avoid using latest in production. Pin to a specific version tag such as ollama/ollama:0.3.14 to prevent unexpected upgrades. Maintain a changelog and test new versions in staging before promoting to production.

Conclusion

Containerizing Ollama for Kubernetes bridges the gap between local LLM experimentation and production-grade inference serving. By packaging Ollama in a container, persisting models with PVCs, defining proper resource limits, and leveraging Kubernetes primitives like Deployments, StatefulSets, and Jobs, you get a resilient, scalable, and observable LLM platform. Whether you run on CPU for smaller models or harness NVIDIA GPUs for larger ones, the manifest-based approach described here gives you a reproducible foundation that fits naturally into GitOps workflows and existing cluster operations. Start with a single replica and the official image, then evolve toward StatefulSets, GPU scheduling, and automated model pulling as your workload grows.