Introduction: The Orchestrator Dilemma
Container orchestration has become the backbone of modern application deployment. Among the many options available, Kubernetes and HashiCorp Nomad stand out as two of the most popular choices. While both are powerful tools for managing containerized workloads, they have different philosophies, capabilities, and trade-offs. This tutorial will help you understand when Kubernetes is the right choice over Nomad, with practical examples to guide your decision-making process.
What Is Kubernetes and What Is Nomad?
Kubernetes Overview
Kubernetes (K8s) is an open-source container orchestration platform originally developed by Google and now maintained by the Cloud Native Computing Foundation (CNCF). It provides a comprehensive system for automating deployment, scaling, and management of containerized applications across clusters of machines.
Nomad Overview
Nomad is a workload orchestrator developed by HashiCorp. It is designed to be simpler and more flexible than Kubernetes, capable of scheduling not just containers but also non-containerized applications like Java JARs, binaries, and virtual machines. Nomad is a single binary that combines scheduling and cluster management.
Why This Decision Matters
Choosing the right orchestrator affects your team's productivity, infrastructure costs, operational complexity, and future scalability. Kubernetes has a steeper learning curve but offers a richer ecosystem. Nomad is easier to adopt but may lack some advanced features. Making the wrong choice can lead to unnecessary complexity or limitations that hinder growth.
When to Choose Kubernetes Over Nomad
1. You Need a Rich Ecosystem and Community Support
Kubernetes has the largest ecosystem in the container orchestration space. The CNCF landscape includes hundreds of projects that integrate natively with Kubernetes, from service meshes (Istio, Linkerd) to monitoring (Prometheus, Grafana) to CI/CD (ArgoCD, Tekton).
If your organization relies on or plans to use cloud-native tools, Kubernetes is the safer bet. Most third-party tools are built for Kubernetes first, and Nomad support is often an afterthought or community-maintained.
2. You Require Advanced Networking Features
Kubernetes has a robust networking model with the Container Network Interface (CNI). It supports network policies for fine-grained traffic control between pods, which is critical for security in multi-tenant environments.
# Example Kubernetes NetworkPolicy restricting access to a database pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-access-policy
namespace: production
spec:
podSelector:
matchLabels:
app: postgres
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 5432
Nomad does not have built-in network policies. You would need to rely on external tools like Consul or CNI plugins to achieve similar functionality, which adds complexity.
3. You Need Built-in Auto-scaling
Kubernetes includes the Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and Cluster Autoscaler out of the box. These features automatically adjust resources based on demand.
# Horizontal Pod Autoscaler example
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web-app-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-app
minReplicas: 3
maxReplicas: 50
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
Nomad does not have native auto-scaling for workloads. You would need to build custom solutions using Nomad's API or integrate with tools like HashiCorp Consul and external autoscalers.
4. You Are Building Cloud-Native Microservices
If your architecture is heavily microservices-based with complex service-to-service communication, Kubernetes provides better tooling. Features like Service resources, Ingress controllers, and service discovery are first-class citizens.
# Kubernetes Service and Ingress example
apiVersion: v1
kind: Service
metadata:
name: api-service
spec:
selector:
app: api
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
5. You Need Multi-Cloud or Hybrid Cloud Portability
Kubernetes provides a consistent API across cloud providers. A deployment that works on AWS EKS can be adapted for Google GKE or Azure AKS with minimal changes. This portability is valuable for organizations avoiding vendor lock-in.
6. You Need Advanced Storage Management
Kubernetes has a mature storage system with PersistentVolumes (PV), PersistentVolumeClaims (PVC), and StorageClasses. It supports dynamic provisioning and a wide range of storage backends.
# StorageClass and PVC example
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp3
fsType: ext4
reclaimPolicy: Retain
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 100Gi
7. You Need Fine-Grained RBAC and Security
Kubernetes has a comprehensive Role-Based Access Control (RBAC) system that allows you to define precise permissions for users and service accounts.
# Kubernetes RBAC example
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: development
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: dev-team-pod-reader
namespace: development
subjects:
- kind: Group
name: dev-team
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
8. You Need Rolling Updates and Rollbacks
Kubernetes Deployments provide declarative rolling updates with built-in rollback capabilities. This is essential for maintaining uptime during application updates.
# Deployment with rolling update strategy
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web
image: myregistry/web-app:v2.1.0
ports:
- containerPort: 8080
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
To roll back a deployment in Kubernetes:
# Rollback to previous version
kubectl rollout undo deployment/web-app
# Rollback to a specific revision
kubectl rollout undo deployment/web-app --to-revision=3
# Check rollout history
kubectl rollout history deployment/web-app
9. You Need Custom Resource Definitions (CRDs)
Kubernetes allows you to extend its API with Custom Resource Definitions. This enables operators and controllers that manage complex applications like databases, message queues, and more.
# Custom Resource Definition example
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.example.com
spec:
group: example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine:
type: string
enum: ["postgres", "mysql"]
version:
type: string
replicas:
type: integer
minimum: 1
maximum: 5
storage:
type: string
scope: Namespaced
names:
plural: databases
singular: database
kind: Database
shortNames:
- db
10. You Need Helm for Package Management
Helm is the package manager for Kubernetes, allowing you to define, install, and upgrade complex applications using charts. Nomad does not have an equivalent packaging system with the same level of community adoption.
# Example Helm Chart structure
my-chart/
├── Chart.yaml
├── values.yaml
├── templates/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── ingress.yaml
└── README.md
# Chart.yaml
apiVersion: v2
name: my-web-app
description: A web application deployment
type: application
version: 1.0.0
appVersion: "2.1.0"
# values.yaml
replicaCount: 3
image:
repository: myregistry/web-app
tag: "2.1.0"
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
When Nomad Might Be the Better Choice
For balance, it is important to acknowledge scenarios where Nomad excels:
- Simplicity: Nomad is a single binary with a simpler operational model, making it easier for small teams to adopt and maintain.
- Mixed workloads: If you need to schedule non-containerized workloads (binaries, Java JARs, virtual machines) alongside containers, Nomad handles this natively.
- Small to medium scale: For clusters with fewer nodes and simpler scheduling needs, Nomad's lightweight footprint is advantageous.
- Existing HashiCorp stack: If you already use Terraform, Consul, and Vault, Nomad integrates seamlessly with these tools.
- Lower resource overhead: Nomad's control plane consumes fewer resources compared to Kubernetes, which can be significant for edge deployments or resource-constrained environments.
How to Migrate from Nomad to Kubernetes
If you have decided that Kubernetes is the right choice, here is a practical approach to migrating from Nomad.
Step 1: Assess Your Current Workloads
Review your Nomad job specifications and identify all workloads, their dependencies, and resource requirements.
# Example Nomad job file
job "web-app" {
datacenters = ["dc1"]
group "web" {
count = 3
network {
port "http" {
to = 8080
}
}
task "server" {
driver = "docker"
config {
image = "myregistry/web-app:v2.1.0"
ports = ["http"]
}
resources {
cpu = 500
memory = 512
}
service {
name = "web-app"
port = "http"
check {
type = "http"
path = "/health"
interval = "10s"
timeout = "2s"
}
}
}
}
}
Step 2: Translate to Kubernetes Resources
Convert the Nomad job into Kubernetes manifests. The example above translates to a Deployment and Service.
# Kubernetes equivalent of the Nomad job above
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
labels:
app: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: server
image: myregistry/web-app:v2.1.0
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 500m
memory: 512Mi
readinessProbe:
httpGet:
path: /health
port: 8080
intervalSeconds: 10
timeoutSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
name: web-app
spec:
selector:
app: web-app
ports:
- port: 80
targetPort: 8080
name: http
Step 3: Handle Service Discovery
Nomad typically uses Consul for service discovery. In Kubernetes, service discovery is built-in through DNS. Update your application configuration to use Kubernetes service names instead of Consul.
# Before (Nomad + Consul)
# Application configured to connect to: postgres.service.consul
# After (Kubernetes)
# Application configured to connect to: postgres.production.svc.cluster.local
Step 4: Migrate Secrets Management
If you use Vault with Nomad, you can continue using Vault with Kubernetes through the Vault Kubernetes integration, or you can migrate to Kubernetes Secrets.
# Using Vault with Kubernetes via CSI driver
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-database
spec:
provider: vault
parameters:
vaultAddress: "https://vault.example.com:8200"
roleName: "database-reader"
objects: |
- objectName: "db-password"
secretPath: "secret/data/database"
secretKey: "password"
Step 5: Set Up CI/CD Pipeline
Update your deployment pipeline to use kubectl or Helm instead of the Nomad CLI.
# Example GitHub Actions workflow for Kubernetes deployment
name: Deploy to Kubernetes
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure kubectl
uses: azure/setup-kubectl@v3
with:
version: 'v1.28.0'
- name: Set up Kubeconfig
run: |
mkdir -p $HOME/.kube
echo "${{ secrets.KUBECONFIG }}" | base64 -d > $HOME/.kube/config
- name: Deploy application
run: |
kubectl apply -f k8s/ -n production
kubectl rollout status deployment/web-app -n production
Best Practices When Choosing and Using Kubernetes
1. Start with Managed Kubernetes
If you are new to Kubernetes, start with a managed service like EKS, GKE, or AKS. Managing the control plane yourself adds significant operational burden that is rarely worth it for most organizations.
2. Use Namespaces for Organization
Organize your workloads using namespaces. This provides logical separation and allows you to apply resource quotas and RBAC policies per namespace.
# ResourceQuota per namespace
apiVersion: v1
kind: ResourceQuota
metadata:
name: production-quota
namespace: production
spec:
hard:
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40Gi
persistentvolumeclaims: "10"
count/deployments: "20"
3. Define Resource Requests and Limits
Always specify resource requests and limits for your containers. Without them, the scheduler cannot make informed decisions, and a single pod can consume all node resources.
4. Use Liveness and Readiness Probes
Probes help Kubernetes understand when your application is ready to receive traffic and when it needs to be restarted.
# Comprehensive probe configuration
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
timeoutSeconds: 3
startupProbe:
httpGet:
path: /startup
port: 8080
failureThreshold: 30
periodSeconds: 10
5. Implement GitOps for Deployments
Use tools like ArgoCD or Flux to manage your Kubernetes manifests through Git. This provides auditability, rollback capabilities, and a single source of truth.
# Example ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/k8s-manifests
targetRevision: HEAD
path: production/web-app
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
6. Monitor and Alert Proactively
Set up monitoring with Prometheus and Grafana. Define alerts for critical metrics like pod restarts, high CPU/memory usage, and node health.
# Prometheus ServiceMonitor example
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: web-app-monitor
namespace: production
labels:
release: prometheus
spec:
selector:
matchLabels:
app: web-app
endpoints:
- port: metrics
interval: 30s
path: /metrics
7. Use Pod Disruption Budgets
Ensure availability during voluntary disruptions like node maintenance by defining Pod Disruption Budgets.
# PodDisruptionBudget example
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-app-pdb
namespace: production
spec:
minAvailable: 2
selector:
matchLabels:
app: web-app
8. Keep Manifests DRY with Kustomize or Helm
Avoid duplicating manifests across environments. Use Kustomize for overlay-based customization or Helm for templated deployments.
# Kustomize directory structure
base/
├── deployment.yaml
├── service.yaml
└── kustomization.yaml
overlays/
├── production/
│ ├── deployment-patch.yaml
│ └── kustomization.yaml
└── staging/
├── deployment-patch.yaml
└── kustomization.yaml
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
- ../../base
patches:
- deployment-patch.yaml
Decision Framework: A Quick Checklist
Use this checklist to determine if Kubernetes is the right choice for your organization:
- You have or plan to have more than 50 microservices
- You need advanced networking with network policies
- You require auto-scaling based on metrics
- You want to leverage the CNCF ecosystem (service meshes, operators, GitOps tools)
- You need multi-cloud or hybrid cloud portability
- You have a dedicated platform team to manage Kubernetes
- You need fine-grained RBAC and security controls
- You plan to use stateful workloads with persistent storage
- You need rolling updates with automatic rollbacks
- You want to extend the platform with custom controllers and CRDs
If you answered yes to five or more of these, Kubernetes is likely the right choice. If you answered yes to fewer than three, Nomad may be sufficient for your needs.
Conclusion
Choosing between Kubernetes and Nomad is not about which tool is objectively better, but about which tool fits your organization's needs, scale, and expertise. Kubernetes shines when you need a rich ecosystem, advanced networking, auto-scaling, fine-grained security, and cloud-native patterns. Its large community and extensive tooling make it the default choice for organizations building complex, distributed systems at scale. Nomad remains an excellent option for teams that value simplicity, need to run mixed workloads, or are already invested in the HashiCorp ecosystem. By carefully evaluating your requirements against the criteria outlined in this tutorial, you can make an informed decision that sets your infrastructure up for long-term success. Remember that the best orchestrator is the one that your team can operate effectively while meeting your application's demands for reliability, scalability, and security.