Scaling AKS: From Prototype to Production

Scaling AKS: From Prototype to Production

Azure Kubernetes Service (AKS) makes it remarkably easy to spin up a cluster and deploy a containerized application in minutes. However, the configuration that works for a weekend prototype is rarely the configuration you want running in production. Scaling AKS from a prototype to a production-grade system involves thoughtful planning around node pools, autoscaling, networking, availability zones, and observability. This tutorial walks you through the journey step by step, with practical commands and manifests you can apply immediately.

Why Scaling AKS Matters

In a prototype, you typically run a single node pool with a small VM size, no autoscaling, and a basic load balancer. This setup is fine for validating functionality, but it breaks down under real-world conditions. Production workloads demand elasticity to handle traffic spikes, high availability to survive failures, cost efficiency to avoid paying for idle resources, and isolation to separate workloads with different performance characteristics. Scaling AKS properly addresses all of these concerns.

The key areas you need to address when moving from prototype to production include:

• Cluster autoscaling and node pool management
• Availability zone distribution for fault tolerance
• Pod-level resource requests and limits
• Horizontal and vertical pod autoscaling
• Network and ingress scaling
• Monitoring and alerting

Starting Point: The Prototype Cluster

Let's begin by looking at a typical prototype cluster creation command. This is what most developers start with:

# Prototype cluster - simple but not production ready
az aks create \
  --resource-group my-prototype-rg \
  --name my-prototype-aks \
  --node-count 2 \
  --node-vm-size Standard_D2s_v3 \
  --generate-ssh-keys

This creates a cluster with two nodes and no autoscaling. If traffic increases, your pods will sit in a Pending state because there is no capacity. If a node fails, you lose half your capacity with no automatic recovery. Let's transform this into a production-ready setup.

Building a Production-Ready Cluster

Creating a Cluster with Autoscaling and Availability Zones

The first step is to create a cluster with the cluster autoscaler enabled and nodes spread across availability zones. Availability zones are physically separate locations within an Azure region, each with independent power, cooling, and networking. Distributing nodes across zones ensures that a single zone failure does not take down your entire cluster.

# Production cluster with autoscaling and zone redundancy
az aks create \
  --resource-group my-production-rg \
  --name my-production-aks \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 10 \
  --zones 1 2 3 \
  --load-balancer-sku standard \
  --enable-managed-identity \
  --enable-addons monitoring \
  --node-osdisk-type Ephemeral \
  --max-pods 110 \
  --generate-ssh-keys

Let's break down the important flags in this command:

• --enable-cluster-autoscaler: Enables automatic scaling of node count based on pod resource demands.
• --min-count and --max-count: Define the bounds within which the autoscaler can add or remove nodes.
• --zones 1 2 3: Spreads the initial nodes across three availability zones.
• --load-balancer-sku standard: Required for zone-redundant load balancers and Standard SKU public IPs.
• --enable-addons monitoring: Enables Azure Monitor for containers out of the box.
• --node-osdisk-type Ephemeral: Uses ephemeral OS disks for faster node provisioning and lower cost.

Adding a Dedicated System Node Pool

In production, you should separate system pods (like CoreDNS, kube-proxy, and the metrics server) from your application workloads. AKS supports this through system node pools. You can taint user node pools so that only your application pods land on them, while system pods stay on the system pool.

# Add a user node pool for application workloads
az aks nodepool add \
  --resource-group my-production-rg \
  --cluster-name my-production-aks \
  --name userpool \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 20 \
  --zones 1 2 3 \
  --mode User \
  --node-taints CriticalAddonsOnly=true:NoSchedule \
  --labels workload=application \
  --max-pods 110

Now update the original system pool to ensure only critical system pods run there:

# Taint the system node pool
az aks nodepool update \
  --resource-group my-production-rg \
  --cluster-name my-production-aks \
  --name nodepool1 \
  --node-taints CriticalAddonsOnly=true:NoSchedule

Configuring Pod Resource Requests and Limits

The cluster autoscaler makes decisions based on pod resource requests, not actual usage. If your pods do not specify resource requests, the scheduler cannot make intelligent placement decisions, and the autoscaler cannot determine when to add nodes. Every production deployment must include resource requests and limits.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      nodeSelector:
        workload: application
      containers:
      - name: web-app
        image: myregistry.azurecr.io/web-app:v1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

Notice the nodeSelector field. This ensures the pod is scheduled only on nodes with the workload: application label, which we applied to the user node pool. This keeps application workloads off the system pool.

Horizontal Pod Autoscaling

While the cluster autoscaler handles node-level scaling, the Horizontal Pod Autoscaler (HPA) handles pod-level scaling. The HPA automatically increases or decreases the number of pod replicas based on observed CPU utilization, memory usage, or custom metrics. Combined with the cluster autoscaler, this gives you a two-tier scaling system: pods scale first, and nodes scale when there is not enough capacity for the new pods.

Before you can use CPU and memory-based autoscaling, you need the Metrics Server deployed in your cluster. AKS clusters created with recent versions include the Metrics Server by default. You can verify this with:

kubectl get pods -n kube-system | grep metrics-server

If it is not present, you can install it:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Now create an HPA for your deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
      - type: Pods
        value: 4
        periodSeconds: 30
      selectPolicy: Max

The behavior section is important for production. The stabilizationWindowSeconds for scale-down prevents flapping by requiring metrics to remain below the threshold for five minutes before scaling down. The scale-up policies allow the HPA to react quickly to traffic spikes by doubling the replica count every 30 seconds or adding 4 pods at a time, whichever is greater.

Vertical Pod Autoscaling

Sometimes you do not need more pods; you need bigger pods. The Vertical Pod Autoscaler (VPA) automatically adjusts CPU and memory requests and limits for your pods based on historical usage. This is useful for workloads that are not horizontally scalable, such as single-instance databases or stateful services.

Install the VPA component:

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Install VPA
./hack/vpa-up.sh

# Verify installation
kubectl get pods -n kube-system | grep vpa

Create a VPA resource for a stateful workload:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: database-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: StatefulSet
    name: postgres-db
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: postgres
      minAllowed:
        cpu: 250m
        memory: 512Mi
      maxAllowed:
        cpu: 2
        memory: 4Gi
      controlledResources: ["cpu", "memory"]

Be cautious with updateMode: Auto because it will evict and recreate pods to apply new resource settings. For critical workloads, consider using updateMode: Off first to collect recommendations, then manually apply them after review.

Scaling the Ingress Layer

As traffic grows, your ingress controller becomes a potential bottleneck. The default NGINX ingress controller deployment often runs a single replica, which is unacceptable in production. You need to scale the ingress controller itself and distribute it across availability zones.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
      containers:
      - name: controller
        image: registry.k8s.io/ingress-nginx/controller:v1.9.0
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "1"
            memory: "512Mi"

The topologySpreadConstraints field ensures that the ingress controller pods are evenly distributed across availability zones. If one zone goes down, the other two zones still have ingress pods to handle traffic.

Configuring Pod Disruption Budgets

When nodes are scaled down or drained for maintenance, Kubernetes evicts pods. Without a Pod Disruption Budget (PDB), too many pods could be evicted simultaneously, causing an outage. A PDB ensures that a minimum number of pods remain available during voluntary disruptions.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-app

This PDB ensures that at least two pods of the web-app deployment are always available. If a node drain would bring the count below two, the eviction request will be denied until new pods are scheduled and ready.

Monitoring and Alerting

Scaling decisions should be driven by data, not guesswork. Azure Monitor for containers, which we enabled during cluster creation, provides out-of-the-box dashboards for node CPU, memory, and pod status. However, you should also set up custom alerts for critical conditions.

Create an alert rule for high node CPU usage using the Azure CLI:

# Create an action group for notifications
az monitor action-group create \
  --name high-cpu-action-group \
  --resource-group my-production-rg \
  --short-name highcpu

# Create a metric alert for node CPU percentage
az monitor metrics alert create \
  --name "High-Node-CPU" \
  --resource-group my-production-rg \
  --scopes "/subscriptions/{subscription-id}/resourceGroups/my-production-rg/providers/Microsoft.ContainerService/managedClusters/my-production-aks" \
  --condition "avg node_cpu_usage_percentage > 80" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action-groups "/subscriptions/{subscription-id}/resourceGroups/my-production-rg/providers/microsoft.insights/actionGroups/high-cpu-action-group" \
  --description "Alert when average node CPU exceeds 80% for 5 minutes"

You should also monitor for pods in a Pending state, which indicates that the cluster needs more capacity:

# Query for pending pods using kubectl
kubectl get pods --all-namespaces \
  --field-selector=status.phase=Pending

# Set up a Prometheus alert rule (if using Prometheus)
# pending-pods-alert.yaml
groups:
- name: pod-status
  rules:
  - alert: PodsPending
    expr: kube_pod_status_phase{phase="Pending"} > 0
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pods are in Pending state"
      description: "{{ $value }} pods have been Pending for more than 10 minutes in cluster {{ $labels.cluster }}."

Best Practices for Production AKS Scaling

Right-Size Your Node VMs

Choosing the right VM size for your node pools has a significant impact on cost and performance. Nodes that are too small lead to fragmentation, where there is enough total CPU and memory but not enough on any single node to schedule a pod. Nodes that are too large waste resources when the cluster scales down. A good starting point for general-purpose workloads is the D-series VMs with at least 4 vCPUs and 16 GB of RAM.

Use Multiple Node Pools for Different Workloads

Different workloads have different requirements. A CPU-intensive API service needs compute-optimized VMs, while an in-memory cache needs memory-optimized VMs. Create separate node pools for each workload type and use taints, tolerations, and node selectors to ensure pods land on the right nodes.

# Add a memory-optimized node pool for caching workloads
az aks nodepool add \
  --resource-group my-production-rg \
  --cluster-name my-production-aks \
  --name mempool \
  --node-count 2 \
  --node-vm-size Standard_E4s_v3 \
  --enable-cluster-autoscaler \
  --min-count 2 \
  --max-count 10 \
  --zones 1 2 3 \
  --mode User \
  --node-taints workload=memory:NoSchedule \
  --labels workload=memory \
  --max-pods 60

Enable Cluster Autoscaler Profile Tuning

The cluster autoscaler has several configuration options that affect how quickly it responds to scaling events. You can tune these settings to match your workload patterns:

# Update cluster autoscaler profile
az aks update \
  --resource-group my-production-rg \
  --name my-production-aks \
  --cluster-autoscaler-profile \
    scan-interval=10s \
    scale-down-delay-after-add=10m \
    scale-down-delay-after-delete=10s \
    scale-down-delay-after-failure=3m \
    scale-down-unneeded-time=10m \
    scale-down-unready-time=20m \
    balance-similar-node-groups=true \
    expander=least-waste

The scan-interval determines how often the autoscaler checks for pending pods or underutilized nodes. The scale-down-delay-after-add prevents the autoscaler from immediately scaling down after adding a node, giving workloads time to stabilize. The expander=least-waste strategy selects the node group that would waste the least resources when scaling up.

Implement Graceful Shutdown

When pods are terminated during scale-down events, they need time to finish processing in-flight requests. Configure graceful shutdown with proper termination grace periods and preStop hooks:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: web-app
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - "sleep 15 && nginx -s quit"

Use Spot Node Pools for Cost-Effective Scaling

For fault-tolerant, batch-processing, or development workloads, Azure Spot VMs offer significant cost savings (up to 90%) in exchange for the possibility of eviction. Add a spot node pool and use tolerations to allow specific workloads to run on it:

# Add a spot node pool
az aks nodepool add \
  --resource-group my-production-rg \
  --cluster-name my-production-aks \
  --name spotpool \
  --node-count 1 \
  --node-vm-size Standard_D4s_v3 \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 10 \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --zones 1 2 3 \
  --mode User \
  --node-taints kubernetes.azure.com/scalesetpriority=spot:NoSchedule \
  --labels workload=spot \
  --max-pods 110

Then add a toleration to workloads that can run on spot nodes:

spec:
  template:
    spec:
      tolerations:
      - key: "kubernetes.azure.com/scalesetpriority"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"
      nodeSelector:
        workload: spot

Validating Your Scaling Configuration

After configuring your production cluster, you should validate that scaling works as expected. Create a load test deployment that can generate enough traffic to trigger the HPA and cluster autoscaler:

# Deploy a load generator
kubectl run load-generator \
  --image=busybox \
  --restart=Never \
  -- /bin/sh -c "while true; do wget -q -O- http://web-app.production.svc.cluster.local:8080; done"

# Watch the HPA in action
kubectl get hpa -n production -w

# Watch for new nodes being added
kubectl get nodes -w

You should see the HPA increase the replica count as CPU utilization rises. Once the existing nodes reach capacity, the cluster autoscaler will add new nodes to accommodate the additional pods. When you stop the load generator, both the HPA and the cluster autoscaler will eventually scale back down after the stabilization windows expire.

Conclusion

Scaling AKS from a prototype to a production-ready system is a multi-faceted process that goes far beyond simply adding more nodes. It requires a thoughtful combination of cluster autoscaling for node-level elasticity, horizontal and vertical pod autoscaling for workload-level responsiveness, availability zone distribution for fault tolerance, proper resource requests and limits for efficient scheduling, and comprehensive monitoring to make data-driven decisions. By following the practices outlined in this tutorial—separating system and user node pools, tuning autoscaler profiles, implementing pod disruption budgets, scaling your ingress layer, and leveraging spot node pools for cost optimization—you can build an AKS environment that handles production traffic reliably and cost-effectively. Remember that scaling is not a one-time configuration task; you should continuously monitor your cluster's performance, review autoscaling behavior, and adjust your configuration as your workloads evolve over time.