Scaling EKS: From Prototype to Production

Amazon Elastic Kubernetes Service (EKS) makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane. However, the configuration that works for a prototype or proof-of-concept is rarely suitable for production workloads. Scaling EKS from a prototype to a production-ready cluster involves careful planning around cluster architecture, node management, autoscaling, networking, security, observability, and cost optimization. This tutorial walks you through the key concepts, practical configurations, and best practices to make that transition successfully.

Why Scaling EKS Matters

In a prototype phase, you typically run a single cluster with a handful of nodes, default networking settings, and minimal observability. As traffic grows and reliability requirements increase, this setup breaks down in several ways. You may encounter IP exhaustion in your VPC subnets, pods stuck in pending states because nodes cannot scale fast enough, noisy neighbor problems on shared nodes, and blind spots when troubleshooting incidents. A production-grade EKS environment must handle sudden traffic spikes, recover gracefully from failures, isolate workloads for security, and provide deep visibility into cluster behavior. Scaling is not just about adding more nodes; it is about building a resilient, observable, and cost-efficient platform.

Cluster Architecture Considerations

Before diving into autoscaling configurations, you need to make foundational architectural decisions. These decisions are expensive to reverse later, so invest time upfront.

Single Cluster vs. Multiple Clusters

For prototypes, a single cluster is fine. For production, consider your isolation requirements. A common pattern is to use separate clusters for different environments (development, staging, production) and sometimes separate clusters per business unit or compliance boundary. Multi-cluster management adds operational overhead but improves blast radius isolation. If you choose a single production cluster, use Kubernetes namespaces, network policies, and RBAC to enforce logical isolation.

Control Plane and Data Plane

EKS manages the control plane for you, and it automatically scales the control plane components based on cluster load. However, you are responsible for the data plane, which consists of your worker nodes. You can choose between managed node groups, self-managed nodes, and Fargate (serverless compute). For most production workloads, managed node groups offer the best balance of control and simplicity.

Cluster Autoscaler vs. Karpenter

The traditional Cluster Autoscaler watches for pending pods and adjusts the size of node groups. Karpenter, developed by AWS, is a newer autoscaler that provisions nodes directly without node groups, offering faster scaling and better bin-packing. For new production clusters, Karpenter is generally recommended.

Setting Up Managed Node Groups

Managed node groups automate the provisioning and lifecycle management of EC2 instances for your EKS cluster. Here is an example of creating a cluster with managed node groups using eksctl:

# cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: production-cluster
  region: us-east-1
  version: "1.29"

managedNodeGroups:
  - name: general-purpose
    instanceTypes: ["m6i.large", "m6i.xlarge"]
    minSize: 3
    maxSize: 10
    desiredCapacity: 3
    volumeSize: 100
    volumeType: gp3
    ssh:
      allow: true
    iam:
      withAddonPolicies:
        autoScaler: true
        cloudWatch: true
    labels:
      role: general
    taints:
      - key: dedicated
        value: general
        effect: NoSchedule
    tags:
      Environment: production
      Team: platform

  - name: compute-intensive
    instanceTypes: ["c6i.2xlarge", "c6i.4xlarge"]
    minSize: 1
    maxSize: 20
    desiredCapacity: 2
    volumeSize: 200
    volumeType: gp3
    labels:
      role: compute
    tags:
      Environment: production
      Team: platform

Apply this configuration with the following command:

eksctl create cluster -f cluster.yaml

This setup creates two node groups: one for general workloads and one for compute-intensive tasks. Having multiple node groups allows you to match instance types to workload requirements and improves cost efficiency.

Implementing Karpenter for Autoscaling

Karpenter is the recommended autoscaling solution for EKS. It provisions nodes in seconds based on pending pod requirements and deprovisions nodes when they are underutilized. Let's walk through installing and configuring it.

Installing Karpenter

First, create the IAM resources and install Karpenter using Helm:

# Create the Karpenter namespace
kubectl create namespace karpenter

# Create the IAM role and service account
eksctl create iamserviceaccount \
  --cluster=production-cluster \
  --namespace=karpenter \
  --name=karpenter \
  --role-name=karpenter-role \
  --attach-policy-arn=arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/KarpenterControllerPolicy \
  --override-existing-serviceaccounts \
  --approve

# Install Karpenter via Helm
helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace karpenter \
  --set serviceAccount.name=karpenter \
  --set clusterName=production-cluster \
  --set clusterEndpoint=https://$(aws eks describe-cluster --name production-cluster --query "cluster.endpoint" --output text) \
  --set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile \
  --version v0.36.0

Creating a NodePool

A NodePool defines the constraints for nodes that Karpenter can provision. Here is a production-ready NodePool configuration:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        nodepool: default
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["m6i.large", "m6i.xlarge", "m6i.2xlarge", "c6i.large", "c6i.xlarge"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      nodeClassRef:
        name: default
      taints:
        - key: example.com/special
          value: "true"
          effect: NoSchedule
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s

Creating an EC2NodeClass

The EC2NodeClass defines how Karpenter creates EC2 instances:

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: production-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: production-cluster
  role: KarpenterNodeRole-production-cluster
  blockDevice:
    deviceName: /dev/xvda
    volume:
      size: 100Gi
      type: gp3
      iops: 3000
      throughput: 125
  tags:
    Environment: production
    ManagedBy: karpenter

Apply both resources:

kubectl apply -f nodepool.yaml
kubectl apply -f ec2nodeclass.yaml

Horizontal Pod Autoscaling

While Karpenter scales your nodes, the Horizontal Pod Autoscaler (HPA) scales the number of pod replicas based on observed metrics. You need both working together for effective scaling. First, ensure the metrics server is installed:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Then create an HPA for your application:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
      selectPolicy: Max

The behavior section is critical for production. The scale-down stabilization window prevents flapping, and the scale-up policies allow rapid response to traffic spikes.

Vertical Pod Autoscaling

The Vertical Pod Autoscaler (VPA) automatically adjusts CPU and memory requests for pods. It is useful for right-sizing workloads but should be used carefully in production. Install it with:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

Here is a VPA configuration that runs in recommendation-only mode, which is safer for production:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 4
          memory: 8Gi

With updateMode: "Off", VPA only provides recommendations without modifying your pods. You can review the recommendations and manually adjust your resource requests over time.

Networking at Scale

Networking is one of the most common bottlenecks when scaling EKS. The default VPC CNI plugin assigns a secondary ENI to each node and uses its IP addresses for pods. This can lead to IP exhaustion in large clusters.

Prefix Delegation

Enable prefix delegation to dramatically increase the number of IPs available per node. With prefix delegation, each ENI receives a /28 prefix (16 IPs) instead of individual IPs:

# Enable prefix delegation on the aws-node DaemonSet
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

# Set the warm prefix target
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Custom Networking

For large clusters, consider custom networking where pods use a separate CIDR range from nodes. This requires additional VPC configuration:

# aws-node configmap patch for custom networking
kubectl patch configmap aws-node -n kube-system --patch '{
  "data": {
    "ENABLE_POD_ENI": "true",
    "CUSTOM_NETWORKING": "true"
  }
}'

# Define a ENIConfig for each subnet
cat <



Security Best Practices

Security becomes more critical as your cluster scales and serves production traffic. Here are the key areas to address.

Pod Security Standards

Enforce pod security standards using the built-in admission controller. Create a baseline policy for your production namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest


Network Policies

Use network policies to restrict pod-to-pod communication. First, ensure you have a CNI that supports network policies, such as the Amazon VPC CNI with network policy support or Calico. Here is a default-deny policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432


IRSA for Pod IAM

Use IAM Roles for Service Accounts (IRSA) instead of attaching policies to node roles. This follows least-privilege principles:

# Create an IAM policy
aws iam create-policy \
  --policy-name S3ReadAccess \
  --policy-document file://s3-read-policy.json

# Create an IAM role with trust relationship for the OIDC provider
eksctl create iamserviceaccount \
  --cluster=production-cluster \
  --namespace=production \
  --name=api-server-sa \
  --role-name=api-server-s3-role \
  --attach-policy-arn=arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/S3ReadAccess \
  --approve


Observability and Monitoring

You cannot scale what you cannot measure. A production EKS cluster needs comprehensive observability covering metrics, logs, and traces.

Installing the AWS Load Balancer Controller

For production ingress, install the AWS Load Balancer Controller which provisions Application Load Balancers:

helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=production-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller


Setting Up Prometheus and Grafana

Install the kube-prometheus-stack for metrics collection and visualization:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=gp3 \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
  --set grafana.persistence.enabled=true \
  --set grafana.persistence.size=10Gi


Key Metrics to Monitor

Set up dashboards and alerts for these critical metrics:


  Node CPU and memory utilization across all node groups
  Pod pending time and scheduling latency
  HPA scaling events and current replica counts
  Karpenter provisioning and deprovisioning events
  API server request latency and error rates
  Pod restart counts and crash loop detection
  Network policy drops and connection tracking
  IP address utilization in VPC subnets


Cost Optimization Strategies

Scaling efficiently means controlling costs. Here are practical strategies.

Using Spot Instances

Spot instances can reduce compute costs by up to 90%. Configure Karpenter to use a mix of on-demand and spot instances, and use the karpenter.sh/capacity-type requirement. For stateful workloads, keep them on on-demand instances:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: stateful
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["r6i.large", "r6i.xlarge"]
      nodeClassRef:
        name: default
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: stateless
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m"]
      nodeClassRef:
        name: default


Resource Requests and Limits

Always set resource requests and limits on your pods. Without requests, the scheduler cannot make good placement decisions, and without limits, a single runaway pod can destabilize a node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: myregistry/api-server:v1.2.0
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20


Disaster Recovery and Upgrades

Production clusters need a disaster recovery strategy. Back up your cluster resources regularly using tools like Velero:

velero install \
  --provider aws \
  --bucket velero-backups-production \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1 \
  --secret-file credentials-velero


Schedule regular backups:

velero schedule create daily-backup \
  --schedule="0 1 * * *" \
  --include-namespaces production \
  --ttl 720h


For cluster upgrades, always follow a blue-green or canary approach. Create a new cluster with the target Kubernetes version, gradually migrate workloads, and decommission the old cluster once all traffic has shifted. Never upgrade a production cluster in place without thorough testing in a staging environment first.

Best Practices Summary


  Use Karpenter instead of the Cluster Autoscaler for faster, more efficient node provisioning
  Run at least three nodes across three Availability Zones for high availability
  Enable prefix delegation to prevent IP exhaustion in large clusters
  Set resource requests and limits on every pod to ensure proper scheduling and prevent resource contention
  Use HPA with custom metrics for application-aware scaling decisions
  Enforce pod security standards and network policies from day one
  Use IRSA for fine-grained IAM access instead of node-level permissions
  Monitor Karpenter events, HPA behavior, and pod scheduling latency as key scaling indicators
  Mix spot and on-demand instances, keeping stateful workloads on on-demand
  Maintain separate clusters for different environments and use infrastructure as code for reproducibility
  Implement a backup strategy with Velero and test your restore process regularly
  Plan cluster upgrades with a blue-green strategy to minimize downtime and risk


Scaling EKS from a prototype to a production environment is a journey that touches every layer of your Kubernetes stack. By investing in proper node management with Karpenter, implementing both horizontal and vertical pod autoscaling, addressing networking capacity proactively, enforcing security through pod security standards and network policies, and building comprehensive observability, you create a foundation that can handle growth gracefully. The transition is not just about handling more traffic; it is about building a platform that is resilient, secure, observable, and cost-efficient. Start with the configurations in this tutorial, monitor your cluster behavior closely, and iterate based on real-world data from your workloads. Remember that production readiness is an ongoing process, not a one-time milestone, and the best scaling strategies evolve as your application and traffic patterns change over time.