Scaling GKE: From Prototype to Production

Introduction to Scaling GKE

Google Kubernetes Engine (GKE) is one of the most popular managed Kubernetes platforms available today. It abstracts away much of the operational complexity of running Kubernetes clusters, but moving from a prototype to a production-grade deployment requires deliberate planning around scaling, reliability, and cost. In this tutorial, we will walk through the key concepts, configurations, and best practices you need to scale a GKE workload from a small prototype to a robust production system.

What Is GKE Scaling?

Scaling in GKE refers to the ability of your cluster and the applications running on it to handle increasing (or decreasing) load efficiently. Scaling happens at multiple layers:

• Pod-level scaling: Adjusting the number of replicas of a workload using the Horizontal Pod Autoscaler (HPA) or Vertical Pod Autoscaler (VPA).
• Node-level scaling: Adding or removing nodes in the cluster using cluster autoscaler or node auto-provisioning.
• Cluster-level scaling: Designing multi-cluster or regional architectures for high availability and geographic distribution.

Each layer interacts with the others. For example, if your HPA scales up pods but there are no available nodes with sufficient resources, those pods will remain in a Pending state until the cluster autoscaler provisions new nodes.

Why Scaling Matters

A prototype often runs on a single small node with a handful of pods. This works fine for demos, but production traffic introduces several challenges:

• Traffic spikes: Sudden increases in demand can overwhelm under-provisioned clusters.
• Cost efficiency: Over-provisioning to handle peak load wastes money when traffic is low.
• High availability: A single node or zone failure should not take down your application.
• Resource contention: Poorly configured workloads can starve each other of CPU and memory.

Proper scaling configuration addresses all of these concerns by making your cluster responsive to demand while keeping costs under control.

Setting Up a Production-Ready Cluster

Choosing Regional Over Zonal Clusters

The first decision when moving to production is whether to use a zonal or regional cluster. A zonal cluster runs its control plane and nodes in a single zone, while a regional cluster spreads the control plane across three zones and can run nodes in multiple zones. For production workloads, regional clusters are strongly recommended because they survive a single-zone failure.

gcloud container clusters create my-prod-cluster \
  --region=us-central1 \
  --num-nodes=1 \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=10 \
  --enable-autorepair \
  --enable-autoupgrade \
  --machine-type=e2-standard-4

This command creates a regional cluster with autoscaling enabled. The --num-nodes flag specifies the number of nodes per zone in the region, so the cluster starts with three nodes total.

Using Node Auto-Provisioning

Standard cluster autoscaler only scales the node pools you define. Node auto-provisioning goes a step further by automatically creating and deleting node pools based on the requirements of pending pods. This is particularly useful when you have workloads with diverse resource needs.

gcloud container clusters update my-prod-cluster \
  --enable-autoprovisioning \
  --min-cpu=4 \
  --max-cpu=100 \
  --min-memory=16 \
  --max-memory=400 \
  --autoprovisioning-scopes=https://www.googleapis.com/auth/cloud-platform

With this configuration, GKE will automatically provision appropriately sized node pools when pods cannot be scheduled, within the CPU and memory limits you specify.

Horizontal Pod Autoscaling

The Horizontal Pod Autoscaler (HPA) automatically scales the number of pod replicas based on observed metrics such as CPU utilization, memory usage, or custom metrics. It is the primary mechanism for scaling applications in response to load.

Basic HPA Configuration

Before configuring HPA, make sure your deployment has resource requests defined. Without requests, the HPA cannot calculate utilization percentages.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
      - name: api
        image: gcr.io/my-project/api-server:v1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi

Now you can create an HPA that targets this deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

This HPA will maintain between 3 and 50 replicas, scaling up when average CPU exceeds 70% or memory exceeds 80% of the requested amounts.

Custom Metrics for Smarter Scaling

CPU and memory are useful signals, but they do not always correlate with application performance. For example, a queue worker should scale based on queue depth, not CPU. GKE supports custom metrics through Cloud Monitoring.

First, deploy the custom metrics adapter:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml

Then configure your HPA to use a custom metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 2
  maxReplicas: 30
  metrics:
  - type: Pods
    pods:
      metric:
        name: queue_depth
      target:
        type: AverageValue
        averageValue: 10

This configuration scales the worker deployment so that each pod handles approximately 10 queue messages. When the queue depth rises, more pods are added.

Vertical Pod Autoscaling

While HPA adjusts the number of replicas, the Vertical Pod Autoscaler (VPA) adjusts the CPU and memory requests and limits of your pods. This is useful for workloads that cannot be easily horizontally scaled, such as databases or stateful services.

Enable VPA on your cluster:

gcloud container clusters update my-prod-cluster \
  --enable-vertical-pod-autoscaling

Create a VPA resource in recommendation mode first, so you can observe suggestions without applying them:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Off"

After running for a while, inspect the recommendations:

kubectl describe vpa api-server-vpa

Once you are confident in the recommendations, switch to Auto mode to let VPA automatically adjust resource requests. Note that VPA in Auto mode will restart pods to apply new resource settings, so it may not be suitable for all workloads.

Cluster Autoscaler Deep Dive

The cluster autoscaler is enabled by default on GKE when you enable autoscaling on a node pool. It works by watching for pods that are in a Pending state due to insufficient resources and then increasing the size of the node pool to accommodate them.

Configuring Autoscaling Profiles

GKE offers two autoscaling profiles: balanced and optimize-utilization. The balanced profile (default) prioritizes keeping some headroom for rapid scale-up. The optimize-utilization profile packs pods more tightly to reduce costs.

gcloud container clusters update my-prod-cluster \
  --autoscaling-profile=optimize-utilization

Use optimize-utilization for cost-sensitive workloads where you can tolerate slightly slower scale-up times.

Scale-Down Configuration

By default, the cluster autoscaler waits 10 minutes before removing underutilized nodes. You can tune this behavior using the cluster-autoscaler ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-status
  namespace: kube-system
data:
  scale-down-unneeded-time: "5m"
  scale-down-delay-after-add: "10m"
  scale-down-unready-time: "20m"
  max-node-provision-time: "15m"

Reducing the scale-down time helps save costs faster, but be careful not to set it too low in environments with bursty traffic, as you may experience constant node churn.

Pod Disruption Budgets for Safe Scaling

When nodes are scaled down or drained for maintenance, Kubernetes evicts pods. Without protection, this can cause service disruptions. Pod Disruption Budgets (PDBs) ensure that a minimum number of pods remain available during voluntary disruptions.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api-server

This PDB ensures that at least two pods of the api-server deployment are always available. The cluster autoscaler and node drain operations will respect this constraint, potentially delaying node removal until enough pods are available elsewhere.

Best Practices for Production GKE Scaling

Always Define Resource Requests and Limits

Resource requests are essential for the scheduler to place pods correctly and for HPA to calculate utilization. Without them, you lose the ability to reason about capacity. Set limits to prevent runaway pods from consuming all node resources, but be aware that hitting CPU limits can cause throttling.

Use Multiple Node Pools for Workload Isolation

Different workloads have different requirements. A web frontend may need many small nodes, while a machine learning batch job may need GPU-enabled nodes. Use separate node pools with appropriate machine types and taints to keep workloads isolated.

gcloud container node-pools create gpu-pool \
  --cluster=my-prod-cluster \
  --region=us-central1 \
  --machine-type=n1-standard-4 \
  --accelerator=type=nvidia-tesla-t4,count=1 \
  --num-nodes=0 \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=4 \
  --node-taints=nvidia.com/gpu=present:NoSchedule

The taint ensures only pods that explicitly tolerate the GPU taint are scheduled on these nodes, preventing non-GPU workloads from wasting expensive resources.

Implement Graceful Shutdowns

When pods are terminated during scale-down events, they need time to finish processing in-flight requests. Configure graceful shutdown with appropriate termination grace periods and preStop hooks.

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: api
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

The preStop hook gives load balancers time to deregister the pod before the container receives a SIGTERM signal.

Monitor and Alert on Scaling Events

Scaling issues often manifest as pending pods, frequent scale-up and scale-down cycles, or pods being OOM-killed. Set up alerts in Cloud Monitoring for these conditions:

• Pods in Pending state for more than 5 minutes
• HPA unable to scale (reaching min or max replicas)
• Node pool at maximum capacity
• High pod restart counts

Use Spot VMs for Cost Optimization

For fault-tolerant workloads like batch processing or stateless web tiers, Spot VMs can dramatically reduce costs. Combine them with a regular node pool for critical workloads.

gcloud container node-pools create spot-pool \
  --cluster=my-prod-cluster \
  --region=us-central1 \
  --machine-type=e2-standard-4 \
  --spot \
  --num-nodes=0 \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=20

Ensure your workloads running on Spot nodes can handle sudden interruptions by implementing proper graceful shutdown and replication across node pools.

Conclusion

Scaling a GKE cluster from prototype to production is a journey that touches every layer of your infrastructure. By starting with a regional cluster, enabling both horizontal and vertical pod autoscaling, configuring the cluster autoscaler thoughtfully, and following best practices around resource management, pod disruption budgets, and monitoring, you can build a system that handles traffic gracefully while keeping costs predictable. The key is to treat scaling as an ongoing process: start with sensible defaults, observe how your workloads behave under real traffic, and continuously refine your autoscaling configurations based on actual data. With the patterns and configurations covered in this tutorial, you are well equipped to take your GKE workloads from a small prototype to a resilient, production-ready deployment.