← Back to DevBytes
Developer Tutorial August 13, 2026 12 min read

Troubleshooting GKE: Common Issues and Solutions

Introduction to Troubleshooting GKE

Google Kubernetes Engine (GKE) is one of the most popular managed Kubernetes platforms, but like any distributed system, it can encounter issues that disrupt workloads. Troubleshooting GKE effectively requires understanding the layers of the stack — from the control plane to nodes, pods, networking, and storage — and knowing which tools and commands to reach for at each layer.

This tutorial walks through the most common GKE issues developers and operators face, explains why they happen, and provides practical, copy-paste-ready solutions. Whether you're running GKE Standard or Autopilot clusters, these techniques will help you diagnose and resolve problems faster.

Why Troubleshooting Skills Matter

Production outages are expensive. A misconfigured pod, a node running out of memory, or a broken ingress can take down an entire service. Knowing how to systematically debug GKE reduces mean time to resolution (MTTR), prevents recurring incidents, and builds confidence when deploying at scale. The key is having a repeatable diagnostic workflow rather than guessing at solutions.

Essential Diagnostic Tools and Commands

Before diving into specific issues, make sure you have the right tooling installed and authenticated. The two primary tools are gcloud (Google Cloud CLI) and kubectl (Kubernetes CLI).

# Authenticate with Google Cloud
gcloud auth login
gcloud config set project YOUR_PROJECT_ID

# Get credentials for your GKE cluster
gcloud container clusters get-credentials CLUSTER_NAME \
  --region=REGION \
  --project=YOUR_PROJECT_ID

# Verify kubectl can reach the cluster
kubectl cluster-info

# Check node status
kubectl get nodes -o wide

# Check all pods across namespaces
kubectl get pods -A -o wide

For deeper cluster-level diagnostics, Google provides the kubectl-gke plugin and Cloud Logging integration. You can also use k9s or stern for faster log tailing across multiple pods.

Issue 1: Pod Stuck in CrashLoopBackOff

CrashLoopBackOff is one of the most common pod states. It means the container is starting, crashing, and Kubernetes is backing off before retrying. The root cause is almost always inside the container itself — an application error, missing config, or failed startup command.

Diagnosing the Crash

# Get pod details and events
kubectl describe pod POD_NAME -n NAMESPACE

# View previous container logs (from before the crash)
kubectl logs POD_NAME -n NAMESPACE --previous

# View current container logs
kubectl logs POD_NAME -n NAMESPACE

# Check container exit codes
kubectl get pod POD_NAME -n NAMESPACE \
  -o jsonpath='{.status.containerStatuses[*].lastState}'

Common Causes and Solutions

# Example: Increase memory limits in a deployment
kubectl set resources deployment/APP_NAME -n NAMESPACE \
  --limits=memory=512Mi,cpu=500m \
  --requests=memory=256Mi,cpu=250m

Issue 2: ImagePullBackOff and ErrImagePull

When a pod can't pull its container image, it enters ImagePullBackOff. This is usually a permissions, naming, or availability issue.

Diagnostic Steps

# Describe the pod to see pull errors
kubectl describe pod POD_NAME -n NAMESPACE

# Look for events like:
# "Failed to pull image ... rpc error: code = Unknown"
# "Failed to pull image ... not found"

Common Causes and Solutions

# Grant Artifact Registry reader role to the node service account
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:NODE_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/artifactregistry.reader"

# Alternatively, create an imagePullSecret for private registries
kubectl create secret docker-registry regcred \
  --docker-server=REGION-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat key.json)" \
  --docker-email=your@email.com

Issue 3: Nodes Not Ready

If a node is in a NotReady state, pods scheduled on it may become unreachable. This often indicates resource exhaustion, disk pressure, or network problems on the node itself.

Checking Node Health

# List nodes with conditions
kubectl get nodes -o wide

# Get detailed conditions for a specific node
kubectl describe node NODE_NAME

# Check for node conditions like DiskPressure, MemoryPressure, PIDPressure
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[-1].type}{"\t"}{.status.conditions[-1].status}{"\n"}{end}'

Common Causes and Solutions

# SSH into a GKE node (Standard clusters only)
gcloud compute ssh NODE_NAME --zone=ZONE

# Once inside, check kubelet
sudo systemctl status kubelet
sudo journalctl -u kubelet --no-pager | tail -50

# Check disk usage
df -h

# Clean up unused container images
sudo crictl rmi --prune

For Autopilot clusters, you cannot SSH into nodes. Instead, rely on Cloud Logging with the filter resource.type="k8s_node" to inspect node-level issues.

Issue 4: Service and Networking Problems

Networking issues in GKE can manifest as pods that can't reach each other, services with no endpoints, or external traffic failing to reach your ingress. GKE uses different CNI plugins depending on your configuration — VPC-native (Alias IP) clusters are the default and recommended.

Debugging Service Connectivity

# Check if a service has endpoints
kubectl get endpoints SERVICE_NAME -n NAMESPACE

# If endpoints are empty, the service selector doesn't match any pods
kubectl get pods -n NAMESPACE --show-labels
kubectl get svc SERVICE_NAME -n NAMESPACE -o jsonpath='{.spec.selector}'

# Test DNS resolution from inside a pod
kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 -it --rm -- bash
# Inside the pod:
nslookup SERVICE_NAME.NAMESPACE.svc.cluster.local

# Test connectivity to a service
kubectl run curl-test --image=curlimages/curl -it --rm -- \
  curl -v http://SERVICE_NAME.NAMESPACE.svc.cluster.local:PORT

Common Networking Issues

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Restart CoreDNS if it's stuck
kubectl rollout restart deployment coredns -n kube-system

# Verify firewall rules for the cluster
gcloud compute firewall-rules list --filter="network:VPC_NETWORK_NAME"

Issue 5: PersistentVolumeClaim (PVC) Stuck in Pending

Stateful workloads depend on persistent storage. When a PVC is stuck in Pending, it means Kubernetes couldn't find or create a matching PersistentVolume.

Diagnosing PVC Issues

# Check PVC status
kubectl get pvc -n NAMESPACE

# Describe the PVC for events
kubectl describe pvc PVC_NAME -n NAMESPACE

# Check available storage classes
kubectl get storageclass

# Check if PersistentVolumes exist
kubectl get pv

Common Causes and Solutions

# Set a default storage class
kubectl patch storageclass standard-rwo -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# Example PVC with explicit storage class
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard-rwo
  resources:
    requests:
      storage: 20Gi
EOF

Issue 6: Cluster Autoscaler Not Scaling

The cluster autoscaler automatically adds or removes nodes based on pod scheduling needs. When it's not working, pods stay in Pending state even though resources are available in the cloud.

Checking Autoscaler Status

# Check for pending pods (unschedulable)
kubectl get pods -A --field-selector=status.phase=Pending

# View scheduler events
kubectl get events -A --field-selector reason=FailedScheduling

# Check autoscaler logs (Standard clusters)
kubectl logs -n kube-system deployment/cluster-autoscaler --tail=50

# Check node pool autoscaling configuration
gcloud container clusters describe CLUSTER_NAME \
  --region=REGION \
  --format="value(nodePools[].name, nodePools[].autoscaling)"

Common Causes and Solutions

# Enable autoscaling on an existing node pool
gcloud container clusters update CLUSTER_NAME \
  --region=REGION \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=10 \
  --node-pool=NODE_POOL_NAME

# Check why a specific pod is pending
kubectl describe pod PENDING_POD_NAME -n NAMESPACE | grep -A 20 Events

Issue 7: IAM and Authentication Errors

GKE integrates with Google Cloud IAM. Misconfigured service accounts or expired tokens can cause workloads to fail when accessing GCP services like Cloud Storage, Pub/Sub, or Cloud SQL.

Diagnosing IAM Issues

# Check which service account a pod is using
kubectl get pod POD_NAME -n NAMESPACE \
  -o jsonpath='{.spec.serviceAccountName}'

# Check the Kubernetes service account's annotation (for Workload Identity)
kubectl get sa KSA_NAME -n NAMESPACE -o yaml

# Verify Workload Identity is enabled on the cluster
gcloud container clusters describe CLUSTER_NAME \
  --region=REGION \
  --format="value(workloadIdentityConfig)"

Common Causes and Solutions

# Set up Workload Identity
# 1. Create or use a Google service account (GSA)
gcloud iam service-accounts create GSA_NAME

# 2. Grant the GSA necessary roles
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:GSA_NAME@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

# 3. Allow the Kubernetes service account (KSA) to impersonate the GSA
gcloud iam service-accounts add-iam-policy-binding \
  GSA_NAME@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:YOUR_PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

# 4. Annotate the KSA
kubectl annotate sa KSA_NAME -n NAMESPACE \
  iam.gke.io/gcp-service-account=GSA_NAME@YOUR_PROJECT_ID.iam.gserviceaccount.com

Issue 8: High Resource Usage and Performance Degradation

Even when pods are running, performance can degrade due to resource contention, noisy neighbors, or inefficient queries. Monitoring and metrics are essential for catching these issues early.

Monitoring Resource Usage

# Check resource usage across all pods
kubectl top pods -A

# Check resource usage per node
kubectl top nodes

# Get detailed resource requests vs limits
kubectl get pods -n NAMESPACE -o custom-columns=\
NAME:.metadata.name,\
CPU_REQ:.spec.containers[*].resources.requests.cpu,\
CPU_LIM:.spec.containers[*].resources.limits.cpu,\
MEM_REQ:.spec.containers[*].resources.requests.memory,\
MEM_LIM:.spec.containers[*].resources.limits.memory

Using Cloud Monitoring for GKE

GKE integrates with Google Cloud Monitoring (formerly Stackdriver). You can query metrics programmatically or use the Cloud Console dashboard. For custom application metrics, use Prometheus and Grafana or Google Managed Prometheus.

# Enable Cloud Operations (if not already enabled during cluster creation)
gcloud container clusters update CLUSTER_NAME \
  --region=REGION \
  --enable-dataplane-observability

# Query GKE metrics using the Monitoring API
gcloud monitoring metrics list \
  --filter="metric.type=starts_with(\"kubernetes.io/\")" \
  --format="value(metric.type)"

Common Performance Issues

Best Practices for GKE Troubleshooting

1. Build a Diagnostic Runbook

Create a standardized checklist for your team. A typical workflow is: check pod status, check events, check logs, check node status, check networking, check IAM. Following the same order every time prevents skipping critical steps.

2. Use Probes Correctly

Always define liveness and readiness probes. Readiness probes prevent traffic from reaching pods that aren't ready, while liveness probes restart unhealthy containers. Without probes, Kubernetes has no way to know if your application is actually serving traffic.

# Example probe configuration
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

3. Structure Logs for Searchability

Output logs in JSON format so Cloud Logging can parse fields. Include correlation IDs, pod names, and timestamps. Avoid logging secrets or sensitive data.

4. Set Up Alerts Before You Need Them

Configure alerting policies in Cloud Monitoring for key signals: high pod restart counts, node CPU utilization above 80%, PVC usage above 90%, and failed scheduling events. Alerts should be actionable — avoid alert fatigue by tuning thresholds.

# Example: Create an alert policy for high pod restarts
cat <<'EOF' > alert.json
{
  "displayName": "High Pod Restart Count",
  "conditions": [{
    "displayName": "Pod restarts > 5 in 10 minutes",
    "conditionThreshold": {
      "filter": "resource.type=\"k8s_pod\" metric.type=\"kubernetes.io/pod/restart_count\"",
      "comparison": "COMPARISON_GT",
      "thresholdValue": 5,
      "duration": "600s"
    }
  }],
  "combiner": "OR"
}
EOF

gcloud alpha monitoring policies create --policy-from-file=alert.json

5. Use Resource Requests and Limits Consistently

Every pod should have resource requests set. Without requests, the scheduler can't make informed placement decisions, leading to node overcommit and unpredictable performance. Limits are optional but recommended for memory to prevent OOM kills from affecting other pods.

6. Keep Clusters Updated

Google automatically manages the control plane, but node pools need manual or scheduled upgrades. Running outdated versions leads to security vulnerabilities and missing bug fixes. Use release channels (Rapid, Regular, Stable) to automate upgrade cadence.

# Check current cluster version and release channel
gcloud container clusters describe CLUSTER_NAME \
  --region=REGION \
  --format="value(currentMasterVersion, releaseChannel.channel)"

# Upgrade a node pool
gcloud container clusters upgrade CLUSTER_NAME \
  --region=REGION \
  --node-pool=NODE_POOL_NAME \
  --cluster-version=VERSION

7. Leverage GKE Diagnostics

Google provides built-in diagnostic tools in the Cloud Console under Kubernetes Engine > Clusters > [Your Cluster] > Diagnostics. This runs automated checks for common misconfigurations and can save significant debugging time.

Conclusion

Troubleshooting GKE effectively comes down to understanding the Kubernetes stack layer by layer and having a systematic approach to diagnosis. Most issues fall into a handful of categories — pod crashes, image pulls, node health, networking, storage, autoscaling, and IAM — and each has well-established diagnostic commands and solutions. By combining kubectl and gcloud with Cloud Monitoring and structured logging, you can quickly pinpoint root causes and restore service. The best defense against production issues is prevention: set resource requests and limits, configure probes, enable alerts, keep clusters updated, and document your runbooks. With these practices in place, your team will be well-equipped to handle whatever GKE throws your way.

— Ad —

Google AdSense will appear here after approval

← Back to all articles