Introduction to EKS Troubleshooting
Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that simplifies running Kubernetes on AWS without needing to install and operate your own control plane. While EKS removes much of the operational overhead of managing Kubernetes, it does not eliminate the complexity of running distributed containerized applications. Clusters can still experience node failures, networking issues, IAM permission problems, and application-level errors that require careful diagnosis.
This tutorial walks through the most common issues developers and operators encounter when running EKS clusters, explains why these problems occur, and provides practical, copy-paste-ready commands and solutions. By the end, you will have a structured troubleshooting methodology and a toolkit of commands you can use to quickly identify and resolve EKS issues.
Why EKS Troubleshooting Matters
When production workloads run on EKS, every minute of downtime can translate into lost revenue, degraded user experience, or compliance violations. Unlike self-managed Kubernetes, EKS abstracts the control plane, which means many traditional debugging techniques must be adapted. You cannot SSH into the API server, and many control plane logs are only available through optional integrations. Understanding the specific failure modes of EKS helps you:
- Reduce mean time to resolution (MTTR) during incidents
- Prevent recurring issues through proactive monitoring
- Build confidence in your cluster operations
- Avoid unnecessary AWS support tickets by self-diagnosing common problems
Prerequisites and Tooling
Before diving into specific issues, ensure you have the right tools installed and configured. Most EKS troubleshooting relies on a combination of AWS CLI, kubectl, and a few specialized utilities.
Essential Tools
# Install or update AWS CLI
pip install --upgrade awscli
# Install kubectl (Linux example)
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/kubectl
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
# Install eksctl for cluster management
curl --silent --location "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
# Verify AWS authentication
aws sts get-caller-identity
Configuring kubectl for EKS
One of the most common stumbling blocks is an improperly configured kubeconfig. Use the AWS CLI to generate the correct context for your cluster:
aws eks update-kubeconfig \
--region us-east-1 \
--name my-eks-cluster
# Verify connectivity
kubectl get nodes
If this command fails, verify that your IAM user or role has the eks:DescribeCluster permission and that the cluster name and region are correct.
Issue 1: Nodes Not Joining the Cluster
One of the most frequent EKS problems is newly launched worker nodes failing to register with the control plane. When you run kubectl get nodes, you see fewer nodes than expected, or nodes remain in a NotReady state.
Diagnosing the Problem
Start by checking the node status and any events that might explain the failure:
# List all nodes and their status
kubectl get nodes -o wide
# Check for node-related events
kubectl get events --sort-by='.lastTimestamp' -A
# Describe a specific problematic node
kubectl describe node <node-name>
If the node is not visible at all, the kubelet on the EC2 instance likely cannot communicate with the EKS API server. SSH into the node (or use Session Manager) and inspect the kubelet logs:
# Connect to the node via AWS Systems Manager Session Manager
aws ssm start-session --target <instance-id>
# Once connected, check kubelet logs
sudo journalctl -u kubelet --no-pager -n 100
# Check the kubelet configuration
sudo cat /etc/kubernetes/kubelet/kubelet-config.json
# Verify the instance can reach the EKS API server
curl -k https://<your-eks-endpoint>/healthz
Common Root Causes and Solutions
1. IAM Role Missing or Incorrect Permissions: The EC2 instance profile attached to the node must have the AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryReadOnly policies. If using a custom role, verify it with:
# Check the instance profile attached to the node
aws ec2 describe-instances \
--instance-ids <instance-id> \
--query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' \
--output text
# List attached policies for the role
aws iam list-attached-role-policies \
--role-name <node-role-name>
2. Incorrect Cluster Endpoint Configuration: If your cluster API server endpoint is set to private-only access, nodes in public subnets or without proper VPC routing cannot connect. Check and update the endpoint access:
# Check current endpoint configuration
aws eks describe-cluster \
--name my-eks-cluster \
--query 'cluster.resourcesVpcConfig.endpointPublicAccess'
# Enable both public and private access
aws eks update-cluster-config \
--name my-eks-cluster \
--resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true
3. Security Group Blocking Traffic: The node security group must allow inbound traffic on port 443 from the control plane security group and the kubelet port 10250. Verify the security group rules:
# Describe the security group attached to your nodes
aws ec2 describe-security-groups \
--group-ids <sg-id> \
--query 'SecurityGroups[0].IpPermissions'
Issue 2: Pods Stuck in Pending State
Pods stuck in Pending state typically indicate a scheduling problem. The Kubernetes scheduler cannot place the pod on any available node, usually due to insufficient resources, taints, or affinity rules.
Diagnosing Pending Pods
# Get all pending pods across namespaces
kubectl get pods -A --field-selector=status.phase=Pending
# Describe the pending pod to see scheduling events
kubectl describe pod <pod-name> -n <namespace>
The describe output's Events section will reveal the specific reason. Common messages include Insufficient cpu, Insufficient memory, node(s) had taints that the pod didn't tolerate, or node(s) didn't match Pod's node affinity.
Solution: Resource Constraints
If the issue is insufficient resources, you need to either add nodes or adjust resource requests. First, check node capacity and allocation:
# Check resource usage across all nodes
kubectl describe nodes | grep -A 5 "Allocated resources"
# Get a detailed view of resource requests vs. limits
kubectl get pods -A -o json | jq '.items[] | {
name: .metadata.name,
namespace: .metadata.namespace,
cpu_request: .spec.containers[].resources.requests.cpu,
mem_request: .spec.containers[].resources.requests.memory
}'
If nodes are genuinely full, scale your node group:
# Scale a managed node group
aws eks update-nodegroup-version \
--cluster-name my-eks-cluster \
--nodegroup-name my-node-group \
--desired-size 5
# Or use eksctl
eksctl scale nodegroup \
--cluster my-eks-cluster \
--name my-node-group \
--nodes 5
Solution: Taints and Tolerations
If nodes have taints that your pods do not tolerate, either remove the taint or add a toleration to your pod spec:
# List taints on all nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
# Remove a taint from a node
kubectl taint nodes <node-name> key=value:NoSchedule-
# Add a toleration to your pod spec
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: tolerant-pod
spec:
containers:
- name: app
image: nginx
tolerations:
- key: "key"
value: "value"
effect: "NoSchedule"
EOF
Issue 3: Pods in CrashLoopBackOff
CrashLoopBackOff indicates that a container is repeatedly crashing and restarting. This is an application-level issue rather than an infrastructure problem, but it is one of the most common EKS support questions.
Diagnosing CrashLoopBackOff
# Get pod status
kubectl get pod <pod-name> -n <namespace>
# View pod events
kubectl describe pod <pod-name> -n <namespace>
# Check container logs (current and previous instance)
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
Common Causes and Fixes
1. Application Exits Immediately: If your container starts and immediately exits, it may be missing a required command or environment variable. Test the image locally first:
# Run the image locally to see if it starts
docker run --rm <image-name>
# Run with an interactive shell to debug
docker run -it --entrypoint /bin/sh <image-name>
2. Missing Configuration or Secrets: A common cause is referencing a ConfigMap or Secret that does not exist. Verify all referenced resources exist:
# Check if referenced ConfigMaps exist
kubectl get configmap -n <namespace>
# Check if referenced Secrets exist
kubectl get secret -n <namespace>
# View the pod spec to see what it references
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A 10 envFrom
3. Liveness Probe Failures: An overly aggressive liveness probe can cause Kubernetes to kill your container before it finishes starting. Check the probe configuration:
# View probe configuration
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].livenessProbe}'
Adjust the initialDelaySeconds to give your application more time to start, and ensure the probe path and port are correct:
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
Issue 4: IAM and Authentication Errors
EKS uses IAM for authentication through the aws-iam-authenticator mechanism (now integrated into the AWS CLI). Authentication errors are among the most confusing EKS issues because they can manifest in many ways.
Common Error Messages
# Typical error when IAM mapping is missing
error: You must be logged in to the server (Unauthorized)
# Error when the IAM entity is not mapped
could not get token: NoCredentialProviders
Diagnosing IAM Issues
# Verify your current AWS identity
aws sts get-caller-identity
# Check the aws-auth ConfigMap (deprecated in newer versions, but still common)
kubectl get configmap aws-auth -n kube-system -o yaml
# For newer clusters using access entries
aws eks list-access-entries --cluster-name my-eks-cluster
Solution: Adding IAM Users or Roles
For clusters using the traditional aws-auth ConfigMap, add your IAM user or role:
# Edit the aws-auth ConfigMap
kubectl edit configmap aws-auth -n kube-system
# Add a mapping for an IAM user
data:
mapUsers: |
- userarn: arn:aws:iam::123456789012:user/developer
username: developer
groups:
- system:masters
# Add a mapping for an IAM role
mapRoles: |
- rolearn: arn:aws:iam::123456789012:role/eks-admin
username: eks-admin
groups:
- system:masters
For newer EKS clusters (Kubernetes 1.29+), use access entries instead:
# Create an access entry for an IAM principal
aws eks create-access-entry \
--cluster-name my-eks-cluster \
--principal-arn arn:aws:iam::123456789012:user/developer
# Associate an access policy
aws eks associate-access-policy \
--cluster-name my-eks-cluster \
--principal-arn arn:aws:iam::123456789012:user/developer \
--policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
--access-scope type=cluster
Issue 5: Networking and DNS Problems
Networking issues in EKS can cause pods to fail to communicate with each other, with services, or with external endpoints. The Amazon VPC CNI plugin manages pod networking, and misconfigurations can lead to pods without IP addresses or connectivity problems.
Diagnosing Networking Issues
# Check if the VPC CNI plugin is running
kubectl get pods -n kube-system -l k8s-app=aws-node
# Check VPC CNI logs
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50
# Test DNS resolution from within a pod
kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- nslookup kubernetes.default
# Test pod-to-pod connectivity
kubectl run nettest --image=busybox:1.28 --rm -it --restart=Never -- wget -qO- http://<target-pod-ip>:8080
Solution: IP Exhaustion
A common EKS networking problem is running out of available IP addresses in subnets. The VPC CNI assigns a secondary ENI to each node and distributes IPs from that ENI to pods. When subnet IPs are exhausted, pods cannot get IPs and remain in ContainerCreating state.
# Check available IP addresses in your subnets
aws ec2 describe-subnets \
--subnet-ids <subnet-id> \
--query 'Subnets[0].AvailableIpAddressCount'
# Check how many IPs each node can allocate
kubectl get nodes -o json | jq '.items[] | {
node: .metadata.name,
maxPods: .status.capacity.pods
}'
# View CNI configuration
kubectl get ds aws-node -n kube-system -o json | jq '.spec.template.spec.containers[0].env'
To mitigate IP exhaustion, consider enabling prefix delegation, which allows each ENI to receive a /28 prefix instead of individual IPs:
# Enable prefix delegation via environment variable
kubectl set env ds aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
# Also set the warm prefix target
kubectl set env ds aws-node -n kube-system WARM_PREFIX_TARGET=1
Solution: CoreDNS Issues
If DNS resolution fails, check CoreDNS health:
# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
# Check CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml
# Check CoreDNS logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Verify the DNS service endpoint
kubectl get svc -n kube-system kube-dns
A common issue is CoreDNS pods being scheduled on nodes that lack proper networking. Ensure CoreDNS has adequate resources and node affinity:
# Scale CoreDNS if needed
kubectl scale deployment coredns -n kube-system --replicas=3
# Check CoreDNS resource limits
kubectl get deployment coredns -n kube-system -o jsonpath='{.spec.template.spec.containers[0].resources}'
Issue 6: Load Balancer and Ingress Problems
EKS integrates with AWS load balancers through the AWS Load Balancer Controller. Problems with service type LoadBalancer or Ingress resources are common and often relate to missing annotations or IAM permissions.
Diagnosing Load Balancer Issues
# Check if the AWS Load Balancer Controller is installed
kubectl get deployment -n kube-system aws-load-balancer-controller
# Check controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller --tail=50
# Check service events
kubectl describe svc <service-name> -n <namespace>
# Check Ingress status and events
kubectl describe ingress <ingress-name> -n <namespace>
Solution: Missing or Incorrect Annotations
The AWS Load Balancer Controller requires specific annotations to provision the correct type of load balancer. Here is a properly annotated service for a Network Load Balancer:
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: default
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
service.beta.kubernetes.io/aws-load-balancer-name: "my-nlb"
spec:
type: LoadBalancer
selector:
app: my-app
ports:
- port: 80
targetPort: 8080
For an Application Load Balancer via Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/abc-123
spec:
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service
port:
number: 80
Solution: IAM Permissions for the Controller
The AWS Load Balancer Controller needs an IAM role with permissions to create and manage load balancers, target groups, and listeners. If the controller logs show AccessDenied errors, verify the IRSA (IAM Roles for Service Accounts) configuration:
# Check the service account used by the controller
kubectl get sa -n kube-system aws-load-balancer-controller -o yaml
# Verify the IAM role exists
aws iam get-role --role-name <controller-role-name>
# Check the trust policy allows the EKS OIDC provider
aws iam get-role --role-name <controller-role-name> \
--query 'Role.AssumeRolePolicyDocument'
Issue 7: Cluster Autoscaler and Karpenter Problems
When workloads scale up but new nodes do not appear, the issue often lies with the Cluster Autoscaler or Karpenter configuration.
Diagnosing Cluster Autoscaler
# Check if Cluster Autoscaler is running
kubectl get deployment -n kube-system cluster-autoscaler
# View autoscaler logs
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=50
# Check for scaling events
kubectl logs -n kube-system -l app=cluster-autoscaler | grep -i "scale"
Common issues include the autoscaler not having the correct --cluster-name flag, or the IAM role lacking the AmazonEKSClusterAutoscalerPolicy. Verify the deployment configuration:
# Check the cluster name flag
kubectl get deployment cluster-autoscaler -n kube-system \
-o jsonpath='{.spec.template.spec.containers[0].command}'
Diagnosing Karpenter
If you use Karpenter instead of the Cluster Autoscaler, check its logs and provisioning status:
# Check Karpenter pods
kubectl get pods -n karpenter
# View Karpenter logs
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=50
# Check NodePools and NodeClaims
kubectl get nodepools
kubectl get nodeclaims
# Describe a specific NodeClaim for detailed status
kubectl describe nodeclaim <nodeclaim-name>
If Karpenter is not provisioning nodes, verify the NodePool and EC2NodeClass configurations:
# Check NodePool configuration
kubectl get nodepool default -o yaml
# Check EC2NodeClass configuration
kubectl get ec2nodeclass default -o yaml
# Verify Karpenter has the correct IAM permissions
kubectl get sa -n karpenter karpenter -o yaml
Best Practices for EKS Troubleshooting
Enable Control Plane Logging
EKS control plane logs are not enabled by default. Enable them to gain visibility into API server, audit, controller manager, and scheduler activity:
# Enable all control plane log types
aws eks update-cluster-config \
--name my-eks-cluster \
--logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
# Verify logging is enabled
aws eks describe-cluster \
--name my-eks-cluster \
--query 'cluster.logging.clusterLogging'
Logs will appear in CloudWatch Logs under the /aws/eks/<cluster-name>/cluster log group.
Implement Proactive Monitoring
Use CloudWatch Container Insights to collect metrics and logs from your cluster:
# Enable Container Insights using the CloudWatch agent
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml | sed "s/{{cluster_name}}/my-eks-cluster/;s/{{region_name}}/us-east-1/" | kubectl apply -f -
Use Resource Requests and Limits Consistently
Always define resource requests and limits for your pods. Without requests, the scheduler cannot make informed placement decisions, and without limits, a single runaway pod can destabilize a node:
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
Regularly Update Addons and Node Versions
Keep your EKS cluster, addons, and node AMIs up to date. Check for available updates regularly:
# Check current Kubernetes version
aws eks describe-cluster --name my-eks-cluster --query 'cluster.version'
# List available addon versions
aws eks describe-addon-versions \
--addon-name vpc-cni \
--kubernetes-version 1.30 \
--query 'addons[].addonVersions[].addonVersion'
# Update an addon
aws eks update-addon \
--cluster-name my-eks-cluster \
--addon-name vpc-cni \
--addon-version v1.18.0-eksbuild.1
Maintain a Troubleshooting Runbook
Document the specific commands and procedures for your environment. A runbook reduces resolution time during incidents and helps team members who may be less familiar with the cluster. Include:
- Cluster access procedures and IAM role mappings
- Standard diagnostic commands for each common issue
- Escalation paths and AWS support contact information
- Known issues and their workarounds
- Backup and restore procedures for critical resources
Conclusion
Troubleshooting EKS requires a systematic approach that combines Kubernetes-native debugging tools with AWS-specific knowledge. By understanding the common failure modes — from nodes not joining the cluster to IAM authentication errors, networking issues, and load balancer misconfigurations — you can dramatically reduce the time it takes to identify and resolve problems. The key is to start with the right diagnostic commands, read the events and logs carefully, and work through root causes methodically rather than making assumptions. Enable control plane logging, implement proactive monitoring, and maintain a runbook so that when issues arise, you and your team are prepared to respond quickly and effectively. With the commands and solutions covered in this tutorial, you now have a solid foundation for diagnosing and resolving the most common EKS issues you will encounter in production.