Introduction to AKS Troubleshooting
Azure Kubernetes Service (AKS) is a managed Kubernetes offering that simplifies deploying, managing, and scaling containerized applications. However, like any complex distributed system, AKS clusters can encounter issues that disrupt workloads, impact performance, or prevent successful deployments. Effective troubleshooting requires a systematic approach, familiarity with Kubernetes primitives, and knowledge of Azure-specific tooling.
This tutorial covers the most common AKS issues developers and operators face, along with practical solutions and diagnostic commands you can run immediately. Whether you are dealing with pod failures, networking problems, node health issues, or authentication errors, this guide will help you identify root causes quickly.
Why Troubleshooting AKS Matters
Production AKS environments host critical workloads where downtime translates directly to business impact. A misconfigured network policy, an exhausted node pool, or a failing container registry credential can cascade into full outages. Understanding how to diagnose and resolve these issues reduces mean time to recovery (MTTR), improves reliability, and builds confidence in your platform operations.
Moreover, AKS integrates multiple layers — Azure infrastructure, Kubernetes control plane, container runtime, networking plugins, and application code. Issues can originate at any layer, so a structured troubleshooting methodology is essential to avoid wasted effort chasing symptoms rather than root causes.
Essential Diagnostic Tooling
Before diving into specific issues, ensure you have the right tools installed and configured. The following commands verify your environment is ready for troubleshooting.
Required CLI Tools
# Install Azure CLI (if not already installed)
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
# Install kubectl
az aks install-cli
# Authenticate to Azure
az login
# Get cluster credentials
az aks get-credentials \
--resource-group myResourceGroup \
--name myAKSCluster
# Verify connectivity
kubectl get nodes
Useful AKS Add-ons
Azure provides several diagnostic features that significantly ease troubleshooting. Enable them proactively rather than reactively.
- Azure Monitor for Containers — collects metrics and logs from your cluster
- AKS Diagnostics — built-in diagnostic tool accessible via the Azure Portal
- Container Insights — provides performance visibility and alerting
- Network Watcher — helps diagnose virtual network and connectivity issues
Issue 1: Pods Stuck in Pending State
One of the most frequent AKS issues is pods remaining in a Pending state. This typically means the Kubernetes scheduler cannot find a suitable node to place the pod, usually due to insufficient resources or scheduling constraints.
Diagnosing the Problem
# List all pods and their statuses
kubectl get pods --all-namespaces
# Describe the problematic pod for events
kubectl describe pod my-app-pod -n default
# Check node resource availability
kubectl describe nodes
# View cluster-wide resource usage
kubectl top nodes
kubectl top pods --all-namespaces
The kubectl describe pod output includes an Events section at the bottom. Look for messages like FailedScheduling, which indicate why the scheduler rejected the pod. Common reasons include insufficient CPU or memory, no matching node selectors, or taints preventing placement.
Common Causes and Solutions
Insufficient Resources: If nodes are at capacity, either reduce resource requests, optimize the workload, or scale the node pool.
# Scale the node pool manually
az aks scale \
--resource-group myResourceGroup \
--name myAKSCluster \
--node-count 5
# Or enable cluster autoscaler
az aks update \
--resource-group myResourceGroup \
--name myAKSCluster \
--enable-cluster-autoscaler \
--min-count 3 \
--max-count 10
Node Selector or Taint Mismatch: Verify that your pod's nodeSelector, nodeAffinity, or tolerations match available nodes. Use the following command to inspect node labels:
# Show labels on all nodes
kubectl get nodes --show-labels
# Add a label to a specific node if needed
kubectl label nodes aks-nodepool1-12345678 vmworkload=highmem
PersistentVolumeClaim Not Bound: If your pod depends on storage that cannot be provisioned, it will remain Pending. Check PVC status:
# Check PVC status
kubectl get pvc -n default
# Check available storage classes
kubectl get storageclass
# Describe the PVC for detailed events
kubectl describe pvc my-pvc -n default
Issue 2: Pods in CrashLoopBackOff
CrashLoopBackOff indicates that a container is repeatedly crashing and Kubernetes is backing off before retrying. This is an application-level issue rather than a scheduling problem, so the focus shifts to container logs and runtime behavior.
Investigating Container Crashes
# Get pod status
kubectl get pod my-app-pod -n default
# View current container logs
kubectl logs my-app-pod -n default
# View previous container instance logs (often more useful)
kubectl logs my-app-pod -n default --previous
# Describe the pod for restart counts and events
kubectl describe pod my-app-pod -n default
The --previous flag is critical because it retrieves logs from the container instance before the last crash, which often contains the actual error that caused the failure.
Common Root Causes
- Application startup errors — missing configuration, invalid environment variables, or failed database connections
- Missing dependencies — the container image lacks required libraries or binaries
- Permission issues — the container runs as a user without access to required files
- OOMKilled — the container exceeds its memory limit and is terminated
Checking for OOMKilled Containers
# Describe the pod and look at the State section
kubectl describe pod my-app-pod -n default
# Look for: Last State: Terminated, Reason: OOMKilled
# Increase memory limits if appropriate
# Edit the deployment
kubectl edit deployment my-app -n default
# Example resource adjustment:
# resources:
# limits:
# memory: "512Mi"
# requests:
# memory: "256Mi"
Debugging with an Interactive Shell
If logs do not reveal the issue, run a temporary debug container in the same pod to investigate interactively:
# Ephemeral debug container (Kubernetes 1.25+)
kubectl debug -it my-app-pod --image=busybox --target=my-container
# Alternatively, run a temporary pod with network access to troubleshoot
kubectl run -it --rm debug --image=alpine --restart=Never -- sh
Issue 3: ImagePullBackOff and ErrImagePull
When Kubernetes cannot pull a container image, pods fail with ImagePullBackOff or ErrImagePull. This is common when using private registries or when images do not exist at the specified path.
Diagnosing Image Pull Failures
# Describe the failing pod
kubectl describe pod my-app-pod -n default
# Look for events like:
# Failed to pull image "myregistry.azurecr.io/myapp:v1": rpc error
Common Causes and Fixes
Incorrect Image Name or Tag: Verify the image exists in your registry and the tag is correct.
# List images in Azure Container Registry
az acr repository list --name myregistry --output table
# Show tags for a specific repository
az acr repository show-tags --name myregistry --repository myapp --output table
Authentication to ACR Failing: AKS needs permission to pull from Azure Container Registry. Attach the ACR to your cluster or create an image pull secret.
# Attach ACR to existing AKS cluster
az aks update \
--resource-group myResourceGroup \
--name myAKSCluster \
--attach-acr myregistry
# Alternatively, create a Kubernetes image pull secret
kubectl create secret docker-registry acr-secret \
--docker-server=myregistry.azurecr.io \
--docker-username=myregistry \
--docker-password=$(az acr login --name myregistry --expose-token --query accessToken -o tsv) \
--docker-email=dev@example.com
# Reference the secret in your deployment
# spec:
# imagePullSecrets:
# - name: acr-secret
Network Connectivity to Registry: If your cluster uses a private endpoint for ACR, ensure the virtual network configuration allows the nodes to reach the registry. Check DNS resolution from a node:
# Run a DNS test from within the cluster
kubectl run dns-test --image=busybox --rm -it --restart=Never -- \
nslookup myregistry.azurecr.io
Issue 4: Networking and Connectivity Problems
AKS networking issues can manifest as services being unreachable, DNS resolution failures, or pods unable to communicate with each other. The complexity arises from the interaction between Azure networking, the CNI plugin, and Kubernetes service discovery.
DNS Resolution Failures
CoreDNS is responsible for service discovery within AKS. If pods cannot resolve service names, applications will fail to connect to dependencies.
# Check CoreDNS pod status
kubectl get pods -n kube-system -l k8s-app=kube-dns
# Test DNS resolution from a pod
kubectl run dns-test --image=busybox --rm -it --restart=Never -- \
nslookup kubernetes.default
# Check CoreDNS logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# View CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml
If CoreDNS pods are crashing or the ConfigMap is misconfigured, restart the CoreDNS deployment after making fixes:
# Restart CoreDNS
kubectl rollout restart deployment coredns -n kube-system
Service Unreachable
When a service is not reachable, verify the service definition, endpoints, and network policies systematically.
# Check the service
kubectl get svc my-service -n default
# Verify endpoints exist (should list pod IPs)
kubectl get endpoints my-service -n default
# If endpoints are empty, check selector matching
kubectl describe svc my-service -n default
kubectl get pods -n default --show-labels
# Test connectivity from a debug pod
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
curl -v http://my-service.default.svc.cluster.local:8080
Empty endpoints usually mean the service selector does not match any pod labels. Compare the Selector field in the service description with the labels on your pods.
Network Policy Blocking Traffic
If you use network policies, they may inadvertently block legitimate traffic. List and inspect all policies:
# List all network policies in a namespace
kubectl get networkpolicy -n default
# Describe a specific policy
kubectl describe networkpolicy deny-all -n default
# Temporarily test by deleting the policy
kubectl delete networkpolicy deny-all -n default
Load Balancer Stuck in Pending
Azure Load Balancer provisioning can fail if the cluster lacks proper permissions or if subnet configuration is incorrect.
# Check the service events
kubectl describe svc my-loadbalancer-service -n default
# Verify the service principal or managed identity has Network Contributor role
az role assignment list \
--assignee <principal-id> \
--role "Network Contributor" \
--scope /subscriptions/<sub-id>/resourceGroups/<rg-name>
Issue 5: Node Not Ready or Unhealthy Nodes
When nodes become NotReady, Kubernetes stops scheduling pods on them and may evict running workloads. Node health issues often stem from resource exhaustion, kubelet problems, or Azure infrastructure failures.
Identifying Unhealthy Nodes
# Check node status
kubectl get nodes -o wide
# Describe a problematic node
kubectl describe node aks-nodepool1-12345678
# Look at conditions: Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable
Common Node Conditions and Meanings
- Ready=False — the kubelet is not responding or the node is shut down
- MemoryPressure=True — node memory is low; pods may be evicted
- DiskPressure=True — node disk is full; image garbage collection triggered
- PIDPressure=True — too many processes running on the node
- NetworkUnavailable=True — CNI has not configured the node network correctly
Investigating Node-Level Issues
# Access the node via a privileged pod
kubectl debug node/aks-nodepool1-12345678 -it --image=ubuntu
# Inside the node shell, check system resources
chroot /host
df -h
free -m
systemctl status kubelet
journalctl -u kubelet --no-pager | tail -50
Dealing with Disk Pressure
Disk pressure often occurs when container images accumulate or logs fill the disk. AKS nodes have automatic garbage collection, but you can manually clean up:
# SSH to the node (or use kubectl debug as above)
# Remove unused container images
crictl rmi --prune
# Check container log sizes
du -sh /var/log/containers/*
# Check kubelet logs
journalctl -u kubelet --since "1 hour ago" | grep -i error
Replacing Unhealthy Nodes
If a node is unrecoverable, drain and delete it. The cluster autoscaler or node pool will provision a replacement.
# Cordon the node to prevent new pods
kubectl cordon aks-nodepool1-12345678
# Drain the node, evicting existing pods
kubectl drain aks-nodepool1-12345678 --ignore-daemonsets --delete-emptydir-data
# Delete the node from the cluster
kubectl delete node aks-nodepool1-12345678
# For managed node pools, the replacement is automatic
# You can also manually trigger node image upgrades
az aks nodepool upgrade \
--resource-group myResourceGroup \
--cluster-name myAKSCluster \
--name nodepool1 \
--node-image-only
Issue 6: Authentication and Authorization Errors
AKS integrates with Azure Active Directory (Entra ID) for cluster authentication. Misconfigurations can prevent users or service principals from accessing the cluster or performing operations.
kubectl Access Denied
# Error: error from server (Forbidden): access denied
# Verify your current Azure context
az account show
# Re-fetch cluster credentials
az aks get-credentials \
--resource-group myResourceGroup \
--name myAKSCluster \
--overwrite-existing
# Check which user you are authenticated as
kubectl auth can-i --list
Checking RBAC Configuration
# List cluster roles and bindings
kubectl get clusterroles
kubectl get clusterrolebindings
# List namespace-level roles and bindings
kubectl get roles -n default
kubectl get rolebindings -n default
# Check what a specific user can do
kubectl auth can-i create deployments --as=user@domain.com -n default
Managed Identity Issues
AKS uses managed identities for cluster operations. If the identity lacks permissions, operations like pulling from ACR or creating load balancers will fail.
# Get the cluster's managed identity
az aks show \
--resource-group myResourceGroup \
--name myAKSCluster \
--query identityProfile.kubeletidentity.clientId -o tsv
# Verify role assignments for the identity
az role assignment list \
--assignee <managed-identity-client-id> \
--output table
Issue 7: Horizontal Pod Autoscaler Not Scaling
The Horizontal Pod Autoscaler (HPA) automatically scales pod replicas based on CPU, memory, or custom metrics. When it fails to scale, workloads may become overloaded.
Diagnosing HPA Issues
# Check HPA status
kubectl get hpa -n default
# Describe the HPA for detailed status and events
kubectl describe hpa my-app-hpa -n default
# Check if metrics server is running
kubectl get deployment metrics-server -n kube-system
# Verify metrics are being collected
kubectl top pods -n default
kubectl top nodes
Common HPA Problems
Missing Resource Requests: HPA relies on resource requests to calculate CPU utilization percentages. If your deployment does not specify resource requests, HPA cannot function.
# Ensure your deployment has resource requests defined
# Example:
# spec:
# containers:
# - name: my-app
# image: my-app:v1
# resources:
# requests:
# cpu: "100m"
# memory: "128Mi"
# limits:
# cpu: "500m"
# memory: "512Mi"
Metrics Server Not Running: The metrics server must be healthy for HPA to receive utilization data.
# Check metrics server logs
kubectl logs -n kube-system -l k8s-app=metrics-server --tail=30
# Restart metrics server if needed
kubectl rollout restart deployment metrics-server -n kube-system
HPA Showing Unknown Status: This usually means the HPA cannot read metrics. Verify the API service is available:
# Check the metrics API service
kubectl get apiservice v1beta1.metrics.k8s.io
# If the service is unavailable, check the service reference
kubectl describe apiservice v1beta1.metrics.k8s.io
Issue 8: Ingress Controller Problems
Ingress controllers route external traffic to your services. The NGINX ingress controller is the most common choice on AKS, and issues typically involve configuration, TLS certificates, or backend service connectivity.
Diagnosing Ingress Issues
# Check ingress resources
kubectl get ingress -n default
# Describe the ingress for events and backend mapping
kubectl describe ingress my-ingress -n default
# Check ingress controller pods
kubectl get pods -n ingress-nginx
# View ingress controller logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50
# Test the ingress controller directly
kubectl exec -it -n ingress-nginx <ingress-pod> -- curl -v http://my-app.default.svc.cluster.local
Common Ingress Problems
Backend Service Not Found: The ingress backend must reference an existing service with matching port. Verify the service name and port in the ingress definition match an actual service.
# Verify the service referenced by ingress exists
kubectl get svc my-app-service -n default
# Check the ingress backend configuration
kubectl get ingress my-ingress -n default -o yaml
TLS Certificate Issues: If HTTPS is not working, check the TLS secret and certificate validity.
# Check the TLS secret
kubectl get secret my-tls-secret -n default
# Decode and inspect the certificate
kubectl get secret my-tls-secret -n default -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
# If using cert-manager, check certificate status
kubectl get certificate -n default
kubectl describe certificate my-cert -n default
Best Practices for AKS Troubleshooting
Implement Proactive Monitoring
Do not wait for issues to surface through user reports. Set up comprehensive monitoring that alerts you before problems impact users.
# Enable Azure Monitor for containers
az aks enable-addons \
--resource-group myResourceGroup \
--name myAKSCluster \
--addons monitoring \
--workspace-resource-id /subscriptions/<sub-id>/resourcegroups/<rg>/providers/microsoft.operationalinsights/workspaces/<workspace>
# Set up alert rules for key metrics
az monitor metrics alert create \
--name "HighCPUUsage" \
--resource-group myResourceGroup \
--condition "avg CPU Usage > 80" \
--scopes <aks-resource-id> \
--window-size 5m \
--evaluation-frequency 1m
Use Resource Requests and Limits Consistently
Always define resource requests and limits for every container. This enables proper scheduling, HPA functionality, and prevents noisy neighbor problems.
Implement Liveness and Readiness Probes
Probes allow Kubernetes to detect and recover from application failures automatically. Configure them for every deployment.
# Example probe configuration
# livenessProbe:
# httpGet:
# path: /health
# port: 8080
# initialDelaySeconds: 30
# periodSeconds: 10
# failureThreshold: 3
# readinessProbe:
# httpGet:
# path: /ready
# port: 8080
# initialDelaySeconds: 5
# periodSeconds: 5
Maintain Runbooks for Common Issues
Document the steps to resolve recurring issues in runbooks. This reduces resolution time and enables team members to respond consistently during incidents.
Use Multiple Namespaces for Isolation
Separate workloads into namespaces to limit the blast radius of misconfigurations and make troubleshooting more focused.
# Create namespaces for different environments
kubectl create namespace development
kubectl create namespace staging
kubectl create namespace production
# Set resource quotas per namespace
kubectl create quota dev-quota \
--namespace development \
--hard=cpu=10,memory=20Gi,pods=50
Regularly Update Node Images
Node image updates include security patches and bug fixes. Schedule regular updates to prevent issues caused by outdated software.
# Check available node image versions
az aks nodepool get-upgrades \
--resource-group myResourceGroup \
--cluster-name myAKSCluster \
--nodepool-name nodepool1
# Perform a node image upgrade
az aks nodepool upgrade \
--resource-group myResourceGroup \
--cluster-name myAKSCluster \
--name nodepool1 \
--node-image-only \
--no-wait
Enable Diagnostic Settings for Log Retention
# Configure diagnostic settings to send logs to a Log Analytics workspace
az monitor diagnostic-settings create \
--resource <aks-resource-id> \
--name aks-diagnostics \
--workspace <workspace-resource-id> \
--logs '[
{"category":"kube-apiserver","enabled":true},
{"category":"kube-controller-manager","enabled":true},
{"category":"kube-scheduler","enabled":true},
{"category":"cluster-autoscaler","enabled":true}
]'
Conclusion
Troubleshooting AKS requires a methodical approach that combines Kubernetes-native tools like kubectl with Azure-specific diagnostics. By understanding the common issues covered in this tutorial — pending pods, crash loops, image pull failures, networking problems, unhealthy nodes, authentication errors, HPA malfunctions, and ingress misconfigurations — you can quickly narrow down the root cause of most problems. The key to effective troubleshooting is starting with the right diagnostic commands, reading events and logs carefully, and addressing root causes rather than symptoms. Pair this knowledge with proactive monitoring, well-defined resource limits, health probes, and documented runbooks, and you will significantly reduce both the frequency and impact of AKS incidents in your environment.