Introduction to ECS Troubleshooting
Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that makes it easy to deploy, manage, and scale containerized applications. While ECS abstracts away much of the complexity of container orchestration, developers and DevOps engineers frequently encounter issues that require careful diagnosis and resolution. Understanding how to troubleshoot ECS effectively is a critical skill for anyone running production workloads on AWS.
What Is ECS Troubleshooting?
ECS troubleshooting is the systematic process of identifying, diagnosing, and resolving problems that occur within your ECS infrastructure. This includes issues with task definitions, container instances, networking, service auto-scaling, load balancing, IAM permissions, and container runtime failures. Troubleshooting ECS requires familiarity with multiple AWS services that integrate with it, including EC2, IAM, CloudWatch, Application Load Balancers, and ECR.
Why It Matters
When ECS tasks fail to start, stop unexpectedly, or cannot communicate with other services, your applications experience downtime or degraded performance. In production environments, every minute of downtime can translate to lost revenue, damaged reputation, and frustrated users. Efficient troubleshooting minimizes mean time to resolution (MTTR), ensures reliability, and helps you build more resilient architectures. Additionally, understanding common failure patterns allows you to implement preventive measures that reduce the frequency of incidents altogether.
Essential Troubleshooting Tools and Commands
Before diving into specific issues, it is important to know the tools available for diagnosing ECS problems. The AWS CLI is your primary interface for inspecting ECS resources, while CloudWatch provides logs and metrics. The following commands form the foundation of any ECS troubleshooting workflow.
Key AWS CLI Commands for ECS
# List all clusters
aws ecs list-clusters
# Describe a specific cluster
aws ecs describe-clusters --clusters my-cluster
# List tasks in a cluster
aws ecs list-tasks --cluster my-cluster
# Describe specific tasks (get stopped reason)
aws ecs describe-tasks \
--cluster my-cluster \
--tasks arn:aws:ecs:us-east-1:123456789012:task/my-cluster/abc123
# List stopped tasks with reasons
aws ecs list-tasks \
--cluster my-cluster \
--desired-status STOPPED
# Describe services
aws ecs describe-services \
--cluster my-cluster \
--services my-service
# List container instances (EC2 launch type)
aws ecs list-container-instances --cluster my-cluster
# Describe container instances
aws ecs describe-container-instances \
--cluster my-cluster \
--container-instances arn:aws:ecs:us-east-1:123456789012:container-instance/my-cluster/def456
Checking Task Stopped Reasons
One of the most valuable pieces of information when troubleshooting is the stoppedReason field. This field often contains a direct explanation of why a task failed.
# Get the stopped reason for a specific task
aws ecs describe-tasks \
--cluster my-cluster \
--tasks arn:aws:ecs:us-east-1:123456789012:task/my-cluster/abc123 \
--query 'tasks[0].{status:lastStatus, reason:stoppedReason, container:containers[0].reason}' \
--output table
Common Issue 1: Tasks Stuck in PENDING State
A task that remains in the PENDING state for an extended period indicates that ECS cannot place or start the task. This is one of the most common issues developers encounter, and it typically stems from resource constraints, placement issues, or container instance registration problems.
Common Causes
- Insufficient resources: Container instances do not have enough CPU, memory, or available ports to satisfy the task definition requirements.
- No registered container instances: For EC2 launch type clusters, no instances are registered or the instances are disconnected.
- Placement constraint mismatches: Task placement constraints or strategies cannot be satisfied by available instances.
- AMI or ECS agent issues: The ECS agent on the container instance is not running or is unhealthy.
Diagnosis and Solution
First, check whether your container instances are registered and have available capacity:
# Check registered container instances
aws ecs list-container-instances --cluster my-cluster
# Check available resources on instances
aws ecs describe-container-instances \
--cluster my-cluster \
--container-instances $(aws ecs list-container-instances --cluster my-cluster --query 'containerInstanceArns' --output text) \
--query 'containerInstances[].{id:ec2InstanceId, status:status, cpu:remainingResources[?name==`CPU`].integerValue | [0], memory:remainingResources[?name==`MEMORY`].integerValue | [0], agent:agentConnected}' \
--output table
If instances show agentConnected: False, SSH into the EC2 instance and restart the ECS agent:
# SSH into the instance and check ECS agent status
sudo systemctl status ecs
# Restart the ECS agent
sudo systemctl restart ecs
# Verify the agent is running
sudo docker ps | grep ecs-agent
# Check agent logs for errors
sudo less /var/log/ecs/ecs-agent.log.*
If the issue is insufficient resources, either scale up by adding more container instances or scale down other tasks. For Fargate launch type, ensure your task definition CPU and memory combinations are valid. Fargate only supports specific CPU-memory pairings:
# Valid Fargate CPU-memory combinations:
# 0.25 vCPU -> 0.5GB, 1GB
# 0.5 vCPU -> 1GB, 2GB, 3GB, 4GB
# 1 vCPU -> 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB
# 2 vCPU -> 4GB through 16GB (in 1GB increments)
# 4 vCPU -> 8GB through 30GB (in 1GB increments)
# 8 vCPU -> 16GB through 60GB (in 4GB increments)
# 16 vCPU -> 32GB through 120GB (in 4GB increments)
Common Issue 2: Task Exits Immediately After Starting
When a task starts but immediately transitions to STOPPED, the container process is failing on startup. The stoppedReason and container-level reason fields are your best starting points.
Diagnosis Steps
# Describe the stopped task to get the reason
aws ecs describe-tasks \
--cluster my-cluster \
--tasks arn:aws:ecs:us-east-1:123456789012:task/my-cluster/abc123 \
--query 'tasks[0].{stoppedReason:stoppedReason, containers:containers[].{name:name, reason:reason, exitCode:exitCode, lastStatus:lastStatus}}' \
--output json
Common Causes and Solutions
Cause 1: Container exits with code 137 (OOM Killed). The container is being killed because it exceeds its memory limit. Increase the memory hard limit in your task definition, or optimize your application's memory usage.
{
"family": "my-app",
"containerDefinitions": [
{
"name": "my-container",
"image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
"memory": 1024,
"memoryReservation": 512,
"cpu": 256
}
]
}
Cause 2: Application error on startup. The application itself is crashing. Enable CloudWatch Logs in your task definition and inspect the logs:
{
"family": "my-app",
"containerDefinitions": [
{
"name": "my-container",
"image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}
]
}
Then query the logs:
# Retrieve recent CloudWatch logs for your task
aws logs get-log-events \
--log-group-name /ecs/my-app \
--log-stream-name ecs/my-container/abc123 \
--limit 50 \
--query 'events[*].message' \
--output text
Cause 3: Missing environment variables or configuration. Ensure all required environment variables are defined in the task definition or referenced from AWS Systems Manager Parameter Store or Secrets Manager:
{
"environment": [
{"name": "DATABASE_URL", "value": "postgresql://..."},
{"name": "NODE_ENV", "value": "production"}
],
"secrets": [
{
"name": "API_KEY",
"valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/my-app/api-key"
}
]
}
Common Issue 3: Tasks Cannot Pull Container Images
If your task fails with a message like CannotPullContainerError, ECS cannot retrieve the Docker image from your registry. This is a frequent issue that can occur due to authentication problems, image availability, or network configuration.
Diagnosis
The stopped reason will typically include one of these messages:
CannotPullContainerError: inspect image has been retried 5 timesCannotPullContainerError: Error response from daemonCannotPullContainerError: API error (500)
Solutions
For ECR images: Ensure the task execution role has the necessary permissions to pull from ECR. The task execution role must include the AmazonECSTaskExecutionRolePolicy or an equivalent custom policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Resource": "*"
}
]
}
For private registries: If you are pulling from a private Docker Hub or other registry, you must store authentication credentials in AWS Secrets Manager and reference them in your task definition:
{
"containerDefinitions": [
{
"name": "my-container",
"image": "private-registry.com/my-app:latest",
"repositoryCredentials": {
"credentialsParameter": "arn:aws:secretsmanager:us-east-1:123456789012:secret:dockerhub-auth-AbCdEf"
}
}
]
}
For Fargate networking issues: If using Fargate with private subnets, ensure your task has a route to the internet through a NAT gateway, or use VPC endpoints for ECR and S3 to avoid NAT costs:
# Required VPC endpoints for Fargate to pull from ECR without NAT
# 1. ecr.api (Interface endpoint)
# 2. ecr.dkr (Interface endpoint)
# 3. s3 (Gateway endpoint)
# Example: Create an interface endpoint for ECR API
aws ec2 create-vpc-endpoint \
--vpc-id vpc-abc123 \
--vpc-endpoint-type Interface \
--service-name com.amazonaws.us-east-1.ecr.api \
--subnet-ids subnet-abc123 subnet-def456 \
--security-group-ids sg-abc123
Image not found: Verify the image tag exists in your repository:
# List images in an ECR repository
aws ecr describe-images \
--repository-name my-app \
--query 'imageDetails[].imageTags' \
--output table
# List images in Docker Hub
docker manifest inspect private-registry.com/my-app:latest
Common Issue 4: Service Tasks Failing Health Checks
When ECS services are configured with a load balancer, tasks that fail health checks are continuously stopped and restarted, creating a loop. This manifests as tasks that start, run briefly, and then get killed by ECS because the load balancer marks them as unhealthy.
Diagnosis
Check the service events for health check failure messages:
# Get recent service events
aws ecs describe-services \
--cluster my-cluster \
--services my-service \
--query 'services[0].events[:10]' \
--output table
You might see messages like:
Service my-service was unable to place tasks because no
container instance met all of its requirements. The closest
matching instance has insufficient CPU/health check failing.
Solutions
Verify the target group health check path: The health check path configured on your Application Load Balancer target group must return an HTTP 200 response. Ensure your application has a dedicated health check endpoint:
# Example Express.js health check endpoint
const express = require('express');
const app = express();
app.get('/health', (req, res) => {
// Check database connection, cache, etc.
const isHealthy = checkDependencies();
if (isHealthy) {
res.status(200).json({ status: 'healthy' });
} else {
res.status(503).json({ status: 'unhealthy' });
}
});
app.listen(3000);
Adjust health check timing: If your application takes time to start, the default health check intervals may be too aggressive. Increase the health check grace period and adjust target group settings:
# Update service with a longer health check grace period
aws ecs update-service \
--cluster my-cluster \
--service my-service \
--health-check-grace-period-seconds 120
# Modify target group health check settings
aws elbv2 modify-target-group-attributes \
--target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-tg/abc123 \
--attributes \
Key=healthy_threshold,Value=3 \
Key=unhealthy_threshold,Value=3 \
Key=health_check_interval,Value=30 \
Key=health_check_timeout,Value=10
Verify security group rules: The container security group must allow inbound traffic from the load balancer security group on the container port. The load balancer security group must allow outbound traffic to the container security group:
# Allow traffic from ALB security group to container security group
aws ec2 authorize-security-group-ingress \
--group-id sg-container \
--protocol tcp \
--port 3000 \
--source-group sg-alb
Common Issue 5: IAM Permission Errors
IAM permission issues are among the most common and frustrating ECS problems. They can manifest in many ways, from task execution failures to application-level access denied errors. The key is identifying which role is involved and what permissions are missing.
Understanding ECS IAM Roles
There are two primary IAM roles in ECS that are often confused:
- Task Execution Role: Used by the ECS agent to pull images, retrieve secrets, and write CloudWatch Logs. This is the
executionRoleArnin your task definition. - Task Role: Used by your application code to access AWS services like S3, DynamoDB, SQS, etc. This is the
taskRoleArnin your task definition.
Diagnosis
Check CloudTrail for AccessDenied events related to your task:
# Query CloudTrail for AccessDenied events
aws logs filter-log-events \
--log-group-name CloudTrail/DefaultLogGroup \
--filter-pattern '{ $.errorCode = "AccessDenied" }' \
--start-time $(date -d '1 hour ago' +%s)000 \
--end-time $(date +%s)000 \
--query 'events[*].{time:eventTime, user:userIdentity.arn, action:eventName, resource:requestParameters}' \
--output table
Common Permission Issues and Fixes
Cannot write to CloudWatch Logs: Add logs permissions to the task execution role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:CreateLogGroup"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}
Cannot retrieve secrets from Secrets Manager or SSM: Add the appropriate permissions to the task execution role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue"
],
"Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-app/*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameters"
],
"Resource": "arn:aws:ssm:us-east-1:123456789012:parameter/my-app/*"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:us-east-1:123456789012:key/abc123"
}
]
}
Application cannot access AWS resources: Ensure the task role (not the execution role) has the necessary permissions. For example, if your application needs to read from S3:
# Create and attach a task role policy for S3 access
aws iam put-role-policy \
--role-name my-app-task-role \
--policy-name S3Access \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::my-app-bucket",
"arn:aws:s3:::my-app-bucket/*"
]
}
]
}'
Common Issue 6: Networking and Connectivity Problems
Networking issues in ECS can be complex because they involve VPC configuration, security groups, route tables, NAT gateways, and DNS resolution. Tasks may start successfully but fail to communicate with databases, external APIs, or other services.
Diagnosis Approach
Use the aws-vpc network mode's ENI information to identify the task's network interface, then verify connectivity:
# Get the ENI ID for a running Fargate task
TASK_ENI=$(aws ecs describe-tasks \
--cluster my-cluster \
--tasks arn:aws:ecs:us-east-1:123456789012:task/my-cluster/abc123 \
--query 'tasks[0].attachments[0].details[?name==`networkInterfaceId`].value' \
--output text)
echo "Task ENI: $TASK_ENI"
# Get the private IP of the task
aws ec2 describe-network-interfaces \
--network-interface-ids $TASK_ENI \
--query 'NetworkInterfaces[0].{ip:PrivateIpAddress, subnet:SubnetId, sg:Groups[*].GroupId}' \
--output table
Common Networking Issues
Cannot reach external services (Fargate in private subnet): Ensure the private subnet has a route to a NAT gateway:
# Check route table for the subnet
aws ec2 describe-route-tables \
--filters "Name=association.subnet-id,Values=subnet-abc123" \
--query 'RouteTables[0].Routes' \
--output table
# Expected: A route with DestinationCidrBlock=0.0.0.0/0
# pointing to a NatGatewayId
Cannot resolve DNS names: Ensure the VPC has DNS resolution and DNS hostnames enabled:
# Check VPC DNS settings
aws ec2 describe-vpc-attribute \
--vpc-id vpc-abc123 \
--attribute enableDnsSupport
aws ec2 describe-vpc-attribute \
--vpc-id vpc-abc123 \
--attribute enableDnsHostnames
# Enable if needed
aws ec2 modify-vpc-attribute \
--vpc-id vpc-abc123 \
--enable-dns-support
aws ec2 modify-vpc-attribute \
--vpc-id vpc-abc123 \
--enable-dns-hostnames
Cannot connect to a database: Verify the database security group allows inbound traffic from the task security group:
# Check inbound rules on the RDS security group
aws ec2 describe-security-groups \
--group-ids sg-database \
--query 'SecurityGroups[0].IpPermissions' \
--output table
# Add rule allowing the task security group to access the database
aws ec2 authorize-security-group-ingress \
--group-id sg-database \
--protocol tcp \
--port 5432 \
--source-group sg-task
Common Issue 7: Auto Scaling Not Working
ECS service auto scaling adjusts the desired task count based on CloudWatch alarms. When auto scaling fails to trigger or behaves unexpectedly, it can lead to under-provisioning or over-provisioning of resources.
Diagnosis
# Check scalable targets
aws application-autoscaling describe-scalable-targets \
--service-namespace ecs \
--resource-ids service/my-cluster/my-service \
--query 'ScalableTargets' \
--output table
# Check scaling policies
aws application-autoscaling describe-scaling-policies \
--service-namespace ecs \
--resource-id service/my-cluster/my-service \
--query 'ScalingPolicies' \
--output table
# Check CloudWatch alarms status
aws cloudwatch describe-alarms \
--alarm-name-prefix my-service \
--query 'MetricAlarms[].{name:AlarmName, state:StateValue, reason:StateReason}' \
--output table
Common Causes and Solutions
Scaling policy not attached: Ensure the scalable target is registered and the scaling policy is attached:
# Register a scalable target
aws application-autoscaling register-scalable-target \
--service-namespace ecs \
--resource-id service/my-cluster/my-service \
--scalable-dimension ecs:service:DesiredCount \
--min-capacity 2 \
--max-capacity 10
# Create a target tracking scaling policy
aws application-autoscaling put-scaling-policy \
--service-namespace ecs \
--resource-id service/my-cluster/my-service \
--scalable-dimension ecs:service:DesiredCount \
--policy-name my-service-cpu-scaling \
--policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration '{
"TargetValue": 70.0,
"PredefinedMetricSpecification": {
"PredefinedMetricType": "ECSServiceAverageCPUUtilization"
},
"ScaleOutCooldown": 60,
"ScaleInCooldown": 300
}'
CloudWatch metrics not available: For custom metrics, ensure your application is emitting them. For standard ECS metrics, verify the cluster and service have enough activity to generate metrics. Note that ECS CPU and memory utilization metrics require the ECSServiceAverageCPUUtilization metric to have data points.
Scheduled actions conflicting: Check if there are scheduled actions that override the auto scaling policy:
# List scheduled actions
aws application-autoscaling describe-scheduled-actions \
--service-namespace ecs \
--resource-id service/my-cluster/my-service \
--query 'ScheduledActions' \
--output table
Common Issue 8: Container Instance Deregistration
For EC2 launch type clusters, container instances can become disconnected from ECS, preventing task placement. This often happens after instance reboots, network disruptions, or agent failures.
Diagnosis
# Check instance connection status
aws ecs describe-container-instances \
--cluster my-cluster \
--container-instances $(aws ecs list-container-instances --cluster my-cluster --query 'containerInstanceArns' --output text) \
--query 'containerInstances[].{id:ec2InstanceId, status:status, connected:agentConnected, running:runningTasksCount, pending:pendingTasksCount}' \
--output table
Solutions
If agentConnected is false, the ECS agent on the instance has lost connection. SSH into the instance and troubleshoot:
# Check ECS agent status
sudo systemctl status ecs
# Check if the agent container is running
sudo docker ps -a | grep ecs-agent
# Restart the agent
sudo systemctl restart ecs
# Check agent introspection API
curl -s http://localhost:51678/v1/metadata | python -m json.tool
# Check agent logs
sudo journalctl -u ecs --since "10 minutes ago" --no-pager
# Verify the instance is registered to the correct cluster
cat /etc/ecs/ecs.config | grep ECS_CLUSTER
If the instance is permanently lost, deregister it and terminate the EC2 instance:
# Deregister a container instance
aws ecs deregister-container-instance \
--cluster my-cluster \
--container-instance arn:aws:ecs:us-east-1:123456789012:container-instance/my-cluster/def456 \
--force
# Terminate the EC2 instance
aws ec2 terminate-instances --instance-ids i-abc123def456
Best Practices for ECS Troubleshooting
Implement Comprehensive Logging
Always configure the awslogs log driver in your task definitions. Without logs, you are troubleshooting blind. Create separate log groups for different environments and applications:
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app-prod",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs",
"awslogs-datetime-format": "[%Y-%m-%d %H:%M:%S]"
}
}
}
Use Structured Logging
Emit structured JSON logs from your application to make querying and filtering easier in CloudWatch Logs Insights:
// Node.js example with structured logging
const logger = {
info: (message, meta = {}) => {
console.log(JSON.stringify({
level: 'info',
message,
timestamp: new Date().toISOString(),
...meta
}));
},
error: (message, meta = {}) => {
console.error(JSON.stringify({
level: 'error',
message,
timestamp: new Date().toISOString(),
...meta
}));
}
};
// Usage
logger.info('Server started', { port: 3000 });
logger.error('Database connection failed', { host: 'db.example.com', error: err.message });
Then query with CloudWatch Logs Insights:
# CloudWatch Logs Insights query for errors
fields @timestamp, level, message, error
| filter level = "error"
| sort @timestamp desc
| limit 100
Set Up CloudWatch Alarms and Dashboards
Proactively monitor your ECS services with alarms for key metrics:
# Create an alarm for high CPU utilization
aws cloudwatch put-metric-alarm \
--alarm-name my-service-high-cpu \
--alarm-description "ECS service CPU above 80%" \
--metric-name CPUUtilization \
--namespace AWS/ECS \
--statistic Average \
--period 60 \
--threshold 80 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 3 \
--dimensions Name=ClusterName,Value=my-cluster Name=ServiceName,Value=my-service \
--alarm-actions arn:aws:sns:us-east-1:123456789012:my-alerts
# Create an alarm for task restart count
aws cloudwatch put-metric-alarm \
--alarm-name my-service-high-restarts \
--alarm-description "ECS service experiencing frequent restarts" \
--metric-name MemoryUtilization \
--namespace AWS/ECS \
--statistic Average \
--period 300 \
--threshold 90 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 2 \
--dimensions Name=ClusterName,Value=my-cluster Name=ServiceName,Value=my-service \
--alarm-actions arn:aws:sns:us-east-1:123456789012:my-alerts
Use Circuit Breakers for Deployment Safety
Enable the ECS deployment circuit breaker to automatically roll back failed deployments:
{
"deploymentConfiguration": {
"deploymentCircuitBreaker": {
"enable": true,
"rollback": true
},
"maximumPercent": 200,
"minimumHealthyPercent": 100
}
}
Tag Everything
Consistent tagging helps you quickly identify and filter resources during troubleshooting:
# Tag an ECS service
aws ecs tag-resource \
--resource-arn arn:aws:ecs:us-east-1:123456789012:service/my-cluster/my-service \
--tags \
key=Environment,value=Production \
key=Application,value=my-app \
key=Team,value=backend \
key=CostCenter,value=12345
Maintain a Runbook
Document common issues and their resolutions in a runbook. Include the exact CLI commands, expected outputs, and resolution steps. This reduces troubleshooting time during incidents and helps onboard new team members.
Conclusion
Troubleshooting ECS requires a methodical approach that combines knowledge of AWS services, container runtime behavior, and networking fundamentals. By understanding the common issues outlined in this tutorial—tasks stuck in pending state, immediate task exits, image pull failures, health check problems, IAM permission errors, networking issues, auto scaling malfunctions, and container instance deregistration—you can diagnose and resolve problems quickly. The key to effective troubleshooting is having comprehensive logging, monitoring, and alerting in place before issues occur, combined with a solid understanding of the tools and commands available. As you gain experience with ECS, you will develop an intuition for where to look first, but always start with the task stopped reason, check CloudWatch logs, verify IAM permissions, and trace network connectivity. With these practices and the command examples provided, you will be well-equipped to keep your ECS workloads running reliably in production.