Troubleshooting EC2: Common Issues and Solutions

Introduction to EC2 Troubleshooting

Amazon Elastic Compute Cloud (EC2) is one of the most widely used compute services in AWS, powering everything from small web applications to large-scale enterprise systems. However, like any infrastructure service, EC2 instances can encounter issues that disrupt availability, performance, and connectivity. Troubleshooting EC2 effectively requires a systematic approach, an understanding of common failure modes, and familiarity with the tools AWS provides to diagnose and resolve problems.

This tutorial covers the most common EC2 issues developers and DevOps engineers face, along with practical solutions and commands you can run to identify and fix them quickly.

Why EC2 Troubleshooting Matters

When an EC2 instance becomes unreachable or behaves unexpectedly, the impact can be immediate and severe. Downtime translates to lost revenue, degraded user experience, and potential SLA violations. Fast, accurate troubleshooting minimizes mean time to recovery (MTTR) and helps maintain trust in your infrastructure. Additionally, understanding root causes prevents recurring issues and informs better architecture decisions.

Issue 1: Instance Not Reachable via SSH or RDP

One of the most frequent problems is being unable to connect to an EC2 instance using SSH (Linux) or RDP (Windows). This can stem from several causes including security group misconfiguration, incorrect key pairs, instance state problems, or operating system-level issues.

Common Causes

• Security group does not allow inbound traffic on port 22 (SSH) or 3389 (RDP)
• Wrong SSH key pair or incorrect username
• Instance is in a stopped or terminated state
• Network ACL blocking traffic
• Instance has run out of CPU credits (t2/t3 burstable instances)
• Operating system firewall (iptables, ufw, Windows Firewall) blocking connections

Diagnostic Steps

First, verify the instance state and status checks using the AWS CLI:

aws ec2 describe-instance-status --instance-ids i-0abcd1234efgh5678 --include-all-instances

Check the security group rules attached to the instance:

aws ec2 describe-security-groups --group-ids sg-0123456789abcdef0

Attempt an SSH connection with verbose output to see where the connection fails:

ssh -i my-key.pem -v ec2-user@<instance-public-ip>

Solutions

If the security group is missing an inbound rule for SSH, add one:

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 22 \
  --cidr 0.0.0.0/0

If the instance has exhausted its CPU credits, you can switch it to unlimited mode or upgrade the instance type:

aws ec2 modify-instance-credit-specification \
  --instance-credit-specifications InstanceId=i-0abcd1234efgh5678,CreditSpecification=CpuCredits=unlimited

If the OS-level firewall is the culprit and you cannot SSH in, you may need to stop the instance, detach the root volume, attach it to another instance, and modify the configuration files directly.

Issue 2: Instance Status Check Failures

AWS performs two types of status checks on EC2 instances: system status checks and instance status checks. System status checks detect problems with the underlying AWS infrastructure, while instance status checks detect problems with the instance itself, such as network configuration issues, exhausted memory, or a corrupted file system.

Identifying the Problem

Use the AWS CLI to check the status of your instance:

aws ec2 describe-instance-status --instance-ids i-0abcd1234efgh5678

The output will show both SystemReachability and InstanceReachability results. If the system status check fails, AWS recommends waiting or restarting the instance. If the instance status check fails, the problem is likely within the operating system.

Resolving System Status Check Failures

For system status check failures, stop and start the instance. This migrates it to a healthy host:

aws ec2 stop-instances --instance-ids i-0abcd1234efgh5678
aws ec2 start-instances --instance-ids i-0abcd1234efgh5678

Note that a simple reboot is not sufficient because it keeps the instance on the same underlying hardware. A stop and start is required to move it to a new host.

Resolving Instance Status Check Failures

For instance status check failures, retrieve the system log to look for OS-level errors:

aws ec2 get-console-output --instance-id i-0abcd1234efgh5678 --output text

Common issues found in console output include kernel panics, file system corruption, and out-of-memory errors. If the instance is unresponsive, you can capture a screenshot of the console (useful for Windows instances):

aws ec2 get-console-screenshot --instance-id i-0abcd1234efgh5678

Issue 3: High CPU or Memory Usage

Performance degradation is another common issue. Instances may become slow or unresponsive due to resource exhaustion. Unlike CPU, memory usage is not directly visible in CloudWatch by default, which makes diagnosis trickier.

Diagnosing High CPU

Check CPU utilization using CloudWatch metrics:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abcd1234efgh5678 \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-01T23:59:59Z \
  --period 300 \
  --statistics Average Maximum

If you can connect to the instance, identify the processes consuming the most CPU:

top -b -n 1 | head -20

Diagnosing High Memory

Since memory metrics are not available by default, you need to install the CloudWatch agent to report them. First, check memory from within the instance:

free -h

To enable memory monitoring in CloudWatch, install the unified CloudWatch agent and configure it:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon-cloudwatch-agent.rpm
sudo rpm -U amazon-cloudwatch-agent.rpm

Create a configuration file for the agent:

{
  "metrics": {
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      }
    }
  }
}

Start the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json \
  -s

Solutions

If the instance is consistently running at high CPU or memory, consider the following approaches:

• Upgrade to a larger instance type with more vCPUs or RAM
• Enable CPU credits to unlimited for burstable instances
• Optimize application code or database queries
• Implement auto-scaling to distribute load across multiple instances
• Add swap space as a temporary measure for memory pressure

To change the instance type, stop the instance, modify it, and start it again:

aws ec2 stop-instances --instance-ids i-0abcd1234efgh5678
aws ec2 modify-instance-attribute \
  --instance-id i-0abcd1234efgh5678 \
  --instance-type "{\"Value\": \"t3.large\"}"
aws ec2 start-instances --instance-ids i-0abcd1234efgh5678

Issue 4: Disk Space and EBS Volume Problems

Running out of disk space is a common issue that can cause applications to fail, logs to stop writing, and databases to crash. EBS volumes can also suffer from performance degradation if they are not properly provisioned.

Checking Disk Space

Connect to the instance and check disk usage:

df -h

Identify large files and directories:

sudo du -sh /* 2>/dev/null | sort -rh | head -10

Find the largest individual files:

sudo find / -type f -exec du -h {} + 2>/dev/null | sort -rh | head -20

Expanding an EBS Volume

If you need more space, increase the size of the EBS volume:

aws ec2 modify-volume --volume-id vol-0abcd1234efgh5678 --size 100

After the volume modification completes, extend the file system at the OS level. For ext4 file systems:

sudo growpart /dev/xvda 1
sudo resize2fs /dev/xvda1

For XFS file systems:

sudo growpart /dev/nvme0n1 1
sudo xfs_growfs /

Checking EBS Performance

If your application is I/O intensive, check the volume's CloudWatch metrics for high queue length or low throughput:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EBS \
  --metric-name VolumeQueueLength \
  --dimensions Name=VolumeId,Value=vol-0abcd1234efgh5678 \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-01T23:59:59Z \
  --period 300 \
  --statistics Average Maximum

If the queue length is consistently high, consider switching to a provisioned IOPS volume (io1 or io2) or increasing the size of a gp3 volume to boost its baseline performance.

Issue 5: Instance Fails to Start After Reboot or Stop/Start

Sometimes an instance fails to boot properly after a reboot or stop/start operation. This can be caused by corrupted boot configurations, full disk partitions, or misconfigured kernel updates.

Retrieving Console Output

The first step is to retrieve the console output to see where the boot process is failing:

aws ec2 get-console-output --instance-id i-0abcd1234efgh5678 --latest --output text | tail -100

Look for error messages such as kernel panics, file system errors, or failed service starts.

Recovering Data Using a Rescue Instance

If the instance cannot boot, you can recover data by detaching the root volume and attaching it to a healthy instance:

# Stop the problematic instance
aws ec2 stop-instances --instance-ids i-0abcd1234efgh5678

# Detach the root volume
aws ec2 detach-volume --volume-id vol-0abcd1234efgh5678

# Attach the volume to a rescue instance
aws ec2 attach-volume \
  --volume-id vol-0abcd1234efgh5678 \
  --instance-id i-0rescue1234efgh5678 \
  --device /dev/sdf

On the rescue instance, mount the volume and investigate:

sudo mkdir /mnt/recovery
sudo mount /dev/xvdf1 /mnt/recovery
cd /mnt/recovery

# Check logs for errors
cat /mnt/recovery/var/log/syslog | tail -50
cat /mnt/recovery/var/log/messages | tail -50

# Check disk space on the recovered volume
df -h /mnt/recovery

After fixing the issue, unmount the volume, detach it, reattach it to the original instance, and start it:

sudo umount /mnt/recovery
aws ec2 detach-volume --volume-id vol-0abcd1234efgh5678
aws ec2 attach-volume \
  --volume-id vol-0abcd1234efgh5678 \
  --instance-id i-0abcd1234efgh5678 \
  --device /dev/xvda
aws ec2 start-instances --instance-ids i-0abcd1234efgh5678

Issue 6: Network Connectivity Problems

Instances may have trouble reaching external services or other instances within a VPC. Network issues can be caused by route table misconfigurations, NAT gateway problems, DNS resolution failures, or VPC peering issues.

Diagnosing Network Issues

Test basic connectivity from within the instance:

# Test DNS resolution
nslookup example.com

# Test internet connectivity
ping -c 4 8.8.8.8

# Trace the network path
traceroute example.com

# Test a specific port
nc -zv example.com 443

Use the AWS VPC Reachability Analyzer to check connectivity between two endpoints:

aws ec2 create-network-insights-path \
  --source i-0abcd1234efgh5678 \
  --destination 0.0.0.0/0 \
  --destination-port 443 \
  --protocol tcp

Common Network Fixes

Verify the route table associated with the instance's subnet has a route to the internet gateway for public instances:

aws ec2 describe-route-tables --route-table-ids rtb-0abcd1234efgh5678

If the instance is in a private subnet, ensure there is a route to a NAT gateway for outbound internet access:

aws ec2 create-route \
  --route-table-id rtb-0abcd1234efgh5678 \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-0abcd1234efgh5678

Check that the instance has a public IP address if it needs to be reachable from the internet:

aws ec2 describe-instances \
  --instance-ids i-0abcd1234efgh5678 \
  --query "Reservations[0].Instances[0].PublicIpAddress"

Issue 7: IAM Role and Permission Errors

Applications running on EC2 instances often need to access other AWS services. If the instance does not have the correct IAM role attached, or if the role lacks the necessary permissions, API calls will fail with access denied errors.

Checking the Attached IAM Role

Verify which IAM role is attached to the instance:

aws ec2 describe-instances \
  --instance-ids i-0abcd1234efgh5678 \
  --query "Reservations[0].Instances[0].IamInstanceProfile.Arn"

Check the policies attached to the role:

aws iam list-attached-role-policies --role-name MyEC2Role
aws iam list-instance-profiles --role-name MyEC2Role

Attaching or Replacing an IAM Role

To attach an IAM role to an instance that does not have one:

aws ec2 associate-iam-instance-profile \
  --instance-id i-0abcd1234efgh5678 \
  --iam-instance-profile Name=MyEC2InstanceProfile

To replace an existing IAM role:

aws ec2 replace-iam-instance-profile-association \
  --iam-instance-profile-association-id iip-assoc-0abcd1234efgh5678 \
  --iam-instance-profile Name=MyNewEC2InstanceProfile

After updating the role, verify that the instance metadata service reflects the new credentials:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/MyEC2Role

Best Practices for EC2 Troubleshooting

Implement Proactive Monitoring

Do not wait for issues to occur before investigating. Set up CloudWatch alarms for key metrics such as CPU utilization, status check failures, and disk space. Create alarms that trigger before resources are exhausted:

aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPU-i-0abcd1234efgh5678" \
  --alarm-description "Alarm when CPU exceeds 80%" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-0abcd1234efgh5678 \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:MyAlertTopic

Use EC2 Systems Manager for Remote Management

Install the SSM Agent on your instances so you can run commands and access sessions without needing SSH or RDP access. This is especially useful when security groups or OS firewalls block direct access:

aws ssm send-command \
  --document-name "AWS-RunShellScript" \
  --instance-ids "i-0abcd1234efgh5678" \
  --parameters 'commands=["df -h", "free -h", "top -b -n 1 | head -20"]'

Maintain Detailed Documentation

Keep records of instance configurations, AMI IDs, user data scripts, and any custom configurations. This documentation is invaluable when troubleshooting or rebuilding instances. Use infrastructure as code tools like CloudFormation or Terraform to ensure configurations are reproducible.

Create AMI Backups Regularly

Take regular AMI snapshots of critical instances so you can quickly launch replacements if an instance becomes unrecoverable:

aws ec2 create-image \
  --instance-id i-0abcd1234efgh5678 \
  --name "MyApp-Backup-2024-01-01" \
  --description "Regular backup of MyApp production instance" \
  --no-reboot

Use Elastic IPs for Stable Public Addresses

If an instance is stopped and started, its public IP address changes. Use an Elastic IP to maintain a consistent public address for instances that external systems depend on:

aws ec2 allocate-address --domain vpc
aws ec2 associate-address --instance-id i-0abcd1234efgh5678 --allocation-id eipalloc-0abcd1234efgh5678

Enable Termination Protection

Prevent accidental termination of critical instances by enabling termination protection:

aws ec2 modify-instance-attribute \
  --instance-id i-0abcd1234efgh5678 \
  --disable-api-termination

Conclusion

Troubleshooting EC2 instances is a critical skill for anyone working with AWS infrastructure. By understanding the common issues covered in this tutorial—connectivity problems, status check failures, resource exhaustion, disk space limitations, boot failures, network misconfigurations, and IAM permission errors—you can diagnose and resolve problems quickly and effectively. The key to efficient troubleshooting is a methodical approach: start by gathering information using the AWS CLI and console output, narrow down the root cause, and apply the appropriate fix. Combine this reactive capability with proactive monitoring, regular backups, and infrastructure as code practices to minimize issues before they impact your applications. With these tools and techniques at your disposal, you can maintain reliable, resilient EC2-based infrastructure that meets the demands of your users and your business.