Introduction to VPC Troubleshooting
Amazon Virtual Private Cloud (VPC) is the foundational networking layer for most AWS deployments. When something breaks in a VPC, the blast radius can be enormous — applications become unreachable, databases lose connectivity, and hybrid VPN tunnels drop without warning. Troubleshooting VPC issues efficiently is therefore a critical skill for any cloud engineer, DevOps practitioner, or platform administrator.
This tutorial walks through the most common VPC problems you will encounter in production, explains why they happen, and shows you how to diagnose and fix them using the AWS CLI, console tools, and time-tested debugging methodologies.
What Is VPC Troubleshooting?
VPC troubleshooting is the systematic process of identifying, isolating, and resolving connectivity, routing, security, and configuration problems within an AWS VPC. It involves analyzing multiple layers — from the physical-like infrastructure of subnets and route tables up to security groups, network ACLs, NAT gateways, VPC endpoints, and peering connections.
Because a VPC is composed of many interdependent resources, a single misconfiguration can cascade into a full outage. Effective troubleshooting requires understanding how traffic flows between these components and knowing which tool to reach for at each layer.
Why It Matters
- Minimizes downtime: Fast, structured diagnosis reduces mean time to recovery (MTTR) during production incidents.
- Prevents recurring issues: Root cause analysis stops the same problem from resurfacing.
- Improves security posture: Many connectivity issues stem from overly permissive or misconfigured security rules that also expose vulnerabilities.
- Reduces cloud costs: Misconfigured NAT gateways, redundant peering, and orphaned resources often silently inflate bills.
- Builds operational confidence: A repeatable troubleshooting playbook empowers on-call engineers to act decisively under pressure.
Essential Troubleshooting Toolkit
Before diving into specific issues, make sure you have the right tools available. Most VPC problems can be diagnosed with a combination of the AWS CLI, VPC Flow Logs, Reachability Analyzer, and standard Linux networking utilities running on an EC2 instance inside the VPC.
Key AWS Services and Features
- VPC Flow Logs: Capture accepted and rejected IP traffic at the subnet, ENI, or VPC level.
- Reachability Analyzer: A static analysis tool that determines whether a source can reach a destination through your VPC topology.
- Route Table Analyzer: Helps inspect route propagation and conflicts.
- CloudWatch Metrics: Monitor NAT gateway bandwidth, VPC endpoint connection counts, and peering traffic.
- AWS CLI: Scriptable access to describe and inspect every VPC resource.
Setting Up a Diagnostic Bastion
It is good practice to maintain a hardened bastion or diagnostic instance in each VPC. Install common network utilities so you can test connectivity on demand:
# Install essential networking tools on Amazon Linux 2
sudo yum install -y tcpdump traceroute mtr nmap-ncip telnet bind-utils iperf3
# Verify AWS CLI is configured with appropriate permissions
aws sts get-caller-identity
# Enable VPC Flow Logs for the entire VPC (if not already enabled)
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-0abc123def456 \
--traffic-type ALL \
--log-destination-type cloud-watch-logs \
--log-group-name /aws/vpc/flowlogs-prod \
--deliver-logs-permission-arn arn:aws:iam::123456789012:role/FlowLogsRole
Issue 1: EC2 Instance Cannot Reach the Internet
This is perhaps the most common VPC complaint. An instance in a private or public subnet cannot download packages, reach external APIs, or communicate with SaaS endpoints. The root cause is almost always one of four things: missing or misconfigured route table entries, a missing Internet Gateway (IGW) or NAT Gateway, a security group blocking egress, or an instance lacking a public IP.
Diagnosis Steps
Start by identifying the subnet and route table associated with the instance, then trace the path outward:
# 1. Find the subnet and VPC for the instance
aws ec2 describe-instances \
--instance-ids i-0abcd1234efgh5678 \
--query 'Reservations[0].Instances[0].[SubnetId,VpcId,PrivateIpAddress,PublicIpAddress]' \
--output table
# 2. Describe the route table for that subnet
aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=subnet-0abc123 \
--query 'RouteTables[0].Routes' \
--output table
# 3. Check if an Internet Gateway is attached to the VPC
aws ec2 describe-internet-gateways \
--filters Name=attachment.vpc-id,Values=vpc-0abc123def456 \
--output table
Common Fixes
For a public subnet: Ensure the route table contains a default route 0.0.0.0/0 pointing to an Internet Gateway, and that the instance has a public IP or Elastic IP attached.
# Attach an Internet Gateway to the VPC (if missing)
aws ec2 attach-internet-gateway \
--internet-gateway-id igw-0abc123 \
--vpc-id vpc-0abc123def456
# Add a default route to the public route table
aws ec2 create-route \
--route-table-id rtb-0abc123 \
--destination-cidr-block 0.0.0.0/0 \
--gateway-id igw-0abc123
# Associate an Elastic IP to the instance
aws ec2 associate-address \
--instance-id i-0abcd1234efgh5678 \
--allocation-id eipalloc-0abc123
For a private subnet: The default route should point to a NAT Gateway, not an Internet Gateway. Verify the NAT Gateway exists and is in an available state:
# Check NAT Gateway status
aws ec2 describe-nat-gateways \
--filter Name=vpc-id,Values=vpc-0abc123def456 \
--query 'NatGateways[*].[NatGatewayId,State,SubnetId]' \
--output table
# Add a default route through the NAT Gateway
aws ec2 create-route \
--route-table-id rtb-private123 \
--destination-cidr-block 0.0.0.0/0 \
--nat-gateway-id nat-0abc123def456
Also verify the outbound security group rules allow HTTPS or the relevant port:
# Describe outbound rules for the instance security group
aws ec2 describe-security-groups \
--group-ids sg-0abc123 \
--query 'SecurityGroups[0].IpPermissionsEgress' \
--output table
Issue 2: Security Group and NACL Conflicts
Security groups are stateful and attached to ENIs, while network ACLs are stateless and applied at the subnet boundary. When both are misconfigured, traffic can be blocked in ways that are confusing to debug because the two layers operate independently.
Understanding the Difference
- Security Groups: Stateful, allow rules only, evaluated per instance, return traffic automatically allowed.
- Network ACLs: Stateless, allow and deny rules, evaluated per subnet, return traffic must be explicitly allowed.
A classic mistake is allowing inbound HTTPS in a security group but forgetting to allow ephemeral outbound ports in the NACL for return traffic. Because NACLs are stateless, the response from the remote server on a high port will be dropped.
Diagnosis and Fix
# List NACL rules for the subnet
aws ec2 describe-network-acls \
--filters Name=association.subnet-id,Values=subnet-0abc123 \
--query 'NetworkAcls[0].Entries' \
--output table
# Add an ephemeral port range for return traffic (1024-65535)
aws ec2 create-network-acl-entry \
--network-acl-id acl-0abc123 \
--rule-number 100 \
--protocol tcp \
--port-range From=1024,To=65535 \
--cidr-block 0.0.0.0/0 \
--rule-action allow \
--egress
Use Reachability Analyzer to confirm whether a security group or NACL is the blocker:
# Create a reachability analysis between two instances
aws ec2 create-network-insights-path \
--source i-0source123 \
--destination i-0dest123 \
--protocol tcp \
--destination-port 443
# Start the analysis using the path ID returned above
aws ec2 start-network-insights-analysis \
--network-insights-path-id nip-0abc123
Issue 3: VPC Peering Connection Not Routing Traffic
VPC peering allows private IPv4 or IPv6 routing between two VPCs. A frequent problem is that the peering connection is active but traffic still does not flow. This is almost always because route tables in one or both VPCs have not been updated to point to the peering connection.
Diagnosis Steps
# Check peering connection status
aws ec2 describe-vpc-peering-connections \
--vpc-peering-connection-ids pcx-0abc123 \
--query 'VpcPeeringConnections[0].[Status.Code,RequesterVpcInfo.VpcId,AccepterVpcInfo.VpcId]' \
--output table
# Check route tables for a peering route
aws ec2 describe-route-tables \
--filters Name=vpc-id,Values=vpc-0requester123 \
--query 'RouteTables[*].[RouteTableId,Routes[?GatewayId==`pcx-0abc123`]]' \
--output table
Fixing the Routes
You must add routes in both VPCs. The requester VPC routes the accepter CIDR through the peering connection, and vice versa:
# Add route in requester VPC to reach accepter CIDR
aws ec2 create-route \
--route-table-id rtb-0requester \
--destination-cidr-block 10.20.0.0/16 \
--vpc-peering-connection-id pcx-0abc123
# Add route in accepter VPC to reach requester CIDR
aws ec2 create-route \
--route-table-id rtb-0accepter \
--destination-cidr-block 10.10.0.0/16 \
--vpc-peering-connection-id pcx-0abc123
Also verify that security groups in both VPCs reference each other. You can use peered security group references in rules:
# Allow inbound from the peered VPC's security group
aws ec2 authorize-security-group-ingress \
--group-id sg-0accepter \
--ip-permissions IpProtocol=tcp,FromPort=443,ToPort=443,UserIdGroupPairs=[{GroupId=sg-0requester,PeeringStatus=active,VpcPeeringConnectionId=pcx-0abc123}]
Issue 4: VPC Endpoint Connectivity Failures
VPC endpoints (Gateway and Interface) let you access AWS services privately without traversing the public internet. When they fail, applications suddenly cannot reach S3, DynamoDB, or other services even though the instance has internet access.
Gateway Endpoint Issues
Gateway endpoints (used for S3 and DynamoDB) require a specific route table entry. If the route is missing or overridden by a more specific route, traffic will not use the endpoint:
# List gateway endpoints and their route tables
aws ec2 describe-vpc-endpoints \
--filters Name=vpc-id,Values=vpc-0abc123 \
--query 'VpcEndpoints[?VpcEndpointType==`Gateway`].[VpcEndpointId,ServiceName,State]' \
--output table
# Verify the route table has a prefix list entry for the endpoint
aws ec2 describe-route-tables \
--route-table-ids rtb-0abc123 \
--query 'RouteTables[0].Routes[?VpcEndpointId!=null]' \
--output table
Interface Endpoint Issues
Interface endpoints create ENIs in your subnets with private IP addresses. Common problems include the endpoint being deployed in the wrong subnets, security groups blocking traffic, or private DNS not being enabled:
# Check interface endpoint subnets and DNS settings
aws ec2 describe-vpc-endpoints \
--vpc-endpoint-ids vpce-0abc123 \
--query 'VpcEndpoints[0].[SubnetIds,PrivateDnsEnabled,Groups]' \
--output table
# Enable private DNS for the interface endpoint
aws ec2 modify-vpc-endpoint \
--vpc-endpoint-id vpce-0abc123 \
--private-dns-enabled
Test connectivity to the endpoint ENI from an instance:
# From a diagnostic instance, test DNS resolution and connectivity
dig ssm.us-east-1.amazonaws.com
nc -zv vpce-0abc123-xyz.us-east-1.vpce.amazonaws.com 443
Issue 5: VPN and Direct Connect Problems
Hybrid connectivity through Site-to-Site VPN or Direct Connect introduces additional complexity. Common symptoms include tunnels flapping, asymmetric routing, and partial reachability where some on-premises subnets work but others do not.
Diagnosing VPN Tunnel Status
# Check VPN connection status and tunnel states
aws ec2 describe-vpn-connections \
--vpn-connection-ids vpn-0abc123 \
--query 'VpnConnections[0].[State,VgwTelemetry]' \
--output table
The VgwTelemetry output shows the status of each tunnel, including whether it is up or down, the last status change, and any error messages. If a tunnel is down, check the customer gateway configuration and ensure both IKE phases complete successfully.
Route Propagation Issues
For VPN connections to carry traffic, routes must be propagated into the route table. If route propagation is disabled, the dynamic routes learned via BGP will not appear:
# Enable route propagation for a VPN gateway
aws ec2 enable-vgw-route-propagation \
--route-table-id rtb-0abc123 \
--gateway-id vgw-0abc123
# Verify propagated routes
aws ec2 describe-route-tables \
--route-table-ids rtb-0abc123 \
--query 'RouteTables[0].Routes[?Origin==`EnableVgwRoutePropagation`]' \
--output table
Asymmetric Routing
Asymmetric routing occurs when outbound traffic leaves through one tunnel but return traffic arrives on another. This often breaks stateful firewalls on the on-premises side. To fix this, use a single active tunnel with the other as standby, or configure equal-cost multipath (ECMP) routing consistently on both sides.
Issue 6: DNS Resolution Failures Inside the VPC
Many applications fail not because of routing but because DNS resolution is broken. The Amazon-provided DNS server at the VPC CIDR plus two (for example, 10.0.0.2 for a 10.0.0.0/16 VPC) must be reachable, and both DNS resolution and DNS hostnames must be enabled on the VPC.
# Check DNS settings on the VPC
aws ec2 describe-vpc-attribute \
--vpc-id vpc-0abc123 \
--attribute enableDnsSupport
aws ec2 describe-vpc-attribute \
--vpc-id vpc-0abc123 \
--attribute enableDnsHostnames
# Enable both if they are false
aws ec2 modify-vpc-attribute \
--vpc-id vpc-0abc123 \
--enable-dns-support
aws ec2 modify-vpc-attribute \
--vpc-id vpc-0abc123 \
--enable-dns-hostnames
If you are using a custom DHCP option set with a third-party DNS server, verify that the security group and NACL allow UDP and TCP port 53 to that server. Also confirm the DHCP option set is associated with the correct VPC:
# Check the DHCP options set associated with the VPC
aws ec2 describe-vpcs \
--vpc-ids vpc-0abc123 \
--query 'Vpcs[0].DhcpOptionsId' \
--output text
Issue 7: NAT Gateway Exhaustion and Port Snat Exhaustion
NAT gateways allocate source ports for outbound connections. Under high concurrency, a single NAT gateway can exhaust its available ports for a given destination, causing intermittent connection failures. This manifests as random timeouts that are difficult to reproduce.
Detecting Port Exhaustion
Monitor the ErrorPortAllocation CloudWatch metric for the NAT gateway:
# Fetch the port allocation error count for the last hour
aws cloudwatch get-metric-statistics \
--namespace AWS/NATGateway \
--metric-name ErrorPortAllocation \
--dimensions Name=NatGatewayId,Value=nat-0abc123 \
--start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 300 \
--statistics Sum \
--output table
Solutions
- Split destinations across multiple NAT gateways using route tables in different subnets.
- Use VPC endpoints for AWS service traffic to bypass the NAT gateway entirely.
- Implement connection pooling in your application to reduce the number of concurrent outbound connections.
- Scale horizontally by distributing workloads across multiple subnets, each with its own NAT gateway.
Best Practices for VPC Troubleshooting
1. Always Enable VPC Flow Logs
Flow logs are your post-mortem record of every packet accepted or rejected. Without them, you are guessing. Send them to CloudWatch Logs or S3 with a retention policy that matches your incident investigation window.
# Publish flow logs directly to S3 for cost-effective long-term storage
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-0abc123def456 \
--traffic-type REJECT \
--log-destination-type s3 \
--log-destination arn:aws:s3:::my-vpc-flowlogs-bucket/prod/
2. Use Infrastructure as Code
When VPC resources are defined in Terraform, CloudFormation, or the CDK, you can diff changes, review pull requests, and roll back quickly. Manual console changes are untraceable and are the leading cause of mysterious outages.
3. Tag Everything
Consistent tagging of subnets, route tables, security groups, and endpoints makes it far easier to identify which resources belong to which application or environment during an incident.
4. Document Your Network Topology
Maintain a current diagram showing VPCs, subnets, CIDR ranges, peering connections, VPN tunnels, and endpoints. Tools like the AWS VPC console visualizer or third-party solutions like Lucidchart can help keep this documentation accurate.
5. Implement a Layered Debugging Methodology
Always troubleshoot from the bottom up: physical/link layer (ENI attached?), network layer (route table correct?), transport layer (security group and NACL allowing ports?), application layer (DNS resolving, service listening?). This prevents you from chasing symptoms at the wrong layer.
6. Automate Connectivity Tests
Use scheduled Lambda functions or ECS tasks to periodically test connectivity between critical endpoints. This catches regressions before users do:
# Example: a simple connectivity test script
#!/bin/bash
ENDPOINTS=("https://s3.amazonaws.com" "https://sts.amazonaws.com" "tcp://10.10.5.10:5432")
for ep in "${ENDPOINTS[@]}"; do
if [[ "$ep" == https://* ]]; then
curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$ep" || echo "FAIL: $ep"
elif [[ "$ep" == tcp://* ]]; then
host=$(echo "$ep" | sed 's|tcp://||; s|:.*||')
port=$(echo "$ep" | sed 's|.*:||')
nc -z -w 5 "$host" "$port" && echo "OK: $ep" || echo "FAIL: $ep"
fi
done
7. Monitor Proactively with CloudWatch Alarms
Set alarms on key VPC-related metrics so you are alerted before users report problems:
# Alarm when a NAT gateway drops packets due to port exhaustion
aws cloudwatch put-metric-alarm \
--alarm-name NATGateway-PortExhaustion-Prod \
--namespace AWS/NATGateway \
--metric-name ErrorPortAllocation \
--dimensions Name=NatGatewayId,Value=nat-0abc123 \
--statistic Sum \
--period 300 \
--threshold 1 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 1 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:network-alerts
Conclusion
Troubleshooting VPC issues is as much about methodology as it is about tooling. By understanding how traffic flows through subnets, route tables, security groups, NACLs, gateways, and endpoints, you can systematically isolate problems instead of making blind changes. The key is to always start with the simplest possible question — is the resource configured correctly at this layer? — and work your way up the stack. Combine that mindset with proactive monitoring through Flow Logs, CloudWatch alarms, and automated connectivity tests, and you will dramatically reduce both the frequency and duration of VPC-related incidents. Remember that every troubleshooting session should end with a documented root cause and a preventive fix, whether that is an infrastructure-as-code update, a new alarm, or a runbook entry for the next on-call engineer.