Introduction to Route 53 Troubleshooting
Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service designed to route end users to Internet applications. Despite its reliability, misconfigurations, propagation delays, and edge-case behaviors can cause frustrating outages. This tutorial walks you through the most common Route 53 issues developers encounter and provides actionable solutions, diagnostic commands, and best practices to keep your DNS infrastructure healthy.
What Is Route 53?
Route 53 performs three primary functions: domain registration, DNS routing, and health checking. It supports multiple routing policies—simple, weighted, latency-based, failover, geolocation, geoproximity, and multivalue answer—each with its own quirks. Understanding how these policies interact with TTLs, health checks, and resolver caching is essential when diagnosing problems.
Why Troubleshooting Matters
DNS issues are notoriously deceptive. A misconfigured record may not surface for hours due to caching, and symptoms often appear in unrelated services—failed TLS handshakes, broken email delivery, or intermittent 502 errors. A systematic troubleshooting approach reduces mean time to resolution (MTTR) and prevents cascading failures across your stack.
Issue 1: DNS Records Not Propagating
The most frequent complaint is that a newly created or modified record is not visible globally. This is usually caused by TTL caching, incorrect record type, or delegation issues.
Diagnosing Propagation
Use dig or nslookup to query Route 53 authoritative servers directly, bypassing recursive resolvers:
# Query the authoritative name servers for your hosted zone
dig NS example.com +short
# Query a specific record directly against an authoritative server
dig @ns-1234.awsdns-58.org api.example.com A +short
# Check current TTL from a public resolver
dig api.example.com A +noall +answer
If the authoritative server returns the expected value but public resolvers do not, the issue is caching. Wait for the previous TTL to expire. If the authoritative server itself returns stale data, verify the record was saved in the correct hosted zone.
Common Causes and Fixes
- Wrong hosted zone: A record created in a private hosted zone will not resolve on the public internet. Confirm the zone ID with
aws route53 list-hosted-zones. - High TTL values: A 24-hour TTL means changes take a full day to propagate. Lower TTLs temporarily before planned changes.
- Delegation not configured: The registrar's NS records must match the Route 53 hosted zone's name servers. Mismatches break the entire delegation chain.
Issue 2: Health Checks Marking Endpoints Unhealthy
Route 53 health checks monitor endpoints and drive failover routing. When health checks fail unexpectedly, traffic stops flowing even if the endpoint is actually healthy.
Verifying Health Check Configuration
# List all health checks
aws route53 list-health-checks
# Get the status of a specific health check
aws route53 get-health-check-status \
--health-check-id abc12345-6789-0abc-def1-234567890abc
# Describe a health check's configuration
aws route53 get-health-check \
--health-check-id abc12345-6789-0abc-def1-234567890abc
Common Health Check Pitfalls
- String matching failures: If you configure a search string, Route 53 expects it in the first 5,120 bytes of the response body. A dynamic page that sometimes omits the string will flap.
- HTTPS without valid certificates: Route 53 validates the certificate chain. Self-signed or expired certificates cause failures. Use a trusted CA and keep certificates renewed.
- Firewall blocking Route 53 IPs: Health checkers originate from a documented range of AWS IP addresses. Ensure your security groups and WAF rules allow them.
- Insufficient failure threshold: The default is 3 consecutive failures (30 seconds). Noisy networks may need a higher threshold to avoid false positives.
Testing Endpoint Reachability
# Simulate a health check request from an external host
curl -v -H "User-Agent: Amazon Route 53 Health Check Service" \
https://api.example.com/health
# Verify the expected response body contains the search string
curl -s https://api.example.com/health | grep -o "HEALTHY"
Issue 3: Alias Records Not Resolving to Load Balancers
Alias records are Route 53-specific extensions that point to AWS resources like Application Load Balancers, CloudFront distributions, or S3 website endpoints. Misconfigured aliases return NXDOMAIN or point to the wrong resource.
Correct Alias Record Syntax
{
"Comment": "Alias to ALB",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "app.example.com.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"DNSName": "dualstack.my-alb-1234567890.us-east-1.elb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
}
]
}
Two critical details often trip up developers:
- HostedZoneId of the target: This is not your hosted zone ID. It is the fixed zone ID of the AWS resource. For ALBs in us-east-1, it is
Z35SXDOTRQ7X7K. Each region and resource type has its own value documented by AWS. - Trailing dot: The
DNSNamemust include the trailing period. Omitting it causes validation errors. - dualstack prefix: For ALBs, prefix the DNS name with
dualstack.to support IPv6 clients.
Applying the Change
aws route53 change-resource-record-sets \
--hosted-zone-id Z1D633PEXAMPLE \
--change-batch file://alias-record.json
Issue 4: Latency Routing Sending Users to Distant Regions
Latency-based routing picks the region with the lowest DNS resolution latency from the user's recursive resolver, not from the user's physical location. This distinction causes surprising behavior.
Why Latency Routing Seems Wrong
If a user in Europe uses a DNS resolver located in the United States (common with corporate networks or some ISPs), Route 53 measures latency to the US resolver and may route the user to a US region. This is by design, not a bug.
Mitigation Strategies
- Combine with geolocation routing when you need deterministic region selection based on user location rather than resolver location.
- Use EDNS Client Subnet (ECS): Some recursive resolvers forward the client subnet prefix to Route 53, improving accuracy. Google Public DNS and OpenDNS support ECS.
- Monitor with CloudWatch: Track health check metrics per region to detect routing imbalances.
Example Latency Record Set
{
"Comment": "Latency routing across regions",
"Changes": [
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "api.example.com.",
"Type": "A",
"SetIdentifier": "us-east-1",
"Region": "us-east-1",
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"DNSName": "dualstack.us-alb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
},
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "api.example.com.",
"Type": "A",
"SetIdentifier": "eu-west-1",
"Region": "eu-west-1",
"AliasTarget": {
"HostedZoneId": "Z32O12XQL9SWCN",
"DNSName": "dualstack.eu-alb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
}
]
}
Issue 5: Private Hosted Zones Not Resolving in VPC
Private hosted zones resolve DNS names within one or more VPCs. When queries fail inside a VPC, the cause is almost always association, overlapping zones, or resolver configuration.
Checking VPC Association
# List private hosted zones
aws route53 list-hosted-zones-by-vpc \
--vpc-id vpc-0abc123def456 \
--vpc-region us-east-1
# Associate a VPC with a private hosted zone
aws route53 associate-vpc-with-private-hosted-zone \
--hosted-zone-id Z1D633PEXAMPLE \
--vpc VPCRegion=us-east-1,VPCId=vpc-0abc123def456
Common Private Zone Problems
- Missing VPC association: The VPC must be explicitly associated. Creating the zone in the same account does not auto-associate.
- Overlapping public and private zones: If you have a public zone for
example.comand a private zone forinternal.example.com, the private zone takes precedence for that subdomain inside the VPC. However, a private zone forexample.comwill shadow the public zone entirely within the VPC. - DHCP Options Set: The VPC must use AmazonProvidedDNS or a custom resolver that forwards to Route 53 Resolver. Verify with
aws ec2 describe-dhcp-options. - Cross-account VPCs: When the VPC is in a different account than the hosted zone, you must create an authorization and association in both accounts.
Cross-Account Association
# Account A (zone owner): authorize Account B's VPC
aws route53 create-vpc-association-authorization \
--hosted-zone-id Z1D633PEXAMPLE \
--vpc VPCRegion=us-east-1,VPCId=vpc-0abc123def456
# Account B (VPC owner): associate the VPC
aws route53 associate-vpc-with-private-hosted-zone \
--hosted-zone-id Z1D633PEXAMPLE \
--vpc VPCRegion=us-east-1,VPCId=vpc-0abc123def456
Issue 6: Failover Routing Not Switching Traffic
Failover routing uses primary and secondary records tied to health checks. When the primary fails, traffic should move to the secondary. If it does not, check the health check binding and record configuration.
Correct Failover Configuration
{
"Comment": "Failover routing",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "app.example.com.",
"Type": "A",
"SetIdentifier": "primary",
"Failover": "PRIMARY",
"HealthCheckId": "abc12345-6789-0abc-def1-234567890abc",
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"DNSName": "dualstack.primary-alb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
},
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "app.example.com.",
"Type": "A",
"SetIdentifier": "secondary",
"Failover": "SECONDARY",
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"DNSName": "dualstack.secondary-alb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
}
]
}
Key points to verify:
- The PRIMARY record must have a health check. Without it, Route 53 always considers it healthy and never fails over.
- The SECONDARY record does not require a health check, but adding one lets you detect total failure.
EvaluateTargetHealthmust betrueon alias targets so Route 53 considers the target's own health.- Failover is not instant. It depends on the health check interval, failure threshold, and the TTL of records served to resolvers.
Issue 7: Query Logging and Auditing
When troubleshooting intermittent issues, query logs are invaluable. Route 53 can log all DNS queries for a hosted zone to CloudWatch Logs, S3, or Kinesis Firehose.
Enabling Query Logging
# Create a CloudWatch Logs resource policy (one-time)
aws route53 create-query-logging-config \
--hosted-zone-id Z1D633PEXAMPLE \
--log-destination-arn arn:aws:logs:us-east-1:123456789012:log-group:/aws/route53/example.com
Analyzing Logs
Query logs include the resolver IP, query name, query type, response code, and the record that was returned. Filter for SERVFAIL or NXDOMAIN responses to find misconfigurations:
# Search CloudWatch Logs for failed queries
aws logs filter-log-events \
--log-group-name /aws/route53/example.com \
--filter-pattern '"SERVFAIL"' \
--limit 50
Best Practices for Route 53 Reliability
- Use Infrastructure as Code: Manage records with Terraform, AWS CDK, or CloudFormation. Manual edits via the console are hard to audit and reproduce.
- Lower TTLs before changes: Reduce TTLs to 60 seconds 24 hours before planned DNS changes, then raise them afterward to reduce resolver load.
- Always configure health checks for production records: Even with simple routing, health checks enable CloudWatch alarms and visibility.
- Enable query logging in production: Logs are cheap relative to the diagnostic value during incidents.
- Use Alias records over CNAME: Alias records have no charge for queries and support zone apex names, unlike CNAME.
- Document routing policies: Complex multi-record configurations with weighted, latency, and geolocation policies are difficult to reason about without documentation.
- Test failover regularly: Simulate primary failure in staging to confirm failover behavior matches expectations.
- Monitor with CloudWatch alarms: Set alarms on health check failure counts and unhealthy child record counts.
CloudWatch Alarm for Health Check Failure
aws cloudwatch put-metric-alarm \
--alarm-name "Route53-HealthCheck-Failure" \
--metric-name HealthCheckPercentageHealthy \
--namespace AWS/Route53 \
--statistic Average \
--period 60 \
--threshold 100 \
--comparison-operator LessThanThreshold \
--evaluation-periods 3 \
--dimensions Name=HealthCheckId,Value=abc12345-6789-0abc-def1-234567890abc \
--alarm-actions arn:aws:sns:us-east-1:123456789012:alerts
Conclusion
Route 53 is a robust DNS service, but its flexibility introduces configuration complexity that can lead to subtle failures. By understanding how TTLs, health checks, routing policies, and VPC associations interact, you can systematically diagnose and resolve issues. The key habits are querying authoritative servers directly, validating health check endpoints from external hosts, managing records as code, and enabling query logging before you need it. With these tools and practices, you can reduce DNS-related downtime and respond confidently when routing problems arise.