Troubleshooting CloudWatch: Common Issues and Solutions

Introduction to CloudWatch Troubleshooting

Amazon CloudWatch is a comprehensive monitoring and observability service provided by AWS that collects and tracks metrics, collects and monitors log files, and sets alarms. While CloudWatch is powerful, developers and DevOps engineers frequently encounter issues related to log delivery, metric visibility, alarm behavior, and IAM permissions. This tutorial covers the most common CloudWatch problems and provides practical solutions with code examples to help you debug and resolve them quickly.

Why CloudWatch Troubleshooting Matters

When CloudWatch is not functioning as expected, it can lead to blind spots in your infrastructure monitoring. Missed alarms, missing logs, or inaccurate metrics can result in undetected outages, security incidents, and degraded application performance. Understanding how to troubleshoot CloudWatch effectively ensures your observability stack remains reliable and your team can respond to incidents proactively.

Common Issue 1: Logs Not Appearing in CloudWatch Logs

One of the most frequent issues developers face is that application or system logs are not showing up in CloudWatch Logs. This problem usually stems from IAM permissions, incorrect log group configuration, or network connectivity issues.

Checking IAM Permissions

The IAM role attached to your EC2 instance, ECS task, or Lambda function must have permissions to create log streams and put log events. A common mistake is granting only logs:CreateLogStream without logs:PutLogEvents.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:*:*:log-group:/my-app/*:*"
      ]
    }
  ]
}

Verifying the CloudWatch Agent Configuration

If you are using the CloudWatch Agent on EC2 or on-premises servers, verify the agent configuration file is correct and the agent service is running.

# Check agent status
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a status

# Start the agent with a specific config
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json \
  -s

A minimal agent configuration for pushing logs looks like this:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/myapp/application.log",
            "log_group_name": "/my-app/application",
            "log_stream_name": "{instance_id}",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          }
        ]
      }
    }
  }
}

Debugging with the AWS CLI

Use the AWS CLI to verify that log groups exist and to manually test log delivery. This helps isolate whether the issue is with the application or with AWS configuration.

# List all log groups
aws logs describe-log-groups --region us-east-1

# Describe log streams in a specific group
aws logs describe-log-streams \
  --log-group-name /my-app/application \
  --region us-east-1

# Manually put a test log event
aws logs put-log-events \
  --log-group-name /my-app/application \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s)000,message="Test log entry" \
  --region us-east-1

Common Issue 2: CloudWatch Alarms Stuck in INSUFFICIENT_DATA

Alarms frequently get stuck in the INSUFFICIENT_DATA state. This means CloudWatch does not have enough data points within the evaluation period to determine the alarm state.

Verifying Metric Existence

First, confirm the metric actually exists and is receiving data points. Use the CLI to check recent metric data:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time $(date -u -v-1H +"%Y-%m-%dT%H:%M:%S") \
  --end-time $(date -u +"%Y-%m-%dT%H:%M:%S") \
  --period 300 \
  --statistics Average \
  --region us-east-1

Checking Alarm Configuration

Review the alarm's period, evaluation periods, and statistic settings. If the period is too short or the evaluation periods are too many, the alarm may never have enough data. Here is how to describe an alarm and inspect its configuration:

aws cloudwatch describe-alarms \
  --alarm-names "HighCPUAlarm" \
  --region us-east-1

A properly configured alarm using Terraform might look like this:

resource "aws_cloudwatch_metric_alarm" "high_cpu" {
  alarm_name          = "HighCPUAlarm"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 300
  statistic           = "Average"
  threshold           = 80
  alarm_description   = "Triggers when CPU exceeds 80% for 10 minutes"
  dimensions = {
    InstanceId = "i-0123456789abcdef0"
  }
  alarm_actions = [aws_sns_topic.alerts.arn]
}

Common Causes of INSUFFICIENT_DATA

• The metric namespace or name is misspelled or uses incorrect casing
• Dimension keys or values do not match what is being published
• The resource being monitored has been stopped or terminated
• Custom metrics are not being published at the expected interval
• The period is shorter than the interval at which data is published

Common Issue 3: Custom Metrics Not Showing Up

When publishing custom metrics using PutMetricData, metrics may not appear immediately in the console. CloudWatch can take a few minutes to index new metrics, but persistent absence usually indicates a code or permissions issue.

Publishing Custom Metrics Correctly

Here is a Python example using boto3 to publish a custom metric:

import boto3
import time

cloudwatch = boto3.client('cloudwatch', region_name='us-east-1')

response = cloudwatch.put_metric_data(
    Namespace='MyApp/Performance',
    MetricData=[
        {
            'MetricName': 'RequestLatency',
            'Dimensions': [
                {
                    'Name': 'Endpoint',
                    'Value': '/api/users'
                }
            ],
            'Value': 150.5,
            'Unit': 'Milliseconds',
            'Timestamp': int(time.time())
        }
    ]
)

print(f"Metric published: {response}")

Validating Metric Data

After publishing, verify the metric is available by listing metrics with the correct namespace:

aws cloudwatch list-metrics \
  --namespace "MyApp/Performance" \
  --region us-east-1

If the metric does not appear, check the following:

• Ensure the IAM user or role has cloudwatch:PutMetricData permission
• Verify the namespace string matches exactly, including case
• Confirm the timestamp is within the last two weeks; older timestamps are rejected
• Check that you are looking in the correct AWS region

Common Issue 4: CloudWatch Logs Insights Queries Returning No Results

Logs Insights is a powerful tool for searching log data, but queries often return no results due to syntax errors, incorrect time ranges, or querying the wrong log group.

Writing Effective Queries

Here is an example of a Logs Insights query that filters for error-level logs and extracts useful fields:

fields @timestamp, @message, level, requestId, errorMessage
| filter level == "ERROR"
| parse @message "userId: *" as userId
| stats count() by userId
| sort @timestamp desc
| limit 100

Running Queries Programmatically

You can also run Logs Insights queries using the AWS SDK. This is useful for automated reporting or alerting:

import boto3
import time

logs = boto3.client('logs', region_name='us-east-1')

query = """
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50
"""

start_query = logs.start_query(
    logGroupName='/my-app/application',
    startTime=int(time.time()) - 3600,
    endTime=int(time.time()),
    queryString=query
)

query_id = start_query['queryId']

# Poll for results
while True:
    results = logs.get_query_results(queryId=query_id)
    if results['status'] == 'Complete':
        for result in results['results']:
            print(result)
        break
    time.sleep(1)

Troubleshooting Empty Results

• Verify the time range covers the period when logs were generated
• Ensure the log group name is correct and contains log streams
• Check that the query syntax is valid; use the console query editor for validation
• Remember that filter is case-sensitive by default
• Use like /pattern/i for case-insensitive matching

Common Issue 5: CloudWatch Agent Not Sending Metrics

The CloudWatch Agent can fail to send metrics due to misconfiguration, missing dependencies, or connectivity problems. Systematic debugging is essential.

Checking Agent Logs

The agent writes its own logs that can reveal configuration errors or connectivity issues:

# View the agent configuration log
sudo cat /opt/aws/amazon-cloudwatch-agent/logs/configuration.log

# View the agent operational log
sudo cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

# Check recent agent errors
sudo grep -i "error" /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log | tail -20

Validating Configuration with the Wizard

Use the configuration wizard to generate a valid config file and compare it with your existing one:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Testing Connectivity

Ensure the instance can reach CloudWatch endpoints. This is especially important in VPC environments without public internet access:

# Test connectivity to CloudWatch endpoints
curl -I https://monitoring.us-east-1.amazonaws.com/

# If using VPC endpoints, verify the endpoint exists
aws ec2 describe-vpc-endpoints \
  --filters Name=service-name,Values=com.amazonaws.us-east-1.monitoring \
  --region us-east-1

Best Practices for CloudWatch Troubleshooting

Use Structured Logging

Structured JSON logs make it significantly easier to query and filter log data in Logs Insights. Instead of writing free-text log messages, emit JSON objects:

import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def handler(event, context):
    log_entry = {
        "level": "INFO",
        "requestId": context.aws_request_id,
        "action": "process_order",
        "orderId": event.get("orderId"),
        "duration_ms": 125
    }
    logger.info(json.dumps(log_entry))
    return {"statusCode": 200, "body": "Order processed"}

Set Up Dashboards for Proactive Monitoring

Create CloudWatch Dashboards that aggregate key metrics across your infrastructure. This gives you a single pane of glass for identifying anomalies before they trigger alarms. Use Infrastructure as Code to manage dashboards:

resource "aws_cloudwatch_dashboard" "main" {
  dashboard_name = "ApplicationDashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", "FunctionName", "my-function"],
            ["AWS/Lambda", "Errors", "FunctionName", "my-function"]
          ]
          period  = 300
          stat    = "Sum"
          region  = "us-east-1"
          title   = "Lambda Invocations and Errors"
        }
      }
    ]
  })
}

Implement Alarm-Based Alerting with SNS

Always connect alarms to SNS topics so your team receives notifications. Route critical alarms to PagerDuty or Slack via SNS subscriptions:

resource "aws_sns_topic" "alerts" {
  name = "cloudwatch-alerts"
}

resource "aws_sns_topic_subscription" "email_alerts" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = "oncall@mycompany.com"
}

Use Metric Math for Advanced Alerting

Metric math allows you to combine multiple metrics into a single alarm expression. For example, you can create an alarm based on an error rate calculated from total requests and error counts:

resource "aws_cloudwatch_metric_alarm" "error_rate" {
  alarm_name          = "HighErrorRate"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  threshold           = 0.05
  alarm_description   = "Error rate exceeds 5%"

  metric_query {
    id          = "errors"
    return_data = false
    metric {
      namespace   = "MyApp"
      metric_name = "Errors"
      period      = 300
      stat        = "Sum"
    }
  }

  metric_query {
    id          = "requests"
    return_data = false
    metric {
      namespace   = "MyApp"
      metric_name = "Requests"
      period      = 300
      stat        = "Sum"
    }
  }

  metric_query {
    id          = "error_rate"
    expression  = "errors / requests"
    return_data = true
  }

  alarm_actions = [aws_sns_topic.alerts.arn]
}

Enable CloudWatch Agent on All Compute Resources

Ensure the CloudWatch Agent is installed and configured on every EC2 instance, ECS container instance, and on-premises server. Use Systems Manager (SSM) to automate agent deployment across your fleet:

resource "aws_ssm_association" "install_cw_agent" {
  name = "AWS-ConfigureAWSPackage"

  targets {
    key    = "tag:Environment"
    values = ["production"]
  }

  parameters = {
    action = "Install"
    name   = "AmazonCloudWatchAgent"
  }

  schedule_expression = "rate(1 day)"
}

Monitor CloudWatch Itself

CloudWatch has its own service quotas and limits. Monitor the number of custom metrics, alarm updates, and API calls to avoid hitting throttling limits. Use the AWS/Usage namespace to track service usage:

aws cloudwatch get-metric-statistics \
  --namespace AWS/Usage \
  --metric-name CallCount \
  --dimensions Name=Service,Value=CloudWatch Name=Resource,Value=PutMetricData \
  --start-time $(date -u -v-1H +"%Y-%m-%dT%H:%M:%S") \
  --end-time $(date -u +"%Y-%m-%dT%H:%M:%S") \
  --period 300 \
  --statistics Sum \
  --region us-east-1

Conclusion

Troubleshooting CloudWatch effectively requires a systematic approach: verify IAM permissions, confirm agent configuration, validate metric and log data through the CLI, and ensure your alarm settings align with your data publishing intervals. By following the best practices outlined in this tutorial—structured logging, Infrastructure as Code for dashboards and alarms, metric math for advanced alerting, and proactive agent deployment—you can build a robust observability foundation that minimizes blind spots and accelerates incident response. Remember that CloudWatch is not just a passive monitoring tool; when configured correctly, it becomes an active participant in keeping your applications healthy and your team informed.