Monitoring RDS: Metrics, Alarms, and Dashboards

What is RDS Monitoring

RDS monitoring encompasses the collection, visualization, and alerting on operational metrics for Amazon Relational Database Service instances. AWS provides three complementary layers of monitoring that give you visibility into your database fleet at different granularities and depths:

• CloudWatch Metrics — automatically collected host-level and database-level metrics at 1-minute intervals (or 1-second with Enhanced Monitoring enabled), covering CPU, memory, disk I/O, network throughput, and database connections
• Enhanced Monitoring — OS-level metrics gathered by an agent running inside the DB instance, exposing process lists, memory paging, swap usage, and CPU distribution at up to 1-second granularity
• Performance Insights — a database engine-level analytics layer that captures SQL query performance, wait events, and execution plans, allowing you to identify the exact queries causing bottlenecks

Together these tools form a comprehensive observability stack that helps you detect problems early, troubleshoot performance regressions, and make data-driven scaling decisions.

Why RDS Monitoring Matters

Databases sit at the heart of most application architectures. A degraded database instance causes cascading failures across every service that depends on it. Without proper monitoring, teams typically discover database issues through customer-facing symptoms — slow page loads, timeouts, or outright errors — rather than proactively. The consequences of inadequate monitoring include:

• Extended outage durations — without alarms, a CPU spike caused by a poorly optimized query can go unnoticed for hours while users suffer degraded experiences
• Missed cost optimization opportunities — over-provisioned instances waste thousands of dollars monthly; monitoring reveals actual utilization patterns so you can right-size
• Slow RCA (root cause analysis) — when an incident happens, lacking historical metrics means you cannot pinpoint what changed, turning a 30-minute investigation into a multi-day effort
• Storage surprises — running out of allocated storage causes database freezing; monitoring storage growth rates lets you take action before hitting limits

Effective monitoring transforms database operations from reactive firefighting into proactive management. The investment in setting up proper metrics, alarms, and dashboards pays for itself the first time you prevent an outage.

CloudWatch Metrics: The Foundation

Default RDS Metrics

Every RDS instance automatically publishes a core set of metrics to CloudWatch under the AWS/RDS namespace. These metrics are available at no additional cost and are retained for 15 months, with decreasing granularity over time. The most important ones are:

• CPUUtilization — percentage of allocated CPU consumed
• DatabaseConnections — active database connections
• FreeableMemory — RAM available (reported in bytes, not percentage)
• ReadIOPS / WriteIOPS — disk operations per second
• ReadLatency / WriteLatency — EBS I/O latency in milliseconds
• DiskQueueDepth — number of pending I/O operations (high values indicate EBS bottleneck)
• NetworkReceiveThroughput / NetworkTransmitThroughput — bandwidth utilization
• FreeStorageSpace — bytes of unused storage; critical for avoiding auto-scaling limits

Retrieving Metrics with the AWS CLI

# Get CPU utilization for a specific DB instance over the last hour
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name CPUUtilization \
  --dimensions Name=DBInstanceIdentifier,Value=mydb-prod-01 \
  --start-time "$(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%SZ')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --period 300 \
  --statistics Average Maximum \
  --output table

# Check free storage space and trigger an alert programmatically
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name FreeStorageSpace \
  --dimensions Name=DBInstanceIdentifier,Value=mydb-prod-01 \
  --start-time "$(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%SZ')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
  --period 3600 \
  --statistics Minimum \
  --query 'Datapoints[0].Minimum' \
  --output text

Custom Metrics via CloudWatch Agent or Lambda

Sometimes you need metrics beyond what RDS provides natively — for example, replication lag in a self-managed read replica, or application-level transaction rates. You can publish custom metrics to CloudWatch using the PutMetricData API:

import boto3
import time
import psycopg2

cloudwatch = boto3.client('cloudwatch')

def publish_replication_lag(db_host, db_name, db_user, db_password):
    conn = psycopg2.connect(
        host=db_host,
        dbname=db_name,
        user=db_user,
        password=db_password
    )
    cursor = conn.cursor()
    
    # Query replication lag in seconds
    cursor.execute("""
        SELECT EXTRACT(epoch FROM (NOW() - pg_last_xact_replay_timestamp())) 
        AS replication_lag_seconds
    """)
    lag = cursor.fetchone()[0] or 0.0
    cursor.close()
    conn.close()
    
    response = cloudwatch.put_metric_data(
        Namespace='MyApplication/Postgres',
        MetricData=[{
            'MetricName': 'ReplicationLagSeconds',
            'Dimensions': [
                {'Name': 'DBInstanceIdentifier', 'Value': 'mydb-prod-01'}
            ],
            'Value': float(lag),
            'Unit': 'Seconds',
            'Timestamp': time.time()
        }]
    )
    print(f"Published replication lag: {lag:.1f}s")
    
# Run this on a schedule (e.g., every 60 seconds via Lambda + EventBridge)
publish_replication_lag('mydb-prod-01.xxxx.us-east-1.rds.amazonaws.com', 
                        'appdb', 'monitor_user', 'securepassword')

Enhanced Monitoring: OS-Level Visibility

What Enhanced Monitoring Provides

Enhanced Monitoring installs a lightweight agent on the underlying DB host. It reports metrics that CloudWatch default metrics cannot capture, such as which specific processes are consuming CPU or which memory allocations are causing swapping. Key Enhanced Monitoring metrics include:

• Process-level CPU and memory — you see exactly which OS processes (mysqld, postgres, aurora, etc.) are consuming resources
• Memory breakdown — total, free, buffers, cached, swap usage, huge pages
• Disk stats — per-device read/write counts and latency
• Load averages — 1-minute, 5-minute, and 15-minute load
• Network stats — per-interface packet counts

Enabling Enhanced Monitoring

# Enable Enhanced Monitoring with 1-second granularity on an existing instance
aws rds modify-db-instance \
  --db-instance-identifier mydb-prod-01 \
  --monitoring-interval 1 \
  --monitoring-role-arn arn:aws:iam::123456789012:role/rds-enhanced-monitoring \
  --apply-immediately

# Create the IAM role for Enhanced Monitoring (one-time setup)
aws iam create-role \
  --role-name rds-enhanced-monitoring \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "monitoring.rds.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

aws iam attach-role-policy \
  --role-name rds-enhanced-monitoring \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonRDSEnhancedMonitoringRole

Querying Enhanced Monitoring Data

# Enhanced Monitoring metrics are published to CloudWatch Logs, not CloudWatch Metrics
# Retrieve the latest log stream for a specific instance

LOG_GROUP="/aws/rds/instance/mydb-prod-01/enhanced"

# List log streams sorted by last event time (most recent first)
aws logs describe-log-streams \
  --log-group-name "$LOG_GROUP" \
  --order-by LastEventTime \
  --descending \
  --limit 1 \
  --query 'logStreams[0].logStreamName' \
  --output text

# Fetch recent entries from that stream
STREAM_NAME=$(aws logs describe-log-streams \
  --log-group-name "$LOG_GROUP" \
  --order-by LastEventTime \
  --descending \
  --limit 1 \
  --query 'logStreams[0].logStreamName' \
  --output text)

aws logs get-log-events \
  --log-group-name "$LOG_GROUP" \
  --log-stream-name "$STREAM_NAME" \
  --limit 10 \
  --output json | jq '.events[] | .message | fromjson | 
    {timestamp: .timestamp, 
     cpu_utilization: .process_list[].cpuUsedPc | add, 
     memory_used_mb: (.memory.total - .memory.free) / 1024}'

Performance Insights: Query-Level Visibility

Understanding Performance Insights

Performance Insights goes beyond infrastructure metrics to analyze what the database engine is actually doing. It shows you the top SQL queries by wait time, CPU consumption, and I/O. This is invaluable because a "high CPU" CloudWatch alarm tells you that something is wrong, but Performance Insights tells you exactly which query and which wait event is responsible.

Performance Insights captures:

• Top SQL — the most impactful queries ranked by total execution time
• Wait events — categorized by type (CPU, IO, Lock, etc.) showing where time is spent
• Execution plans — for supported engines, the actual query plans used
• Hourly snapshots — up to 7 days of historical query performance data in the free tier, extendable to 2 years with paid retention

Programmatic Access to Performance Insights

import boto3
import json
from datetime import datetime, timedelta

pi = boto3.client('pi')  # Performance Insights client

# Fetch top queries by total time for the last hour
response = pi.get_resource_metrics(
    ServiceType='RDS',
    Identifier='db-ABCDEFGHIJKLMNOP',  # DB instance resource ID, not the friendly name
    StartTime=datetime.utcnow() - timedelta(hours=1),
    EndTime=datetime.utcnow(),
    PeriodInSeconds=3600,
    MetricQueries=[{
        'Metric': 'db.sql_tokenized.stats.sum_exec_time',
        'GroupBy': {
            'Group': 'db.sql_tokenized.statement',
            'Dimensions': ['db.sql_tokenized.statement'],
            'Limit': 5
        }
    }]
)

for metric in response['Metrics']:
    print(f"\nMetric: {metric['Metric']}")
    for datapoint in metric['DataPoints']:
        print(f"  Time: {datapoint['Timestamp']}")
        for dimension_data in datapoint['DimensionData']:
            query_text = dimension_data.get('Value', 'unknown')
            # Decode the statement from Performance Insights format
            print(f"  Query: {query_text[:200]}...")

# Get dimension details for a specific SQL statement to see full query text
response = pi.get_dimension_details(
    ServiceType='RDS',
    Identifier='db-ABCDEFGHIJKLMNOP',
    Group='db.sql_tokenized.statement',
    GroupIdentifier='SELECT/*...*/',  # From the previous response
    RequestedDimensions=['db.sql.statement.statement']
)

Common Performance Insights Wait Events

Understanding wait events is key to diagnosing issues quickly. Here are the most common ones you will encounter:

• CPU — the query is actively using CPU; if dominant, look for missing indexes or inefficient joins
• IO:DataRead — reading data from disk; indicates queries scanning large tables that do not fit in the buffer pool
• IO:DataWrite — writing data to disk; common during bulk inserts or checkpoint operations
• Lock:tuple (Postgres) / Lock:rowlock (MySQL) — row-level contention; indicates concurrent transactions touching the same rows
• LWLock:buffer_content (Postgres) — contention for shared buffer pages; often caused by many concurrent reads of hot data
• synch/mutex/innodb/aurora (MySQL) — contention on internal mutexes; may require engine tuning

Setting Up CloudWatch Alarms

Core Alarms Every RDS Instance Needs

Alarms transform passive monitoring into active alerting. When a metric crosses a threshold you define, CloudWatch Alarms can send notifications via SNS, trigger Auto Scaling actions, or invoke Lambda functions for automated remediation. The following alarms should be considered the minimum baseline for any production RDS instance:

• High CPU — sustained CPU above 80-90% for multiple periods
• Low Freeable Memory — less than 10-20% of instance memory remains free
• Low Storage Space — less than 10-15% free storage or an absolute threshold like 100GB remaining
• High Connection Count — approaching the instance's max_connections limit
• High Read/Write Latency — EBS latency above 10-20ms sustained
• Disk Queue Depth — sustained queue depth above 1-2 indicates EBS bottleneck

Creating Alarms via CloudFormation

# CloudFormation snippet for a production-grade alarm suite
# Save as rds-alarms.yaml and deploy with:
# aws cloudformation deploy --template-file rds-alarms.yaml --stack-name rds-alarms-prod

Parameters:
  DBInstanceIdentifier:
    Type: String
    Default: 'mydb-prod-01'
  SNSTopicArn:
    Type: String
    Description: SNS topic ARN for alarm notifications
  InstanceMemoryGB:
    Type: Number
    Default: 8
    Description: Total memory of the RDS instance in GB

Resources:
  # --- High CPU Alarm ---
  HighCPUAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${DBInstanceIdentifier}-HighCPU"
      AlarmDescription: !Sub "CPU utilization exceeds 85% for ${DBInstanceIdentifier}"
      Namespace: AWS/RDS
      MetricName: CPUUtilization
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref DBInstanceIdentifier
      Statistic: Average
      Period: 300
      EvaluationPeriods: 3
      Threshold: 85
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref SNSTopicArn
      TreatMissingData: notBreaching

  # --- Low Memory Alarm (percentage-based using math expressions) ---
  LowMemoryAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${DBInstanceIdentifier}-LowMemory"
      AlarmDescription: !Sub "Freeable memory drops below 10% of total for ${DBInstanceIdentifier}"
      Metrics:
        - Id: freeable
          Namespace: AWS/RDS
          MetricName: FreeableMemory
          Dimensions:
            - Name: DBInstanceIdentifier
              Value: !Ref DBInstanceIdentifier
          Statistic: Minimum
          Period: 300
        - Id: total_memory
          Expression: !Sub "${InstanceMemoryGB * 1024 * 1024 * 1024}"
          Label: TotalMemoryBytes
          ReturnData: false
        - Id: threshold_10_percent
          Expression: !Sub "${InstanceMemoryGB * 1024 * 1024 * 1024 * 0.10}"
          Label: TenPercentOfTotal
          ReturnData: false
      EvaluationPeriods: 3
      Threshold: 0
      ComparisonOperator: LessThanThreshold
      AlarmActions:
        - !Ref SNSTopicArn
      TreatMissingData: notBreaching

  # --- Low Storage Space Alarm ---
  LowStorageAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${DBInstanceIdentifier}-LowStorage"
      AlarmDescription: !Sub "Free storage space below 50GB for ${DBInstanceIdentifier}"
      Namespace: AWS/RDS
      MetricName: FreeStorageSpace
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref DBInstanceIdentifier
      Statistic: Minimum
      Period: 300
      EvaluationPeriods: 2
      Threshold: 50000000000  # 50GB in bytes
      ComparisonOperator: LessThanThreshold
      AlarmActions:
        - !Ref SNSTopicArn
      TreatMissingData: breaching

  # --- High Disk Queue Depth ---
  HighDiskQueueAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${DBInstanceIdentifier}-HighDiskQueue"
      AlarmDescription: !Sub "Disk queue depth exceeds 2 for ${DBInstanceIdentifier}"
      Namespace: AWS/RDS
      MetricName: DiskQueueDepth
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref DBInstanceIdentifier
      Statistic: Average
      Period: 300
      EvaluationPeriods: 3
      Threshold: 2
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref SNSTopicArn
      TreatMissingData: notBreaching

  # --- High Database Connections (percentage of max) ---
  HighConnectionsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${DBInstanceIdentifier}-HighConnections"
      AlarmDescription: !Sub "Database connections exceed 80% of maximum"
      Namespace: AWS/RDS
      MetricName: DatabaseConnections
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref DBInstanceIdentifier
      Statistic: Maximum
      Period: 300
      EvaluationPeriods: 2
      Threshold: 200  # Adjust based on your instance's max_connections
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref SNSTopicArn
      TreatMissingData: notBreaching

  # --- Read Latency Alarm ---
  HighReadLatencyAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub "${DBInstanceIdentifier}-HighReadLatency"
      AlarmDescription: !Sub "EBS read latency exceeds 20ms for ${DBInstanceIdentifier}"
      Namespace: AWS/RDS
      MetricName: ReadLatency
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref DBInstanceIdentifier
      Statistic: Average
      Period: 300
      EvaluationPeriods: 3
      Threshold: 0.02  # 20ms in seconds (CloudWatch reports latency in seconds)
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref SNSTopicArn
      TreatMissingData: notBreaching

Creating Alarms with Terraform

# terraform snippet: rds_alarms.tf
resource "aws_cloudwatch_metric_alarm" "high_cpu" {
  alarm_name          = "${var.db_instance_identifier}-HighCPU"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "CPUUtilization"
  namespace           = "AWS/RDS"
  period              = 300
  statistic           = "Average"
  threshold           = 85
  alarm_description   = "CPU utilization exceeds 85% for ${var.db_instance_identifier}"
  alarm_actions       = [var.sns_topic_arn]
  
  dimensions = {
    DBInstanceIdentifier = var.db_instance_identifier
  }
}

resource "aws_cloudwatch_metric_alarm" "low_storage" {
  alarm_name          = "${var.db_instance_identifier}-LowStorage"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 2
  metric_name         = "FreeStorageSpace"
  namespace           = "AWS/RDS"
  period              = 300
  statistic           = "Minimum"
  threshold           = 50000000000  # 50GB
  alarm_description   = "Free storage below 50GB for ${var.db_instance_identifier}"
  alarm_actions       = [var.sns_topic_arn]
  
  dimensions = {
    DBInstanceIdentifier = var.db_instance_identifier
  }
}

resource "aws_cloudwatch_metric_alarm" "high_read_latency" {
  alarm_name          = "${var.db_instance_identifier}-HighReadLatency"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "ReadLatency"
  namespace           = "AWS/RDS"
  period              = 300
  statistic           = "Average"
  threshold           = 0.02  # 20ms
  alarm_description   = "Read latency exceeds 20ms"
  alarm_actions       = [var.sns_topic_arn]
  
  dimensions = {
    DBInstanceIdentifier = var.db_instance_identifier
  }
}

Creating an SNS Topic for Alarm Delivery

# Create an SNS topic and subscribe your team's email or Slack webhook
aws sns create-topic --name rds-alarms-prod

# Subscribe email endpoint
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:rds-alarms-prod \
  --protocol email \
  --notification-endpoint dba-team@example.com

# For Slack integration, use a Lambda subscriber that transforms
# SNS messages into Slack webhook payloads
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:rds-alarms-prod \
  --protocol lambda \
  --notification-endpoint arn:aws:lambda:us-east-1:123456789012:function:sns-to-slack

Building CloudWatch Dashboards

Why Dashboards Matter

Alarms alert you to problems, but dashboards give you situational awareness before alarms fire. A well-designed dashboard lets an operator glance at a single screen and instantly understand the health of every database instance — CPU trends, memory pressure, storage growth, and query performance — without running ad-hoc CLI commands.

Dashboard Design Principles

• One dashboard per environment — production, staging, and development each get their own dashboard to avoid noise
• Group metrics by instance — vertically stack related metrics (CPU, memory, I/O) for each DB instance so the eye can scan vertically
• Use period-appropriate time ranges — default view shows last 3 hours; provide quick links for 1 day, 1 week, 1 month
• Include annotation widgets — mark deployment events and configuration changes directly on the timeline
• Surface Performance Insights data — embed the top queries widget so operators can immediately see what queries are running when metrics spike

Creating a Dashboard via CloudFormation

# CloudFormation snippet for a comprehensive RDS monitoring dashboard
Resources:
  RDSMonitoringDashboard:
    Type: AWS::CloudWatch::Dashboard
    Properties:
      DashboardName: !Sub "${EnvironmentName}-rds-monitoring"
      DashboardBody: !Sub |
        {
          "widgets": [
            {
              "type": "metric",
              "x": 0, "y": 0, "width": 24, "height": 6,
              "properties": {
                "title": "${DBInstanceIdentifier} - Overview",
                "metrics": [
                  ["AWS/RDS", "CPUUtilization", 
                   "DBInstanceIdentifier", "${DBInstanceIdentifier}",
                   {"stat": "Average", "period": 60}],
                  [".", "FreeableMemory", ".", ".", 
                   {"stat": "Average", "period": 60}],
                  [".", "DatabaseConnections", ".", ".", 
                   {"stat": "Maximum", "period": 60}]
                ],
                "view": "timeSeries",
                "stacked": false,
                "region": "${AWS::Region}",
                "period": 3600,
                "stat": "Average"
              }
            },
            {
              "type": "metric",
              "x": 0, "y": 6, "width": 12, "height": 6,
              "properties": {
                "title": "${DBInstanceIdentifier} - Disk I/O",
                "metrics": [
                  ["AWS/RDS", "ReadIOPS", 
                   "DBInstanceIdentifier", "${DBInstanceIdentifier}",
                   {"stat": "Sum", "period": 60}],
                  [".", "WriteIOPS", ".", ".", 
                   {"stat": "Sum", "period": 60, "color": "#d62728"}]
                ],
                "view": "timeSeries",
                "region": "${AWS::Region}"
              }
            },
            {
              "type": "metric",
              "x": 12, "y": 6, "width": 12, "height": 6,
              "properties": {
                "title": "${DBInstanceIdentifier} - EBS Latency",
                "metrics": [
                  ["AWS/RDS", "ReadLatency", 
                   "DBInstanceIdentifier", "${DBInstanceIdentifier}",
                   {"stat": "Average", "period": 60}],
                  [".", "WriteLatency", ".", ".", 
                   {"stat": "Average", "period": 60, "color": "#d62728"}]
                ],
                "view": "timeSeries",
                "region": "${AWS::Region}",
                "yAxis": {"left": {"label": "Seconds", "min": 0}}
              }
            },
            {
              "type": "metric",
              "x": 0, "y": 12, "width": 8, "height": 5,
              "properties": {
                "title": "${DBInstanceIdentifier} - Storage",
                "metrics": [
                  ["AWS/RDS", "FreeStorageSpace", 
                   "DBInstanceIdentifier", "${DBInstanceIdentifier}",
                   {"stat": "Minimum", "period": 3600}]
                ],
                "view": "singleValue",
                "region": "${AWS::Region}",
                "period": 3600
              }
            },
            {
              "type": "metric",
              "x": 8, "y": 12, "width": 8, "height": 5,
              "properties": {
                "title": "${DBInstanceIdentifier} - Disk Queue Depth",
                "metrics": [
                  ["AWS/RDS", "DiskQueueDepth", 
                   "DBInstanceIdentifier", "${DBInstanceIdentifier}",
                   {"stat": "Average", "period": 300}]
                ],
                "view": "singleValue",
                "region": "${AWS::Region}",
                "period": 3600
              }
            },
            {
              "type": "metric",
              "x": 16, "y": 12, "width": 8, "height": 5,
              "properties": {
                "title": "${DBInstanceIdentifier} - Network (bytes/sec)",
                "metrics": [
                  ["AWS/RDS", "NetworkReceiveThroughput", 
                   "DBInstanceIdentifier", "${DBInstanceIdentifier}",
                   {"stat": "Average", "period": 300}],
                  [".", "NetworkTransmitThroughput", ".", ".", 
                   {"stat": "Average", "period": 300, "color": "#d62728"}]
                ],
                "view": "timeSeries",
                "region": "${AWS::Region}"
              }
            },
            {
              "type": "text",
              "x": 0, "y": 17, "width": 24, "height": 2,
              "properties": {
                "markdown": "## Alarm Status\n\nHigh CPU: `${DBInstanceIdentifier}-HighCPU` | Low Storage: `${DBInstanceIdentifier}-LowStorage` | Low Memory: `${DBInstanceIdentifier}-LowMemory`"
              }
            }
          ]
        }

Creating Dashboards Programmatically with Python

import boto3
import json

cloudwatch = boto3.client('cloudwatch')

dashboard_name = "MyApp-Production-RDS"
instances = ["mydb-prod-01", "mydb-prod-02", "mydb-replica-01"]

widgets = []

# Create a row of CPU widgets for each instance
for idx, instance in enumerate(instances):
    y_position = idx * 8  # Stack vertically, 8 height units per instance block
    
    widgets.append({
        "type": "metric",
        "x": 0, "y": y_position, "width": 24, "height": 6,
        "properties": {
            "title": f"{instance} — CPU, Memory, Connections",
            "metrics": [
                ["AWS/RDS", "CPUUtilization", "DBInstanceIdentifier", instance,
                 {"stat": "Average", "period": 60}],
                [".", "FreeableMemory", ".", ".",
                 {"stat": "Average", "period": 60}],
                [".", "DatabaseConnections", ".", ".",
                 {"stat": "Maximum", "period": 60}]
            ],
            "view": "timeSeries",
            "region": "us-east-1",
            "period": 3600
        }
    })
    
    widgets.append({
        "type": "metric",
        "x": 0, "y": y_position + 6, "width": 24, "height": 2,
        "properties": {
            "title": f"{instance} — Storage & Queue Depth",
            "metrics": [
                ["AWS/RDS", "FreeStorageSpace", "DBInstanceIdentifier", instance,
                 {"stat": "Minimum", "period": 3600}],
                [".", "DiskQueueDepth", ".", ".",
                 {"stat": "Average", "period": 300, "yAxis": "right"}]
            ],
            "view": "timeSeries",
            "region": "us-east-1",
            "period": 3600
        }
    })

dashboard_body = json.dumps({"widgets": widgets}, indent=2)

cloudwatch.put_dashboard(
    DashboardName=dashboard_name,
    DashboardBody=dashboard_body
)

print(f"Dashboard '{dashboard_name}' created/updated successfully.")

Composite Alarms: Reducing Alert Fatigue

The Problem with Simple Threshold Alarms

A single metric crossing a threshold once can generate false positives from transient spikes — a brief CPU burst during a cron job, a momentary connection surge during a deploy. Simple threshold alarms configured with short evaluation periods create alert fatigue, and the team eventually starts ignoring them. Composite alarms solve this by combining multiple conditions into a single alarm that fires only when several things are wrong simultaneously.

Creating a Composite Alarm

# Create two underlying metric alarms first
# Alarm 1: High CPU
aws cloudwatch put-metric-alarm \
  --alarm-name mydb-prod-01-HighCPU-Underlying \
  --alarm-description "CPU > 90% for 5 minutes" \
  --namespace AWS/RDS \
  --metric-name CPUUtilization \
  --dimensions Name=DBInstanceIdentifier,Value=mydb-prod-01 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 90 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data missing

# Alarm 2: High Disk Queue Depth (indicates real workload, not idle CPU)
aws cloudwatch put-metric-alarm \
  --alarm-name mydb-prod-01-HighDiskQueue-Underlying \
  --alarm-description "Disk queue depth > 2 for 5 minutes" \
  --namespace AWS/RDS \
  --metric-name DiskQueueDepth \
  --dimensions Name=DBInstanceIdentifier,Value=mydb-prod-01 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 2 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data missing

# Composite alarm: fires only when BOTH CPU is high AND disk queue is deep
# This filters out CPU spikes from idle compute but not from real database load
aws cloudwatch put-composite-alarm \
  --alarm-name mydb-prod-01-GenuineHighLoad \
  --alarm-description "High CPU with concurrent high disk I/O — real database load" \
  --alarm-rule "ALARM(mydb-prod-01-HighCPU-Underlying) AND ALARM(mydb-prod-01-HighDiskQueue-Underlying)" \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:rds-alarms-prod \
  --treat-missing-data notBreaching

Anomaly Detection Alarms

When Static Thresholds Fail

Workloads often have natural cyclical patterns — higher CPU during business hours, lower on weekends. A static threshold of 80% CPU