Introduction to S3 Monitoring
Amazon S3 monitoring encompasses the collection, visualization, and alerting on operational metrics that reveal how your storage infrastructure is performing. AWS provides native tools — CloudWatch metrics, alarms, and dashboards — that give you deep visibility into bucket operations, request patterns, error rates, and data transfer volumes. Effective monitoring transforms S3 from a simple storage service into an observable, production-grade component of your application architecture.
What Are S3 CloudWatch Metrics?
Every S3 bucket automatically publishes a core set of metrics to CloudWatch at no additional cost. These are called default metrics and include:
- BucketSizeBytes — Total storage size by storage class (Standard, IA, Glacier, etc.)
- NumberOfObjects — Total object count per storage class
- AllRequests — Total requests across all request types
- GetRequests, PutRequests, DeleteRequests, HeadRequests, ListRequests — Per-operation request counts
- BytesUploaded, BytesDownloaded — Total ingress and egress bytes
- 4xxErrors, 5xxErrors — Client and server error counts
- FirstByteLatency, TotalRequestLatency — Performance timing metrics
These metrics are published at one-minute intervals and retained for 15 months, though data older than 15 days is aggregated. For more granular visibility, you can enable request-level metrics (also called "CloudWatch request metrics") on prefixes or entire buckets. These deliver per-request detail at 1-minute granularity but incur CloudWatch custom metric charges.
Enabling Request-Level Metrics
Request-level metrics can be enabled via the S3 console, AWS CLI, or SDK. When enabled, S3 sends additional metrics such as per-prefix request counts and latency distributions that help pinpoint performance hotspots.
# Enable request metrics on a bucket using AWS CLI
aws s3api put-bucket-metrics-configuration \
--bucket my-application-bucket \
--id EntireBucket \
--metrics-configuration '{
"Id": "EntireBucket",
"MetricsFilter": {
"Prefix": ""
},
"Filter": {
"Prefix": ""
}
}'
# Enable metrics for a specific prefix only
aws s3api put-bucket-metrics-configuration \
--bucket my-application-bucket \
--id UploadsPrefix \
--metrics-configuration '{
"Id": "UploadsPrefix",
"MetricsFilter": {
"Prefix": "uploads/"
},
"Filter": {
"Prefix": "uploads/"
}
}'
Why S3 Monitoring Matters
Monitoring S3 is critical for several reasons:
- Cost control — Unmonitored buckets can accumulate stale objects, orphaned multipart uploads, or unintended storage class transitions that inflate costs silently.
- Performance optimization — Latency metrics reveal whether your application is experiencing S3 throttling or if request patterns would benefit from prefix distribution.
- Error detection — A spike in 4xx or 5xx errors often indicates IAM policy misconfigurations, expired presigned URLs, or S3 service degradation.
- Capacity planning — BucketSizeBytes trends help forecast storage growth and inform lifecycle policy tuning.
- Security auditing — Abnormal request patterns can signal unauthorized access attempts or data exfiltration.
Setting Up CloudWatch Alarms
CloudWatch Alarms watch a single metric over a time window and trigger when a threshold is breached. For S3, alarms are essential for catching error spikes, unusual latency, or unexpected storage growth.
Creating an Alarm for High Error Rates
# Create an alarm that triggers when 5xx errors exceed 10 per minute
aws cloudwatch put-metric-alarm \
--alarm-name "S3-High-5xx-Errors" \
--alarm-description "Triggers when S3 5xx errors spike" \
--namespace "AWS/S3" \
--metric-name "5xxErrors" \
--statistic "Sum" \
--period 300 \
--evaluation-periods 2 \
--threshold 10 \
--comparison-operator "GreaterThanThreshold" \
--dimensions "Name=BucketName,Value=my-application-bucket" \
--alarm-actions "arn:aws:sns:us-east-1:123456789012:ops-alerts" \
--treat-missing-data "notBreaching"
Alarm on Bucket Storage Growth
# Alarm when Standard storage exceeds 1 TB (in bytes)
aws cloudwatch put-metric-alarm \
--alarm-name "S3-Storage-Exceeds-1TB" \
--alarm-description "Alert when bucket storage crosses 1TB" \
--namespace "AWS/S3" \
--metric-name "BucketSizeBytes" \
--statistic "Average" \
--period 86400 \
--evaluation-periods 1 \
--threshold 1099511627776 \
--comparison-operator "GreaterThanThreshold" \
--dimensions "Name=BucketName,Value=my-application-bucket" \
"Name=StorageType,Value=StandardStorage" \
--alarm-actions "arn:aws:sns:us-east-1:123456789012:cost-alerts"
Latency-Based Alarm
# Alarm when p99 total request latency exceeds 500ms
aws cloudwatch put-metric-alarm \
--alarm-name "S3-High-Latency-p99" \
--alarm-description "p99 latency above 500ms threshold" \
--namespace "AWS/S3" \
--metric-name "TotalRequestLatency" \
--extended-statistic "p99" \
--period 300 \
--evaluation-periods 3 \
--threshold 500 \
--comparison-operator "GreaterThanThreshold" \
--dimensions "Name=BucketName,Value=my-application-bucket" \
--alarm-actions "arn:aws:sns:us-east-1:123456789012:performance-alerts"
Building S3 Dashboards
CloudWatch Dashboards provide a visual canvas for S3 metrics. You can build widgets that display time-series graphs, single-value summaries, and stacked area charts — all in a single pane of glass.
Creating a Dashboard via CLI
# Create a comprehensive S3 monitoring dashboard
aws cloudwatch put-dashboard \
--dashboard-name "S3-Operations-Dashboard" \
--dashboard-body '{
"widgets": [
{
"type": "metric",
"x": 0,
"y": 0,
"width": 12,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "GetRequests", {"stat": "Sum"}],
["AWS/S3", "PutRequests", {"stat": "Sum"}],
["AWS/S3", "DeleteRequests", {"stat": "Sum"}],
["AWS/S3", "HeadRequests", {"stat": "Sum"}]
],
"period": 300,
"region": "us-east-1",
"title": "Request Counts by Operation",
"view": "timeSeries",
"stacked": false
}
},
{
"type": "metric",
"x": 12,
"y": 0,
"width": 12,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "4xxErrors", {"stat": "Sum"}],
["AWS/S3", "5xxErrors", {"stat": "Sum"}]
],
"period": 300,
"region": "us-east-1",
"title": "Error Rates",
"view": "timeSeries",
"stacked": true,
"yAxis": {
"left": {"label": "Error Count", "min": 0}
}
}
},
{
"type": "metric",
"x": 0,
"y": 6,
"width": 8,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "TotalRequestLatency", {"stat": "p90"}],
["AWS/S3", "TotalRequestLatency", {"stat": "p99"}]
],
"period": 300,
"region": "us-east-1",
"title": "Request Latency (ms)",
"view": "timeSeries"
}
},
{
"type": "metric",
"x": 8,
"y": 6,
"width": 8,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "BytesDownloaded", {"stat": "Sum"}],
["AWS/S3", "BytesUploaded", {"stat": "Sum"}]
],
"period": 300,
"region": "us-east-1",
"title": "Data Transfer (Bytes)",
"view": "timeSeries",
"stacked": false
}
},
{
"type": "metric",
"x": 16,
"y": 6,
"width": 8,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "BucketSizeBytes", {"stat": "Average"}]
],
"period": 86400,
"region": "us-east-1",
"title": "Total Bucket Size (Bytes)",
"view": "singleValue"
}
}
]
}'
Using Python (boto3) to Build Dashboards Programmatically
import boto3
import json
cloudwatch = boto3.client('cloudwatch', region_name='us-east-1')
dashboard_body = {
"widgets": [
{
"type": "metric",
"x": 0,
"y": 0,
"width": 24,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "AllRequests", {
"stat": "Sum",
"label": "Total Requests"
}]
],
"period": 60,
"region": "us-east-1",
"title": "S3 Request Volume (Per Minute)",
"view": "timeSeries",
"stacked": False
}
},
{
"type": "metric",
"x": 0,
"y": 6,
"width": 12,
"height": 6,
"properties": {
"metrics": [
["AWS/S3", "FirstByteLatency", {
"stat": "p50",
"label": "Median"
}],
["AWS/S3", "FirstByteLatency", {
"stat": "p95",
"label": "p95"
}],
["AWS/S3", "FirstByteLatency", {
"stat": "p99",
"label": "p99"
}]
],
"period": 60,
"region": "us-east-1",
"title": "First Byte Latency Distribution",
"view": "timeSeries",
"yAxis": {
"left": {"label": "Milliseconds", "min": 0}
}
}
}
]
}
response = cloudwatch.put_dashboard(
DashboardName='S3-Performance-Analysis',
DashboardBody=json.dumps(dashboard_body)
)
print(f"Dashboard created: {response['DashboardARN']}")
Automating Alarm Deployment with CloudFormation
For production environments, alarms should be deployed as infrastructure-as-code. Here is a CloudFormation snippet that provisions S3 error alarms alongside the bucket itself.
AWSTemplateFormatVersion: '2010-09-09'
Description: 'S3 bucket with monitoring alarms'
Resources:
ApplicationBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: app-data-prod
VersioningConfiguration:
Status: Enabled
MetricsConfigurations:
- Id: RequestMetrics
MetricsFilter:
Prefix: ''
High5xxAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub "${ApplicationBucket}-High5xxErrors"
AlarmDescription: "Alert on high 5xx errors for app-data-prod"
Namespace: "AWS/S3"
MetricName: "5xxErrors"
Statistic: "Sum"
Period: 300
EvaluationPeriods: 2
Threshold: 5
ComparisonOperator: "GreaterThanThreshold"
Dimensions:
- Name: "BucketName"
Value: !Ref ApplicationBucket
AlarmActions:
- !Ref AlertTopic
TreatMissingData: "notBreaching"
HighLatencyAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub "${ApplicationBucket}-HighLatency"
AlarmDescription: "Alert on p99 latency exceeding 300ms"
Namespace: "AWS/S3"
MetricName: "TotalRequestLatency"
ExtendedStatistic: "p99"
Period: 300
EvaluationPeriods: 3
Threshold: 300
ComparisonOperator: "GreaterThanThreshold"
Dimensions:
- Name: "BucketName"
Value: !Ref ApplicationBucket
AlarmActions:
- !Ref AlertTopic
AlertTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: "S3-Alerts"
Subscription:
- Endpoint: "ops-team@company.com"
Protocol: "email"
Monitoring Replication and Lifecycle Metrics
S3 also publishes replication-specific metrics when replication rules are configured. These include:
- OperationReplication — Tracks replication success/failure counts
- ReplicationLatency — Measures the time from upload to replication completion
Lifecycle actions are not directly metered as CloudWatch metrics, but you can monitor their effects indirectly through BucketSizeBytes transitions between storage classes and NumberOfObjects changes. For direct lifecycle visibility, enable S3 event notifications to Lambda and log transition events to CloudWatch Logs.
Querying S3 Metrics with CloudWatch Metric Insights
# Use metric math to calculate error rate percentage
aws cloudwatch get-metric-data \
--metric-data-queries '[
{
"Id": "errors",
"MetricStat": {
"Metric": {
"Namespace": "AWS/S3",
"MetricName": "5xxErrors",
"Dimensions": [
{"Name": "BucketName", "Value": "my-application-bucket"}
]
},
"Period": 300,
"Stat": "Sum"
},
"ReturnData": false
},
{
"Id": "total",
"MetricStat": {
"Metric": {
"Namespace": "AWS/S3",
"MetricName": "AllRequests",
"Dimensions": [
{"Name": "BucketName", "Value": "my-application-bucket"}
]
},
"Period": 300,
"Stat": "Sum"
},
"ReturnData": false
},
{
"Id": "errorRate",
"Expression": "(errors / total) * 100",
"Label": "ErrorRatePercentage",
"ReturnData": true
}
]' \
--start-time "$(date -d '3 hours ago' +%Y-%m-%dT%H:%M:%S)" \
--end-time "$(date +%Y-%m-%dT%H:%M:%S)" \
--region us-east-1
Best Practices for S3 Monitoring
- Enable request metrics selectively — Enable them on high-traffic prefixes rather than entire buckets to control costs. Each prefix configuration counts as a distinct custom metric.
- Use composite alarms — Combine multiple metrics into a single alarm rule to reduce noise. For example, trigger only when both error count is high AND error rate percentage exceeds a threshold.
- Set baseline thresholds empirically — Run your workload for at least a week to establish normal operating ranges before configuring thresholds. What constitutes "high" latency depends entirely on your application profile.
- Monitor storage class transitions — Create separate widgets or alarms tracking BucketSizeBytes filtered by StorageType dimension (StandardStorage, IntelligentTieringFA, GlacierInstantRetrieval, etc.) to catch lifecycle policy issues.
- Include S3 alarms in your incident response runbooks — Document which alarms correspond to which failure modes. A 403 spike suggests IAM issues; a 503 spike suggests S3 throttling or service limits.
- Tag-driven dashboard organization — Use resource tags to create dashboards that aggregate across multiple buckets belonging to the same application or environment.
- Monitor multipart uploads — Orphaned multipart uploads accumulate silently. Create a custom metric using S3 Inventory or a periodic Lambda that counts incomplete multipart uploads and alarms when the count exceeds a threshold.
- Use anomaly detection — CloudWatch anomaly detection can automatically learn metric baselines and alert on deviations without manual threshold setting. Enable it on BucketSizeBytes and AllRequests for automatic pattern deviation alerts.
Example: Anomaly Detection Alarm
# Enable anomaly detection on request volume
aws cloudwatch put-anomaly-detector \
--namespace "AWS/S3" \
--metric-name "AllRequests" \
--stat "Sum" \
--dimensions "Name=BucketName,Value=my-application-bucket" \
--anomaly-detector-configuration '{
"ExcludedTimeRanges": [],
"MetricTimezone": "UTC"
}'
# Create alarm based on anomaly band
aws cloudwatch put-metric-alarm \
--alarm-name "S3-Anomalous-Request-Volume" \
--alarm-description "Triggers on unusual request patterns" \
--namespace "AWS/S3" \
--metric-name "AllRequests" \
--statistic "Sum" \
--period 300 \
--evaluation-periods 2 \
--threshold 3 \
--comparison-operator "GreaterThanUpperThreshold" \
--threshold-metric-id "anomalyDetectorBand" \
--dimensions "Name=BucketName,Value=my-application-bucket" \
--alarm-actions "arn:aws:sns:us-east-1:123456789012:ops-alerts"
Integrating with External Observability Platforms
While CloudWatch dashboards are powerful, many organizations stream S3 metrics to external platforms like Datadog, Grafana, or Splunk. The pattern typically involves a Lambda function that periodically pulls CloudWatch metrics and pushes them to the external service. Alternatively, S3 server access logs (which capture every request in detail) can be shipped to these platforms for richer analysis than aggregated CloudWatch metrics alone.
# Python Lambda snippet to pull S3 metrics and forward to external service
import boto3
import requests
import os
from datetime import datetime, timedelta
def lambda_handler(event, context):
cloudwatch = boto3.client('cloudwatch')
external_endpoint = os.environ['METRICS_ENDPOINT']
response = cloudwatch.get_metric_data(
MetricDataQueries=[
{
'Id': 'requests',
'MetricStat': {
'Metric': {
'Namespace': 'AWS/S3',
'MetricName': 'AllRequests',
'Dimensions': [
{'Name': 'BucketName', 'Value': 'my-application-bucket'}
]
},
'Period': 300,
'Stat': 'Sum'
}
}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow()
)
# Forward metric data to external platform
for metric_result in response['MetricDataResults']:
payload = {
"metric": metric_result['Label'],
"timestamps": [t.isoformat() for t in metric_result['Timestamps']],
"values": metric_result['Values']
}
requests.post(external_endpoint, json=payload)
return {"statusCode": 200}
Conclusion
Monitoring S3 through CloudWatch metrics, alarms, and dashboards transforms opaque storage infrastructure into a transparent, observable service. Default metrics give you a free baseline of request counts, errors, and storage size. Request-level metrics — though paid — unlock per-prefix granularity essential for performance tuning. Alarms close the loop by notifying you before minor issues become outages, while dashboards provide the visual context your operations team needs during incidents and capacity planning sessions. By following the practices outlined here — enabling selective metrics, setting empirically-derived thresholds, leveraging anomaly detection, and codifying alarm infrastructure — you build a robust observability layer that keeps your S3-based applications reliable, performant, and cost-effective.