Introduction to Amazon SNS
Amazon Simple Notification Service (SNS) is a fully managed messaging service designed for application-to-application (A2A) and application-to-person (A2P) communication. It enables developers to decouple microservices, distribute system events, and send messages to subscribers through multiple protocols including HTTP, HTTPS, email, SMS, and AWS Lambda. At its core, SNS operates on a publish-subscribe model, where publishers send messages to topics and subscribers receive those messages through their chosen endpoints.
Whether you are building an event-driven architecture, sending transactional emails, or triggering downstream processing pipelines, SNS provides a scalable, highly available, and cost-effective backbone for asynchronous communication across distributed systems.
Why SNS Matters
In modern cloud architectures, services need to communicate without tight coupling. SNS solves several critical challenges:
- Decoupling: Publishers do not need to know about subscribers, enabling independent scaling and deployment.
- Fan-out: A single message can be delivered to multiple subscribers simultaneously, such as triggering a Lambda function, sending an email, and queuing a message in SQS all at once.
- Durability: SNS stores messages across multiple availability zones, ensuring high availability.
- Multi-protocol delivery: Subscribers can choose how they receive messages, from webhooks to mobile push notifications.
- Integration with AWS ecosystem: SNS works seamlessly with Lambda, SQS, Kinesis, and CloudWatch for end-to-end event processing.
Core Concepts and Architecture
Before diving into setup, it is important to understand the fundamental building blocks of SNS:
- Topic: A communication channel where publishers send messages. Topics can be Standard (at-least-once delivery, high throughput) or FIFO (first-in-first-out ordering, exactly-once processing).
- Publisher: The entity that publishes messages to a topic. This can be an application, an AWS service, or a user.
- Subscriber: The endpoint that receives messages from a topic. Subscriptions can be to SQS queues, Lambda functions, HTTP/HTTPS endpoints, email addresses, or mobile devices.
- Message: The payload sent through the topic, which can include a subject, body, and message attributes for filtering.
- Subscription Filter Policy: A JSON-based policy that allows subscribers to receive only messages matching specific attributes.
Setting Up SNS Using the AWS Console
The AWS Management Console provides a visual interface for creating and configuring SNS topics. Here is a step-by-step walkthrough:
Step 1: Create a Topic
Navigate to the SNS service in the AWS Console, click "Topics" in the left navigation, then click "Create topic". Choose between Standard and FIFO based on your ordering requirements. Provide a meaningful name and optional display name. For FIFO topics, the name must end with ".fifo".
Step 2: Configure Access Policies
The access policy determines who can publish to and subscribe to your topic. By default, only the topic owner has access. For cross-account or service-level access, you will need to modify the policy JSON.
Step 3: Create Subscriptions
After creating a topic, click "Create subscription". Select the protocol (SQS, Lambda, HTTP, email, etc.) and provide the endpoint. For email and HTTP subscriptions, confirmation is required before messages are delivered.
Setting Up SNS Using the AWS CLI
For infrastructure-as-code workflows and automation, the AWS CLI is the preferred method. Below is a complete setup using CLI commands.
Creating a Standard Topic
# Create a standard SNS topic
aws sns create-topic \
--name order-events-topic \
--region us-east-1
# The command returns the Topic ARN
# Example output: arn:aws:sns:us-east-1:123456789012:order-events-topic
Creating a FIFO Topic
# Create a FIFO topic with content-based deduplication
aws sns create-topic \
--name order-events-topic.fifo \
--attributes "FifoTopic=true,ContentBasedDeduplication=true" \
--region us-east-1
Creating Subscriptions
# Subscribe an SQS queue to the topic
aws sns subscribe \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--protocol sqs \
--notification-endpoint arn:aws:sqs:us-east-1:123456789012:order-processing-queue
# Subscribe a Lambda function
aws sns subscribe \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--protocol lambda \
--notification-endpoint arn:aws:lambda:us-east-1:123456789012:function:order-processor
# Subscribe an email endpoint (requires confirmation)
aws sns subscribe \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--protocol email \
--notification-endpoint developer@example.com
Publishing a Message
# Publish a simple message
aws sns publish \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--subject "New Order Created" \
--message '{"orderId":"ORD-12345","customerId":"CUST-67890","total":129.99}'
# Publish with message attributes for filtering
aws sns publish \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--message '{"orderId":"ORD-12346","status":"shipped"}' \
--message-attributes '{"orderStatus":{"DataType":"String","StringValue":"shipped"}}'
Setting Up SNS Using AWS SDK (Python Boto3)
For application-level integration, the AWS SDK provides programmatic access to SNS. Below is a complete Python example using Boto3.
Installation and Configuration
# Install Boto3
pip install boto3
# Configure AWS credentials (if not already configured)
# Option 1: Using AWS CLI
aws configure
# Option 2: Using environment variables
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export AWS_DEFAULT_REGION=us-east-1
Complete SNS Manager Class
import boto3
import json
from botocore.exceptions import ClientError
class SNSManager:
def __init__(self, region_name='us-east-1'):
"""Initialize the SNS client."""
self.sns_client = boto3.client('sns', region_name=region_name)
self.sns_resource = boto3.resource('sns', region_name=region_name)
def create_topic(self, topic_name, is_fifo=False, deduplication=False):
"""Create an SNS topic (standard or FIFO)."""
attributes = {}
if is_fifo:
if not topic_name.endswith('.fifo'):
topic_name += '.fifo'
attributes['FifoTopic'] = 'true'
if deduplication:
attributes['ContentBasedDeduplication'] = 'true'
try:
response = self.sns_client.create_topic(
Name=topic_name,
Attributes=attributes
)
topic_arn = response['TopicArn']
print(f"Created topic: {topic_arn}")
return topic_arn
except ClientError as e:
print(f"Error creating topic: {e}")
raise
def subscribe(self, topic_arn, protocol, endpoint):
"""Subscribe an endpoint to a topic."""
try:
response = self.sns_client.subscribe(
TopicArn=topic_arn,
Protocol=protocol,
Endpoint=endpoint,
ReturnSubscriptionArn=True
)
subscription_arn = response['SubscriptionArn']
print(f"Subscription created: {subscription_arn}")
return subscription_arn
except ClientError as e:
print(f"Error creating subscription: {e}")
raise
def publish_message(self, topic_arn, message, subject=None, attributes=None):
"""Publish a message to a topic with optional attributes."""
kwargs = {
'TopicArn': topic_arn,
'Message': message if isinstance(message, str) else json.dumps(message)
}
if subject:
kwargs['Subject'] = subject
if attributes:
message_attributes = {}
for key, value in attributes.items():
message_attributes[key] = {
'DataType': 'String',
'StringValue': str(value)
}
kwargs['MessageAttributes'] = message_attributes
try:
response = self.sns_client.publish(**kwargs)
message_id = response['MessageId']
print(f"Message published with ID: {message_id}")
return message_id
except ClientError as e:
print(f"Error publishing message: {e}")
raise
def publish_fifo_message(self, topic_arn, message, message_group_id,
message_deduplication_id=None, attributes=None):
"""Publish a message to a FIFO topic (requires MessageGroupId)."""
kwargs = {
'TopicArn': topic_arn,
'Message': message if isinstance(message, str) else json.dumps(message),
'MessageGroupId': message_group_id
}
if message_deduplication_id:
kwargs['MessageDeduplicationId'] = message_deduplication_id
if attributes:
message_attributes = {}
for key, value in attributes.items():
message_attributes[key] = {
'DataType': 'String',
'StringValue': str(value)
}
kwargs['MessageAttributes'] = message_attributes
try:
response = self.sns_client.publish(**kwargs)
message_id = response['MessageId']
print(f"FIFO message published with ID: {message_id}")
return message_id
except ClientError as e:
print(f"Error publishing FIFO message: {e}")
raise
def set_filter_policy(self, subscription_arn, filter_policy):
"""Set a subscription filter policy for message filtering."""
try:
self.sns_client.set_subscription_attributes(
SubscriptionArn=subscription_arn,
AttributeName='FilterPolicy',
AttributeValue=json.dumps(filter_policy)
)
print(f"Filter policy set for subscription: {subscription_arn}")
except ClientError as e:
print(f"Error setting filter policy: {e}")
raise
def list_subscriptions(self, topic_arn):
"""List all subscriptions for a topic."""
try:
response = self.sns_client.list_subscriptions_by_topic(
TopicArn=topic_arn
)
return response['Subscriptions']
except ClientError as e:
print(f"Error listing subscriptions: {e}")
raise
def delete_subscription(self, subscription_arn):
"""Delete a subscription."""
try:
self.sns_client.unsubscribe(SubscriptionArn=subscription_arn)
print(f"Subscription deleted: {subscription_arn}")
except ClientError as e:
print(f"Error deleting subscription: {e}")
raise
def delete_topic(self, topic_arn):
"""Delete a topic and all its subscriptions."""
try:
self.sns_client.delete_topic(TopicArn=topic_arn)
print(f"Topic deleted: {topic_arn}")
except ClientError as e:
print(f"Error deleting topic: {e}")
raise
# Usage example
if __name__ == '__main__':
manager = SNSManager(region_name='us-east-1')
# Create a standard topic
topic_arn = manager.create_topic('ecommerce-events')
# Subscribe an SQS queue
sqs_endpoint = 'arn:aws:sqs:us-east-1:123456789012:order-queue'
sub_arn = manager.subscribe(topic_arn, 'sqs', sqs_endpoint)
# Set a filter policy to only receive 'order.created' events
manager.set_filter_policy(sub_arn, {
'event_type': ['order.created']
})
# Publish a message with attributes
manager.publish_message(
topic_arn=topic_arn,
message={'orderId': 'ORD-99999', 'amount': 49.99},
subject='Order Created',
attributes={'event_type': 'order.created', 'priority': 'high'}
)
# Clean up (uncomment to delete)
# manager.delete_subscription(sub_arn)
# manager.delete_topic(topic_arn)
Setting Up SNS with Terraform
For reproducible infrastructure, Terraform is the industry standard. Below is a complete Terraform configuration for an SNS topic with an SQS subscription and a Lambda subscription.
# variables.tf
variable "aws_region" {
description = "AWS region for resources"
type = string
default = "us-east-1"
}
variable "topic_name" {
description = "Name of the SNS topic"
type = string
default = "app-events"
}
variable "queue_name" {
description = "Name of the SQS queue"
type = string
default = "event-processing-queue"
}
# main.tf
provider "aws" {
region = var.aws_region
}
# SNS Topic
resource "aws_sns_topic" "event_topic" {
name = var.topic_name
display_name = "Application Events Topic"
kms_master_key_id = "alias/aws/sns"
tags = {
Environment = "production"
Project = "event-driven-app"
}
}
# SQS Queue for subscription
resource "aws_sqs_queue" "event_queue" {
name = var.queue_name
visibility_timeout_seconds = 300
message_retention_seconds = 1209600 # 14 days
kms_master_key_id = "alias/aws/sqs"
tags = {
Environment = "production"
Project = "event-driven-app"
}
}
# SQS Queue Policy allowing SNS to send messages
resource "aws_sqs_queue_policy" "queue_policy" {
queue_url = aws_sqs_queue.event_queue.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = "*"
Action = "sqs:SendMessage"
Resource = aws_sqs_queue.event_queue.arn
Condition = {
ArnEquals = {
"aws:SourceArn" = aws_sns_topic.event_topic.arn
}
}
}
]
})
}
# SNS Subscription to SQS
resource "aws_sns_topic_subscription" "sqs_subscription" {
topic_arn = aws_sns_topic.event_topic.arn
protocol = "sqs"
endpoint = aws_sqs_queue.event_queue.arn
raw_message_delivery = true
filter_policy = jsonencode({
event_type = ["order.created", "order.updated"]
})
}
# SNS Topic Policy allowing CloudWatch alarms to publish
resource "aws_sns_topic_policy" "topic_policy" {
arn = aws_sns_topic.event_topic.arn
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = { Service = "cloudwatch.amazonaws.com" }
Action = "sns:Publish"
Resource = aws_sns_topic.event_topic.arn
},
{
Effect = "Allow"
Principal = "*"
Action = ["sns:Subscribe", "sns:Receive"]
Resource = aws_sns_topic.event_topic.arn
}
]
})
}
# CloudWatch alarm that publishes to SNS
resource "aws_cloudwatch_metric_alarm" "error_alarm" {
alarm_name = "high-error-rate"
alarm_description = "Alerts when error rate exceeds threshold"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 2
metric_name = "Errors"
namespace = "AWS/Lambda"
period = 300
statistic = "Sum"
threshold = 10
alarm_actions = [aws_sns_topic.event_topic.arn]
dimensions = {
FunctionName = "order-processor"
}
}
# Outputs
output "sns_topic_arn" {
value = aws_sns_topic.event_topic.arn
}
output "sqs_queue_url" {
value = aws_sqs_queue.event_queue.url
}
output "sqs_queue_arn" {
value = aws_sqs_queue.event_queue.arn
}
Message Filtering with Subscription Filter Policies
One of the most powerful features of SNS is message filtering. Instead of every subscriber receiving every message, filter policies allow subscribers to receive only the messages they care about. This reduces unnecessary processing and network traffic.
Filter Policy Examples
# Simple string-based filter
{
"event_type": ["order.created", "order.updated"]
}
# Numeric range filter
{
"order_amount": [{"numeric": [">=", 100]}]
}
# Multiple attribute filter (AND logic)
{
"event_type": ["order.created"],
"store": ["us-east", "us-west"]
}
# Prefix-based filter
{
"device_id": [{"prefix": "ios-"}]
}
# Exists-based filter (message must have the attribute)
{
"priority": [{"exists": true}]
}
# Anything-but filter
{
"event_type": [{"anything-but": ["order.cancelled"]}]
}
Applying a Filter Policy with Boto3
import boto3
import json
sns_client = boto3.client('sns', region_name='us-east-1')
subscription_arn = 'arn:aws:sns:us-east-1:123456789012:app-events:abc123-def456'
# Apply a filter policy
filter_policy = {
"event_type": ["order.created"],
"order_amount": [{"numeric": [">=", 50.00]}],
"store_region": ["us-east-1", "us-east-2"]
}
response = sns_client.set_subscription_attributes(
SubscriptionArn=subscription_arn,
AttributeName='FilterPolicy',
AttributeValue=json.dumps(filter_policy)
)
print(f"Filter policy applied: {json.dumps(filter_policy, indent=2)}")
Integrating SNS with Lambda
Lambda is one of the most common SNS subscribers. When SNS triggers a Lambda function, it passes the message as an event payload. Below is a complete Lambda handler that processes SNS messages.
# lambda_handler.py
import json
import logging
from typing import Dict, Any
# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
"""
Process SNS messages triggered by topic subscription.
Expected event structure:
{
"Records": [
{
"EventSource": "aws:sns",
"Sns": {
"MessageId": "...",
"TopicArn": "...",
"Subject": "...",
"Message": "...",
"MessageAttributes": {...},
"Timestamp": "..."
}
}
]
}
"""
logger.info(f"Received event with {len(event.get('Records', []))} records")
processed_count = 0
errors = []
for record in event.get('Records', []):
try:
sns_data = record.get('Sns', {})
message_id = sns_data.get('MessageId')
topic_arn = sns_data.get('TopicArn')
subject = sns_data.get('Subject', 'No Subject')
raw_message = sns_data.get('Message')
message_attributes = sns_data.get('MessageAttributes', {})
# Parse the message body (typically JSON)
try:
message_body = json.loads(raw_message)
except json.JSONDecodeError:
message_body = {'raw': raw_message}
logger.info(f"Processing message {message_id} from {topic_arn}")
logger.info(f"Subject: {subject}")
logger.info(f"Message: {json.dumps(message_body, indent=2)}")
# Extract message attributes
extracted_attrs = {}
for attr_name, attr_data in message_attributes.items():
attr_type = attr_data.get('Type')
attr_value = attr_data.get('Value')
extracted_attrs[attr_name] = attr_value
logger.info(f"Attribute {attr_name} ({attr_type}): {attr_value}")
# Business logic: process the order
if 'orderId' in message_body:
process_order(message_body, extracted_attrs)
processed_count += 1
else:
logger.warning(f"Message {message_id} missing orderId, skipping")
except Exception as e:
logger.error(f"Error processing record: {str(e)}", exc_info=True)
errors.append({
'message_id': sns_data.get('MessageId', 'unknown'),
'error': str(e)
})
result = {
'statusCode': 200 if not errors else 207,
'processed': processed_count,
'errors': errors,
'total_records': len(event.get('Records', []))
}
logger.info(f"Processing complete: {json.dumps(result)}")
return result
def process_order(order_data: Dict[str, Any], attributes: Dict[str, Any]) -> None:
"""Business logic for processing an order."""
order_id = order_data.get('orderId')
amount = order_data.get('amount', 0)
priority = attributes.get('priority', 'normal')
logger.info(f"Processing order {order_id}, amount: ${amount}, priority: {priority}")
# Simulate order processing
if priority == 'high':
logger.info(f"Expedited processing for order {order_id}")
# Add high-priority processing logic here
else:
logger.info(f"Standard processing for order {order_id}")
# Add standard processing logic here
# In a real application, you might:
# - Write to DynamoDB
# - Call another microservice
# - Publish to another SNS topic
# - Send a notification
SNS-to-SQS Fan-Out Pattern
The fan-out pattern is one of the most common SNS architectures. A single message published to an SNS topic is delivered to multiple SQS queues, each processing the message independently. This enables parallel processing without coupling.
# fanout_setup.py
import boto3
import json
sns = boto3.client('sns', region_name='us-east-1')
sqs = boto3.client('sqs', region_name='us-east-1')
ACCOUNT_ID = '123456789012'
REGION = 'us-east-1'
def setup_fanout_pattern():
"""Set up an SNS-to-SQS fan-out pattern with multiple queues."""
# Step 1: Create the SNS topic
topic_response = sns.create_topic(Name='order-events-fanout')
topic_arn = topic_response['TopicArn']
print(f"Created topic: {topic_arn}")
# Step 2: Create multiple SQS queues
queues = [
'inventory-update-queue',
'notification-queue',
'analytics-queue',
'audit-log-queue'
]
queue_arns = {}
for queue_name in queues:
# Create the queue
queue_response = sqs.create_queue(QueueName=queue_name)
queue_url = queue_response['QueueUrl']
# Get the queue ARN
attrs = sqs.get_queue_attributes(
QueueUrl=queue_url,
AttributeNames=['QueueArn']
)
queue_arn = attrs['Attributes']['QueueArn']
queue_arns[queue_name] = {'arn': queue_arn, 'url': queue_url}
print(f"Created queue: {queue_name} -> {queue_arn}")
# Step 3: Set queue policy to allow SNS to send messages
queue_policy = {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "sns.amazonaws.com"},
"Action": "sqs:SendMessage",
"Resource": queue_arn,
"Condition": {
"ArnEquals": {
"aws:SourceArn": topic_arn
}
}
}]
}
sqs.set_queue_attributes(
QueueUrl=queue_url,
Attributes={
'Policy': json.dumps(queue_policy)
}
)
# Step 4: Subscribe the queue to the topic
sub_response = sns.subscribe(
TopicArn=topic_arn,
Protocol='sqs',
Endpoint=queue_arn,
ReturnSubscriptionArn=True
)
print(f"Subscribed {queue_name} to topic: {sub_response['SubscriptionArn']}")
# Step 5: Apply different filter policies to each queue
filter_policies = {
'inventory-update-queue': {'event_type': ['order.created', 'order.cancelled']},
'notification-queue': {'event_type': ['order.created', 'order.shipped']},
'analytics-queue': {}, # Receive all messages
'audit-log-queue': {} # Receive all messages
}
for queue_name, policy in filter_policies.items():
queue_arn = queue_arns[queue_name]['arn']
# List subscriptions to find the subscription ARN
subs = sns.list_subscriptions_by_topic(TopicArn=topic_arn)
for sub in subs['Subscriptions']:
if sub['Endpoint'] == queue_arn:
if policy:
sns.set_subscription_attributes(
SubscriptionArn=sub['SubscriptionArn'],
AttributeName='FilterPolicy',
AttributeValue=json.dumps(policy)
)
print(f"Filter policy set for {queue_name}: {policy}")
else:
print(f"No filter policy for {queue_name} (receives all)")
return topic_arn, queue_arns
def publish_fanout_message(topic_arn):
"""Publish a message that fans out to all queues."""
message = {
'orderId': 'ORD-FANOUT-001',
'customerId': 'CUST-12345',
'items': [
{'sku': 'ITEM-001', 'quantity': 2, 'price': 29.99},
{'sku': 'ITEM-002', 'quantity': 1, 'price': 49.99}
],
'total': 109.97,
'shippingAddress': {
'street': '123 Main St',
'city': 'Seattle',
'state': 'WA',
'zip': '98101'
}
}
response = sns.publish(
TopicArn=topic_arn,
Message=json.dumps(message),
Subject='Order Created - Fanout',
MessageAttributes={
'event_type': {'DataType': 'String', 'StringValue': 'order.created'},
'priority': {'DataType': 'String', 'StringValue': 'high'}
}
)
print(f"Published fanout message: {response['MessageId']}")
return response['MessageId']
# Run the setup
if __name__ == '__main__':
topic_arn, queues = setup_fanout_pattern()
publish_fanout_message(topic_arn)
Dead-Letter Queues and Message Delivery Status
When SNS fails to deliver a message to an endpoint (such as an HTTP webhook that is down), it retries according to a retry policy. If delivery continues to fail, the message can be sent to a Dead-Letter Queue (DLQ) for later analysis.
Configuring Delivery Status Logging
# Configure delivery status logging for HTTP subscriptions
aws sns set-topic-attributes \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--attribute-name HTTPSuccessFeedbackSampleRate \
--attribute-value 100
aws sns set-topic-attributes \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--attribute-name HTTPFailureFeedbackRoleArn \
--attribute-value arn:aws:iam::123456789012:role/SNSDeliveryFailureRole
aws sns set-topic-attributes \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--attribute-name HTTPFailureFeedbackSampleRate \
--attribute-value 100
Setting Up a DLQ with Terraform
# DLQ for SNS subscription
resource "aws_sqs_queue" "sns_dlq" {
name = "sns-subscription-dlq"
message_retention_seconds = 1209600 # 14 days
kms_master_key_id = "alias/aws/sqs"
}
# IAM role for SNS to write to CloudWatch Logs
resource "aws_iam_role" "sns_delivery_status_role" {
name = "sns-delivery-status-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "sns.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy" "sns_delivery_status_policy" {
name = "sns-delivery-status-policy"
role = aws_iam_role.sns_delivery_status_role.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
Resource = "*"
},
{
Effect = "Allow"
Action = ["sqs:SendMessage"]
Resource = aws_sqs_queue.sns_dlq.arn
}
]
})
}
# Apply delivery status configuration to the topic
resource "aws_sns_topic" "topic_with_dlq" {
name = "topic-with-dlq"
}
# Configure the topic to log failures
resource "aws_sns_topic_subscription" "subscription_with_dlq" {
topic_arn = aws_sns_topic.topic_with_dlq.arn
protocol = "sqs"
endpoint = aws_sqs_queue.sns_dlq.arn
}
Security Best Practices
Securing your SNS topics is critical to prevent unauthorized access and data leaks. Below are the key security practices every developer should implement.
1. Use KMS Encryption
# Create a topic with KMS encryption using Boto3
import boto3
sns = boto3.client('sns', region_name='us-east-1')
response = sns.create_topic(
Name='secure-topic',
Attributes={
'KmsMasterKeyId': 'arn:aws:kms:us-east-1:123456789012:key/abcd1234-5678-90ef-ghij-klmnopqrstuv'
}
)
print(f"Created encrypted topic: {response['TopicArn']}")
2. Restrict Topic Access with IAM Policies
# IAM policy that restricts publishing to a specific topic
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["sns:Publish"],
"Resource": "arn:aws:sns:us-east-1:123456789012:order-events-topic",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
}
}
},
{
"Effect": "Allow",
"Action": ["sns:Subscribe", "sns:ListSubscriptionsByTopic"],
"Resource": "arn:aws:sns:us-east-1:123456789012:order-events-topic"
},
{
"Effect": "Deny",
"Action": ["sns:CreateTopic", "sns:DeleteTopic", "sns:SetTopicAttributes"],
"Resource": "*"
}
]
}
3. Enable Raw Message Delivery for SQS Subscriptions
# Raw message delivery strips SNS metadata, reducing payload size
# and simplifying parsing in the consumer
aws sns subscribe \
--topic-arn arn:aws:sns:us-east-1:123456789012:order-events-topic \
--protocol sqs \
--notification-endpoint arn:aws:sqs:us-east-1:123456789012:order-queue \
--attributes RawMessageDelivery=true
4. Use Cross-Account Access Carefully
# Topic policy allowing cross-account publishing
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::987654321098:root"
},
"Action": "sns:Publish",
"Resource": "arn:aws:sns:us-east-1:123456789012:order-events-topic",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "987654321098"
}
}
}
]
}
Monitoring and Observability
Monitoring SNS is essential for ensuring message delivery and detecting issues early. AWS provides several built-in metrics through CloudWatch.
Key CloudWatch Metrics to Monitor
- NumberOfMessagesPublished: Number of messages published to the topic.
- NumberOfNotificationsDelivered: Number of messages successfully delivered to subscribers.
- NumberOfNotificationsFailed: Number of messages that failed delivery.
- PublishSize: Average message size in bytes.
- NumberOfNotificationsFilteredOut: Messages filtered out by subscription filter policies.
CloudWatch Alarm Configuration
# alarm.tf - Monitor SNS delivery failures
resource "aws_cloudwatch_metric_alarm" "sns_delivery_failures" {
alarm_name = "sns-delivery-failures"
alarm_description = "Alerts when SNS message delivery failures exceed threshold"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
metric_name = "NumberOfNotificationsFailed"
namespace = "AWS/SNS"
period = 300
statistic = "Sum"
threshold = 5
alarm_actions = [aws_sns_topic.alert_topic.arn]
dimensions = {
TopicName = aws_sns_topic.event_topic.name
}
}
# Monitor for no messages published (potential publisher outage)
resource "aws_cloudwatch_metric_alarm" "sns_no_messages" {
alarm_name = "sns-no-messages-published"
alarm_description = "Alerts when no messages have been published in 1 hour"
comparison_operator = "LessThanThreshold"
evaluation_periods = 1
metric_name = "NumberOfMessagesPublished"
namespace = "AWS/SNS"
period = 3600
statistic = "Sum"
threshold = 1
alarm_actions = [aws_sns_topic.alert_topic.arn]
dimensions = {
TopicName = aws_sns_topic.event_topic.name
}
}
Cost Optimization
SNS pricing is based on the number of requests, the number of deliveries, and data transfer. Here are strategies to optimize costs:
- Use filter policies aggressively: Filtering messages at the SNS level prevents unnecessary SQS processing and Lambda invocations downstream.
- Batch publish