← Back to DevBytes
Developer Tutorial August 13, 2026 8 min read

Troubleshooting SQS: Common Issues and Solutions

Introduction to SQS Troubleshooting

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables decoupling and scaling of microservices, distributed systems, and serverless applications. While SQS is highly reliable, developers frequently encounter issues related to message delivery, visibility timeouts, permissions, and dead-letter queues. This tutorial covers the most common SQS problems and provides practical solutions with code examples to help you debug and resolve them quickly.

Why Troubleshooting SQS Matters

Message queues are the backbone of event-driven architectures. When SQS behaves unexpectedly, it can cause silent data loss, duplicate processing, delayed messages, or cascading failures across your entire system. Understanding how to diagnose and fix these issues is critical for maintaining system reliability, data integrity, and application performance. A single misconfigured visibility timeout or IAM policy can bring down a production pipeline, so mastering SQS troubleshooting is an essential skill for any cloud developer.

Common SQS Issues and Solutions

1. Messages Stuck in Queue (Not Being Consumed)

One of the most frequent issues is messages sitting in the queue without being picked up by consumers. This typically happens due to incorrect IAM permissions, misconfigured event source mappings, or consumer application failures.

Common causes:

Solution: First, verify the IAM policy attached to your consumer. Here is an example of a correct IAM policy for an SQS consumer:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}

Next, check if your consumer is actively polling. Here is a Python (boto3) example that demonstrates proper long polling:

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

response = sqs.receive_message(
    QueueUrl=queue_url,
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20,  # Long polling reduces empty responses
    VisibilityTimeout=30
)

if 'Messages' in response:
    for message in response['Messages']:
        print(f"Processing message: {message['Body']}")
        # Process the message here
        sqs.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message['ReceiptHandle']
        )
else:
    print("No messages received")

If you are using AWS Lambda as the consumer, verify the event source mapping is enabled:

aws lambda list-event-source-mappings \
  --function-name my-consumer-function \
  --region us-east-1

# Enable if disabled
aws lambda update-event-source-mapping \
  --uuid  \
  --enabled true

2. Messages Being Processed Multiple Times (Duplicates)

SQS provides "at-least-once" delivery, meaning duplicates can occur. If you are seeing excessive duplicate processing, the likely culprit is a visibility timeout that is too short for your processing time.

How it works: When a consumer receives a message, SQS hides it from other consumers for the duration of the visibility timeout. If the consumer does not delete the message before the timeout expires, the message becomes visible again and another consumer can pick it up.

Solution: Set the visibility timeout to be longer than your maximum processing time. A good rule of thumb is to set it to at least 1.5 times your expected processing duration.

# Update queue visibility timeout
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes VisibilityTimeout=120

# Or set it per-message when receiving
response = sqs.receive_message(
    QueueUrl=queue_url,
    VisibilityTimeout=120,  # Override queue default
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20
)

Additionally, implement idempotency in your consumer logic to handle duplicates gracefully:

import hashlib

def process_message(message_body, message_id, processed_cache):
    # Generate a unique key for this message
    dedup_key = hashlib.md5(message_id.encode()).hexdigest()
    
    if dedup_key in processed_cache:
        print(f"Duplicate message {message_id} detected, skipping")
        return
    
    # Process the message
    print(f"Processing: {message_body}")
    
    # Mark as processed
    processed_cache[dedup_key] = True

3. Messages Going to Dead-Letter Queue (DLQ) Unexpectedly

A Dead-Letter Queue receives messages that failed processing after a specified number of attempts. If messages are landing in your DLQ sooner than expected, check your maxReceiveCount setting and the redrive policy.

Solution: Review and update the redrive policy:

# Check current redrive policy
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attribute-names RedrivePolicy

# Update redrive policy with higher maxReceiveCount
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:my-dlq\",\"maxReceiveCount\":\"5\"}"
  }'

To reprocess messages from the DLQ back to the main queue, you can use this script:

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')
dlq_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-dlq'
main_queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

while True:
    response = sqs.receive_message(
        QueueUrl=dlq_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=5
    )
    
    if 'Messages' not in response:
        break
    
    for message in response['Messages']:
        # Send back to main queue
        sqs.send_message(
            QueueUrl=main_queue_url,
            MessageBody=message['Body']
        )
        # Delete from DLQ
        sqs.delete_message(
            QueueUrl=dlq_url,
            ReceiptHandle=message['ReceiptHandle']
        )
        print(f"Requeued message {message['MessageId']}")

4. FIFO Queue Messages Not in Order

FIFO queues guarantee order within a message group. If messages appear out of order, it usually means you are not using message group IDs correctly, or you have multiple message groups interleaving.

Solution: Always specify a MessageGroupId when sending to FIFO queues. Messages with the same group ID are processed in order:

# Sending ordered messages to a FIFO queue
response = sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/my-fifo-queue.fifo',
    MessageBody='{"event": "order_created", "order_id": 12345}',
    MessageGroupId='order-12345',  # All messages for this order share this ID
    MessageDeduplicationId='order-12345-created-1700000000'  # Unique per message
)

Important note: If a single consumer is processing messages from the same message group sequentially, a slow or failed message will block all subsequent messages in that group. Consider using multiple message groups to parallelize processing while maintaining order within each group.

5. High API Costs from Excessive Polling

If your SQS bill is higher than expected, you may be performing short polling too frequently. Each ReceiveMessage call is billed, and short polling with no messages still counts as a request.

Solution: Use long polling by setting WaitTimeSeconds to a value between 1 and 20. This reduces the number of empty receives and lowers costs:

# Long polling example - waits up to 20 seconds for messages
response = sqs.receive_message(
    QueueUrl=queue_url,
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20  # Long polling
)

# Also set ReceiveMessageWaitTimeSeconds at the queue level
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes ReceiveMessageWaitTimeSeconds=20

6. Permission Errors When Accessing SQS

Access denied errors are common when the service role or user lacks the necessary permissions. This is especially common with cross-account access or when using SQS with other AWS services like SNS or Lambda.

Solution: For cross-account access, you need a queue policy that grants access to the other account:

{
  "Version": "2012-10-17",
  "Id": "SQSCrossAccountPolicy",
  "Statement": [
    {
      "Sid": "AllowCrossAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::987654321098:root"
      },
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}

Apply this policy to the queue:

aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes file://queue-policy.json

7. Lambda Function Not Triggered by SQS

When using Lambda with SQS, messages may not trigger the function due to several reasons including batch size settings, concurrency limits, or the event source mapping being in a paused state.

Solution: Check the event source mapping status and look for FunctionResponseTypes and batching settings:

# Check event source mapping details
aws lambda get-event-source-mapping \
  --uuid 

# Update batch size and enable report batch item failures
aws lambda update-event-source-mapping \
  --uuid  \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5 \
  --function-response-types ReportBatchItemFailures

When using ReportBatchItemFailures, your Lambda function should return partial batch failures so only failed messages are retried:

import json

def lambda_handler(event, context):
    batch_item_failures = []
    
    for record in event['Records']:
        try:
            message_body = json.loads(record['body'])
            process_message(message_body)
        except Exception as e:
            print(f"Failed to process message {record['messageId']}: {str(e)}")
            batch_item_failures.append({
                'itemIdentifier': record['messageId']
            })
    
    return {
        'batchItemFailures': batch_item_failures
    }

def process_message(message):
    # Your processing logic here
    pass

Best Practices for SQS Reliability

Monitoring and Alerting

Set up CloudWatch alarms for key SQS metrics to catch issues before they impact users:

# Create CloudWatch alarm for approximate number of messages visible
aws cloudwatch put-metric-alarm \
  --alarm-name "SQS-High-Message-Count" \
  --alarm-description "Alert when queue has more than 1000 messages" \
  --metric-name ApproximateNumberOfMessagesVisible \
  --namespace AWS/SQS \
  --statistic Average \
  --period 300 \
  --threshold 1000 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=QueueName,Value=my-queue \
  --evaluation-periods 2 \
  --alarm-actions "arn:aws:sns:us-east-1:123456789012:alerts"

Key metrics to monitor:

Always Use Dead-Letter Queues

Configure a DLQ for every production queue to capture messages that fail processing. This prevents poison pill messages from blocking the queue indefinitely and gives you a way to inspect and reprocess failures.

Implement Proper Error Handling

Always handle exceptions in your consumer and only delete messages after successful processing:

def safe_process_message(queue_url, receipt_handle, message_body):
    try:
        result = process_business_logic(message_body)
        if result.success:
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=receipt_handle
            )
            print("Message processed and deleted")
        else:
            print("Processing failed, message will be retried after visibility timeout")
    except Exception as e:
        print(f"Error processing message: {e}")
        # Do NOT delete the message - it will be retried
        # After maxReceiveCount, it goes to DLQ

Use Deduplication for FIFO Queues

For FIFO queues, always provide a MessageDeduplicationId to prevent duplicate messages within the 5-minute deduplication window. If you do not provide one, SQS uses the message body hash, which may not be sufficient for your use case.

Right-Size Your Visibility Timeout

Calculate your visibility timeout based on your actual processing time. If processing takes 30 seconds on average with a maximum of 60 seconds, set the visibility timeout to at least 90 seconds to account for retries and edge cases.

Debugging Checklist

When troubleshooting SQS issues, follow this systematic checklist:

Conclusion

Troubleshooting SQS effectively requires a solid understanding of how message visibility, delivery semantics, permissions, and queue configurations interact. By following the solutions and best practices outlined in this tutorial, you can quickly diagnose common issues like stuck messages, duplicate processing, unexpected DLQ routing, and permission errors. Remember to always implement idempotent consumers, configure appropriate visibility timeouts, use dead-letter queues for every production workload, and set up CloudWatch monitoring to catch problems early. With these tools and patterns in your toolkit, you can build robust, reliable message-driven architectures on AWS SQS that scale gracefully and recover automatically from failures.

— Ad —

Google AdSense will appear here after approval

← Back to all articles