← Back to DevBytes
Developer Tutorial August 04, 2026 14 min read

Scaling SQS: From Prototype to Production

Introduction to Scaling SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. While getting started with SQS is straightforward—often just a few lines of code—scaling it from a prototype to a production-grade system requires careful consideration of throughput, latency, cost, error handling, and observability.

This tutorial walks through the journey of taking an SQS-based architecture from a quick prototype to a robust production deployment. We'll cover the fundamentals, common pitfalls, scaling strategies, and best practices that will help you build reliable systems on top of SQS.

What Is SQS and Why Scaling Matters

SQS is a distributed message queue that offers at-least-once delivery semantics. It comes in two flavors:

Scaling matters because the patterns that work at low volume—polling a single queue with a single consumer—often break down as traffic grows. You may encounter throttling, increased costs from excessive polling, message backlog growth, duplicate processing, or visibility timeout issues that cause messages to be processed multiple times.

Starting Point: The Prototype

Let's begin with a typical prototype. You have a producer sending messages and a consumer polling for them. Here's a minimal example using the AWS SDK for Node.js (v3):

// producer.js
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";

async function sendMessage(orderId, payload) {
  const command = new SendMessageCommand({
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify({ orderId, payload }),
  });
  const result = await sqs.send(command);
  console.log("Sent message:", result.MessageId);
}

// Simulate sending a few messages
for (let i = 0; i < 10; i++) {
  await sendMessage(`order-${i}`, { amount: i * 10 });
}

// consumer.js
import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";

async function poll() {
  while (true) {
    const result = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20,
    }));

    if (!result.Messages || result.Messages.length === 0) {
      continue;
    }

    for (const message of result.Messages) {
      console.log("Processing:", message.Body);
      await sqs.send(new DeleteMessageCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle,
      }));
    }
  }
}

poll();

This prototype works fine for a few messages per second. But what happens when you need to process thousands of messages per second? Let's explore the scaling strategies.

Scaling Strategy 1: Batch Operations

The single most impactful change you can make is to use batch APIs. SQS supports sending and receiving up to 10 messages per API call. Batching reduces API calls by up to 10x, which directly reduces costs and increases effective throughput.

Batch Sending

// batch-producer.js
import { SQSClient, SendMessageBatchCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";

async function sendBatch(messages) {
  // SQS allows max 10 messages per batch
  const batches = [];
  for (let i = 0; i < messages.length; i += 10) {
    batches.push(messages.slice(i, i + 10));
  }

  for (const batch of batches) {
    const command = new SendMessageBatchCommand({
      QueueUrl: QUEUE_URL,
      Entries: batch.map((msg, idx) => ({
        Id: `msg-${idx}`,
        MessageBody: JSON.stringify(msg),
      })),
    });

    const result = await sqs.send(command);
    if (result.Failed && result.Failed.length > 0) {
      console.error("Failed messages:", result.Failed);
      // Implement retry logic for failed messages
    }
  }
}

await sendBatch(
  Array.from({ length: 100 }, (_, i) => ({ orderId: `order-${i}` }))
);

Batch Receiving and Deleting

// batch-consumer.js
import {
  SQSClient,
  ReceiveMessageCommand,
  DeleteMessageBatchCommand,
} from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";

async function pollBatch() {
  while (true) {
    const result = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20,
      VisibilityTimeout: 60,
    }));

    if (!result.Messages || result.Messages.length === 0) continue;

    const processedEntries = [];

    for (const message of result.Messages) {
      try {
        await processMessage(JSON.parse(message.Body));
        processedEntries.push({
          Id: message.MessageId,
          ReceiptHandle: message.ReceiptHandle,
        });
      } catch (err) {
        console.error("Failed to process message:", message.MessageId, err);
        // Leave message invisible; it will become visible again after
        // the visibility timeout expires and can be retried
      }
    }

    if (processedEntries.length > 0) {
      await sqs.send(new DeleteMessageBatchCommand({
        QueueUrl: QUEUE_URL,
        Entries: processedEntries,
      }));
    }
  }
}

async function processMessage(body) {
  // Your business logic here
  console.log("Processing:", body.orderId);
}

pollBatch();

Scaling Strategy 2: Concurrency and Parallel Consumers

A single consumer process polling with long polling can handle roughly 10 messages per poll cycle. To scale horizontally, you need multiple consumers running in parallel. There are two main approaches:

Multiple Pollers in a Single Process

// concurrent-consumer.js
import {
  SQSClient,
  ReceiveMessageCommand,
  DeleteMessageBatchCommand,
} from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";
const POLLER_COUNT = 5;

async function poller(workerId) {
  while (true) {
    try {
      const result = await sqs.send(new ReceiveMessageCommand({
        QueueUrl: QUEUE_URL,
        MaxNumberOfMessages: 10,
        WaitTimeSeconds: 20,
        VisibilityTimeout: 60,
      }));

      if (!result.Messages || result.Messages.length === 0) continue;

      // Process messages concurrently within this poller
      const results = await Promise.allSettled(
        result.Messages.map((msg) => processMessage(msg, workerId))
      );

      const toDelete = [];
      results.forEach((r, idx) => {
        if (r.status === "fulfilled") {
          toDelete.push({
            Id: result.Messages[idx].MessageId,
            ReceiptHandle: result.Messages[idx].ReceiptHandle,
          });
        }
      });

      if (toDelete.length > 0) {
        await sqs.send(new DeleteMessageBatchCommand({
          QueueUrl: QUEUE_URL,
          Entries: toDelete,
        }));
      }
    } catch (err) {
      console.error(`Worker ${workerId} error:`, err);
      await new Promise((r) => setTimeout(r, 1000));
    }
  }
}

async function processMessage(message, workerId) {
  const body = JSON.parse(message.Body);
  console.log(`Worker ${workerId} processing ${body.orderId}`);
  // Simulate work
  await new Promise((r) => setTimeout(r, 100));
}

// Launch multiple pollers
for (let i = 0; i < POLLER_COUNT; i++) {
  poller(i);
}

Scaling with AWS Lambda

For many workloads, AWS Lambda is the simplest way to scale SQS consumers. Lambda automatically scales the number of concurrent functions based on the number of messages in the queue. Here's a Lambda handler:

// lambda-handler.js
export const handler = async (event) => {
  const results = [];

  for (const record of event.Records) {
    try {
      const body = JSON.parse(record.body);
      await processMessage(body);
      results.push({ messageId: record.messageId, status: "success" });
    } catch (err) {
      console.error("Failed:", record.messageId, err);
      // Do not throw — throwing causes the entire batch to retry.
      // Instead, handle partial failures explicitly.
      results.push({ messageId: record.messageId, status: "failed", error: err.message });
    }
  }

  // If all messages failed, throw to trigger retry
  const allFailed = results.every((r) => r.status === "failed");
  if (allFailed && results.length > 0) {
    throw new Error("All messages in batch failed");
  }

  return results;
};

async function processMessage(body) {
  console.log("Processing:", body.orderId);
}

When using Lambda with SQS, configure the following settings carefully:

Scaling Strategy 3: Visibility Timeout Tuning

The visibility timeout is one of the most critical configuration parameters. When a consumer receives a message, SQS hides it from other consumers for the duration of the visibility timeout. If the consumer doesn't delete the message before the timeout expires, the message becomes visible again and may be picked up by another consumer—resulting in duplicate processing.

Guidelines for setting visibility timeout:

// Extending visibility timeout for long-running processing
import {
  SQSClient,
  ChangeMessageVisibilityCommand,
} from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";

async function processWithHeartbeat(receiptHandle, workFn) {
  const HEARTBEAT_INTERVAL = 30000; // 30 seconds
  let timeout = 120; // initial visibility timeout in seconds

  const heartbeat = setInterval(async () => {
    timeout += 60;
    await sqs.send(new ChangeMessageVisibilityCommand({
      QueueUrl: QUEUE_URL,
      ReceiptHandle: receiptHandle,
      VisibilityTimeout: timeout,
    }));
    console.log("Extended visibility timeout to", timeout);
  }, HEARTBEAT_INTERVAL);

  try {
    await workFn();
  } finally {
    clearInterval(heartbeat);
  }
}

Scaling Strategy 4: Dead Letter Queues

In production, some messages will always fail to process—whether due to malformed payloads, downstream service outages, or bugs. A Dead Letter Queue (DLQ) captures messages that exceed a maximum number of receive attempts, preventing poison pills from blocking your pipeline.

// Configure DLQ using AWS CDK (TypeScript)
import * as sqs from "aws-cdk-lib/aws-sqs";

const dlq = new sqs.Queue(scope, "MyDLQ", {
  queueName: "my-queue-dlq",
  retentionPeriod: cdk.Duration.days(14), // Keep failed messages longer
});

const mainQueue = new sqs.Queue(scope, "MyQueue", {
  queueName: "my-queue",
  visibilityTimeout: cdk.Duration.seconds(120),
  deadLetterQueue: {
    queue: dlq,
    maxReceiveCount: 5, // After 5 failed attempts, move to DLQ
  },
});

Best practices for DLQs:

Scaling Strategy 5: Cost Optimization

SQS pricing is based on API requests, not message size or throughput. Every SendMessage, ReceiveMessage, and DeleteMessage counts as a request. Here are key cost optimization strategies:

S3 Payload Offloading Example

// Large message handling with S3 offloading
import { S3Client, PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const s3 = new S3Client({ region: "us-east-1" });
const sqs = new SQSClient({ region: "us-east-1" });
const BUCKET = "my-message-payloads";
const QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue";

async function sendLargeMessage(messageData) {
  const key = `messages/${Date.now()}-${Math.random().toString(36).slice(2)}`;

  // Upload large payload to S3
  await s3.send(new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    Body: JSON.stringify(messageData),
    ContentType: "application/json",
  }));

  // Send lightweight reference message to SQS
  await sqs.send(new SendMessageCommand({
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify({
      s3Bucket: BUCKET,
      s3Key: key,
    }),
    MessageAttributes: {
      PayloadLocation: {
        DataType: "String",
        StringValue: "s3",
      },
    },
  }));
}

async function receiveLargeMessage(sqsMessage) {
  const { s3Bucket, s3Key } = JSON.parse(sqsMessage.Body);
  const response = await s3.send(new GetObjectCommand({
    Bucket: s3Bucket,
    Key: s3Key,
  }));
  const payload = await response.Body.transformToString();
  return JSON.parse(payload);
}

Scaling Strategy 6: FIFO Queue Scaling

FIFO queues are constrained to 300 TPS by default. If you need higher throughput, you must use message groups. Each message group is processed in order, but different message groups can be processed in parallel. With batching, FIFO queues can scale to 3,000 TPS.

// FIFO queue with message groups
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: "us-east-1" });
const FIFO_QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo";

async function sendFIFOMessage(orderId, userId, payload) {
  await sqs.send(new SendMessageCommand({
    QueueUrl: FIFO_QUEUE_URL,
    MessageBody: JSON.stringify({ orderId, payload }),
    MessageGroupId: `user-${userId}`, // Orders for same user are ordered
    MessageDeduplicationId: `order-${orderId}`, // Prevents duplicates within 5 min
  }));
}

The key insight is that MessageGroupId determines parallelism. If all messages share the same group ID, you're limited to sequential processing. Distribute messages across many group IDs to maximize throughput while preserving per-group ordering.

Observability and Monitoring

Production SQS deployments require robust monitoring. SQS publishes several CloudWatch metrics that you should alert on:

// CloudWatch alarm for queue depth using AWS CDK
import * as cloudwatch from "aws-cdk-lib/aws-cloudwatch";

const queueDepthAlarm = new cloudwatch.Alarm(scope, "QueueDepthAlarm", {
  metric: mainQueue.metricApproximateNumberOfMessagesVisible({
    period: cdk.Duration.minutes(1),
    statistic: "Average",
  }),
  threshold: 10000,
  evaluationPeriods: 5,
  alarmDescription: "Queue depth exceeds 10k messages for 5 minutes",
});

const oldestMessageAlarm = new cloudwatch.Alarm(scope, "OldestMessageAlarm", {
  metric: mainQueue.metricApproximateAgeOfOldestMessage({
    period: cdk.Duration.minutes(1),
    statistic: "Maximum",
  }),
  threshold: 600, // 10 minutes
  evaluationPeriods: 3,
  alarmDescription: "Oldest message is older than 10 minutes",
});

Idempotency: Handling Duplicate Messages

SQS provides at-least-once delivery, which means your consumers will occasionally see the same message more than once. Your processing logic must be idempotent—processing the same message multiple times should produce the same result as processing it once.

// Idempotent consumer using a deduplication store
import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";

const ddb = new DynamoDBClient({ region: "us-east-1" });
const DEDUP_TABLE = "message-dedup";

async function processMessageIdempotently(message) {
  const messageId = message.MessageId;

  // Try to insert with a condition — if it already exists, we've processed it
  try {
    await ddb.send(new PutItemCommand({
      TableName: DEDUP_TABLE,
      Item: {
        messageId: { S: messageId },
        processedAt: { S: new Date().toISOString() },
      },
      ConditionExpression: "attribute_not_exists(messageId)",
    }));
  } catch (err) {
    if (err.name === "ConditionalCheckFailedException") {
      console.log("Duplicate message, skipping:", messageId);
      return;
    }
    throw err;
  }

  // Safe to process — this is the first time we've seen this message
  const body = JSON.parse(message.Body);
  await doWork(body);
}

For FIFO queues, you can rely on MessageDeduplicationId for producer-side deduplication, but consumer-side idempotency is still recommended as a defense-in-depth measure.

Production Architecture Example

Putting it all together, here's what a production SQS architecture looks like:

// infrastructure.ts (AWS CDK)
import * as cdk from "aws-cdk-lib";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as lambdaEventSources from "aws-cdk-lib/aws-lambda-event-sources";
import * as cloudwatch from "aws-cdk-lib/aws-cloudwatch";
import * as s3 from "aws-cdk-lib/aws-s3";

export class ProductionSqsStack extends cdk.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);

    // S3 bucket for large payload offloading
    const payloadBucket = new s3.Bucket(this, "PayloadBucket", {
      lifecycleRules: [{ expiration: cdk.Duration.days(7) }],
    });

    // Dead Letter Queue
    const dlq = new sqs.Queue(this, "DLQ", {
      queueName: "orders-dlq",
      retentionPeriod: cdk.Duration.days(14),
    });

    // Main queue
    const queue = new sqs.Queue(this, "OrdersQueue", {
      queueName: "orders",
      visibilityTimeout: cdk.Duration.seconds(300),
      deadLetterQueue: {
        queue: dlq,
        maxReceiveCount: 4,
      },
    });

    // Consumer Lambda
    const consumer = new lambda.Function(this, "OrderConsumer", {
      runtime: lambda.Runtime.NODEJS_20_X,
      handler: "handler.handler",
      code: lambda.Code.fromAsset("./dist"),
      timeout: cdk.Duration.seconds(60),
      reservedConcurrentExecutions: 50, // Cap concurrency
      environment: {
        QUEUE_URL: queue.queueUrl,
        PAYLOAD_BUCKET: payloadBucket.bucketName,
      },
    });

    // Wire SQS to Lambda
    consumer.addEventSource(new lambdaEventSources.SqsEventSource(queue, {
      batchSize: 10,
      maxBatchingWindow: cdk.Duration.seconds(5),
      reportBatchItemFailures: true, // Enable partial batch failure reporting
    }));

    // Alarms
    new cloudwatch.Alarm(this, "QueueDepthAlarm", {
      metric: queue.metricApproximateNumberOfMessagesVisible(),
      threshold: 5000,
      evaluationPeriods: 3,
    });

    new cloudwatch.Alarm(this, "DLQAlarm", {
      metric: dlq.metricApproximateNumberOfMessagesVisible(),
      threshold: 1,
      evaluationPeriods: 1,
    });
  }
}

Partial Batch Failure Reporting

One of the most important production features is partial batch failure reporting. Without it, if one message in a Lambda batch fails, the entire batch is retried—including the messages that succeeded. With partial batch failure reporting, you return which specific messages failed, and only those are retried.

// Lambda handler with partial batch failure reporting
export const handler = async (event) => {
  const batchItemFailures = [];

  await Promise.allSettled(
    event.Records.map(async (record) => {
      try {
        await processMessage(JSON.parse(record.body));
      } catch (err) {
        console.error(`Failed to process ${record.messageId}:`, err);
        batchItemFailures.push({ itemIdentifier: record.messageId });
      }
    })
  );

  return { batchItemFailures };
};

async function processMessage(body) {
  // Business logic
  console.log("Processing order:", body.orderId);
}

Best Practices Summary

Conclusion

Scaling SQS from a prototype to production is less about the queue itself—SQS handles virtually unlimited throughput on its own—and more about how you interact with it. The choices you make around batching, concurrency, visibility timeouts, dead letter queues, idempotency, and observability determine whether your system will handle production load gracefully or buckle under pressure. By following the strategies and patterns outlined in this tutorial, you can build SQS-based architectures that are cost-efficient, resilient to failures, and capable of scaling to meet demanding workloads. Start with batching and long polling, add a DLQ early, implement idempotent consumers, and invest in monitoring before you need it—these foundational practices will serve you well regardless of how large your system grows.

— Ad —

Google AdSense will appear here after approval

← Back to all articles