Migrating from SQS to Pub/Sub: Step-by-Step Guide

What is SQS to Pub/Sub Migration?

Migrating from AWS Simple Queue Service (SQS) to Google Cloud Pub/Sub involves transitioning your messaging infrastructure from a managed message queue system to a fully-managed, scalable pub/sub messaging platform. SQS operates on a pull-based queue model where consumers poll for messages, while Pub/Sub uses a push-or-pull subscription model that decouples publishers from subscribers more completely. This migration is not merely a lift-and-shift operation — it requires rethinking message delivery patterns, acknowledging the architectural differences, and adapting your application code accordingly.

At its core, Pub/Sub offers a global, real-time messaging bus where publishers send messages to topics and subscribers receive them via subscriptions. Unlike SQS queues that are tied to a single consumer group, Pub/Sub allows multiple independent subscriptions per topic, enabling fan-out patterns natively without the need for SNS-SQS combinations.

Why Migrate from SQS to Pub/Sub?

Several compelling reasons drive teams to migrate from SQS to Pub/Sub:

• Multi-cloud or GCP-native strategy: If your organization is moving workloads to Google Cloud, consolidating on Pub/Sub reduces cross-cloud data transfer costs and operational complexity.
• Native fan-out capabilities: With SQS, broadcasting a message to multiple services requires chaining SNS to multiple SQS queues. Pub/Sub handles this natively with multiple subscriptions on a single topic.
• Push subscriptions: Pub/Sub can push messages directly to HTTP endpoints, Cloud Functions, or Cloud Run services, eliminating the need for persistent polling infrastructure.
• Stronger ordering guarantees: Pub/Sub offers ordered delivery with consistent key-based ordering, while SQS FIFO queues have throughput limitations.
• Global scale: Pub/Sub topics are global resources, not tied to a specific region, which simplifies multi-region architectures.
• Dead letter topic integration: Pub/Sub's dead letter topic feature is built directly into the subscription model without requiring separate queue setup.

Key Architectural Differences

Before diving into the migration steps, understanding the conceptual mapping between the two services is critical. The table below outlines the core differences that will shape your migration approach.

Message Lifecycle

In SQS, a message is produced to a queue and remains there until a consumer explicitly deletes it after processing. Visibility timeouts prevent other consumers from re-processing the same message. In Pub/Sub, a message published to a topic is forwarded to all attached subscriptions. Each subscription maintains its own acknowledgment state independently — a single published message can be processed by multiple subscriber applications without coordination.

Delivery Models

SQS is exclusively pull-based. Consumers must poll the queue repeatedly, which can introduce latency and wasted compute cycles. Pub/Sub supports both pull-based delivery (similar to SQS) and push-based delivery, where messages are automatically delivered to a configured endpoint via HTTP. Push subscriptions can significantly simplify consumer architecture, especially for serverless deployments.

Ordering and Deduplication

SQS FIFO queues provide exactly-once processing with ordered delivery but are limited to 3,000 messages per second (with batching). Pub/Sub's ordered message feature uses an ordering key to guarantee that messages with the same key are delivered in order within a subscription, with higher throughput ceilings. Deduplication in Pub/Sub is handled via message IDs and can be configured at the topic level.

Step-by-Step Migration Guide

Step 1: Inventory Your SQS Infrastructure

Begin by cataloging every SQS queue in your system. For each queue, document:

• The producer services that send messages to it
• The consumer services that read from it
• Whether it uses standard or FIFO ordering
• The configured visibility timeout and maximum retention period
• Any dead letter queue associations and redrive policies
• Message attributes and body formats in use
• Encryption settings (SSE-KMS keys)

Use the AWS CLI to export queue attributes for documentation:

aws sqs get-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
    --attribute-names All --output json > sqs-queue-inventory.json

Step 2: Provision Pub/Sub Resources

For each SQS queue you identified, create a corresponding Pub/Sub topic. If an SQS queue receives messages from multiple producer types that should remain logically separated, consider whether separate topics are warranted. Use the Google Cloud Console or gcloud CLI:

gcloud pubsub topics create order-processing-topic \
    --message-retention-duration=7d \
    --topic-encryption-key=projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key

Next, create subscriptions. Each SQS consumer group maps to a Pub/Sub subscription. For a standard SQS queue with one consumer, create a single pull subscription:

gcloud pubsub subscriptions create order-processing-sub \
    --topic=order-processing-topic \
    --ack-deadline=600s \
    --message-retention-duration=7d \
    --enable-exactly-once-delivery

For scenarios where SNS fans out to multiple SQS queues, simply create multiple subscriptions on the same Pub/Sub topic — each independent subscription receives every message published to the topic.

Step 3: Adapt Producer Code

Producers in SQS use the SendMessage or SendMessageBatch API. In Pub/Sub, the equivalent is publishing to a topic. Below is a Python example showing both the SQS producer (left) and its Pub/Sub equivalent (right).

Original SQS Producer (Python with boto3):

import boto3
import json
import uuid

sqs = boto3.client('sqs', region_name='us-east-1')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789012/order-queue'

def publish_order(order_data):
    response = sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps(order_data),
        MessageAttributes={
            'orderId': {
                'DataType': 'String',
                'StringValue': order_data['orderId']
            },
            'source': {
                'DataType': 'String',
                'StringValue': 'web-app'
            }
        },
        MessageGroupId=order_data['customerId'],  # FIFO only
        MessageDeduplicationId=str(uuid.uuid4())   # FIFO only
    )
    return response['MessageId']

Migrated Pub/Sub Publisher (Python with google-cloud-pubsub):

from google.cloud import pubsub_v1
import json

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path('my-project', 'order-processing-topic')

def publish_order(order_data):
    # Pub/Sub attributes map to SQS message attributes
    future = publisher.publish(
        topic_path,
        data=json.dumps(order_data).encode('utf-8'),
        orderId=order_data['orderId'],     # attribute key-value
        source='web-app',
        # ordering_key enables ordered delivery per key
        ordering_key=order_data['customerId']
    )
    message_id = future.result()
    return message_id

Key points in the producer migration:

• MessageBody maps to the data parameter in Pub/Sub
• MessageAttributes become keyword arguments passed directly to publish()
• MessageGroupId in SQS FIFO maps to ordering_key in Pub/Sub
• Deduplication in Pub/Sub can be handled by the service if you enable exactly-once delivery on the subscription, rather than requiring a per-message deduplication ID
• Pub/Sub publish() returns a future that resolves to the message ID — use future.result() to ensure synchronous confirmation

Step 4: Adapt Consumer Code

SQS consumers poll the queue in a loop, receive messages, process them, and then delete them. Pub/Sub pull consumers follow a similar pattern but use a streaming pull for higher throughput and must acknowledge messages rather than delete them.

Original SQS Consumer:

import boto3
import time

sqs = boto3.client('sqs', region_name='us-east-1')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789012/order-queue'

def process_messages():
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20,  # long polling
            VisibilityTimeout=300
        )
        messages = response.get('Messages', [])
        for msg in messages:
            try:
                body = msg['Body']
                handle_order(body)
                sqs.delete_message(
                    QueueUrl=QUEUE_URL,
                    ReceiptHandle=msg['ReceiptHandle']
                )
            except Exception:
                # Message will become visible again after timeout
                print(f"Failed to process {msg['MessageId']}")
        if not messages:
            time.sleep(1)

Migrated Pub/Sub Consumer (Synchronous Pull):

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    'my-project', 'order-processing-sub'
)

def process_messages():
    while True:
        response = subscriber.pull(
            subscription=subscription_path,
            max_messages=10,
            timeout=30.0   # long-polling timeout in seconds
        )
        received = response.received_messages
        if not received:
            continue

        ack_ids = []
        for msg in received:
            try:
                data = msg.message.data.decode('utf-8')
                handle_order(data)
                ack_ids.append(msg.ack_id)
            except Exception:
                # Do not acknowledge — message will be redelivered
                print(f"Failed to process {msg.message.message_id}")

        if ack_ids:
            subscriber.acknowledge(
                subscription=subscription_path,
                ack_ids=ack_ids
            )

Migrated Pub/Sub Consumer (Streaming Pull — Recommended):

from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    'my-project', 'order-processing-sub'
)

def handle_message(message):
    try:
        data = message.data.decode('utf-8')
        handle_order(data)
        message.ack()   # acknowledge on success
    except Exception:
        message.nack()  # nack triggers redelivery
        print(f"Failed to process {message.message_id}")

# Streaming pull manages long-lived connection
streaming_pull_future = subscriber.subscribe(
    subscription_path,
    callback=handle_message,
    flow_control=pubsub_v1.types.FlowControl(
        max_messages=10
    )
)

# Block indefinitely or wrap in try/except for graceful shutdown
try:
    streaming_pull_future.result()
except KeyboardInterrupt:
    streaming_pull_future.cancel()

Important consumer migration notes:

• Replace delete_message with acknowledge — only acknowledged messages are removed from the subscription
• Replace visibility timeout with ack_deadline configured at the subscription level
• Streaming pull (subscriber.subscribe) maintains a persistent gRPC connection, delivering lower latency than the polling loop
• Use nack() to explicitly signal that a message could not be processed and should be redelivered sooner

Step 5: Migrate Dead Letter Handling

SQS dead letter queues (DLQs) require you to create a separate queue and configure a redrive policy. Pub/Sub integrates dead letter topics natively into subscriptions. You create a dead letter topic and configure the subscription to forward messages that exceed the maximum delivery attempts.

# Create the dead letter topic
gcloud pubsub topics create order-dlq-topic

# Update subscription with dead letter policy
gcloud pubsub subscriptions update order-processing-sub \
    --dead-letter-topic=order-dlq-topic \
    --max-delivery-attempts=5 \
    --min-dead-letter-delivery-retry=600s

To replay dead-lettered messages back to the original topic (equivalent to SQS redrive), use the Pub/Sub replay functionality:

# Create a subscription on the DLQ topic to pull dead-lettered messages
gcloud pubsub subscriptions create dlq-replay-sub --topic=order-dlq-topic

# Use the subscription to pull and re-publish messages to the original topic
# This is a simplified replay script

from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
subscriber = pubsub_v1.SubscriberClient()
dlq_sub_path = subscriber.subscription_path('my-project', 'dlq-replay-sub')
original_topic_path = publisher.topic_path('my-project', 'order-processing-topic')

def replay_dead_letters():
    while True:
        response = subscriber.pull(
            subscription=dlq_sub_path,
            max_messages=50,
            timeout=30.0
        )
        ack_ids = []
        for msg in response.received_messages:
            # Re-publish to original topic
            publisher.publish(
                original_topic_path,
                data=msg.message.data,
                **dict(msg.message.attributes)
            )
            ack_ids.append(msg.ack_id)

        if ack_ids:
            subscriber.acknowledge(
                subscription=dlq_sub_path,
                ack_ids=ack_ids
            )
        else:
            break

Step 6: Handle Push Subscriptions for Serverless Consumers

If your SQS consumers run as Lambda functions triggered by SQS events, the closest Pub/Sub equivalent is a push subscription targeting a Cloud Function or Cloud Run service. This eliminates the polling layer entirely — Pub/Sub delivers messages via HTTP POST.

gcloud pubsub subscriptions create order-push-sub \
    --topic=order-processing-topic \
    --push-endpoint=https://us-central1-my-project.cloudfunctions.net/orderHandler \
    --push-auth-service-account=order-handler-sa@my-project.iam.gserviceaccount.com \
    --ack-deadline=600s \
    --max-delivery-attempts=5 \
    --dead-letter-topic=order-dlq-topic

The Cloud Function receives the message in the standard Pub/Sub push format:

def order_handler(event, context):
    # Pub/Sub push delivers base64-encoded data
    import base64, json

    pubsub_message = event['data']
    message_data = base64.b64decode(pubsub_message).decode('utf-8')
    attributes = event.get('attributes', {})

    order = json.loads(message_data)
    process_order(order)
    # No explicit ack needed — HTTP 200 response acknowledges the message
    return 'OK', 200

Step 7: Testing and Cutover Strategy

A dual-write approach minimizes risk during migration. Modify your producers to publish to both SQS and Pub/Sub during a transition period, while consumers gradually switch over. Validate message integrity, ordering, and processing correctness before decommissioning SQS.

# Dual-write pattern during migration
def publish_order_dual(order_data):
    # Write to SQS (existing path)
    sqs.send_message(
        QueueUrl=SQS_QUEUE_URL,
        MessageBody=json.dumps(order_data),
        MessageAttributes={...}
    )
    # Write to Pub/Sub (new path)
    publisher.publish(
        topic_path,
        data=json.dumps(order_data).encode('utf-8'),
        ordering_key=order_data['customerId'],
        **attributes_dict
    )

Run validation checks comparing SQS and Pub/Sub message counts, consumer throughput, and error rates over a window of at least 24 hours before cutting over entirely. Use Cloud Monitoring to track subscription metrics like subscription/ack_message_count and subscription/dead_letter_message_count.

Best Practices for a Smooth Migration

• Start with low-risk workloads: Migrate non-critical, stateless message pipelines first to build operational familiarity with Pub/Sub.
• Leverage exactly-once delivery: Enable --enable-exactly-once-delivery on subscriptions to match SQS FIFO semantics and avoid duplicate processing.
• Tune ack deadlines carefully: SQS visibility timeouts directly map to Pub/Sub ack deadlines. Set the ack deadline to at least double your expected processing time to account for variability.
• Use ordering keys intentionally: If migrating from SQS FIFO, map MessageGroupId to ordering_key. Messages with different ordering keys are delivered in parallel, preserving throughput.
• Implement idempotent consumers: Even with exactly-once delivery, design your consumers to be idempotent by checking message IDs or business keys before processing.
• Monitor subscription backlogs: Watch subscription/backlog_bytes and subscription/oldest_unacked_message_age to detect consumer stalls early.
• Automate DLQ replay: Build automated replay tooling for dead-lettered messages early, as operational incidents will inevitably require it.
• Clean up SQS resources post-migration: After the cutover, purge remaining SQS queues and delete them to avoid lingering costs and confusion.

Conclusion

Migrating from SQS to Pub/Sub is a strategic move that unlocks native fan-out, push-based delivery, global topic management, and tighter integration with Google Cloud's serverless ecosystem. The migration process — inventorying queues, provisioning topics and subscriptions, adapting producer and consumer code, and handling dead letter patterns — is methodical but manageable with a dual-write transition strategy. By understanding the conceptual mapping between SQS queues and Pub/Sub topics/subscriptions, and by leveraging streaming pull, exactly-once delivery, and push subscriptions where appropriate, you can complete the migration with minimal disruption while positioning your messaging infrastructure for greater scale and flexibility.