← Back to DevBytes
Developer Tutorial August 04, 2026 10 min read

Scaling Pub/Sub: From Prototype to Production

Scaling Pub/Sub: From Prototype to Production

Pub/Sub (Publish-Subscribe) is one of the most powerful architectural patterns for building decoupled, scalable systems. It works beautifully in a prototype — a single broker, a handful of subscribers, and everything just hums along. But the moment you push it toward production traffic, the cracks appear: message ordering breaks down, consumers fall behind, retries cause duplicate processing, and a single slow subscriber can poison the entire pipeline. This tutorial walks through the journey of taking a Pub/Sub system from a working prototype to a resilient, production-grade platform.

What Is Pub/Sub?

Pub/Sub is a messaging pattern where publishers emit events to a topic without knowing who will consume them. Subscribers express interest in topics and receive events asynchronously. This decoupling — both in time and in identity — is what makes Pub/Sub so attractive for distributed systems.

Common implementations include cloud-managed services like Google Cloud Pub/Sub, Amazon SNS/SQS, and Azure Service Bus, as well as self-hosted options like Apache Kafka, NATS, Redis Pub/Sub, and RabbitMQ. While the specifics differ, the scaling challenges are remarkably similar across all of them.

Why Scaling Matters

In a prototype, you typically have one publisher, one topic, and one or two subscribers running on a single machine. Throughput is low, failures are rare, and the happy path is all you test. In production, you face:

Without deliberate design, each of these will eventually bite you. Let's build up a system step by step.

The Prototype: A Simple Pub/Sub Setup

Here's a typical prototype using Node.js with Google Cloud Pub/Sub. It publishes user activity events and has a single consumer that logs them.

// publisher.js
const { PubSub } = require('@google-cloud/pubsub');
const pubsub = new PubSub();
const topic = pubsub.topic('user-events');

async function publishEvent(event) {
  const dataBuffer = Buffer.from(JSON.stringify(event));
  await topic.publishMessage({ data: dataBuffer });
  console.log('Published:', event.type);
}

publishEvent({ type: 'signup', userId: 'u123', timestamp: Date.now() });

// subscriber.js
const { PubSub } = require('@google-cloud/pubsub');
const pubsub = new PubSub();
const subscription = pubsub.subscription('user-events-sub');

subscription.on('message', async (message) => {
  const event = JSON.parse(message.data.toString());
  console.log('Received:', event.type, 'for user', event.userId);
  message.ack();
});

console.log('Listening for messages...');

This works. It's clean, it's readable, and it will serve you well for a demo. But it has several latent problems: there's no retry logic beyond the broker defaults, no ordering key, no batching, no dead-letter handling, and no observability. Let's address each.

Step 1: Add Batching and Ordering

The first production concern is throughput. Publishing one message at a time introduces significant overhead. Most brokers support batching — grouping multiple messages into a single network call. Additionally, if your consumers need ordered processing per entity (e.g., all events for a given user processed in order), you need an ordering key.

// publisher.js — production version
const { PubSub } = require('@google-cloud/pubsub');
const pubsub = new PubSub();

const topic = pubsub.topic('user-events', {
  batching: {
    maxMessages: 1000,
    maxMilliseconds: 100,
  },
  gaxOpts: {
    timeout: 30000,
  },
});

async function publishEvent(event) {
  const dataBuffer = Buffer.from(JSON.stringify(event));
  await topic.publishMessage({
    data: dataBuffer,
    orderingKey: event.userId, // ensures per-user ordering
    attributes: {
      eventType: event.type,
      version: '1.0',
    },
  });
}

// Simulate a burst of events
const events = [];
for (let i = 0; i < 5000; i++) {
  events.push({
    type: 'page_view',
    userId: `user_${i % 100}`,
    timestamp: Date.now(),
    page: `/product/${i}`,
  });
}

await Promise.all(events.map(publishEvent));
console.log('All events published');

Key changes: the batching config groups up to 1000 messages or waits 100ms before flushing. The orderingKey ensures that all events for the same userId are delivered to the same consumer in order. Note that ordering requires the topic to be configured with message ordering enabled, and it slightly reduces throughput because the broker must route messages with the same key to the same partition.

Step 2: Build a Resilient Consumer

The prototype consumer ack()s every message immediately. In production, you need to process the message first, handle failures gracefully, and avoid getting stuck on poison messages. Here's a more robust consumer:

// subscriber.js — production version
const { PubSub } = require('@google-cloud/pubsub');
const pubsub = new PubSub();

const subscription = pubsub.subscription('user-events-sub', {
  flowControl: {
    maxMessages: 100,        // process at most 100 concurrently
    maxBytes: 10 * 1024 * 1024, // 10MB in flight
  },
  ackDeadline: 60, // seconds before broker redelivers
});

let processed = 0;
let errors = 0;

async function processEvent(event) {
  // Simulate business logic
  if (event.type === 'unknown') {
    throw new Error('Unknown event type');
  }
  // ... do real work here, e.g., write to database
  return;
}

subscription.on('message', async (message) => {
  try {
    const event = JSON.parse(message.data.toString());
    await processEvent(event);
    message.ack();
    processed++;
  } catch (err) {
    errors++;
    console.error('Processing failed:', err.message);

    // If the message has been retried too many times, let it
    // route to a dead-letter topic instead of looping forever.
    const deliveryAttempt = message.deliveryAttempt || 0;
    if (deliveryAttempt >= 5) {
      console.error('Moving message to DLQ after', deliveryAttempt, 'attempts');
      message.ack(); // ack to stop redelivery; DLQ handles it
    } else {
      message.nack(); // redeliver
    }
  }
});

subscription.on('error', (err) => {
  console.error('Subscription error:', err);
});

// Health metrics endpoint (simplified)
setInterval(() => {
  console.log(`Processed: ${processed}, Errors: ${errors}`);
}, 10000);

console.log('Consumer running with flow control...');

Several production patterns are at work here:

Step 3: Dead-Letter Queues and Retry Strategy

A dead-letter queue (DLQ) is a separate topic or queue where messages that repeatedly fail processing are sent for inspection. Without a DLQ, a single malformed message can cause infinite redelivery loops that waste resources and block the pipeline. Most managed services support DLQs natively. Here's how to configure one on Google Cloud Pub/Sub using the gcloud CLI:

# Create the dead-letter topic
gcloud pubsub topics create user-events-dlq

# Create a subscription on the DLQ for inspection
gcloud pubsub subscriptions create user-events-dlq-sub \
  --topic=user-events-dlq

# Attach a dead-letter policy to the main subscription
gcloud pubsub subscriptions update user-events-sub \
  --dead-letter-topic=user-events-dlq \
  --max-delivery-attempts=10 \
  --dead-letter-topic-project=my-project-id

With this in place, the broker itself handles moving failed messages after 10 delivery attempts. Your consumer code can then focus on happy-path processing and let the infrastructure manage retries. You should also build a small tool or dashboard to inspect the DLQ so that operations teams can diagnose and replay failed messages.

Step 4: Horizontal Consumer Scaling

A single consumer process will eventually become a bottleneck. To scale horizontally, you run multiple instances of the same consumer, all pulling from the same subscription. The broker distributes messages across them. The key considerations are:

Here's an example of an idempotent consumer that uses a database to track processed events:

// idempotent-consumer.js
const { PubSub } = require('@google-cloud/pubsub');
const pubsub = new PubSub();
const subscription = pubsub.subscription('user-events-sub');

// Pseudo-database client
const db = require('./db');

async function processEvent(event, messageId) {
  // Check if we already processed this exact message
  const alreadyProcessed = await db.query(
    'SELECT 1 FROM processed_events WHERE message_id = $1',
    [messageId]
  );
  if (alreadyProcessed.rows.length > 0) {
    console.log('Skipping duplicate:', messageId);
    return;
  }

  // Do the real work
  await db.query(
    'INSERT INTO user_activity (user_id, event_type, created_at) VALUES ($1, $2, $3)',
    [event.userId, event.type, new Date(event.timestamp)]
  );

  // Record that we processed this message
  await db.query(
    'INSERT INTO processed_events (message_id, processed_at) VALUES ($1, NOW())',
    [messageId]
  );
}

subscription.on('message', async (message) => {
  try {
    const event = JSON.parse(message.data.toString());
    await processEvent(event, message.id);
    message.ack();
  } catch (err) {
    console.error('Error:', err.message);
    message.nack();
  }
});

This pattern — check, process, record — ensures that even if a message is redelivered to a different consumer instance, it won't be processed twice. The processed_events table acts as a deduplication ledger. For higher throughput, you can use a TTL on these records so the table doesn't grow forever, since redelivery windows are typically minutes, not days.

Step 5: Observability and Monitoring

A production Pub/Sub system without observability is a black box. You need visibility into publish rates, delivery latencies, ack rates, and backlog depth. At minimum, track these metrics:

Here's a simple Prometheus-compatible metrics exporter for a consumer:

// metrics.js
const promClient = require('prom-client');
const express = require('express');

const register = new promClient.Registry();
promClient.collectDefaultMetrics({ register });

const messagesProcessed = new promClient.Counter({
  name: 'pubsub_messages_processed_total',
  help: 'Total messages processed',
  labelNames: ['status'],
  registers: [register],
});

const processingDuration = new promClient.Histogram({
  name: 'pubsub_processing_duration_seconds',
  help: 'Time spent processing a message',
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 5, 10],
  registers: [register],
});

const backlogGauge = new promClient.Gauge({
  name: 'pubsub_backlog_messages',
  help: 'Number of unacked messages',
  registers: [register],
});

const app = express();
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(9090, () => {
  console.log('Metrics server on :9090');
});

module.exports = { messagesProcessed, processingDuration, backlogGauge };

Then instrument your consumer to record these metrics on every message:

const { messagesProcessed, processingDuration } = require('./metrics');

subscription.on('message', async (message) => {
  const start = Date.now();
  try {
    const event = JSON.parse(message.data.toString());
    await processEvent(event, message.id);
    message.ack();
    messagesProcessed.inc({ status: 'success' });
  } catch (err) {
    message.nack();
    messagesProcessed.inc({ status: 'error' });
  } finally {
    processingDuration.observe((Date.now() - start) / 1000);
  }
});

With these metrics flowing into Prometheus and visualized in Grafana, you can set alerts: page the on-call engineer if backlog exceeds 10,000 messages for more than 5 minutes, or if error rate exceeds 5%.

Step 6: Schema Management and Evolution

As your system grows, multiple teams will publish and consume events from the same topics. Without schema management, a producer changing a field can silently break dozens of consumers. Use a schema registry (like Confluent Schema Registry for Kafka, or Google Cloud's schema support for Pub/Sub) to enforce contracts.

Here's an example Avro schema for user events:

// user-events-schema.avsc
{
  "type": "record",
  "name": "UserEvent",
  "namespace": "com.example.events",
  "fields": [
    { "name": "userId", "type": "string" },
    { "name": "type", "type": "string" },
    { "name": "timestamp", "type": "long" },
    { "name": "properties", "type": { "type": "map", "values": "string" }, "default": {} },
    { "name": "version", "type": "int", "default": 1 }
  ]
}

Follow these evolution rules to maintain backward compatibility:

Best Practices Summary

Conclusion

Scaling a Pub/Sub system from prototype to production is less about the broker and more about the patterns you build around it. Batching and ordering keys solve throughput and correctness. Flow control and dead-letter queues solve resilience. Idempotency solves the reality of at-least-once delivery. Observability and schema management solve the human problems of operating a system that many teams depend on. By incrementally adding each of these layers — rather than trying to build them all on day one — you can ship your prototype quickly while having a clear roadmap for hardening it as real traffic arrives. The result is a messaging backbone that stays stable under load, fails gracefully when things go wrong, and scales horizontally as your business grows.

— Ad —

Google AdSense will appear here after approval

← Back to all articles