Monitoring Pub/Sub: Metrics, Alarms, and Dashboards

Understanding Pub/Sub Monitoring

Pub/Sub (publish/subscribe) messaging is the backbone of many event-driven architectures, microservices communication, and real-time data pipelines. Monitoring your Pub/Sub infrastructure means continuously observing its health, performance, and data flow to ensure messages are delivered reliably and on time. This tutorial covers the essential metrics, how to set up alarms, and how to build dashboards that give you complete visibility into your messaging system.

What Is Pub/Sub Monitoring?

Pub/Sub monitoring involves collecting, analyzing, and visualizing operational data from your messaging topics, subscriptions, and the delivery pipeline. In platforms like Google Cloud Pub/Sub, Apache Kafka, or RabbitMQ, monitoring typically exposes metrics such as:

Message throughput – publish and pull rates.
Backlog size – number of unacknowledged messages.
Acknowledgment latency – time taken to process messages.
Error rates – failed publishes, dead-letter deliveries.
Resource utilization – CPU/memory for self-managed brokers.

These metrics are surfaced via a monitoring service (like Cloud Monitoring, Prometheus, or Datadog) and used to create alarms and dashboards.

Why It Matters

Without monitoring, a Pub/Sub system can silently degrade:

A subscriber lag can grow unnoticed until the backlog becomes unmanageable.
A sudden spike in publish errors might indicate a downstream service outage.
Message duplication or expired messages can corrupt business logic.

Proper monitoring helps you detect anomalies early, maintain SLOs (Service Level Objectives), and troubleshoot issues before they impact users. It also enables capacity planning by tracking trends in message volume and system load.

Key Metrics to Track

For a managed service like Google Cloud Pub/Sub, the following metrics are crucial:

Undelivered messages (subscription backlog) – metric subscription/num_unacked_messages or subscription/backlog_bytes.
Published message count – topic/send_message_request_count.
Pull/Acknowledge request count – subscription/pull_request_count, subscription/acknowledge_request_count.
Acknowledgment latency – subscription/ack_latencies (distribution).
Dead-letter message count – subscription/dead_letter_message_count.
Topic/subsciption message size – topic/send_message_size.

For self-hosted systems (Kafka, RabbitMQ), you'll also monitor broker health, partition lag, and connection counts.

Setting Up Metrics Collection

Google Cloud Pub/Sub Monitoring with Cloud Monitoring

Cloud Monitoring automatically collects Pub/Sub metrics. You can view them in the Metrics Explorer or via the API. Here’s an example of reading the undelivered messages metric using the Cloud Monitoring API in Python:


from google.cloud import monitoring_v3
from google.protobuf import timestamp_pb2
import time

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{PROJECT_ID}"

# Build the time series query for subscription backlog
interval = monitoring_v3.TimeInterval({
    "end_time": timestamp_pb2.Timestamp(seconds=int(time.time())),
    "start_time": timestamp_pb2.Timestamp(seconds=int(time.time()) - 3600)
})

query = monitoring_v3.QueryTimeSeriesRequest(
    name=project_name,
    query='fetch subscription::pubsub.googleapis.com/subscription/num_unacked_messages'
          '| filter resource.label.subscription_id = "my-subscription"'
          '| every 1m',
    interval=interval
)

for ts in client.query_time_series(request=query):
    for point in ts.points:
        print(f"Time: {point.interval.end_time}, Unacked: {point.value.int64_value}")

Custom Metrics and Export

If you need business-specific metrics (e.g., number of orders processed per second), you can write custom metrics using Cloud Monitoring’s API or OpenTelemetry. Exporting metrics to external tools like Grafana or Datadog can be achieved via Google Cloud’s cloud-monitoring-export sidecar or Prometheus federation.

Creating Alarms

Alarms (alerting policies) automatically notify you when a metric crosses a defined threshold. You can create them via the Cloud Console, gcloud, or Terraform.

Alerting on Undelivered Messages

An increasing backlog often means a subscriber is down or slow. The following Terraform snippet creates an alert when the number of unacknowledged messages exceeds 1000 for more than 5 minutes:


resource "google_monitoring_alert_policy" "pubsub_backlog_alert" {
  display_name = "High Pub/Sub Backlog"
  combiner     = "OR"
  conditions {
    display_name = "Unacked messages > 1000"
    condition_threshold {
      filter          = "metric.type=\"pubsub.googleapis.com/subscription/num_unacked_messages\" resource.label.subscription_id=\"orders-sub\""
      comparison      = "COMPARISON_GT"
      threshold_value = 1000
      duration {
        seconds = 300
      }
      aggregations {
        alignment_period {
          seconds = 300
        }
        per_series_aligner = "ALIGN_MEAN"
      }
    }
  }
  notification_channels = [google_monitoring_notification_channel.email.id]
}

resource "google_monitoring_notification_channel" "email" {
  display_name = "DevOps Email"
  type         = "email"
  labels = {
    email_address = "devops@example.com"
  }
}

Alerting on Subscription Backlog with gcloud

Alternatively, use the gcloud CLI to quickly create an alert:


gcloud alpha monitoring policies create \
  --display-name="Backlog Alert" \
  --condition='metric.type="pubsub.googleapis.com/subscription/num_unacked_messages" resource.label.subscription_id="my-sub" > 500 for 5 min' \
  --notification-channel="projects/my-project/notificationChannels/1234567890"

Always test alerts by simulating load or temporarily stopping a subscriber.

Building Dashboards

Dashboards provide real-time visibility and historical context. You can use Cloud Monitoring dashboards, Grafana, or custom solutions.

Cloud Monitoring Dashboards

Define a dashboard declaratively with a JSON layout or via the UI. Below is a dashboard snippet that shows published messages and backlog side by side:


gcloud monitoring dashboards create my-pubsub-dashboard.json

Example my-pubsub-dashboard.json:


{
  "displayName": "Pub/Sub Overview",
  "dashboardFilters": [],
  "layouts": [
    {
      "widget": {
        "title": "Published Messages (1 hour)",
        "xyChart": {
          "dataSets": [
            {
              "timeSeriesQuery": {
                "timeSeriesQuery": "fetch pubsub.googleapis.com/topic/send_message_request_count | every 1m | group_by(resource.topic_id)"
              }
            }
          ],
          "chartOptions": {
            "mode": "LINE"
          }
        }
      }
    },
    {
      "widget": {
        "title": "Subscription Backlog",
        "xyChart": {
          "dataSets": [
            {
              "timeSeriesQuery": {
                "timeSeriesQuery": "fetch pubsub.googleapis.com/subscription/num_unacked_messages | every 1m"
              }
            }
          ],
          "chartOptions": {
            "mode": "AREA"
          }
        }
      }
    }
  ]
}

Grafana Integration

To use Grafana with Cloud Monitoring, set up the Google Cloud Monitoring data source. Then import a dashboard like:


apiVersion: 1
datasources:
  - name: GoogleCloudMonitoring
    type: stackdriver
    orgId: 1
    jsonData:
      authenticationType: jwt
      defaultProject: your-project-id
      tokenUri: https://oauth2.googleapis.com/token
      clientEmail: grafana@your-project.iam.gserviceaccount.com

In Grafana, create panels with queries like pubsub.googleapis.com/subscription/num_unacked_messages and set up alerts directly from Grafana.

Best Practices

Alert on symptoms, not causes – Focus on user-visible impact like message processing delays, not internal metrics like CPU unless they directly correlate.
Set meaningful thresholds – Use baseline data to determine normal backlog ranges; avoid static thresholds that trigger for every small fluctuation.
Implement multi-channel notifications – Combine email, Slack, and PagerDuty for critical alerts to ensure on-call engineers respond promptly.
Use composite alerts – Combine multiple conditions (e.g., high backlog AND subscriber health check failing) to reduce noise.
Monitor end-to-end latency – Track the time from publish to final acknowledgment, including any processing logic.
Dashboard everything, alert on exceptions – Dashboards give context; alerts demand action. Keep dashboards comprehensive but alerts precise.
Regularly review and update – As traffic patterns change, revisit thresholds and alert policies to avoid fatigue.

Conclusion

Monitoring Pub/Sub is not a one-time setup but an evolving practice. By tracking the right metrics, setting actionable alarms, and building informative dashboards, you ensure that your messaging layer remains robust and your event-driven applications stay healthy. Start with the built-in metrics of your chosen platform, create alerts for critical conditions like growing backlogs, and gradually refine your dashboards to provide the operational insights your team needs. With these tools in place, you’ll catch issues before they cascade, maintain high reliability, and keep your data flowing smoothly.