← Back to DevBytes
Developer Tutorial July 17, 2026 13 min read

Monitoring BigQuery: Metrics, Alarms, and Dashboards

Understanding BigQuery Monitoring

Monitoring BigQuery means systematically tracking the health, performance, and cost of your data warehouse workloads. Unlike traditional databases where you manage infrastructure, BigQuery is a fully managed, serverless platform — but that doesn't eliminate the need for observability. In fact, because costs scale directly with usage, monitoring becomes a critical discipline for both engineering and finance teams.

At its core, BigQuery monitoring encompasses three pillars:

Together, these pillars form a feedback loop: metrics inform alarms, alarms trigger investigations, and dashboards provide the context needed to diagnose and resolve issues quickly.

Why BigQuery Monitoring Matters

The serverless nature of BigQuery means you never have to provision or resize clusters. However, this abstraction also obscures operational details that would be visible in a self-managed system. Without monitoring, you risk:

In short, monitoring transforms BigQuery from a black box into a transparent, controllable service. It empowers teams to optimize continuously rather than reactively firefighting.

Key Metrics You Should Track

Query Performance Metrics

BigQuery exposes query-level metrics through the INFORMATION_SCHEMA views and Cloud Monitoring. The most important query metrics include:

Slot Utilization Metrics

Slots are BigQuery's unit of computational capacity. Monitoring slot usage is essential for capacity planning and cost control:

Storage Metrics

Cost Metrics

Querying Metrics Using INFORMATION_SCHEMA

BigQuery's INFORMATION_SCHEMA provides native SQL access to job history, reservations, and capacity data. These views are your primary tool for building custom monitoring logic without external instrumentation.

Analyzing Query Performance History

The INFORMATION_SCHEMA.JOBS view contains execution details for completed queries. The following query retrieves the top 10 most expensive queries in the past 24 hours:


SELECT
  creation_time,
  project_id,
  job_id,
  user_email,
  query,
  total_bytes_processed / POW(1024, 4) AS tb_processed,
  total_slot_ms / 1000 AS slot_seconds,
  SAFE_DIVIDE(total_slot_ms, TIMESTAMP_DIFF(end_time, start_time, MILLISECOND)) AS avg_active_slots,
  end_time - start_time AS duration,
  cache_hit,
  error_result.message AS error_message
FROM
  `region-us`.INFORMATION_SCHEMA.JOBS
WHERE
  creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
  AND statement_type = 'SELECT'
  AND state = 'DONE'
ORDER BY
  total_bytes_processed DESC
LIMIT 10;

This query gives you immediate visibility into heavy hitters. Pay attention to queries with high avg_active_slots — they may be consuming disproportionate compute resources.

Tracking Slot Utilization Over Time

For reservation-based pricing, monitoring slot utilization across time windows helps identify peak usage periods:


SELECT
  TIMESTAMP_TRUNC(period_start, MINUTE) AS time_minute,
  SUM(period_slot_ms) / (60 * 1000) AS average_slots_in_that_minute,
  MAX(period_slot_ms) / 1000 AS peak_slot_demand_that_minute
FROM
  `region-us`.INFORMATION_SCHEMA.JOBS_TIMELINE
WHERE
  period_start >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 6 HOUR)
  AND job_type = 'QUERY'
GROUP BY
  time_minute
ORDER BY
  time_minute ASC;

The JOBS_TIMELINE view breaks each job into periodic slices, showing how slot consumption evolves throughout query execution. This granularity is invaluable for understanding concurrency patterns.

Monitoring Reservation Capacity

If you use BigQuery Reservations, the RESERVATIONS view reveals current assignment and utilization:


SELECT
  project_id,
  reservation_name,
  slot_capacity,
  baseline_slot_capacity,
  target_job_concurrency,
  autoscale.current_slots AS autoscaled_slots,
  autoscale.max_slots AS max_autoscale_slots
FROM
  `region-us`.INFORMATION_SCHEMA.RESERVATIONS
WHERE
  reservation_name = 'my-production-reservation';

Identifying Queries Blocked by Slot Contention

When demand exceeds capacity, queries queue. This query finds jobs that waited unusually long before execution:


SELECT
  job_id,
  creation_time,
  start_time,
  TIMESTAMP_DIFF(start_time, creation_time, SECOND) AS queue_wait_seconds,
  total_bytes_processed / POW(1024, 4) AS tb_processed,
  query
FROM
  `region-us`.INFORMATION_SCHEMA.JOBS
WHERE
  creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  AND state = 'DONE'
  AND TIMESTAMP_DIFF(start_time, creation_time, SECOND) > 60
ORDER BY
  queue_wait_seconds DESC
LIMIT 20;

A queue wait exceeding a few seconds during business hours often signals that reservations need expansion or workload scheduling needs adjustment.

Cloud Monitoring Metrics for BigQuery

Google Cloud Monitoring automatically collects BigQuery metrics and makes them available through the Metrics Explorer, Alerting, and Dashboards services. These metrics require no setup — they exist as soon as you start using BigQuery.

Key Cloud Monitoring metric names include:

These metrics support filtering by project_id, dataset_id, reservation_name, and priority, enabling targeted analysis of specific workloads.

Setting Up Alarms and Alerts

Alerting on Slot Utilization Spikes

Configure an alert when average slot utilization exceeds 85% of capacity for a sustained period. In the Cloud Console, navigate to Monitoring → Alerting → Create Policy, or use the following gcloud command:


gcloud alpha monitoring policies create \
  --display-name="BigQuery High Slot Utilization" \
  --condition='resource.type="bigquery_project" AND metric.type="bigquery.googleapis.com/query/cpu_usage" AND metric.labels.reservation="default"' \
  --condition-threshold="aggregation.alignmentPeriod=300s, aggregation.perSeriesAligner=ALIGN_MEAN, aggregation.crossSeriesReducer=REDUCE_MEAN, comparison=COMPARISON_GT, threshold=0.85, duration=300s" \
  --notification-channel="projects/my-project/notificationChannels/1234567890"

This creates an alert that fires when slot utilization remains above 85% for at least 5 minutes. The duration clause prevents flapping from momentary spikes.

Alerting on Excessive Data Scanned

To catch queries that scan abnormally large volumes — a common source of cost surprises — create an alert on the scanned_bytes metric:


gcloud alpha monitoring policies create \
  --display-name="BigQuery Excessive Scan Volume" \
  --condition='resource.type="bigquery_project" AND metric.type="bigquery.googleapis.com/query/scanned_bytes"' \
  --condition-threshold="aggregation.alignmentPeriod=300s, aggregation.perSeriesAligner=ALIGN_SUM, comparison=COMPARISON_GT, threshold=10995116277760, duration=0s" \
  --notification-channel="projects/my-project/notificationChannels/1234567890"

The threshold of 10 TB (10,995,116,277,760 bytes) within a 5-minute window catches queries that could individually cost tens of dollars. Adjust the threshold based on your tolerance and typical workload patterns.

Alerting on Query Failure Rate

Monitor the ratio of failed queries to total queries. A sudden increase often indicates upstream schema changes, permission revocations, or data corruption:


-- Use Cloud Monitoring Query Language (MQL) for ratio-based alerts
-- This MQL expression computes failure rate over a 10-minute window
fetch bigquery.googleapis.com/job/job_count
| filter metric.job_state = 'FAILED'
| align delta 600s
| value val() / (
    fetch bigquery.googleapis.com/job/job_count
    | filter metric.job_state = 'DONE'
    | align delta 600s
    | value val()
  )
| condition gt(val(), 0.05)

This MQL snippet can be used directly in the Cloud Monitoring alerting UI under the Query Editor tab. It triggers when more than 5% of queries fail within a 10-minute period.

Programmatic Alert Creation with Python

For teams managing alerts as code, the Cloud Monitoring API offers full control. Here's how to create an alert using Python:


from google.cloud import monitoring_v3
from google.protobuf import duration_pb2
import google.api_core.protobuf_helpers

client = monitoring_v3.AlertPolicyServiceClient()
project_name = f"projects/{project_id}"

# Define the condition: average slot utilization > 80%
condition = monitoring_v3.AlertPolicy.Condition()
condition.display_name = "High slot utilization"
condition.condition_threshold = monitoring_v3.AlertPolicy.Condition.MetricThreshold()
condition.condition_threshold.filter = (
    'resource.type="bigquery_project" AND '
    'metric.type="bigquery.googleapis.com/query/cpu_usage"'
)
condition.condition_threshold.aggregations.append(
    monitoring_v3.Aggregation(
        alignment_period=duration_pb2.Duration(seconds=300),
        per_series_aligner=monitoring_v3.Aggregation.Aligner.ALIGN_MEAN,
        cross_series_reducer=monitoring_v3.Aggregation.Reducer.REDUCE_MEAN,
    )
)
condition.condition_threshold.comparison = monitoring_v3.ComparisonType.COMPARISON_GT
condition.condition_threshold.threshold_value = 0.80
condition.condition_threshold.duration = duration_pb2.Duration(seconds=300)

# Create the alert policy
policy = monitoring_v3.AlertPolicy()
policy.display_name = "BigQuery - High Slot Utilization"
policy.conditions.append(condition)
policy.notification_channels = [f"projects/{project_id}/notificationChannels/{channel_id}"]

created_policy = client.create_alert_policy(
    name=project_name,
    alert_policy=policy,
)
print(f"Created alert policy: {created_policy.name}")

This approach integrates naturally with CI/CD pipelines, allowing alert definitions to live alongside infrastructure code in version control.

Building Effective Dashboards

Cloud Monitoring Custom Dashboards

The Cloud Monitoring Dashboards service lets you build visualization widgets without leaving the GCP console. A well-structured BigQuery dashboard should include:

To create a dashboard widget for slot utilization via the gcloud CLI:


gcloud monitoring dashboards create my-bigquery-dashboard.json \
  --description="BigQuery operational dashboard"

Where the JSON configuration file might contain:


{
  "displayName": "BigQuery Operations",
  "dashboardFilters": [],
  "mosaicLayout": {
    "columns": 12,
    "tiles": [
      {
        "title": "Slot Utilization (Average)",
        "dataSource": {
          "type": "TIME_SERIES",
          "timeSeriesQuery": {
            "timeSeriesFilter": {
              "filter": "metric.type=\"bigquery.googleapis.com/query/cpu_usage\" resource.type=\"bigquery_project\"",
              "aggregation": {
                "alignmentPeriod": "300s",
                "perSeriesAligner": "ALIGN_MEAN"
              }
            }
          }
        },
        "chartOptions": {
          "displayVisualization": "LINE"
        }
      }
    ]
  }
}

Building Rich Dashboards with Looker Studio

For more sophisticated visualizations and cross-service reporting, Looker Studio (formerly Data Studio) connects directly to BigQuery and Cloud Monitoring. A common pattern is to create a monitoring dataset containing aggregated metrics from INFORMATION_SCHEMA and then visualize them in Looker Studio.

First, create a scheduled query that materializes aggregated metrics every hour:


-- Scheduled query: populate monitoring_agg table every hour
INSERT INTO `my-project.monitoring.query_metrics_1h`
SELECT
  TIMESTAMP_TRUNC(creation_time, HOUR) AS hour,
  user_email,
  COUNT(*) AS query_count,
  SUM(total_bytes_processed) / POW(1024, 4) AS total_tb_processed,
  SUM(total_slot_ms) / 1000 AS total_slot_seconds,
  AVG(TIMESTAMP_DIFF(end_time, start_time, MILLISECOND)) / 1000 AS avg_duration_seconds,
  COUNTIF(error_result IS NOT NULL) AS error_count
FROM
  `region-us`.INFORMATION_SCHEMA.JOBS
WHERE
  creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
  AND statement_type = 'SELECT'
  AND state = 'DONE'
GROUP BY
  hour, user_email;

Then connect Looker Studio to this table and create:

Real-Time Dashboard with Streaming Metrics

For near-real-time visibility, stream job completion events into a Pub/Sub topic and consume them in a lightweight dashboard. The BigQuery audit logs feed can be routed to Pub/Sub via Cloud Logging sinks:


# Create a log sink that routes BigQuery audit logs to Pub/Sub
gcloud logging sinks create bq-job-sink \
  pubsub.googleapis.com/projects/my-project/topics/bq-jobs \
  --log-filter='resource.type="bigquery.googleapis.com/Job" AND protoPayload.methodData.jobInsertRequest.resource.jobConfiguration.query IS NOT NULL'

Then a small Python service can subscribe to this topic, compute rolling metrics, and serve them via a web dashboard or push updates to a monitoring backend:


from google.cloud import pubsub_v1
import json

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path('my-project', 'bq-job-sub')

def callback(message):
    data = json.loads(message.data.decode('utf-8'))
    job_stats = data.get('protoPayload', {}).get('methodData', {}).get('jobInsertResponse', {})
    
    if job_stats:
        billed_bytes = int(job_stats.get('totalBilledBytes', 0))
        slot_ms = int(job_stats.get('totalSlotMs', 0))
        
        # Update your metrics store (Prometheus, custom DB, etc.)
        # For example, increment a counter and record a histogram
        print(f"Job completed: {billed_bytes} bytes, {slot_ms} slot_ms")
    
    message.ack()

subscriber.subscribe(subscription_path, callback=callback)
print("Listening for BigQuery job events...")

Best Practices for BigQuery Monitoring

1. Establish Baseline Thresholds from Historical Data

Before setting alerts, collect at least two weeks of metrics to understand your workload's natural rhythm. Query volume and slot usage often follow predictable daily and weekly patterns. Alerts configured without this context will generate noise and lead to alert fatigue.

2. Tier Your Alerts by Severity

Not all anomalies require immediate pager response. Structure alerts in tiers:

3. Monitor Per-User and Per-Dataset Granularity

Aggregate metrics can hide problematic behavior. A single user scanning petabytes may be invisible in a project-wide average. Build dashboards that break down metrics by user_email, dataset_id, and reservation_name. Use the INFORMATION_SCHEMA.JOBS view to attribute costs accurately.

4. Automate Response Actions

When an alert fires, automate the first diagnostic step. For example, on a high slot utilization alert, automatically run a query that identifies the top-consuming queries at that moment and logs them to a Slack channel or incident ticket. This reduces mean time to resolution.

5. Version-Control Your Monitoring Configuration

Treat alert policies, dashboard definitions, and scheduled monitoring queries as code. Store them in Git, review changes through pull requests, and deploy via CI/CD. This prevents configuration drift and ensures monitoring evolves alongside your data infrastructure.

6. Leverage the JOBS_TIMELINE for Concurrency Analysis

The JOBS_TIMELINE view is underutilized but extremely powerful. It shows how individual queries consume slots over their lifetime. Use it to identify queries that start with high parallelism and then drop to a single thread — classic indicators of skewed join keys or inefficient UDFs.

7. Set Storage Lifecycle Policies Based on Monitoring Data

Don't guess at storage lifecycle rules. Monitor actual access patterns using INFORMATION_SCHEMA.TABLE_STORAGE combined with query logs to identify tables not queried in 90+ days. Apply lifecycle policies surgically rather than blanket time-to-live rules.

8. Correlate BigQuery Metrics with Upstream Systems

BigQuery performance issues often originate upstream. Correlate slot spikes with Airflow DAG executions, Kafka throughput drops, or Fivetran sync failures. A holistic dashboard that overlays BigQuery metrics with pipeline events provides much richer diagnostic context.

Putting It All Together: A Complete Monitoring Stack

Here's a reference architecture for a production-grade BigQuery monitoring stack:


# Architecture components:
# 1. Scheduled INFORMATION_SCHEMA queries → aggregated metrics tables
# 2. Cloud Monitoring native metrics → Alert policies
# 3. Log sink → Pub/Sub → custom metrics processor → Looker Studio
# 4. Terraform-managed alert policies and dashboards

# Example Terraform snippet for an alert policy
resource "google_monitoring_alert_policy" "high_slot_usage" {
  display_name = "BigQuery High Slot Usage"
  combiner     = "OR"

  conditions {
    display_name = "Slot utilization above threshold"
    condition_threshold {
      filter          = "metric.type=\"bigquery.googleapis.com/query/cpu_usage\" resource.type=\"bigquery_project\""
      duration        = "300s"
      comparison      = "COMPARISON_GT"
      threshold_value = 0.85
      aggregations {
        alignment_period   = "300s"
        per_series_aligner = "ALIGN_MEAN"
      }
    }
  }

  notification_channels = [google_monitoring_notification_channel.email.id]

  alert_strategy {
    notification_rate_limit {
      period = "300s"
    }
  }
}

Conclusion

Monitoring BigQuery is not an optional add-on — it's a foundational practice that separates efficient, cost-effective data platforms from those plagued by surprises and waste. By instrumenting query performance, slot utilization, storage growth, and cost metrics, you gain the observability needed to operate at scale. The native tools — INFORMATION_SCHEMA views, Cloud Monitoring metrics, and Looker Studio dashboards — provide a comprehensive toolkit that requires no third-party integration to deliver immediate value. Start with the queries and alert configurations outlined in this tutorial, establish your baselines, and iterate your thresholds as your workload evolves. The investment in monitoring pays for itself many times over through prevented cost overruns, faster incident resolution, and continuously optimized query performance.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles