← Back to DevBytes

Monitoring CosmosDB: Metrics, Alarms, and Dashboards

Understanding CosmosDB Monitoring

Monitoring Azure CosmosDB is the practice of collecting, analyzing, and acting on telemetry data from your database operations. This encompasses three core pillars: metrics (numerical measurements of system behavior over time), alarms (automated notifications when thresholds are breached), and dashboards (visual canvases that unify metrics and alerts into a single pane of glass). Together, these pillars form a comprehensive observability strategy that ensures your globally distributed NoSQL database remains performant, available, and cost-efficient.

What Constitutes CosmosDB Monitoring Data

Azure CosmosDB emits a rich stream of telemetry across multiple layers:

All this data flows through Azure Monitor, where it can be visualized, queried via Kusto Query Language (KQL), and wired to automated alert rules.

Why Monitoring CosmosDB Matters

CosmosDB operates as a fully managed service, which might lull teams into believing monitoring is optional. In reality, the shared responsibility model means Microsoft guarantees the platform uptime while you own workload health. Without proactive monitoring, several costly scenarios can unfold silently:

Cost Spikes from Unoptimized Throughput

CosmosDB bills provisioned throughput (RU/s) at container or database level. A single misconfigured autoscale container can silently ramp to 10x baseline RU/s if queries lack efficient indexing or contain cross-partition scans. Monitoring the NormalizedRUConsumption metric reveals whether you're paying for capacity you don't need or throttling under sudden load.

Silent 429 Throttling

When request volume exceeds provisioned throughput, CosmosDB returns HTTP 429 status codes. Client SDKs retry these transparently with exponential backoff, so without metrics, your application may appear "slower" but not broken. Over time, retries inflate latency, burn SDK retry budgets, and cascade into upstream timeouts. Monitoring TotalRequests grouped by status code exposes exactly how many requests are throttled per second.

Partition Hot Spots

CosmosDB distributes data across logical partitions. If your partition key design funnels 80% of requests into a single partition, that partition's throughput ceiling caps total container performance regardless of overall RU/s provisioned. Metrics like PartitionKeyStatistics (available via diagnostic logs) and per-partition throughput consumption help detect skew before it causes outages.

Unexpected Storage Costs

Analytical store, time-to-live (TTL) misses, and unmonitored index growth can bloat storage. Monitoring IndexSize and DataSize separately helps pinpoint whether your indexing policy is indexing fields you never query.

Key Metrics Every Developer Should Track

Azure Monitor surfaces CosmosDB metrics under the Microsoft.DocumentDB/databaseAccounts namespace. Here are the most actionable ones:

Enabling Diagnostic Settings for Deep Insights

While Azure Monitor platform metrics give you numerical aggregates, diagnostic settings unlock granular, queryable logs that capture every request's metadata. This is where you find partition key distribution, detailed latency breakdowns, and operation-level diagnostics.

Step-by-Step: Enabling Diagnostic Logs

You configure diagnostic settings at the CosmosDB account level, routing logs to Log Analytics Workspace, Storage Account, or Event Hub.

# Azure CLI: Enable diagnostic settings for CosmosDB
az monitor diagnostic-settings create \
  --resource "/subscriptions/{subscriptionId}/resourceGroups/{rg}/providers/Microsoft.DocumentDB/databaseAccounts/{accountName}" \
  --name "cosmos-logs-to-la" \
  --workspace "/subscriptions/{subscriptionId}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}" \
  --logs '[
    {"category": "DataPlaneRequests", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "MongoRequests", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "QueryRuntimeStatistics", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "PartitionKeyStatistics", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "PartitionKeyRUConsumption", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "GremlinRequests", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "CassandraRequests", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "TableApiRequests", "enabled": true, "retentionPolicy": {"enabled": false}}
  ]' \
  --metrics '[
    {"category": "Requests", "enabled": true, "retentionPolicy": {"enabled": false}},
    {"category": "MongoRequests", "enabled": true, "retentionPolicy": {"enabled": false}}
  ]'

The critical log categories to enable are:

Querying Metrics with KQL in Log Analytics

Once diagnostic logs flow into Log Analytics, you can craft Kusto queries to surface patterns invisible in aggregate metrics. Below are practical queries you can run directly in the Azure Portal's Log Analytics blade.

Detecting Hot Partitions

This query aggregates request count per partition within a container over the last hour. Partitions with disproportionately high counts indicate skew.

// Hot partition detection over the last 1 hour
PartitionKeyStatistics
| where TimeGenerated >= ago(1h)
| where DatabaseName == "orders-db" and CollectionName == "orders-container"
| summarize RequestCount = sum(RequestCharge) by PartitionKey
| order by RequestCount desc
| project PartitionKey, RequestCount, PercentageOfTotal = round(RequestCount / sum(RequestCount) * 100, 2)
| take 10

429 Throttling Rate by Operation

Track how many requests receive 429 responses, broken down by operation type and minute granularity.

// 429 throttling rate per minute by operation type
DataPlaneRequests
| where TimeGenerated >= ago(1h)
| where StatusCode == 429
| summarize ThrottledCount = count() by bin(TimeGenerated, 1m), OperationType
| order by ThrottledCount desc
| project TimeGenerated, OperationType, ThrottledCount

P99 Latency for Read Operations

Calculate the 99th percentile latency for read operations, which helps identify tail latency issues affecting the slowest 1% of users.

// P99 read latency in milliseconds
DataPlaneRequests
| where TimeGenerated >= ago(1h)
| where OperationType == "Read"
| summarize 
    P50 = percentile(DurationMs, 50),
    P95 = percentile(DurationMs, 95),
    P99 = percentile(DurationMs, 99),
    Avg = avg(DurationMs),
    Max = max(DurationMs)
    by bin(TimeGenerated, 5m)
| order by TimeGenerated desc

RU Consumption by Partition (Diagnostic Logs)

Identify which partitions consume the most RU/s, helping tune partition key strategy.

// RU consumption per partition
PartitionKeyRUConsumption
| where TimeGenerated >= ago(1h)
| where DatabaseName == "inventory-db" and CollectionName == "products"
| summarize TotalRU = sum(RequestCharge) by PartitionKey
| order by TotalRU desc
| project PartitionKey, TotalRU, Percentage = round(TotalRU / sum(TotalRU) * 100, 2)
| take 20

Building Alarms That Prevent Outages

Alarms transform passive metrics into active guardrails. The key is to design alerts that fire before user-impacting degradation, not after.

Types of Alert Rules in Azure Monitor

Creating a Metric Alert for RU Consumption

This alert fires when normalized RU consumption exceeds 80% for 5 consecutive minutes, indicating potential throttling soon.

# Azure CLI: Create metric alert for normalized RU consumption
az monitor metrics alert create \
  --name "cosmos-high-ru-alert" \
  --resource-group "prod-rg" \
  --scopes "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.DocumentDB/databaseAccounts/{accountName}" \
  --description "Alert when normalized RU exceeds 80% for 5 minutes" \
  --condition "where NormalizedRUConsumption > 80" \
  --evaluation-frequency 1m \
  --window-size 5m \
  --severity 2 \
  --action "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.Insights/actionGroups/oncall-pager"

Creating a Log Alert for Sustained 429 Rate

When more than 3% of requests receive 429 status codes over a 5-minute window, trigger an alert. This catches throttling before retry storms cascade.

# Azure CLI: Create log alert for high 429 rate
az monitor scheduled-query create \
  --name "cosmos-high-429-rate" \
  --resource-group "prod-rg" \
  --scopes "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.DocumentDB/databaseAccounts/{accountName}" \
  --description "Alert when 429 rate exceeds 3% over 5 minutes" \
  --query '
    let totalRequests = toscalar(DataPlaneRequests 
      | where TimeGenerated >= ago(5m) 
      | summarize count());
    let throttledRequests = toscalar(DataPlaneRequests 
      | where TimeGenerated >= ago(5m) and StatusCode == 429 
      | summarize count());
    print ThrottleRate = round((throttledRequests * 1.0 / totalRequests) * 100, 2)
      | where ThrottleRate > 3.0
  ' \
  --evaluation-frequency 5m \
  --severity 1 \
  --action-groups "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.Insights/actionGroups/oncall-pager"

Action Groups: Where Alerts Deliver Value

Alerts are useless if nobody sees them. Action groups define notification channels:

# Create an action group for email and webhook
az monitor action-group create \
  --name "oncall-pager" \
  --resource-group "prod-rg" \
  --action email "ops-team" ops-team@company.com \
  --action webhook "pagerduty" "https://events.pagerduty.com/integration/{key}/enqueue" \
  --short-name "pager"

Designing Effective Dashboards

Dashboards transform raw metrics and alert states into visual narratives that different audiences can consume quickly. A well-designed dashboard answers three questions in five seconds: "Is my database healthy?", "Where should I look next?", and "Is this a new or known issue?"

Azure Portal Native Dashboards

The simplest approach is Azure's shared dashboard service. You pin metric charts, log query results, and alert tiles onto a customizable grid. Dashboards support role-based access control, so you can create separate views for developers (detailed latency and RU charts) and executives (availability SLA and cost trends).

To pin a metric chart programmatically, you first define the metric tile as JSON and then create or update the dashboard via ARM template or CLI.

// ARM template snippet for a dashboard tile showing RU consumption
{
  "type": "Microsoft.Portal/dashboards",
  "apiVersion": "2020-09-01-preview",
  "name": "cosmos-operations-dashboard",
  "properties": {
    "lenses": [
      {
        "order": 0,
        "parts": [
          {
            "position": { "x": 0, "y": 0, "colSpan": 6, "rowSpan": 4 },
            "metadata": {
              "type": "Extension/Microsoft_Azure_Monitoring/PartType/MetricsChartPart",
              "settings": {
                "content": {
                  "title": "Normalized RU Consumption (5m)",
                  "chartType": "Line",
                  "resourceId": "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.DocumentDB/databaseAccounts/{accountName}",
                  "metricQuery": {
                    "resourceId": "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.DocumentDB/databaseAccounts/{accountName}",
                    "metricNames": ["NormalizedRUConsumption"],
                    "aggregation": "Average",
                    "timeRange": "PT1H",
                    "timeGrain": "PT5M"
                  }
                }
              }
            }
          }
        ]
      }
    ]
  }
}

Building Grafana Dashboards for CosmosDB

For teams managing multiple data sources, Azure Monitor serves as a Grafana data source. This allows you to combine CosmosDB metrics with application metrics from Prometheus, logs from Elasticsearch, and infrastructure data from other clouds — all on one pane.

To connect Grafana to Azure Monitor, configure the managed identity or app registration and then use the Azure Monitor data source plugin. Below is a sample Grafana dashboard JSON panel definition querying CosmosDB latency.

// Grafana dashboard panel JSON for CosmosDB P99 latency
{
  "datasource": {
    "type": "grafana-azure-monitor-datasource",
    "uid": "azure-monitor-uid"
  },
  "targets": [
    {
      "azureMonitor": {
        "metricDefinition": "Microsoft.DocumentDB/databaseAccounts",
        "metricName": "Latency",
        "aggregation": "Percentile99",
        "timeGrain": "5m",
        "resource": "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.DocumentDB/databaseAccounts/{accountName}",
        "dimensionFilters": [
          {
            "dimension": "OperationType",
            "operator": "eq",
            "values": ["Read"]
          }
        ]
      },
      "refId": "A"
    }
  ],
  "title": "CosmosDB P99 Read Latency",
  "type": "timeseries",
  "fieldConfig": {
    "defaults": {
      "unit": "ms",
      "thresholds": {
        "steps": [
          { "value": 0, "color": "green" },
          { "value": 100, "color": "orange" },
          { "value": 200, "color": "red" }
        ]
      }
    }
  }
}

Structuring Your Dashboard Layout

A production-grade CosmosDB dashboard should follow this logical flow:

Automating Monitoring Deployment with ARM and Bicep

Manually configuring alerts and dashboards per environment doesn't scale. Infrastructure-as-code templates ensure every CosmosDB instance — dev, staging, production — ships with the same monitoring baseline.

Bicep Template for Alert Rules

Below is a reusable Bicep module that creates a metric alert for normalized RU consumption on any CosmosDB account.

// monitoring-alerts.bicep — reusable alert module
param cosmosAccountName string
param location string
param actionGroupResourceId string

resource cosmosAccount 'Microsoft.DocumentDB/databaseAccounts@2023-04-01' existing = {
  name: cosmosAccountName
}

resource ruAlert 'Microsoft.Insights/metricAlerts@2021-09-01-preview' = {
  name: '${cosmosAccountName}-high-ru-alert'
  location: location
  properties: {
    description: 'Alert when normalized RU consumption exceeds 80% for 5 minutes'
    severity: 2
    enabled: true
    evaluationFrequency: 'PT1M'
    windowSize: 'PT5M'
    scopes: [
      {
        id: cosmosAccount.id
      }
    ]
    criteria: {
      'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria': {
        allOf: [
          {
            name: 'HighRU'
            metricName: 'NormalizedRUConsumption'
            metricNamespace: 'Microsoft.DocumentDB/databaseAccounts'
            operator: 'GreaterThan'
            threshold: 80
            timeAggregation: 'Average'
            dimensions: []
          }
        ]
      }
    }
    actions: [
      {
        actionGroupId: actionGroupResourceId
      }
    ]
  }
}

// Output the alert resource ID for composability
output alertResourceId string = ruAlert.id

Deploying the Bicep Module

# Deploy monitoring baseline for a CosmosDB account
az deployment group create \
  --resource-group "prod-rg" \
  --template-file monitoring-alerts.bicep \
  --parameters cosmosAccountName="orders-cosmos-prod" \
               location="westus2" \
               actionGroupResourceId="/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.Insights/actionGroups/oncall-pager"

Best Practices for CosmosDB Monitoring

1. Right-Size Alert Thresholds Using Historical Baselines

Don't guess thresholds. Deploy your workload, let it run for at least two weeks, then query the NormalizedRUConsumption 95th percentile over that period. Set your alert threshold slightly above that baseline (e.g., baseline 45% + 30% headroom = alert at 75%). This prevents false positives from normal fluctuations while catching genuine anomalies.

// Determine 95th percentile RU consumption baseline
AzureMetrics
| where MetricName == "NormalizedRUConsumption"
| where TimeGenerated >= ago(14d)
| summarize P95 = percentile(Average, 95) by bin(TimeGenerated, 1h)
| summarize BaselineP95 = percentile(P95, 50)

2. Alert on Latency, Not Just Throughput

RU overage alerts fire before throttling, but latency spikes can occur even when RU headroom exists — due to cross-partition queries, large payloads, or SDK connection pool exhaustion. Always pair a throughput alert with a P99 latency alert targeting your SLA (e.g., P99 read latency > 100ms).

3. Separate Alerts per Environment

A dev environment hitting 80% RU consumption might be normal during load testing; the same in production is an emergency. Use Azure tags (environment=prod) and scope alerts accordingly, or maintain separate action groups that route dev alerts to Slack and prod alerts to PagerDuty.

4. Monitor the Monitoring Pipeline

If your Log Analytics workspace ingestion drops to zero or the diagnostic setting gets accidentally disabled, all log-based alerts go silent. Create a "heartbeat" log alert that fires when DataPlaneRequests row count over the past hour is zero — this indicates either a real outage or a monitoring gap.

// Heartbeat check: no data plane logs in the last hour
DataPlaneRequests
| where TimeGenerated >= ago(1h)
| summarize RowCount = count()
| where RowCount == 0

5. Use Azure Policy to Enforce Diagnostic Settings

Instead of relying on manual configuration, deploy Azure Policy that automatically enables diagnostic settings on all CosmosDB accounts in a subscription, routing logs to a central Log Analytics workspace.

// Azure Policy definition to enforce diagnostic settings on CosmosDB
{
  "policyRule": {
    "if": {
      "field": "type",
      "equals": "Microsoft.DocumentDB/databaseAccounts"
    },
    "then": {
      "effect": "deployIfNotExists",
      "details": {
        "type": "Microsoft.Insights/diagnosticSettings",
        "existenceCondition": {
          "field": "Microsoft.Insights/diagnosticSettings/logs[*].enabled",
          "equals": "true"
        },
        "roleDefinitionIds": [
          "/providers/Microsoft.Authorization/roleDefinitions/b24988ac-2480-4640-8644-5379e9aeea2b"
        ],
        "deployment": {
          "properties": {
            "mode": "incremental",
            "template": {
              "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
              "contentVersion": "1.0.0.0",
              "resources": [
                {
                  "type": "Microsoft.Insights/diagnosticSettings",
                  "apiVersion": "2021-05-01-preview",
                  "name": "autoDiagnosticSetting",
                  "properties": {
                    "workspaceId": "[parameters('workspaceId')]",
                    "logs": [
                      { "category": "DataPlaneRequests", "enabled": true },
                      { "category": "PartitionKeyStatistics", "enabled": true },
                      { "category": "PartitionKeyRUConsumption", "enabled": true },
                      { "category": "QueryRuntimeStatistics", "enabled": true }
                    ]
                  }
                }
              ]
            }
          }
        }
      }
    }
  }
}

6. Budget Alerts Are Non-Negotiable

CosmosDB costs can spiral from misconfigured autoscale or runaway storage. Configure Azure Cost Management budget alerts scoped to your CosmosDB resource group, firing at 50%, 75%, and 100% of monthly forecast. Cross-reference these with throughput metrics to correlate cost anomalies with usage patterns.

7. Document Runbooks Alongside Alerts

Every production alert should link to a runbook — a concise Markdown document stored in your team's knowledge base. The runbook describes: what the alert means, the likely root causes (e.g., "sudden traffic spike from marketing campaign"), immediate triage steps (check 429 rate, check partition distribution), and escalation path. Embed the runbook URL in the alert's description field so it appears in the notification.

8. Leverage Workbooks for Interactive Troubleshooting

Azure Monitor Workbooks are interactive reports that combine text, metrics, and KQL queries into a notebook-like experience. Create a "CosmosDB Troubleshooting Workbook" that guides engineers through a decision tree: "Is the issue throughput, latency, or errors?" → drill into relevant metrics → view correlated logs. Workbooks can be shared as a link and require no dashboard setup.

Putting It All Together: A Complete Monitoring Deployment Script

Below is an end-to-end PowerShell script that deploys diagnostic settings, metric alerts, log alerts, an action group, and pins key metrics to a dashboard — all for an existing CosmosDB account.

# Complete CosmosDB monitoring deployment script
# Requirements: Az PowerShell module, logged into Azure context

param(
  [string]$ResourceGroup = "prod-rg",
  [string]$CosmosAccountName = "orders-cosmos-prod",
  [string]$Location = "westus2",
  [string]$LogAnalyticsWorkspaceId = "/subscriptions/{subId}/resourceGroups/prod-rg/providers/Microsoft.OperationalInsights/workspaces/central-logs",
  [string]$AlertEmail = "ops-team@company.com"
)

$cosmosResourceId = "/subscriptions/{subId}/resourceGroups/$ResourceGroup/providers/Microsoft.DocumentDB/databaseAccounts/$CosmosAccountName"

# Step 1: Create Action Group
Write-Host "Creating action group..."
$actionGroup = az monitor action-group create `
  --name "cosmos-oncall" `
  --resource-group $ResourceGroup `
  --action email "ops-email" $AlertEmail `
  --action webhook "pagerduty" "https://events.pagerduty.com/v2/enqueue" `
  --short-name "cosmos" `
  --query "id" -o tsv

# Step 2: Enable Diagnostic Settings
Write-Host "Enabling diagnostic settings..."
az monitor diagnostic-settings create `
  --resource $cosmosResourceId `
  --name "cosmos-full-logs" `
  --workspace $LogAnalyticsWorkspaceId `
  --logs '[
    {"category":"DataPlaneRequests","enabled":true},
    {"category":"PartitionKeyStatistics","enabled":true},
    {"category":"PartitionKeyRUConsumption","enabled":true},
    {"category":"QueryRuntimeStatistics","enabled":true}
  ]'

# Step 3: Metric Alert - High RU Consumption
Write-Host "Creating RU consumption alert..."
az monitor metrics alert create `
  --name "cosmos-high-ru" `
  --resource-group $ResourceGroup `
  --scopes $cosmosResourceId `
  --description "NormalizedRUConsumption exceeds 80% for 5 minutes" `
  --condition "where NormalizedRUConsumption > 80" `
  --evaluation-frequency 1m `
  --window-size 5m `
  --severity 2 `
  --action $actionGroup

# Step 4: Metric Alert - P99 Latency
Write-Host "Creating P99 latency alert..."
az monitor metrics alert create `
  --name "cosmos-high-latency" `
  --resource-group $ResourceGroup `
  --scopes $cosmosResourceId `
  --description "P99 read latency exceeds 150ms for 3 minutes" `
  --condition "where Latency > 150 and OperationType == Read" `
  --evaluation-frequency 1m `
  --window-size 3m `
  --severity 1 `
  --action $actionGroup

# Step 5: Log Alert - High 429 Rate
Write-Host "Creating 429 rate log alert..."
az monitor scheduled-query create `
  --name "cosmos-429-surge" `
  --resource-group $ResourceGroup `
  --scopes $cosmosResourceId `
  --description "429 rate exceeds 3% in 5-minute window" `
  --query '
    let total = toscalar(DataPlaneRequests | where TimeGenerated >= ago(5m) | count);
    let throttled = toscalar(DataPlaneRequests | where TimeGenerated >= ago(5m) and StatusCode == 429 | count);
    print Rate = round((throttled * 1.0 / total) * 100, 2) | where Rate > 3.0
  ' `
  --evaluation-frequency 5m `
  --severity 1 `
  --action-groups $actionGroup

# Step 6: Log Alert - Monitoring Heartbeat
Write-Host "Creating monitoring heartbeat alert..."
az monitor scheduled-query create `
  --name "cosmos-no-logs" `
  --resource-group $ResourceGroup `
  --scopes $cosmosResourceId `
  --description "No DataPlaneRequests logs in the past hour - possible monitoring gap" `
  --query 'DataPlaneRequests | where TimeGenerated >= ago(1h) | count | where Count == 0' `
  --evaluation-frequency 1h `
  --severity 2 `
  --action-groups $actionGroup

Write-Host "CosmosDB monitoring baseline deployed successfully for $CosmosAccountName"

Conclusion

Monitoring CosmosDB is not a one-time configuration task — it is an evolving practice that matures alongside your workload. Start by enabling platform metrics and diagnostic logs on every account. Build metric alerts for the two signals that precede most incidents: sustained high RU consumption and rising P99 latency. Augment those with log-based alerts for 429 throttling rates and partition hot spots that aggregate metrics cannot reveal. Invest in dashboards structured by audience and urgency, and codify everything — alerts, diagnostic settings, dashboards — into infrastructure-as-code templates so that every new CosmosDB instance inherits the same observability baseline. Finally, treat your monitoring pipeline itself as critical infrastructure: heartbeat alerts, Azure Policy enforcement of diagnostic settings, and budget alerts ensure that your guardrails remain intact. With these practices, you transform CosmosDB from a black-box cloud service into a transparent, predictable, and cost-optimized component of your application architecture.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles