Monitoring Cloud Functions: Metrics, Alarms, and Dashboards

AWS CloudWatch Alarm via CloudFormation / Terraform

resource "aws_cloudwatch_metric_alarm" "lambda_error_rate" {
  alarm_name          = "lambda-error-rate-high"
  alarm_description   = "Triggers when Lambda error rate exceeds 5% over 5 minutes"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 5
  period              = 60
  threshold           = 0.05
  namespace           = "AWS/Lambda"
  metric_name         = "Errors"
  statistic           = "Sum"
  dimensions = {
    FunctionName = "processOrder"
  }
  alarm_actions = [aws_sns_topic.alerts.arn]
  treat_missing_data = "notBreaching"
}

Alarm Design Principles

• Alert on symptoms, not causes: Alert on elevated error rates (symptom), not on increased CPU usage (cause). Symptoms directly map to user pain.
• Include runbooks in alarm documentation: Every alarm should link to a specific playbook. "Check the payment-gateway health dashboard and examine recent deploys" is infinitely better than "Something went wrong."
• Use multi-condition alerting: A single condition on error rate is noisy. Combine error rate > 5% AND invocation count > 10 (to eliminate low-traffic noise) AND duration p95 > 500ms.
• Implement dead-man's switch alarms: Create an alarm that triggers if a function stops being invoked entirely for an hour. Silence can be catastrophic—it often means the trigger (pub/sub topic, cron scheduler, API gateway) is broken.

Building Dashboards That Reveal Insight

Dashboards transform raw metrics into visual narratives. A good dashboard lets you answer "Is the system healthy?" in under five seconds and then drill into "Which function is responsible for the latency spike?" in under thirty.

Essential Dashboard Layout for Cloud Functions

Organize your dashboard in a top-down flow: overall health → per-function detail → dependency drill-down.

• Row 1: Global Health Summary — A single-stat row showing total invocation rate, aggregate error percentage, and a heat map of function latency by region. Use sparklines to show 24-hour trends.
• Row 2: Per-Function Latency Profiles — Line charts plotting p50, p95, and p99 duration for your top five most-invoked functions. Overlay cold-start latency as a separate series. Use log-scale Y-axis to make spikes visible.
• Row 3: Error Breakdown — Stacked bar chart of error counts per function, color-coded by error type (timeout, unhandled exception, out-of-memory). Cross-filter with deployment markers.
• Row 4: Resource Utilization — Memory usage vs. allocated memory for each function. This directly informs cost optimization: you are paying for allocated memory, not used memory.
• Row 5: Dependency Health — Custom metrics for external API latency and cache hit ratios. These are your leading indicators—they degrade before your function metrics do.

Creating a GCP Custom Dashboard Programmatically

// Create a Monitoring Dashboard via the API
const {DashboardServiceClient} = require('@google-cloud/monitoring-dashboards').v1;
const dashboardClient = new DashboardServiceClient();

async function createFunctionDashboard(projectId) {
  const dashboard = {
    displayName: 'Cloud Functions Health Dashboard',
    gridLayout: {
      columns: 2,
      widgets: [
        // Widget 1: Invocation rate (top row, spans full width)
        {
          title: 'Total Invocation Rate (req/s)',
          xyChart: {
            dataSets: [{
              timeSeriesQuery: {
                timeSeriesFilter: {
                  filter: 'resource.type="cloud_function" metric.type="cloudfunctions.googleapis.com/function/invocation_count"',
                  aggregation: {
                    alignmentPeriod: '60s',
                    perSeriesAligner: 'ALIGN_RATE',
                  },
                },
              },
              plotType: 'LINE',
            }],
            yAxis: { label: 'invocations/s' },
          },
        },
        // Widget 2: p95 Duration per function
        {
          title: 'p95 Execution Duration by Function',
          xyChart: {
            dataSets: [{
              timeSeriesQuery: {
                timeSeriesFilter: {
                  filter: 'resource.type="cloud_function" metric.type="cloudfunctions.googleapis.com/function/duration"',
                  aggregation: {
                    alignmentPeriod: '300s',
                    perSeriesAligner: 'ALIGN_PERCENTILE_95',
                  },
                  secondaryAggregation: {
                    crossSeriesReducer: 'REDUCE_NONE',
                    perSeriesAligner: 'ALIGN_MEAN',
                  },
                },
              },
              plotType: 'LINE',
            }],
            yAxis: { label: 'ms' },
          },
        },
        // Widget 3: Error count by function
        {
          title: 'Errors by Function (5m rate)',
          xyChart: {
            dataSets: [{
              timeSeriesQuery: {
                timeSeriesFilter: {
                  filter: 'resource.type="cloud_function" metric.type="cloudfunctions.googleapis.com/function/invocation_count" metric.labels.status!="ok"',
                  aggregation: {
                    alignmentPeriod: '300s',
                    perSeriesAligner: 'ALIGN_RATE',
                  },
                },
              },
              plotType: 'STACKED_BAR',
            }],
          },
        },
        // Widget 4: Custom metric – cache hit ratio
        {
          title: 'Cache Hit Ratio',
          xyChart: {
            dataSets: [{
              timeSeriesQuery: {
                timeSeriesFilter: {
                  filter: 'metric.type="custom.googleapis.com/functions/cache_hit_ratio"',
                  aggregation: {
                    alignmentPeriod: '300s',
                    perSeriesAligner: 'ALIGN_MEAN',
                  },
                },
              },
              plotType: 'LINE',
            }],
            yAxis: { label: 'ratio', minValue: 0, maxValue: 1 },
          },
        },
      ],
    },
  };

  const request = {
    parent: `projects/${projectId}`,
    dashboard: dashboard,
  };

  const [created] = await dashboardClient.createDashboard(request);
  console.log(`Dashboard created: ${created.name}`);
  return created;
}

AWS CloudWatch Dashboard via JSON Template

{
  "widgets": [
    {
      "type": "metric",
      "x": 0, "y": 0, "width": 24, "height": 6,
      "properties": {
        "metrics": [
          ["AWS/Lambda", "Invocations", "FunctionName", "processOrder", {"stat": "Sum", "period": 60}],
          ["AWS/Lambda", "Errors", "FunctionName", "processOrder", {"stat": "Sum", "period": 60}]
        ],
        "view": "timeSeries",
        "stacked": false,
        "region": "us-east-1",
        "title": "processOrder Invocations vs Errors",
        "yAxis": { "left": { "label": "Count" } }
      }
    },
    {
      "type": "metric",
      "x": 0, "y": 6, "width": 12, "height": 6,
      "properties": {
        "metrics": [
          ["AWS/Lambda", "Duration", "FunctionName", "processOrder", {"stat": "p95", "period": 300}],
          ["AWS/Lambda", "Duration", "FunctionName", "processOrder", {"stat": "p50", "period": 300}]
        ],
        "view": "timeSeries",
        "stacked": false,
        "region": "us-east-1",
        "title": "processOrder Duration p50/p95",
        "yAxis": { "left": { "label": "Milliseconds" } }
      }
    },
    {
      "type": "metric",
      "x": 12, "y": 6, "width": 12, "height": 6,
      "properties": {
        "metrics": [
          [{ "expression": "m1/m2*100", "label": "ErrorRatePercent", "id": "e1" }],
          ["AWS/Lambda", "Errors", "FunctionName", "processOrder", {"stat": "Sum", "period": 300, "id": "m1"}],
          ["AWS/Lambda", "Invocations", "FunctionName", "processOrder", {"stat": "Sum", "period": 300, "id": "m2"}]
        ],
        "view": "timeSeries",
        "stacked": false,
        "region": "us-east-1",
        "title": "processOrder Error Rate %",
        "yAxis": { "left": { "label": "Percent", "min": 0, "max": 100 } }
      }
    }
  ]
}

Best Practices for Cloud Function Monitoring

1. Define and Track SLIs Rigorously

Your Service Level Indicators (SLIs) should be derived directly from metrics. For a Cloud Function that serves an API endpoint, your primary SLI is the proportion of invocations that return a successful HTTP status within an acceptable latency threshold. Define this in your monitoring system as a composite metric: invocations with status='ok' AND duration < 500ms divided by total invocations. Alert on this SLI breaching your SLO (Service Level Objective), not on raw error counts.

2. Avoid Alert Fatigue with Proper Thresholding

Set alerts to trigger after a sustained breach (at least 5 minutes), not on instantaneous spikes. Use percentile-based thresholds (p95 duration > 2 seconds) rather than mean-based ones, because the mean obscures tail latency that impacts individual users. For low-traffic functions, add a minimum invocation count condition to prevent alerting on statistical noise when only three invocations occurred and one failed.

3. Embrace Log-Based Metrics for High-Cardinality Data

Platform metrics have limited label cardinality. When you need to slice by customer ID, request trace ID, or specific error codes, create log-based metrics. In GCP, write structured logs and define metrics that extract values from log fields. In AWS, use CloudWatch Logs Insights to generate metrics from log patterns. This lets you answer questions like "Which customer is experiencing the most 504 errors from the payments function?"

// Structured logging for log-based metrics (Node.js)
// Each log entry becomes a data point when Cloud Monitoring extracts metric values

console.log(JSON.stringify({
  severity: 'INFO',
  message: 'External API call completed',
  functionName: process.env.FUNCTION_NAME,
  apiLatencyMs: 145,
  apiName: 'payment-gateway',
  statusCode: 200,
  customerId: 'cust_12345',
  traceId: req.headers['x-cloud-trace-context'] || '',
}));

// Cloud Monitoring log-based metric configuration would extract:
// - metric type: custom.googleapis.com/functions/api_latency
// - label: apiName, statusCode
// - value: from field apiLatencyMs

4. Correlate Metrics with Deployment Events

Overlay deployment markers on every dashboard. When duration spikes at 14:32 and your deployment pipeline shows a new version rolled out at 14:30, you have eliminated 80% of the diagnostic effort. In GCP, Cloud Deploy automatically pushes deployment markers; in AWS, CodeDeploy does the same. Use these markers religiously.

5. Monitor Cold Starts Separately

Cold starts are not a continuous metric but they dominate latency outliers. In GCP, examine the cloudfunctions.googleapis.com/function/duration metric alongside instance start timestamps from logs. Create a derived metric that flags invocations where duration exceeds 2× the recent p50—these are almost certainly cold starts. Track the cold start ratio over time; a sudden increase indicates either a traffic surge or that instances are being recycled too aggressively.

6. Right-Size Memory Using Observed Metrics

Cloud Functions bill based on allocated memory, not used memory. The built-in user_memory_bytes metric shows actual peak usage. If your function consistently uses 180 MB but is allocated 512 MB, you are paying for 332 MB of waste. Set up a monthly review process: query the 95th percentile of memory usage over a rolling 30-day window and adjust the allocation to 1.2× that value as a safety buffer.

7. Implement Multi-Channel Notifications with Escalation

A single email alert is insufficient. Structure your notification channels into tiers:

• Tier 1 (immediate, non-paging): Slack/Teams channel notification for all alerts. This is the team's always-visible information radiator.
• Tier 2 (paging, 5-minute sustain): PagerDuty/OpsGenie for error rate > 5% sustained over 5 minutes during business hours.
• Tier 3 (paging, 1-minute sustain, 24/7): On-call escalation for error rate > 10% or complete function silence (zero invocations for 15 minutes when expected traffic is non-zero).

8. Version-Pin Your Monitoring Configuration

Dashboards, alerting policies, and metric definitions are infrastructure. Store them in version control alongside your function code. Use Terraform, Pulumi, or equivalent IaC tools. This ensures that when you roll back a function deployment, the corresponding monitoring configuration is also rolled back. It also creates an audit trail for who changed an alert threshold and why.

Conclusion

Monitoring Cloud Functions is not a peripheral activity—it is a core engineering practice that directly impacts cost, reliability, and your team's ability to debug under pressure. The built-in platform metrics give you immediate visibility into invocation patterns, errors, and resource usage. Custom metrics fill the gap by surfacing business-level signals and dependency health. Alarms convert these signals into actionable notifications when configured with realistic thresholds, sustained breach durations, and clear runbook documentation. Dashboards knit everything together into a visual narrative that lets you assess system health in seconds and drill into root causes in minutes. By following the practices outlined here—defining SLIs, avoiding alert fatigue, leveraging log-based metrics, correlating deployments, right-sizing memory, and managing monitoring as infrastructure code—you transform your serverless functions from opaque black boxes into transparent, manageable, and continuously improving components of your production system.