← Back to DevBytes
Developer Tutorial July 14, 2026 14 min read

KubeState Metrics: Complete Implementation Guide

Introduction to KubeState Metrics

KubeState Metrics (often abbreviated as KSM) is a Kubernetes-native service that generates Prometheus-compatible metrics about the state of objects in a Kubernetes cluster. Unlike metrics-server or cAdvisor, which focus on resource utilization like CPU and memory consumption, KubeState Metrics exposes rich, object-level metadata about the state of Kubernetes resources — Deployments, Pods, Nodes, PersistentVolumes, StatefulSets, and many more.

It works by listening to the Kubernetes API server, watching object changes, and transforming the current state into Prometheus metrics. It does not export metrics about itself, nor does it require any agent to be installed inside pods. It simply runs as a standalone deployment or as a DaemonSet and scrapes the entire cluster state into a flat metric endpoint.

Originally created by the Kubernetes SIG-Instrumentation group, KubeState Metrics has become a foundational component of any serious Kubernetes monitoring stack. It is maintained under the kubernetes/kube-state-metrics GitHub repository and ships as a standalone binary or container image.

Why KubeState Metrics Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Without KubeState Metrics, Prometheus can only see raw container metrics and node-level data. But understanding what is running, how it is deployed, and whether it is healthy requires object-level state information. Here is why KSM is indispensable:

These metrics enable operators to build precise alerts — for example, "alert when any Deployment has fewer ready replicas than desired for more than 5 minutes" — which is impossible with only CPU/memory metrics.

Architecture and How It Works

Internal Design

KubeState Metrics runs as a single binary that:

  1. Connects to the Kubernetes API server using a ServiceAccount token (typically via the default cluster RBAC).
  2. Discovers all configured resource types based on command-line flags or a configuration file.
  3. Establishes list-and-watch connections for each resource type (Pods, Deployments, Nodes, etc.).
  4. Maintains an in-memory cache of all objects and their state transitions.
  5. On each Prometheus scrape request to /metrics, iterates over the cached objects and generates metric lines.
  6. Exposes a /healthz and /livez endpoint for health checks.

The critical design insight is that KSM does not store time-series data. It only holds the current state. Prometheus scrapes this state periodically and stores the time series in its TSDB. This separation keeps KSM lightweight and focused.

Metric Generation Model

Each Kubernetes object produces several metrics. For example, a Deployment generates:

All metrics carry Kubernetes metadata as Prometheus labels: namespace, deployment, and sometimes uid, resource_version, or condition. This labeling scheme is what makes PromQL queries so expressive.

Installation and Deployment

Option 1: Static Manifest Deployment

The simplest method is to apply the upstream static manifest directly. This deploys a ClusterRole, ServiceAccount, ClusterRoleBinding, and Deployment for KSM:

# Download and apply the official manifest
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kube-state-metrics/main/examples/standard/service-account.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kube-state-metrics/main/examples/standard/cluster-role.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kube-state-metrics/main/examples/standard/cluster-role-binding.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kube-state-metrics/main/examples/standard/deployment.yaml

# Verify it is running
kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-state-metrics

Option 2: Custom Deployment with Fine-Grained Configuration

For production, you often want to customize the metrics exposed, add resource limits, and tune scrape behavior. Here is a complete Deployment manifest with all essential configurations:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - pods
      - services
      - persistentvolumeclaims
      - persistentvolumes
      - namespaces
      - configmaps
      - secrets
      - resourcequotas
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - deployments
      - daemonsets
      - statefulsets
      - replicasets
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - jobs
      - cronjobs
    verbs: ["list", "watch"]
  - apiGroups: ["autoscaling"]
    resources:
      - horizontalpodautoscalers
    verbs: ["list", "watch"]
  - apiGroups: ["policy"]
    resources:
      - poddisruptionbudgets
    verbs: ["list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources:
      - ingresses
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
  - kind: ServiceAccount
    name: kube-state-metrics
    namespace: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: "2.13.0"
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
        - name: kube-state-metrics
          image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0
          args:
            - "--port=8080"
            - "--telemetry-port=8081"
            - "--metric-labels-allowlist=pods=[*]"
            - "--resources=certificatesigningrequests,cronjobs,daemonsets,deployments,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,nodes,persistentvolumeclaims,persistentvolumes,pods,poddisruptionbudgets,replicasets,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
          ports:
            - name: metrics
              containerPort: 8080
              protocol: TCP
            - name: telemetry
              containerPort: 8081
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /livez
              port: telemetry
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /healthz
              port: telemetry
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 65534
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-state-metrics
spec:
  ports:
    - name: metrics
      port: 8080
      targetPort: metrics
      protocol: TCP
    - name: telemetry
      port: 8081
      targetPort: telemetry
      protocol: TCP
  selector:
    app.kubernetes.io/name: kube-state-metrics

Option 3: Helm Installation

The kube-prometheus-stack Helm chart bundles KSM by default. If you prefer a standalone Helm installation:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --create-namespace \
  --set image.tag=v2.13.0 \
  --set resources.requests.cpu=100m \
  --set resources.requests.memory=256Mi \
  --set resources.limits.cpu=500m \
  --set resources.limits.memory=512Mi

Configuration Deep Dive

Selecting Resources to Monitor

By default, KSM watches a broad set of resources. In large clusters (thousands of pods, hundreds of deployments), this can produce an enormous number of metrics, causing high cardinality and memory pressure. You can restrict which resources are watched using the --resources flag:

# Only watch pods, deployments, and nodes
--resources=pods,deployments,nodes

Alternatively, use the --resource-denylist flag to exclude specific resources while keeping everything else:

# Exclude secrets and configmaps to reduce cardinality
--resource-denylist=secrets,configmaps

Controlling Metric Label Cardinality

Label cardinality is the single biggest operational concern with KSM. Every unique combination of label values creates a new time series in Prometheus. By default, KSM attaches many labels. The --metric-labels-allowlist flag lets you precisely control which labels appear on metrics for each resource type:

# For pods, only include namespace, pod, and uid labels
--metric-labels-allowlist=pods=[namespace,pod,uid]

You can also use wildcards:

# Allow all labels for pods
--metric-labels-allowlist=pods=[*]
# Allow only namespace and name for deployments
--metric-labels-allowlist=deployments=[namespace,deployment]

For production, a sensible starting configuration is:

args:
  - "--metric-labels-allowlist=pods=[namespace,pod,uid,node,phase]"
  - "--metric-labels-allowlist=deployments=[namespace,deployment]"
  - "--metric-labels-allowlist=statefulsets=[namespace,statefulset]"
  - "--metric-labels-allowlist=nodes=[node]"
  - "--metric-labels-allowlist=namespaces=[namespace]"
  - "--metric-labels-allowlist=persistentvolumes=[persistentvolume,storageclass]"
  - "--metric-labels-allowlist=jobs=[namespace,job]"
  - "--metric-labels-allowlist=cronjobs=[namespace,cronjob]"
  - "--metric-labels-allowlist=daemonsets=[namespace,daemonset]"

Using a Configuration File (v2.6+)

Since version 2.6, KSM supports a YAML configuration file passed via --config. This provides more granular control over which metrics and labels are enabled per resource:

apiVersion: kube-state-metrics/v2alpha1
kind: ResourceConfig
resources:
  pods:
    metrics:
      - kube_pod_status_phase
      - kube_pod_status_reason
      - kube_pod_container_status_restarts_total
    labels:
      - namespace
      - pod
      - uid
      - node
  deployments:
    metrics:
      - kube_deployment_status_replicas
      - kube_deployment_status_replicas_ready
      - kube_deployment_status_replicas_available
    labels:
      - namespace
      - deployment
  nodes:
    metrics:
      - kube_node_status_condition
      - kube_node_spec_unschedulable
    labels:
      - node
      - condition

Apply this configuration by mounting it as a ConfigMap and passing --config=/etc/kube-state-metrics/config.yaml.

Sharding for Large Clusters

In clusters with tens of thousands of pods, a single KSM instance can consume gigabytes of memory. Sharding distributes the workload across multiple KSM instances. Each instance watches only a subset of namespaces based on a hash ring:

# Instance 1 of 3
args:
  - "--shard=1"
  - "--shards=3"
# Instance 2 of 3
args:
  - "--shard=2"
  - "--shards=3"
# Instance 3 of 3
args:
  - "--shard=3"
  - "--shards=3"

Each instance exposes the /metrics endpoint independently. Prometheus must be configured to scrape all instances. The sharding is namespace-based: each namespace is assigned to exactly one shard using consistent hashing, ensuring no duplicate metrics.

Key Metrics Reference

Pod Metrics

Pod metrics are the most heavily used in alerting. Here are the essential ones:

Deployment Metrics

Node Metrics

Job and CronJob Metrics

PersistentVolume Metrics

Integrating with Prometheus and Alerting

Prometheus Scrape Configuration

Add a scrape job to your Prometheus configuration to collect KSM metrics:

scrape_configs:
  - job_name: 'kube-state-metrics'
    scrape_interval: 30s
    scrape_timeout: 25s
    honor_labels: true
    static_configs:
      - targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):.*'
        target_label: instance
        replacement: '${1}'

If you are using sharding, add all shard instances:

scrape_configs:
  - job_name: 'kube-state-metrics'
    scrape_interval: 30s
    honor_labels: true
    static_configs:
      - targets:
          - 'kube-state-metrics-shard-1.monitoring.svc.cluster.local:8080'
          - 'kube-state-metrics-shard-2.monitoring.svc.cluster.local:8080'
          - 'kube-state-metrics-shard-3.monitoring.svc.cluster.local:8080'

For Kubernetes service discovery, use:

scrape_configs:
  - job_name: 'kube-state-metrics'
    scrape_interval: 30s
    honor_labels: true
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
            - monitoring
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: 'kube-state-metrics'
        action: keep
      - source_labels: [__meta_kubernetes_endpoints_name]
        regex: 'kube-state-metrics'
        action: keep

Essential PromQL Queries

Here are battle-tested queries for common operational needs:

# Count pods in CrashLoopBackOff state
count(
  kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}
) by (namespace, pod, container)

# Deployments with fewer ready replicas than desired
(
  kube_deployment_spec_replicas
  !=
  kube_deployment_status_replicas_ready
) > 0

# Pods not ready for more than 5 minutes
(
  kube_pod_status_ready{condition="true"} == 0
) and (
  (time() - kube_pod_status_scheduled_time) > 300
)

# Nodes with DiskPressure condition active
kube_node_status_condition{condition="DiskPressure",status="true"} == 1

# Failed jobs in the last hour
count(
  kube_job_status_failed{job="kube-state-metrics"}
  > 0
) by (namespace, job_name)

# CronJobs that have not been scheduled in the last 2 hours
(
  time() - kube_cronjob_status_last_schedule_time
) > 7200

# PVCs not bound to a PV
kube_persistentvolumeclaim_status_phase{phase!="Bound"}

# Top 10 namespaces by running pod count
topk(10,
  sum(kube_pod_status_phase{phase="Running"}) by (namespace)
)

# Percentage of ready pods per deployment
(
  sum(kube_deployment_status_replicas_ready) by (namespace, deployment)
  /
  sum(kube_deployment_spec_replicas) by (namespace, deployment)
) * 100

# Node allocatable CPU remaining (summarized)
sum(
  kube_node_status_allocatable{resource="cpu"}
) by (node)
-
sum(
  kube_pod_container_resource_requests{resource="cpu"}
) by (node)

Alerting Rules

Convert the queries above into Prometheus alerting rules. Create a file named kube-state-metrics-alerts.yaml:

groups:
  - name: kube-state-metrics
    rules:
      - alert: DeploymentReplicasMismatch
        expr: |
          (
            kube_deployment_spec_replicas
            !=
            kube_deployment_status_replicas_ready
          ) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Deployment {{ $labels.deployment }} in {{ $labels.namespace }} has mismatched replicas"
          description: "Desired: {{ $value }} replicas, Ready: {{ $labels.kube_deployment_status_replicas_ready }}"

      - alert: PodCrashLooping
        expr: |
          kube_pod_container_status_restarts_total > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod {{ $labels.pod }} container {{ $labels.container }} is crash looping"
          description: "Container has restarted {{ $value }} times in namespace {{ $labels.namespace }}"

      - alert: NodeDiskPressure
        expr: |
          kube_node_status_condition{condition="DiskPressure",status="true"} == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.node }} has DiskPressure condition"
          description: "Immediate disk cleanup or expansion required on node {{ $labels.node }}"

      - alert: JobFailed
        expr: |
          kube_job_status_failed > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Job {{ $labels.job_name }} in {{ $labels.namespace }} has failed"
          description: "Job has {{ $value }} failed executions"

      - alert: CronJobSuspended
        expr: |
          kube_cronjob_spec_suspend == 1
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "CronJob {{ $labels.cronjob }} in {{ $labels.namespace }} is suspended"
          description: "CronJob has been suspended for over an hour, may need manual intervention"

      - alert: PVReleasedOrFailed
        expr: |
          kube_persistentvolume_status_phase{phase=~"Released|Failed"} > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "PersistentVolume {{ $labels.persistentvolume }} is in {{ $labels.phase }} phase"
          description: "PV may need manual cleanup or investigation"

      - alert: PodUnschedulable
        expr: |
          kube_pod_status_scheduled{condition="false"} == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} cannot be scheduled"
          description: "Check node capacity, affinity rules, or tolerations for pod {{ $labels.pod }}"

Load these rules into Prometheus by referencing the file in your Prometheus configuration:

rule_files:
  - '/etc/prometheus/rules/kube-state-metrics-alerts.yaml'

Grafana Dashboard Integration

KSM metrics shine when visualized. The Kubernetes-mixin project provides pre-built Grafana dashboards and Prometheus alerts specifically designed around KSM metrics. To import them:

# Clone the kubernetes-monitoring/kubernetes-mixin repository
git clone https://github.com/kubernetes-monitoring/kubernetes-mixin.git
cd kubernetes-mixin

# Install jsonnet-bundler and dependencies
jb install

# Generate dashboards and alerts
make dashboards alerts

# The generated dashboards are in:
# dashboards/kube-state-metrics.json
# dashboards/nodes.json
# dashboards/pods.json
# dashboards/deployments.json

Import these JSON files into Grafana via the UI or provision them automatically through Grafana's dashboard provisioning mechanism.

Best Practices

1. Control Label Cardinality Aggressively

This is the number one rule. By default, KSM can generate millions of unique time series in large clusters. Always use --metric-labels-allowlist to restrict labels. Never export kube_pod_labels or kube_pod_annotations unless you absolutely need them and have a Prometheus instance sized to handle the cardinality. A single pod with many labels can explode into thousands of time series.

2. Set Appropriate Resource Limits

KSM memory usage scales with the number of objects it watches. Start with 256Mi requests and 512Mi limits, then adjust based on observation. In clusters with more than 5000 pods, consider 1Gi+ limits or sharding. CPU usage is moderate but spikes during initial sync after a restart — set limits at 500m minimum.

3. Use Sharding for Clusters Over 5000 Pods

Sharding is not optional at scale. A rule of thumb: when total pod count exceeds 5000, deploy at least 3 shards. Monitor each shard's memory usage and add shards if any instance exceeds 2Gi of memory.

4. Separate Scrape Intervals for Different Use Cases

KSM metrics change relatively slowly compared to container metrics. A 30s or 60s scrape interval is sufficient for most alerting needs. If you are using KSM metrics only for dashboards, 120s is acceptable and reduces load on both KSM and Prometheus.

5. Monitor KSM Itself

KSM exposes its own metrics on the telemetry port (default 8081). Scrape these to monitor KSM health:

scrape_configs:
  - job_name: 'kube-state-metrics-self'
    scrape_interval: 30s
    static_configs:
      - targets: ['kube-state-metrics.monitoring.svc.cluster.local:8081']

Key self-metrics to watch:

6. Avoid Duplicate Scraping

If you run both KSM and kube-prometheus-stack, ensure you are not scraping KSM twice. The kube-prometheus-stack includes KSM by default with its own ServiceMonitor. If you deploy a standalone KSM, either disable the bundled one or adjust scrape configurations to avoid metric duplication.

7. Use honor_labels: true

Always set honor_labels: true in your Prometheus scrape job for KSM. This preserves the labels KSM attaches (like namespace, deployment, pod) instead of overwriting them with scrape target labels. Without this, your queries will break because critical labels get overwritten.

8. Leverage KSM for Capacity Planning

KSM metrics are invaluable for forecasting. Use kube_node_status_capacity and kube_pod_container_resource_requests to compute resource headroom across the cluster. Combine with kube_persistentvolume_capacity_bytes for storage forecasting. These metrics, aggregated over time, give you a precise picture of cluster growth trends.

9. Pin Version and Test Upgrades

KSM metric names and labels occasionally change between major versions. Always pin to a specific version tag (e.g., v2.13.0) rather than using latest. Before upgrading, review the release notes for metric deprecations or label changes, and test in a staging cluster to verify that your PromQL queries and alerting rules still work correctly.

10. Secure the Metrics Endpoint

The KSM metrics endpoint contains sensitive information about your cluster's configuration and state. Apply network policies to restrict access to the metrics port only to your Prometheus instances. Do not expose KSM to the internet. Consider enabling TLS on the metrics endpoint for additional security in shared or multi-tenant clusters.

Conclusion

KubeState Metrics is a cornerstone of Kubernetes observability. It transforms the raw object state from the Kubernetes API into structured Prometheus metrics, enabling precise alerting, insightful dashboards, and informed capacity planning. By understanding its architecture, carefully managing label cardinality, leveraging sharding at scale, and integrating it thoughtfully with Prometheus and Grafana, you can build a monitoring stack that truly understands your cluster's operational state — not just its resource consumption. The investment in

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles