← Back to DevBytes

Metrics Server: Complete Implementation Guide

What is Metrics Server?

The Kubernetes Metrics Server is a cluster-wide aggregator of resource usage data. It collects CPU and memory metrics from the Kubelets running on each node and exposes them through the Kubernetes API server. Unlike a full monitoring system like Prometheus, Metrics Server is designed to be lightweight, fast, and focused exclusively on serving the metrics needed by core Kubernetes autoscaling pipelines—namely the Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA).

At its core, Metrics Server discovers all nodes in the cluster, queries each node's Kubelet for pod and node-level resource utilization, and caches the results in memory for a short period. It then serves these metrics via the /apis/metrics.k8s.io/v1beta1 API endpoint, making them available to kubectl top commands and autoscaler controllers.

Key Characteristics

Why Metrics Server Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Without Metrics Server installed, several critical Kubernetes features simply do not work. Understanding the concrete dependencies will help you prioritize its deployment in every cluster you manage.

Enabling Horizontal Pod Autoscaling

The Horizontal Pod Autoscaler (HPA) relies on Metrics Server to make scaling decisions. When you define an HPA targeting CPU or memory utilization, the controller queries the Metrics API to get current pod usage. If Metrics Server is absent, HPA objects will remain in a perpetual "unavailable" state with errors like:

failed to get metrics: unable to fetch metrics from resource metrics API

Powering kubectl top Commands

The familiar kubectl top pod and kubectl top node commands depend entirely on Metrics Server. These commands are the developer's first line of investigation when debugging resource contention. Without Metrics Server, you lose this essential diagnostic capability and must fall back to shelling into nodes or parsing cAdvisor output manually.

Supporting Vertical Pod Autoscaling

The Vertical Pod Autoscaler (VPA) uses Metrics Server data to recommend resource requests and limits based on historical usage patterns. While VPA can optionally use Prometheus for richer history, Metrics Server provides the baseline real-time metrics that feed its recommender engine.

Cluster Capacity Planning

Operators use node-level metrics from Metrics Server to understand cluster-wide utilization, identify resource hotspots, and make informed decisions about node scaling or pod eviction policies. Combined with kubectl top node, you can quickly assess whether your cluster is over-provisioned or approaching capacity limits.

Architecture Deep Dive

Understanding Metrics Server's internal architecture helps you troubleshoot issues and tune its behavior effectively. The system consists of three logical layers that work together in a straightforward pipeline.

Component Breakdown

Data Flow

Here is the complete data flow from container to API consumer:

Container Runtime (e.g., containerd)
    │
    ▼
cAdvisor (embedded in Kubelet) ← collects CPU/memory per container
    │
    ▼
Kubelet /metrics/resource endpoint ← serves scraped data
    │
    ▼
Metrics Server ← scrapes all Kubelets every scrape interval
    │
    ▼
In-memory cache ← stores latest metrics, timestamped
    │
    ▼
Kubernetes API Server ← proxies /apis/metrics.k8s.io requests
    │
    ▼
Consumers (HPA, kubectl top, VPA, custom controllers)

Scrape Mechanism

Metrics Server connects directly to each Kubelet's /metrics/resource endpoint on port 10250. It authenticates using TLS with the Kubelet's serving certificate. By default, it scrapes every 60 seconds, but this interval is configurable via the --metric-resolution flag. Each scrape fetches node-level metrics and pod-level metrics for all pods running on that node. The server then merges these into a single cluster-wide view and caches the result.

Installation Guide

Installing Metrics Server correctly requires attention to networking, security, and version compatibility. The following steps cover the most common installation scenarios.

Prerequisites

Method 1: Using the Official Manifest (Recommended)

The simplest approach is applying the official release manifest directly. This installs everything in the kube-system namespace.

# Download and install the latest stable version
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Verify installation
kubectl get deployment metrics-server -n kube-system
kubectl get apiservice v1beta1.metrics.k8s.io

Method 2: Customized Deployment with Kubelet TLS Overrides

In development clusters, self-signed Kubelet certificates often cause TLS verification failures. The following manifest adds the --kubelet-insecure-tls flag. Use this only in development or testing environments.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    app: metrics-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: metrics-server
  template:
    metadata:
      labels:
        app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: registry.k8s.io/metrics-server/metrics-server:v0.7.1
        args:
          - --cert-dir=/tmp
          - --secure-port=10250
          - --kubelet-use-node-status-port
          - --metric-resolution=15s
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        ports:
        - containerPort: 10250
          protocol: TCP
        resources:
          requests:
            cpu: 50m
            memory: 100Mi
          limits:
            cpu: 100m
            memory: 200Mi
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-server
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/metrics", "pods", "pods/metrics"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    app: metrics-server
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 10250
  selector:
    app: metrics-server
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

Method 3: Helm Installation

If your organization standardizes on Helm, the Metrics Server community maintains an official Helm chart.

# Add the metrics-server Helm repository
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm repo update

# Install with custom values
helm upgrade --install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --set args[0]=--kubelet-insecure-tls \
  --set args[1]=--metric-resolution=30s \
  --set replicas=2

Verifying the Installation

After deploying, wait approximately 30–60 seconds for the first scrape cycle to complete, then test:

# Check if the API service is available
kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

# Check pod metrics
kubectl top pod --all-namespaces

# Check node metrics
kubectl top node

# Check Metrics Server pod logs
kubectl logs -n kube-system deployment/metrics-server

A healthy installation will show output similar to:

NAMESPACE     NAME                          CPU(cores)   MEMORY(bytes)
kube-system   coredns-78fcd6995c-abc12      2m           12Mi
kube-system   metrics-server-xyz789         3m           18Mi
default       my-app-456def7890-ghi34       15m          128Mi

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-1         256m         12%    2048Mi          28%
node-2         189m         9%     1800Mi          25%

Configuration Reference

Metrics Server exposes numerous command-line flags that control its behavior. Understanding these flags is essential for tuning performance and resolving common issues in heterogeneous cluster environments.

Complete Flag Reference

# Core flags
--metric-resolution=60s         # How often to scrape Kubelets (minimum 15s)
--secure-port=10250             # Port for serving metrics to API server
--cert-dir=/tmp                 # Directory for TLS certificates

# Kubelet connectivity
--kubelet-use-node-status-port  # Use the port from node status (default 10250)
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
--kubelet-insecure-tls          # Skip Kubelet TLS verification (dev only)
--kubelet-client-certificate    # Path to client cert for Kubelet auth
--kubelet-client-key            # Path to client key for Kubelet auth

# Resource limits
--requestheader-allowed-names   # Comma-separated list of allowed CNs
--requestheader-extra-headers-prefix
--requestheader-group-headers
--requestheader-username-headers

# High availability
--replicas=1                    # Number of replicas (set in deployment, not flag)
--leader-elect                  # Enable leader election for multiple replicas

Address Type Priority

The --kubelet-preferred-address-types flag determines how Metrics Server resolves node addresses when connecting to Kubelets. The order matters—the server tries each type in sequence until it finds a reachable address.

# Prefer InternalIP first, then ExternalIP, then Hostname
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

# For clusters where nodes only have ExternalIP reachable from control plane
--kubelet-preferred-address-types=ExternalIP,InternalIP

Configuring TLS Properly for Production

For production clusters, avoid --kubelet-insecure-tls. Instead, configure proper Kubelet client certificates:

args:
  - --cert-dir=/tmp
  - --secure-port=10250
  - --kubelet-use-node-status-port
  - --metric-resolution=30s
  - --kubelet-preferred-address-types=InternalIP
  - --kubelet-client-certificate=/etc/metrics-server/kubelet-client.crt
  - --kubelet-client-key=/etc/metrics-server/kubelet-client.key
volumeMounts:
  - name: kubelet-client-certs
    mountPath: /etc/metrics-server
    readOnly: true
volumes:
  - name: kubelet-client-certs
    secret:
      secretName: metrics-server-kubelet-client-cert

Using Metrics Server Data

Once Metrics Server is running, you can consume its data through several interfaces. The most common access patterns are covered below.

CLI Access with kubectl top

The kubectl top command provides immediate visibility into resource consumption across your cluster.

# View all pods in the current namespace
kubectl top pod

# View pods across all namespaces
kubectl top pod --all-namespaces

# View pods filtered by label selector
kubectl top pod -l app=my-service

# View pods in a specific namespace
kubectl top pod -n production

# View node metrics
kubectl top node

# View node metrics sorted by CPU usage
kubectl top node --sort-by=cpu

# View node metrics sorted by memory usage
kubectl top node --sort-by=memory

# Watch metrics in real-time (requires watch utility)
watch -n 5 kubectl top pod -n production

Programmatic Access via Metrics API

You can query the Metrics API directly using standard Kubernetes API client libraries or raw HTTP requests.

# Get pod metrics via kubectl raw API
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

# Get metrics for a specific pod
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods/default/my-pod-abc123

# Get node metrics
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes

# Get metrics for a specific node
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/node-1

The API returns JSON responses structured as follows:

{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "name": "my-pod-abc123",
      "namespace": "default",
      "containers": [
        {
          "name": "app-container",
          "usage": {
            "cpu": "150m",
            "memory": "256Mi"
          }
        },
        {
          "name": "sidecar",
          "usage": {
            "cpu": "5m",
            "memory": "32Mi"
          }
        }
      ],
      "timestamp": "2025-01-15T10:30:00Z",
      "window": "15s"
    }
  ]
}

Golang Client Example

Here is a complete Go program that fetches pod metrics using the official Kubernetes client library:

package main

import (
    "context"
    "fmt"
    "os"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
    metricsv1beta1 "k8s.io/metrics/pkg/client/clientset/versioned"
)

func main() {
    // Load kubeconfig from default location
    loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
    configOverrides := &clientcmd.ConfigOverrides{}
    kubeConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
        loadingRules,
        configOverrides,
    )

    config, err := kubeConfig.ClientConfig()
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error creating client config: %v\n", err)
        os.Exit(1)
    }

    // Create metrics client
    metricsClient, err := metricsv1beta1.NewForConfig(config)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error creating metrics client: %v\n", err)
        os.Exit(1)
    }

    // Fetch pod metrics across all namespaces
    podMetrics, err := metricsClient.MetricsV1beta1().
        PodMetricses("").  // empty string = all namespaces
        List(context.TODO(), metav1.ListOptions{})
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error fetching pod metrics: %v\n", err)
        os.Exit(1)
    }

    fmt.Println("Pod Metrics:")
    for _, pod := range podMetrics.Items {
        fmt.Printf("Pod: %s/%s\n", pod.Namespace, pod.Name)
        fmt.Printf("  Timestamp: %s\n", pod.Timestamp.Format(time.RFC3339))
        for _, container := range pod.Containers {
            cpu := container.Usage.Cpu().MilliValue()
            mem := container.Usage.Memory().Value()
            fmt.Printf("  Container: %s -> CPU: %dm, Memory: %d bytes\n",
                container.Name, cpu, mem)
        }
    }

    // Fetch node metrics
    nodeMetrics, err := metricsClient.MetricsV1beta1().
        NodeMetricses().
        List(context.TODO(), metav1.ListOptions{})
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error fetching node metrics: %v\n", err)
        os.Exit(1)
    }

    fmt.Println("\nNode Metrics:")
    for _, node := range nodeMetrics.Items {
        cpu := node.Usage.Cpu().MilliValue()
        mem := node.Usage.Memory().Value()
        fmt.Printf("Node: %s -> CPU: %dm, Memory: %d bytes\n",
            node.Name, cpu, mem)
    }
}

Python Client Example

For Python developers, here is an equivalent implementation using the official Kubernetes Python client:

from kubernetes import client, config
from kubernetes.client import api_client
from kubernetes.metrics import MetricsApi

def fetch_pod_metrics():
    # Load authentication from default kubeconfig
    config.load_kube_config()

    # Initialize the metrics API client
    api = MetricsApi()

    # Fetch pod metrics for all namespaces
    pod_metrics = api.get_pod_metrics(
        namespace="",
        label_selector="",
        field_selector=""
    )

    print("Pod Metrics:")
    for pod in pod_metrics.items:
        print(f"Pod: {pod.metadata.namespace}/{pod.metadata.name}")
        print(f"  Timestamp: {pod.timestamp}")
        for container in pod.containers:
            cpu = container.usage.get('cpu', '0')
            mem = container.usage.get('memory', '0')
            print(f"  Container: {container.name} -> "
                  f"CPU: {cpu}, Memory: {mem}")

    # Fetch node metrics
    node_metrics = api.get_node_metrics()
    print("\nNode Metrics:")
    for node in node_metrics.items:
        cpu = node.usage.get('cpu', '0')
        mem = node.usage.get('memory', '0')
        print(f"Node: {node.metadata.name} -> "
              f"CPU: {cpu}, Memory: {mem}")

if __name__ == "__main__":
    fetch_pod_metrics()

Integrating with HPA

The primary consumer of Metrics Server is the Horizontal Pod Autoscaler. Here is a complete example demonstrating the integration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-intensive-app
  labels:
    app: cpu-intensive-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cpu-intensive-app
  template:
    metadata:
      labels:
        app: cpu-intensive-app
    spec:
      containers:
      - name: app
        image: registry.k8s.io/hpa-example
        resources:
          requests:
            cpu: 200m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
        ports:
        - containerPort: 80
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-intensive-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: cpu-intensive-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

Verify the HPA is receiving metrics:

kubectl describe hpa cpu-intensive-app-hpa

# Expected output snippet:
# Metrics:                                               (current / target)
#   resource cpu on pods  (as a percentage of request):  35% / 50%
#   resource memory on pods (as a percentage of request): 45% / 70%

High Availability and Scaling

While a single Metrics Server replica is sufficient for most clusters, large-scale deployments benefit from multiple replicas with leader election to prevent duplicate scraping and ensure availability during pod disruptions.

Multi-Replica Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: metrics-server
  template:
    metadata:
      labels:
        app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: registry.k8s.io/metrics-server/metrics-server:v0.7.1
        args:
          - --cert-dir=/tmp
          - --secure-port=10250
          - --kubelet-use-node-status-port
          - --metric-resolution=30s
          - --leader-elect
          - --leader-elect-lease-duration=30s
          - --leader-elect-renew-deadline=20s
          - --leader-elect-retry-period=5s
        resources:
          requests:
            cpu: 50m
            memory: 100Mi
          limits:
            cpu: 200m
            memory: 300Mi

Leader Election Configuration

The leader election parameters control how quickly a new leader takes over when the current leader fails:

With these defaults, a leader failure will be detected within approximately 30 seconds, and a new leader will take over within an additional 5 seconds. This means metrics may be briefly unavailable during a leader transition.

Best Practices

1. Set Appropriate Metric Resolution

The --metric-resolution flag determines scrape frequency. A shorter interval provides more recent data but increases load on Kubelets and the Metrics Server itself.

2. Always Set Resource Requests and Limits

Metrics Server is a critical infrastructure component. Without resource limits, it could be starved during cluster-wide resource contention, creating a circular dependency where HPA cannot scale because Metrics Server is throttled.

resources:
  requests:
    cpu: 50m
    memory: 100Mi
  limits:
    cpu: 200m
    memory: 300Mi

3. Configure Kubelet Address Types Thoughtfully

In cloud environments, the control plane often cannot reach node InternalIPs due to network topology. Always test connectivity from a Metrics Server-like pod before finalizing configuration.

# Common cloud provider configurations:
# AWS EKS: InternalIP works if VPC networking allows pod-to-node traffic
# GKE: InternalIP typically works within the same cluster
# Azure AKS: May require ExternalIP or specific network plugin configuration
# On-premises: InternalIP or Hostname depending on DNS setup

4. Monitor Metrics Server Itself

Metrics Server exposes its own metrics at /metrics on port 10250. Scrape these with Prometheus to detect issues proactively:

# Sample Prometheus scrape config
scrape_configs:
- job_name: metrics-server
  static_configs:
  - targets:
    - metrics-server.kube-system.svc.cluster.local:10250
  scheme: https
  tls_config:
    insecure_skip_verify: true

Key metrics to monitor:

5. Handle Kubelet TLS Properly

Never use --kubelet-insecure-tls in production. Instead, issue proper client certificates signed by the cluster's CA, or configure the Kubelet's certificate to be verifiable by Metrics Server. The flag exists purely for development convenience and bypasses critical security controls.

6. Run Regular Connectivity Checks

Periodically validate that Metrics Server can reach all Kubelets. A single unreachable node will cause missing metrics for all pods on that node:

# Check Metrics Server logs for scrape errors
kubectl logs -n kube-system deployment/metrics-server | grep -i error

# Common error patterns:
# - "unable to fetch metrics from node X": Network or TLS issue
# - "node X not found": Node object missing or deleted
# - "context deadline exceeded": Kubelet overloaded or unreachable

7. Namespace Isolation for Critical Pods

Keep Metrics Server in kube-system and use RBAC to restrict access to the metrics API if needed. The APIService registration allows fine-grained access control via standard Kubernetes RBAC on the metrics.k8s.io API group.

Troubleshooting Common Issues

Issue: kubectl top returns "metrics not available"

Symptoms: Running kubectl top pod returns:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

Diagnosis:

# Check if the APIService is healthy
kubectl get apiservice v1beta1.metrics.k8s.io

# Expected output when healthy:
# NAME                     SERVICE                      AVAILABLE   AGE
# v1beta1.metrics.k8s.io   kube-system/metrics-server   True        5m

# If AVAILABLE is False, check the APIService details
kubectl describe apiservice v1beta1.metrics.k8s.io

Common causes:

Issue: Missing metrics for specific pods

Symptoms: Some pods appear in kubectl top pod but others are missing entirely.

Diagnosis:

# Look for node-level scrape failures in Metrics Server logs
kubectl logs -n kube-system deployment/metrics-server | grep "unable to fetch"

# Check if the pod's node is reachable from Metrics Server
kubectl get pod -o wide  # note the NODE column
kubectl top node  # check if that node has metrics

Common causes:

Issue: HPA shows "unavailable" for metrics

Symptoms: HPA describe shows:

Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  failed to get cpu utilization: unable to fetch metrics

Resolution steps:

# 1. Verify Metrics Server is running
kubectl get deployment metrics-server -n kube-system

# 2. Verify the HPA target resource has requests set
kubectl get deployment cpu-intensive-app -o yaml | grep -A 5 resources
# HPA requires resource requests to calculate utilization percentages

# 3. Check if metrics are available for the target pods
kubectl top pod -l app=cpu-intensive-app

# 4. If metrics exist but HPA still fails, check API service availability
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods 2>&1

Issue: High Metrics Server CPU Usage

Symptoms: Metrics Server pod consistently using >200m CPU.

Resolution:

# Increase metric resolution to reduce scrape frequency
args:
  - --metric-resolution=60s  # was 15s

# Reduce the number of pods being scraped by using node filters
# (if using a dedicated Metrics Server per node group)
# Consider increasing the deployment replicas to distribute load

Conclusion

Metrics Server is a foundational component of the Kubernetes ecosystem that bridges the gap between raw container metrics and actionable cluster automation. Its deliberate simplicity—in-memory storage, short retention, minimal resource footprint—makes it the right tool for real-time metric aggregation at the cluster level. By understanding its architecture, configuring it appropriately for your environment, and integrating it with autoscalers and diagnostic workflows, you unlock essential Kubernetes capabilities that keep your applications properly sized and responsive to changing load conditions. Whether you are a platform engineer deploying a new cluster or a developer debugging resource contention, a properly functioning Metrics Server is indispensable. Invest the time to install it correctly, monitor its health, and treat it as a critical production dependency—because in modern Kubernetes operations, it is.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles