Introduction to cAdvisor
cAdvisor (Container Advisor) is an open-source tool originally developed by Google to provide resource usage and performance characteristics of running containers. It collects, aggregates, processes, and exports metrics about containers in real time, making it a foundational component in many observability stacks. While it was originally designed for Docker containers, cAdvisor now supports a wide range of container runtimes and orchestration platforms, including Kubernetes, where it is embedded directly into the Kubelet.
For developers and platform engineers, cAdvisor serves as the bridge between low-level kernel cgroup data and high-level monitoring systems. It exposes metrics through a clean HTTP API and a Prometheus-compatible endpoint, allowing teams to build dashboards, alerts, and capacity planning workflows around container behavior.
Why cAdvisor Matters
Containers are ephemeral by nature, and understanding how they consume CPU, memory, network, and storage resources is critical for reliability and cost optimization. cAdvisor matters because it provides this visibility without requiring applications to instrument themselves. It operates at the host level, observing all containers from the outside, which means you get consistent metrics regardless of the language or framework your application uses.
- Runtime agnostic: Works with Docker, containerd, CRI-O, and raw cgroups.
- Low overhead: Written in Go, it runs as a lightweight daemon with minimal CPU and memory footprint.
- Prometheus native: Exposes metrics in a format that Prometheus can scrape directly.
- Historical context: Provides per-container time series data, not just instantaneous snapshots.
- Kubernetes integration: Embedded in the Kubelet, so metrics are available out of the box in any Kubernetes cluster.
How cAdvisor Works
At its core, cAdvisor reads data from the Linux kernel's cgroup filesystem and the proc filesystem. Cgroups track resource accounting for process groups, which is exactly how containers are isolated on Linux. cAdvisor parses these files, enriches them with metadata from the container runtime, and exposes the results through an HTTP server.
The architecture is straightforward. A single cAdvisor process runs on each host and monitors all containers on that host. It maintains an in-memory time series database for recent history and exposes both a web UI and machine-readable endpoints. When integrated with Prometheus, the time series storage is handled externally, allowing for long-term retention and querying.
Installing cAdvisor as a Standalone Daemon
While cAdvisor is embedded in Kubernetes, running it as a standalone daemon is useful for bare-metal hosts, Docker Swarm clusters, or development environments. The simplest way to run it is via the official Docker image.
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
--privileged \
--device=/dev/kmsg \
gcr.io/cadvisor/cadvisor:v0.49.1
Each volume mount serves a specific purpose. The root filesystem is mounted read-only so cAdvisor can inspect disk usage. The Docker socket and library path allow it to query the Docker daemon for container metadata. The sysfs mount provides cgroup data, and the disk mount enables filesystem metrics. The /dev/kmsg device gives access to kernel logs, which cAdvisor uses for container event tracking.
Once running, you can access the web UI at http://localhost:8080. This interface shows per-container CPU, memory, network, and filesystem usage with real-time charts. For programmatic access, the metrics endpoint is available at http://localhost:8080/metrics.
Running cAdvisor with Docker Compose
For development environments, a Docker Compose file makes it easier to manage cAdvisor alongside other monitoring components. Here is a complete setup that includes cAdvisor, Prometheus, and Grafana.
version: "3.8"
services:
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.49.1
container_name: cadvisor
ports:
- "8080:8080"
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
devices:
- /dev/kmsg
privileged: true
restart: unless-stopped
prometheus:
image: prom/prometheus:v2.51.0
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
restart: unless-stopped
grafana:
image: grafana/grafana:10.4.0
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
restart: unless-stopped
The corresponding Prometheus configuration file should include a scrape job for cAdvisor. This tells Prometheus to poll the metrics endpoint at a regular interval.
global:
scrape_interval: 15s
scrape_configs:
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
With this setup, you can launch the entire stack with a single command and immediately start collecting container metrics.
docker compose up -d
Deploying cAdvisor on Kubernetes
In Kubernetes, cAdvisor is already embedded in the Kubelet, and its metrics are exposed at the Kubelet's /metrics/cadvisor endpoint. However, some teams prefer to run a standalone cAdvisor DaemonSet for more control over configuration, or because they want metrics in a specific format or with custom labels.
Here is a complete DaemonSet manifest that deploys cAdvisor to every node in a Kubernetes cluster.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cadvisor
namespace: monitoring
labels:
app: cadvisor
spec:
selector:
matchLabels:
app: cadvisor
template:
metadata:
labels:
app: cadvisor
spec:
hostNetwork: true
containers:
- name: cadvisor
image: gcr.io/cadvisor/cadvisor:v0.49.1
ports:
- containerPort: 8080
name: http
volumeMounts:
- name: rootfs
mountPath: /rootfs
readOnly: true
- name: var-run
mountPath: /var/run
readOnly: true
- name: sys
mountPath: /sys
readOnly: true
- name: docker
mountPath: /var/lib/docker
readOnly: true
- name: disk
mountPath: /dev/disk
readOnly: true
volumes:
- name: rootfs
hostPath:
path: /
- name: var-run
hostPath:
path: /var/run
- name: sys
hostPath:
path: /sys
- name: docker
hostPath:
path: /var/lib/docker
- name: disk
hostPath:
path: /dev/disk
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
The hostNetwork: true setting allows cAdvisor to access the host's network namespace directly, which is necessary for accurate network metrics. The toleration ensures cAdvisor also runs on control-plane nodes, giving you full cluster visibility.
Understanding Key Metrics
cAdvisor exposes a large number of metrics, all prefixed with container_. Understanding the most important ones is essential for building effective dashboards and alerts.
CPU Metrics
The primary CPU metric is container_cpu_usage_seconds_total, a counter that represents the total CPU time consumed by a container in seconds. Because it is a counter, you typically use the rate() function in PromQL to get CPU usage as a percentage of a core.
# CPU usage as a fraction of a single core
rate(container_cpu_usage_seconds_total{name!="",}[5m])
# CPU usage as a percentage of allocated limit
rate(container_cpu_usage_seconds_total[5m]) / on(name) container_spec_cpu_quota * 100
Memory Metrics
Memory metrics are gauges, meaning they represent current values rather than cumulative counts. The most important ones are container_memory_usage_bytes for total memory usage and container_memory_working_set_bytes for the working set, which is the memory that cannot be evicted without causing the container to be OOM-killed.
# Memory usage per container
container_memory_usage_bytes{name!=""}
# Memory usage as percentage of limit
container_memory_working_set_bytes / on(name) container_spec_memory_limit_bytes * 100
Network Metrics
Network metrics are counters that track bytes received and transmitted per network interface. The relevant metrics are container_network_receive_bytes_total and container_network_transmit_bytes_total.
# Network receive rate in bytes per second
rate(container_network_receive_bytes_total[5m])
Filesystem Metrics
Filesystem metrics track disk usage within the container's writable layer and mounted volumes. container_fs_usage_bytes shows current usage, while container_fs_reads_bytes_total and container_fs_writes_bytes_total track I/O.
Querying Metrics with PromQL
Once Prometheus is scraping cAdvisor, you can write PromQL queries to analyze container behavior. Here are several practical examples that cover common use cases.
# Top 5 containers by CPU usage
topk(5, rate(container_cpu_usage_seconds_total{name!=""}[5m]))
# Total memory used across all containers
sum(container_memory_working_set_bytes{name!=""})
# Containers approaching their memory limit (above 80%)
(container_memory_working_set_bytes / on(name) container_spec_memory_limit_bytes) * 100 > 80
# Network throughput per container in megabytes per second
rate(container_network_receive_bytes_total[5m]) / 1024 / 1024
These queries can be used directly in Grafana panels or saved as Prometheus alert rules. For alerting, you would typically define thresholds and notification channels in your Alertmanager configuration.
Setting Up Alerts
Alerts help you catch problems before they impact users. Here is a Prometheus alerting rules file that covers common container failure modes.
groups:
- name: container_alerts
rules:
- alert: ContainerHighCpuUsage
expr: rate(container_cpu_usage_seconds_total[5m]) * 100 > 80
for: 10m
labels:
severity: warning
annotations:
summary: "Container {{ $labels.name }} has high CPU usage"
description: "Container {{ $labels.name }} is using more than 80% CPU for 10 minutes."
- alert: ContainerHighMemoryUsage
expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) * 100 > 85
for: 5m
labels:
severity: critical
annotations:
summary: "Container {{ $labels.name }} is near memory limit"
description: "Container {{ $labels.name }} is using more than 85% of its memory limit."
- alert: ContainerOomKilled
expr: increase(container_memory_failcnt[5m]) > 0
labels:
severity: critical
annotations:
summary: "Container {{ $labels.name }} was OOM killed"
description: "Container {{ $labels.name }} has been OOM killed in the last 5 minutes."
Building Grafana Dashboards
Grafana provides the visualization layer on top of Prometheus and cAdvisor. A well-structured dashboard typically includes panels for CPU, memory, network, and filesystem metrics, filtered by container name or namespace. You can import the official cAdvisor dashboard from the Grafana dashboard marketplace using ID 14282, or build your own.
Here is an example of a Grafana panel JSON configuration for a CPU usage panel that shows per-container CPU consumption as a stacked graph.
{
"title": "CPU Usage by Container",
"type": "timeseries",
"datasource": "Prometheus",
"targets": [
{
"expr": "rate(container_cpu_usage_seconds_total{name!=\"\"}[5m])",
"legendFormat": "{{name}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "cores"
}
},
"options": {
"stacking": {
"mode": "normal"
}
}
}
Best Practices
Running cAdvisor effectively requires attention to a few operational details that can significantly impact the quality and reliability of your metrics.
- Use resource limits on cAdvisor itself: Even though cAdvisor is lightweight, you should set CPU and memory limits to prevent it from competing with your workloads during spikes.
- Filter out the root cgroup: Many queries include
name!=""to exclude the root container, which aggregates all containers on the host and can produce misleading results. - Adjust scrape intervals thoughtfully: A 15-second scrape interval is a good default. Shorter intervals increase storage costs, while longer intervals reduce alert responsiveness.
- Secure the metrics endpoint: In production, put cAdvisor behind a reverse proxy with authentication, or restrict access using network policies in Kubernetes.
- Monitor cAdvisor itself: Use a health check or uptime monitor to ensure cAdvisor is running on every host. A missing cAdvisor means blind spots in your observability.
- Use labels consistently: In Kubernetes, ensure your pods have consistent labels for app, team, and environment so you can aggregate metrics meaningfully.
- Be aware of metric cardinality: cAdvisor generates a high number of time series, especially on busy hosts. Monitor your Prometheus storage usage and use relabeling to drop unnecessary metrics if needed.
Advanced Configuration
cAdvisor supports a range of command-line flags that allow you to customize its behavior. Here are some of the most useful ones.
# Run cAdvisor with custom port and disabled web UI
cadvisor \
--port=9090 \
--disable_metrics=disk,tcp,udp \
--housekeeping_interval=30s \
--max_housekeeping_interval=60s \
--allow_dynamic_housekeeping=true
The --disable_metrics flag is particularly valuable for reducing metric cardinality. If you do not need disk or network protocol metrics, disabling them reduces the number of time series Prometheus has to store. The housekeeping interval controls how often cAdvisor polls the kernel for updates; increasing it reduces CPU usage at the cost of metric freshness.
You can also use relabeling in Prometheus to drop metrics before they are stored. This example drops all filesystem metrics, which are often the highest cardinality.
scrape_configs:
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
metric_relabel_configs:
- source_labels: [__name__]
regex: "container_fs_.*"
action: drop
Troubleshooting Common Issues
When cAdvisor is not reporting metrics correctly, the problem usually falls into one of a few categories. Understanding these will save you significant debugging time.
Missing container metadata: If cAdvisor shows containers but without names or image information, it likely cannot access the container runtime socket. Verify that the Docker socket or containerd socket is mounted correctly and that cAdvisor has permission to read it.
Zero or negative CPU values: This can happen when the cgroup hierarchy is not properly mounted or when cAdvisor is running inside a container without access to the host's cgroup filesystem. Ensure the /sys mount is read-only but accessible.
High memory usage by cAdvisor: On hosts with many containers, cAdvisor's in-memory storage can grow large. Reduce the storage duration with the --storage_duration flag, or rely entirely on Prometheus for historical data and set the duration to a short value.
Prometheus not scraping: Check that the cAdvisor container is reachable from the Prometheus container. In Docker Compose, both services must be on the same network. In Kubernetes, verify that the Service or pod IP is correct in the scrape configuration.
Conclusion
cAdvisor remains one of the most reliable and widely used tools for container resource monitoring. Its tight integration with Kubernetes, native Prometheus support, and low operational overhead make it an excellent choice for teams at any scale. By understanding how cAdvisor collects data, which metrics matter most, and how to configure it for your specific environment, you can build a robust observability foundation that gives you deep insight into how your containers consume resources. Combined with Prometheus for storage and Grafana for visualization, cAdvisor forms the backbone of a complete container monitoring stack that will serve you well from development through production.