Kubernetes Server Performance: Profiling and Optimization

Understanding Kubernetes Server Performance Profiling

Kubernetes server performance profiling refers to the systematic process of collecting, analyzing, and interpreting runtime data from Kubernetes control plane and node components to identify bottlenecks, resource contention, latency spikes, and inefficient code paths. This encompasses profiling the kube-apiserver, kube-scheduler, controller-manager, kubelet, kube-proxy, and the underlying container runtime as well as the etcd datastore.

Profiling in Kubernetes typically involves two complementary approaches: application-level profiling using Go's built-in pprof tooling for components written in Go, and system-level profiling using Linux performance analysis tools like perf, eBPF, and ftrace to examine kernel-space and hardware-level behavior. Together, these provide a comprehensive view of what your cluster is doing at any given moment and where cycles, memory, or I/O are being consumed.

Why Performance Profiling Matters

Kubernetes clusters running at scale inevitably encounter performance degradation that manifests as slow API responses, delayed pod scheduling, sluggish container starts, or unexplained node instability. Profiling matters because:

• Cluster Stability: A slow API server can cause cascading failures as controllers time out, leases expire, and nodes become unreachable. Profiling helps catch these issues before they cause outages.
• Cost Efficiency: Over-provisioning compute resources to compensate for poor performance wastes cloud spend. Profiling reveals where you can right-size or optimize rather than simply adding more nodes.
• Developer Productivity: Slow scheduling or container startup times directly impact CI/CD pipeline durations and developer iteration speed.
• SLA Compliance: Many organizations have internal SLOs for API response times and pod readiness. Profiling provides the data needed to meet these targets consistently.
• Capacity Planning: Understanding the performance envelope of each component allows accurate predictions about when clusters will need to scale or be replaced.

Enabling and Accessing pprof Endpoints

Most Kubernetes core components are written in Go and ship with net/http/pprof handlers. These are often disabled by default in production builds for security reasons but can be enabled via command-line flags. For profiling, you need to expose these endpoints and then use the go tool pprof CLI to collect profiles.

Enabling pprof on the API Server

To enable profiling on kube-apiserver, add the following flag to the API server manifest (typically located at /etc/kubernetes/manifests/kube-apiserver.yaml on control plane nodes managed by kubeadm):

spec:
  containers:
  - command:
    - kube-apiserver
    - --profiling=true
    - --profiling-port=6060
    - --profiling-address=0.0.0.0
    # ... other flags

If you are running a self-managed cluster, you can also set this via systemd service arguments. After restarting the API server, verify the endpoint is accessible:

# Test from a control plane node
curl -s http://localhost:6060/debug/pprof/ | head -n 20

# Expected output includes links like:
# /debug/pprof/profile
# /debug/pprof/heap
# /debug/pprof/goroutine
# /debug/pprof/threadcreate
# /debug/pprof/block
# /debug/pprof/mutex

Enabling pprof on the Kubelet

The kubelet also supports pprof profiling. Edit the kubelet configuration or pass the flag:

# In kubelet systemd drop-in or configuration file
KUBELET_EXTRA_ARGS="--profiling=true --profiling-address=0.0.0.0 --profiling-port=6061"

Restart the kubelet and confirm the endpoint:

curl -s http://localhost:6061/debug/pprof/

Collecting and Analyzing Profiles

CPU Profiling

A CPU profile samples the call stack at a configurable rate (default 100 Hz) and shows where the process spends its time. This is the most useful profile for identifying hot functions. Collect a 30-second CPU profile from the API server:

# Collect CPU profile over 30 seconds
curl -s http://localhost:6060/debug/pprof/profile?seconds=30 -o api-server-cpu.pprof

# Open the interactive visualization
go tool pprof api-server-cpu.pprof

# Inside pprof interactive mode, useful commands:
# (pprof) top20          # show top 20 functions by CPU usage
# (pprof) list serveRequest  # show source code annotated with CPU time
# (pprof) web            # open call graph in browser (requires graphviz)

You can also collect profiles directly via the go tool pprof command without manually curling:

# Stream CPU profile for 30 seconds from remote endpoint
go tool pprof -seconds 30 http://localhost:6060/debug/pprof/profile

# This opens the interactive pprof shell automatically

Heap (Memory) Profiling

Heap profiles reveal memory allocation hotspots and can help detect memory leaks. Collect a heap profile:

# Grab current heap snapshot
curl -s http://localhost:6060/debug/pprof/heap -o api-server-heap.pprof

# Analyze in pprof
go tool pprof api-server-heap.pprof

# Common commands:
# (pprof) top20 -cum      # top allocations including downstream calls
# (pprof) list NewManager # allocation detail for a specific function
# (pprof) web             # visual call graph

To compare two heap profiles over time (for leak detection), collect snapshots several minutes apart and use the -base flag:

# Collect first snapshot
curl -s http://localhost:6060/debug/pprof/heap -o heap-1.pprof

# Wait 10 minutes, collect second snapshot
curl -s http://localhost:6060/debug/pprof/heap -o heap-2.pprof

# Compare to see what grew between snapshots
go tool pprof -base heap-1.pprof heap-2.pprof
# (pprof) top20  # shows allocations that increased

Goroutine Profiling

A goroutine profile shows all goroutines with their stack traces. This is invaluable for finding goroutine leaks, deadlocks, or runaway concurrency:

# Collect goroutine profile
curl -s http://localhost:6060/debug/pprof/goroutine?debug=2 -o goroutines.txt

# The debug=2 format provides full stack traces in text
# Search for common patterns like stuck HTTP handlers
grep -c "chan receive" goroutines.txt
grep -c "IO wait" goroutines.txt
grep -c "select" goroutines.txt

# For pprof format analysis
curl -s http://localhost:6060/debug/pprof/goroutine -o goroutine.pprof
go tool pprof goroutine.pprof

Block and Mutex Profiling

Block profiling shows where goroutines spend time blocked on synchronization primitives (channels, mutexes). Mutex profiling shows contention on specific mutexes. These require explicit enabling via API server flags:

# Add these to kube-apiserver flags
--contention-profiling=true
--profiling=true

Then collect:

# Block profile
curl -s http://localhost:6060/debug/pprof/block -o block.pprof
go tool pprof block.pprof

# Mutex profile
curl -s http://localhost:6060/debug/pprof/mutex -o mutex.pprof
go tool pprof mutex.pprof

System-Level Profiling with eBPF and perf

Application-level profiling only tells half the story. Kubernetes nodes also suffer from kernel-level bottlenecks, I/O contention, and hardware resource saturation. System-level profiling tools fill this gap.

Using perf on Kubernetes Nodes

The perf tool profiles the entire system, including kernel and userspace. It's particularly useful for understanding why a node's CPU is saturated:

# Record system-wide CPU profile for 30 seconds
sudo perf record -a -g -o perf.data -- sleep 30

# Generate a report
sudo perf report -i perf.data

# For flame graph visualization (requires FlameGraph scripts)
sudo perf script -i perf.data | stackcollapse-perf.pl | flamegraph.pl > flamegraph.svg

To profile a specific process like the kubelet or container runtime:

# Find PID of kubelet
KUBELET_PID=$(pgrep -f kubelet | head -1)

# Profile that PID with call graphs
sudo perf record -p $KUBELET_PID -g -o kubelet-perf.data -- sleep 30
sudo perf report -i kubelet-perf.data

eBPF-Based Profiling with BCC Tools

eBPF provides low-overhead, programmable profiling. The BCC toolkit includes ready-made scripts ideal for Kubernetes node analysis:

# Install BCC tools on the node
sudo apt-get install bpfcc-tools linux-headers-$(uname -r)  # Ubuntu/Debian
# or for Red Hat-based systems
sudo dnf install bcc-tools kernel-devel

# Profile file I/O latency distribution (find slow disk)
sudo /usr/share/bcc/tools/fileslower 1

# Profile block I/O by process
sudo /usr/share/bcc/tools/biolatency -m

# Trace container I/O usage by cgroup
sudo /usr/share/bcc/tools/runqlat    # scheduler latency
sudo /usr/share/bcc/tools/cpudist    # CPU usage distribution

For profiling network performance of kube-proxy or CNI components:

# Trace TCP connections and their durations
sudo /usr/share/bcc/tools/tcptracer

# Profile network softirq latency
sudo /usr/share/bcc/tools/netqslat

Profiling etcd Performance

etcd is the backbone of Kubernetes state storage. Profiling etcd directly is critical because API server slowness often traces back to etcd latency. etcd exposes Prometheus metrics and a dedicated pprof endpoint.

# Enable pprof on etcd (in etcd manifest or systemd config)
--enable-pprof=true
--profiling-port=2379

# Collect etcd CPU profile
curl -s http://localhost:2379/debug/pprof/profile?seconds=30 -o etcd-cpu.pprof
go tool pprof etcd-cpu.pprof

For deeper etcd performance analysis, use the built-in etcdctl metrics and the Prometheus endpoint:

# Check etcd metrics endpoint
curl -s http://localhost:2379/metrics | grep -E "etcd_server_(proposals|apply|read)_" | head

# Key metrics to watch:
# etcd_server_proposals_committed_total   - write throughput
# etcd_disk_wal_fsync_duration_seconds    - disk fsync latency
# etcd_disk_backend_commit_duration_seconds - backend commit latency
# etcd_network_client_grpc_sent_bytes_total - network pressure

Optimization Strategies Based on Profiling Results

Once profiles are collected, the data guides targeted optimizations. Here are common findings and their remediation strategies.

API Server Optimization

If CPU profiles reveal heavy time spent in encoding/json or runtime.mallocgc, consider:

• Reducing the number of watches by using informers with resync periods tuned appropriately
• Enabling --watch-cache-sizes to buffer watch responses and reduce etcd round-trips
• Increasing --max-requests-inflight and --max-mutating-requests-inflight to allow higher concurrency (if etcd can keep up)
• Using --request-timeout to prevent long-running requests from consuming handler goroutines indefinitely

If heap profiles show memory growth in cacher or storage packages:

# Adjust watch cache sizes per resource
--watch-cache-sizes=pod#1000,node#500,deployment#500
# Lower values reduce memory but increase etcd load — balance is key

etcd Optimization

When etcd profiles show high fsync latency (etcd_disk_wal_fsync_duration_seconds in high percentiles), the disk is the bottleneck. Solutions include:

• Using faster SSD/NVMe storage with high IOPS for etcd data directory
• Separating etcd WAL directory from data directory on different disks using --wal-dir
• Increasing --backend-batch-interval and --backend-batch-limit to batch writes more aggressively
• Adjusting --snapshot-count to control compaction frequency

# Example etcd tuning flags
--wal-dir=/var/lib/etcd/wal
--backend-batch-interval=50ms
--backend-batch-limit=2048
--snapshot-count=50000
--quota-backend-bytes=8589934592  # 8GiB

Scheduler Optimization

Scheduler profiling often reveals time spent in node filtering and scoring algorithms. Optimizations include:

• Reducing the number of scheduler plugins enabled if certain predicates are not needed
• Increasing --kube-api-qps and --kube-api-burst to allow faster binding
• Using --percentage-of-nodes-to-find-viable to limit the search space in large clusters

# Example scheduler flags for large clusters
--kube-api-qps=100
--kube-api-burst=200
--percentage-of-nodes-to-find-viable=50

Kubelet and Container Runtime Optimization

If node-level profiling shows high CPU in runc or containerd, or high I/O wait from container operations:

• Tune container runtime configuration (e.g., containerd's containerd.service for concurrent operations)
• Adjust kubelet sync frequencies: --node-status-update-frequency, --sync-frequency
• Increase --max-pods carefully, balanced against node resources
• Use image pulling concurrency limits via --serialize-image-pulls and --max-parallel-image-pulls

# Example kubelet performance flags
--sync-frequency=1m
--node-status-update-frequency=10s
--max-parallel-image-pulls=5
--serialize-image-pulls=false

Continuous Profiling in Production

Rather than ad-hoc profiling during incidents, consider implementing continuous profiling using tools like Parca, Pyroscope, or Google's cloud-profiler. These integrate with Kubernetes and collect profiles on a schedule, storing them for comparative analysis over time.

# Example: deploying Parca agent as DaemonSet for continuous profiling
# The Parca agent collects pprof profiles from all nodes automatically
kubectl apply -f https://github.com/parca-dev/parca-agent/releases/latest/download/deploy.yaml

# Parca server stores and serves profiles with a web UI for analysis
# It supports comparing profiles across time ranges to detect regressions

Continuous profiling allows you to:

• Establish performance baselines for every component
• Detect gradual performance regressions before they trigger alerts
• Compare profiles before and after configuration changes or upgrades
• Attribute performance costs to specific code changes in your CI/CD pipeline

Best Practices for Kubernetes Performance Profiling

• Profile in Representative Environments: Development clusters often lack the load and scale of production. Whenever possible, profile in staging environments with production-like traffic patterns generated via load testing tools such as k6 or vegeta.
• Enable Profiling Selectively: Keep pprof endpoints disabled by default and enable them temporarily via flags when needed. If continuous profiling is required, restrict access using network policies or binding to localhost only, then use port-forwarding for access.
• Correlate Application and System Profiles: A CPU spike in the API server's pprof output that coincides with high disk latency in etcd's perf data tells a complete story. Always cross-reference multiple data sources.
• Profile During Peak Load: Collect CPU and heap profiles during peak traffic hours or during load tests. Idle profiles are largely uninteresting and miss contention patterns that only appear under concurrency.
• Watch Profile Overhead: CPU profiling typically imposes 1–3% overhead. Heap profiling can trigger stop-the-world pauses during collection. In production, keep sampling intervals short (15–30 seconds) and space collections apart.
• Use Flame Graphs for Communication: Flame graphs generated from pprof or perf data are more accessible to teams than raw profiling output. Use pprof -http=:8080 to launch a web UI with flame graph visualization built in.
• Automate Profile Collection During Incidents: Integrate profile collection into your incident response runbooks. A 30-second CPU profile collected during a latency spike often reveals the root cause faster than log analysis alone.
• Version-Tag Profiles: Kubernetes performance characteristics change between versions. Tag all profiles with the Kubernetes version, component versions, and configuration hashes to make historical comparisons meaningful.

Conclusion

Kubernetes server performance profiling is not a one-time debugging exercise but an essential operational discipline for clusters running at any meaningful scale. By combining Go pprof for application-level insight with system tools like perf and eBPF for kernel and hardware visibility, operators gain a complete understanding of where latency, CPU, and memory are being consumed. Profiling data directly informs optimization decisions—from etcd disk configuration and API server watch cache sizing to scheduler algorithm tuning and kubelet sync frequency adjustments. The practice of continuous profiling elevates performance from a reactive firefighting activity to a proactive engineering capability, enabling teams to detect regressions early, right-size infrastructure confidently, and deliver a consistently responsive Kubernetes experience to developers and end users alike.