Troubleshooting Cilium Service Mesh: Common Issues and Fixes

Introduction to Cilium Service Mesh Troubleshooting

Cilium Service Mesh is a modern, eBPF-based service mesh implementation that leverages the Linux kernel's extended Berkeley Packet Filter to provide observability, security, and networking for Kubernetes workloads. Unlike traditional sidecar-based service meshes, Cilium operates at the kernel level, eliminating the need for per-pod sidecar proxies and significantly reducing resource overhead.

However, like any distributed system component, Cilium Service Mesh can encounter issues that affect connectivity, observability, and security. This tutorial walks you through the most common problems developers and platform engineers face when operating Cilium Service Mesh, along with practical diagnostic techniques and fixes.

Why Troubleshooting Cilium Matters

Because Cilium operates deep in the networking stack and relies on eBPF programs attached to various kernel hooks, debugging requires a different mindset than traditional sidecar-based meshes. Misconfigurations can manifest as silent packet drops, partial connectivity failures, or missing observability data. Understanding how to systematically diagnose these issues is critical for maintaining a healthy service mesh.

Prerequisites and Diagnostic Tooling

Before diving into specific issues, ensure you have the right tooling installed. Cilium ships with a powerful CLI that provides deep visibility into the data plane.

# Install the Cilium CLI
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}

# Verify installation
cilium version

Additionally, you should have access to kubectl and basic familiarity with Kubernetes networking concepts. The following diagnostic commands form the backbone of Cilium troubleshooting:

• cilium status — overall cluster health
• cilium service list — view load-balanced services
• cilium endpoint list — inspect per-pod endpoints
• cilium policy get — review applied network policies
• cilium monitor — real-time packet and event tracing
• cilium bgp peers — BGP peering status (if enabled)

Issue 1: Cilium Pods Not Reaching Ready State

One of the most common issues after installation is Cilium agent pods stuck in a non-ready state. This typically indicates a fundamental problem with the eBPF data plane or kernel configuration.

Diagnosing the Problem

# Check Cilium pod status
kubectl get pods -n kube-system -l k8s-app=cilium

# Describe a failing pod for events
kubectl describe pod -n kube-system <cilium-pod-name>

# Check Cilium status from within the agent
cilium status --wait=false

Common root causes include insufficient kernel versions, missing eBPF support, or conflicting kube-proxy configurations.

Fix: Kernel Requirements

Cilium requires a minimum kernel version of 5.4 for full functionality, with 5.10 or later recommended. Verify your kernel version:

# On each node
uname -r

# Check eBPF support
ls /sys/fs/bpf/

# Verify required kernel config options
zcat /proc/config.gz | grep -E 'CONFIG_BPF|CONFIG_XDP|CONFIG_NET_CLS_BPF'

If the kernel is too old, you must upgrade your nodes. For managed Kubernetes offerings, this means selecting a newer node image or instance type.

Fix: kube-proxy Conflict

If you installed Cilium in kube-proxy replacement mode but kube-proxy is still running, you will see conflicts. Remove kube-proxy and clean up its iptables rules:

# Delete kube-proxy DaemonSet
kubectl delete daemonset kube-proxy -n kube-system

# Clean up iptables rules on each node (run via a privileged pod or SSH)
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X

# Ensure Cilium is configured for kube-proxy replacement
helm upgrade cilium cilium/cilium -n kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_IP> \
  --set k8sServicePort=6443

Issue 2: Pod-to-Pod Connectivity Failures

When pods cannot communicate with each other across nodes, the issue often lies in the underlying tunnel or routing configuration, or in network policies blocking traffic.

Diagnosing with Cilium Monitor

The cilium monitor command is your best friend for tracing packet flow through the eBPF data plane:

# Open a shell in a Cilium pod
kubectl exec -it -n kube-system <cilium-pod> -- cilium monitor --type drop

# Trace traffic from a specific endpoint
kubectl exec -it -n kube-system <cilium-pod> -- cilium monitor --from <endpoint-id>

# Verbose tracing with verdicts
kubectl exec -it -n kube-system <cilium-pod> -- cilium monitor -v

Look for Policy denied or Drop verdicts, which indicate packets being intentionally dropped by Cilium's policy enforcement.

Fix: Network Policy Misconfiguration

Cilium enforces both Kubernetes NetworkPolicies and its own CRD-based CiliumNetworkPolicies. A common mistake is creating a default-deny policy without corresponding allow rules:

# Check applied policies
cilium policy get

# View CiliumNetworkPolicies in a namespace
kubectl get cnp -n <namespace>

# View Kubernetes NetworkPolicies
kubectl get netpol -n <namespace>

If you have a default-deny policy, ensure you have explicit allow rules. Here is an example of a properly structured CiliumNetworkPolicy that allows traffic from a specific namespace:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: backend
spec:
  endpointSelector: {}
  ingress:
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP

Apply and verify the policy is correctly enforced:

kubectl apply -f allow-frontend-to-backend.yaml

# Verify the policy is loaded
cilium policy get | grep allow-frontend-to-backend

Fix: Tunnel vs Native Routing Issues

If pods on different nodes cannot communicate, verify the routing mode. Cilium supports both tunnel mode (VXLAN or Geneve) and native routing. A mismatch between node configurations can cause silent failures:

# Check the routing mode
kubectl exec -it -n kube-system <cilium-pod> -- cilium config | grep -E 'tunnel|routing'

# Verify tunnel interface exists (if tunnel mode)
ip link show cilium_vxlan

# Check routes on a node
ip route | grep cilium

If using native routing, ensure the underlying network allows the pod CIDR ranges. If using tunnel mode, verify UDP port 8472 (VXLAN) or 6081 (Geneve) is not blocked by security groups or firewalls.

Issue 3: Service Load Balancing Not Working

Cilium replaces kube-proxy for service load balancing using eBPF. When service traffic is not being distributed correctly, the issue usually involves the eBPF service map or node-level configuration.

Diagnosing Service Load Balancing

# List all services known to Cilium
cilium service list

# Get details for a specific service
cilium service get <service-id>

# Check if a specific pod is a backend for a service
cilium endpoint get <endpoint-id> | grep -A5 "Service"

If a service is missing from the list, Cilium may not be syncing with the Kubernetes API server properly. Check the agent logs:

kubectl logs -n kube-system <cilium-pod> | grep -i "service\|sync"

Fix: Enable Socket-Level Load Balancing

For optimal performance and correctness, ensure socket-level load balancing is enabled. This intercepts connect() syscalls and redirects directly to a backend pod, bypassing intermediate hops:

# Check current configuration
kubectl exec -it -n kube-system <cilium-pod> -- cilium config | grep sock

# Enable via Helm values
helm upgrade cilium cilium/cilium -n kube-system \
  --set socketLB.enabled=true \
  --set nodePort.enableHealthCheck=true

Fix: Stale Service Entries

Sometimes, after deleting and recreating services, stale entries remain in the eBPF maps. Force a resync:

# Restart Cilium agents to force full resync
kubectl rollout restart daemonset/cilium -n kube-system

# Wait for rollout to complete
kubectl rollout status daemonset/cilium -n kube-system

Issue 4: Missing or Incomplete Observability Data

Cilium provides rich observability through Hubble, its observability layer. If you are not seeing flow data or metrics, several configuration issues could be at play.

Diagnosing Hubble Issues

# Check Hubble status
cilium hubble status

# Try to retrieve flows
cilium hubble port-forward &
cilium hubble observe --verdict DROPPED

# Check Hubble Relay logs
kubectl logs -n kube-system -l k8s-app=hubble-relay

Fix: Enable Hubble Metrics

Hubble metrics must be explicitly enabled. Configure them via Helm:

helm upgrade cilium cilium/cilium -n kube-system \
  --set hubble.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

After enabling, verify the metrics endpoint is serving data:

# Port-forward to a Cilium pod
kubectl port-forward -n kube-system <cilium-pod> 9965:9965

# In another terminal
curl http://localhost:9965/metrics | grep hubble

Fix: Hubble Relay Connectivity

If Hubble UI or CLI cannot connect to Relay, check the TLS certificates and peer service:

# Verify Hubble Relay is running
kubectl get pods -n kube-system -l k8s-app=hubble-relay

# Check the peer service
kubectl get svc -n kube-system hubble-relay

# Verify certificates are present
kubectl get secret -n kube-system hubble-relay-client-certs
kubectl get secret -n kube-system hubble-server-certs

If certificates are missing or expired, regenerate them:

cilium connectivity test --hubble

# Or manually regenerate
cilium hubble enable --ui

Issue 5: mTLS and L7 Policy Not Enforced

Cilium Service Mesh supports mutual TLS and L7 (HTTP/gRPC/Kafka) policy enforcement without sidecars. If these features are not working, it is usually because they require the Envoy-based proxy or specific feature flags.

Diagnosing L7 Policy Enforcement

# Check if Envoy is running
kubectl exec -it -n kube-system <cilium-pod> -- cilium status | grep -i envoy

# Check L7 policy configuration
kubectl exec -it -n kube-system <cilium-pod> -- cilium config | grep -i envoy

Fix: Enable L7 Proxy and mTLS

L7 policies and mTLS require the embedded Envoy proxy. Enable it via Helm:

helm upgrade cilium cilium/cilium -n kube-system \
  --set envoy.enabled=true \
  --set ingressController.enabled=true \
  --set gatewayAPI.enabled=true \
  --set authentication.mutual.spire.enabled=true \
  --set authentication.mutual.spire.install.enabled=true

After enabling, create an L7-aware CiliumNetworkPolicy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: http-l7-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: web-frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /api/v1/health$
        - method: POST
          path: /api/v1/users$

Verify the policy is enforced at L7 by checking Hubble flows for HTTP-level verdicts:

cilium hubble observe --type l7 --verdict DROPPED

Issue 6: DNS Resolution Failures

Pods may fail to resolve service names, which can be caused by Cilium's DNS proxy, CoreDNS configuration, or network policy blocking DNS traffic.

Diagnosing DNS Issues

# Test DNS from within a pod
kubectl exec -it <pod-name> -- nslookup kubernetes.default

# Check if DNS proxy is enabled
kubectl exec -it -n kube-system <cilium-pod> -- cilium config | grep -i dns

# Observe DNS flows in Hubble
cilium hubble observe --type dns

Fix: Allow DNS Traffic in Policies

If you have a default-deny policy, you must explicitly allow DNS traffic to CoreDNS:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  endpointSelector: {}
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s:k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
      - port: "53"
        protocol: TCP
      rules:
        dns:
        - matchPattern: "*"

Fix: Enable DNS Proxy for Visibility

For enhanced DNS visibility and policy enforcement, enable the DNS proxy:

helm upgrade cilium cilium/cilium -n kube-system \
  --set dnsProxy.enableTransparentMode=true

Issue 7: Performance Degradation

While Cilium is generally high-performance, certain configurations can lead to increased latency or CPU usage.

Diagnosing Performance Issues

# Check Cilium resource usage
kubectl top pods -n kube-system -l k8s-app=cilium

# Check for BPF program complexity
kubectl exec -it -n kube-system <cilium-pod> -- cilium bpf metrics list

# Monitor for drops
kubectl exec -it -n kube-system <cilium-pod> -- cilium monitor --type drop

Fix: Optimize BPF Map Sizes

If you have a large cluster, default BPF map sizes may be insufficient, causing evictions and performance issues:

helm upgrade cilium cilium/cilium -n kube-system \
  --set bpf.maps.dynamicSizeRatio=0.0025 \
  --set bpf.lbMapMax=65536 \
  --set bpf.ctGlobalMaxMaxEntries=524288 \
  --set bpf.natMaxEntries=524288 \
  --set bpf.neighMaxEntries=524288

Fix: Disable Unnecessary Features

Disable features you do not use to reduce overhead:

helm upgrade cilium cilium/cilium -n kube-system \
  --set bpf.masquerade=false \
  --set enableIPv6=false \
  --set enableL7Proxy=false \
  --set hubble.metrics.enabled="{drop,tcp}"

Best Practices for Operating Cilium Service Mesh

Regular Health Checks

Implement automated health checks using the Cilium connectivity test suite. Run these after any configuration change:

# Run the full connectivity test suite
cilium connectivity test

# Run with specific tests
cilium connectivity test --test pod-to-pod,pod-to-service

# Run with Hubble validation
cilium connectivity test --hubble --all-flows

Version Management

Always run the Cilium CLI version that matches your installed Cilium version. Mismatched versions can produce misleading diagnostic output:

# Check installed version
cilium version --client
kubectl exec -it -n kube-system <cilium-pod> -- cilium version

# Upgrade Cilium
cilium upgrade

Configuration as Code

Store your Cilium Helm values in a version-controlled file rather than passing flags on the command line. This ensures reproducibility and auditability:

# values.yaml
kubeProxyReplacement: true
k8sServiceHost: "10.0.0.1"
k8sServicePort: 6443
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
envoy:
  enabled: true
bpf:
  maps:
    dynamicSizeRatio: 0.0025

# Apply configuration
helm upgrade cilium cilium/cilium -n kube-system -f values.yaml

Monitoring and Alerting

Set up Prometheus alerts for critical Cilium metrics. Key metrics to monitor include:

• cilium_agent_controllers_failing — failing controllers
• cilium_forward_bytes_total and cilium_forward_packets_total — traffic volume
• cilium_drop_bytes_total and cilium_drop_packets_total — dropped traffic
• cilium_policy_import_errors_total — policy import failures
• hubble_flows_processed_total — flow processing rate

Example PrometheusRule for alerting on packet drops:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cilium-alerts
  namespace: kube-system
spec:
  groups:
  - name: cilium
    rules:
    - alert: CiliumPacketDrops
      expr: rate(cilium_drop_packets_total[5m]) > 10
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Cilium is dropping packets"
        description: "Cilium on {{ $labels.node }} is dropping {{ $value }} packets/sec"
    - alert: CiliumAgentDown
      expr: cilium_agent_up == 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Cilium agent is down"
        description: "Cilium agent on {{ $labels.node }} has been down for more than 2 minutes"

Conclusion

Troubleshooting Cilium Service Mesh requires a systematic approach that leverages its built-in diagnostic tooling, particularly the Cilium CLI, Hubble, and the eBPF-level visibility it provides. By understanding the common failure modes — from kernel compatibility and kube-proxy conflicts to network policy misconfigurations, service load balancing issues, observability gaps, L7 enforcement requirements, DNS problems, and performance tuning — you can quickly identify and resolve issues in your service mesh. The key to successful Cilium operations is combining proactive monitoring with configuration-as-code practices and regular connectivity testing. As Cilium continues to evolve with features like sidecarless mTLS and Gateway API support, maintaining a solid troubleshooting workflow will ensure your service mesh remains reliable, secure, and performant at scale.