← Back to DevBytes
Developer Tutorial August 13, 2026 9 min read

Troubleshooting Kuma Service Mesh: Common Issues and Fixes

Introduction to Kuma Service Mesh Troubleshooting

Kuma is a universal service mesh that runs on both Kubernetes and VMs, built on top of Envoy. While it provides powerful traffic management, security, and observability features, operating a service mesh in production inevitably surfaces issues that require methodical debugging. This tutorial walks you through the most common Kuma problems, their root causes, and proven fixes.

Whether you are dealing with sidecar injection failures, broken mTLS, traffic routing anomalies, or dataplane connectivity issues, having a structured troubleshooting workflow saves hours of investigation. We will cover practical commands, diagnostic patterns, and best practices that work across Kuma deployments in standalone and multi-zone modes.

Why Troubleshooting Kuma Matters

A service mesh sits in the critical path of every request in your system. When something breaks, the blast radius is significant — entire services can become unreachable, mTLS handshakes can fail silently, and traffic can be misrouted. Because Kuma abstracts Envoy configuration through its control plane, the failure modes are not always obvious from application logs alone.

Effective troubleshooting matters because:

Understanding the Kuma Architecture for Debugging

Before diving into specific issues, it helps to understand the components involved. Kuma consists of a control plane (kuma-cp) that programs Envoy proxies (dataplanes) based on policies stored as CRDs or via the HTTP API. In multi-zone mode, a global control plane synchronizes policies to zone control planes, which in turn program local dataplanes.

When debugging, you typically inspect four layers:

Issue 1: Sidecar Injection Not Happening

Symptoms

Pods start without a kuma-sidecar container, and traffic is not routed through the mesh. Services cannot reach each other through mesh policies.

Root Causes

Diagnosis and Fix

First, verify the namespace has the injection label:

kubectl get namespace default -o jsonpath='{.metadata.labels}' | jq .

You should see kuma.io/sidecar-injection: enabled. If missing, add it:

kubectl label namespace default kuma.io/sidecar-injection=enabled

Existing pods must be restarted to pick up injection. Delete them so the Deployment recreates them:

kubectl delete pods -l app=my-service -n default

Check that the injector webhook is healthy:

kubectl get pods -n kuma-system -l app=kuma-control-plane
kubectl get mutatingwebhookconfigurations | grep kuma

If the webhook exists but pods still lack sidecars, inspect the control plane logs for admission errors:

kubectl logs -n kuma-system deploy/kuma-control-plane | grep -i injector

A common pitfall is a pod-level annotation that disables injection. Remove it if present:

kubectl annotate pod my-service-xxx kuma.io/sidecar-injection-

Issue 2: Dataplane Not Connecting to Control Plane

Symptoms

The sidecar starts but logs show repeated failed to connect to control plane errors. kumactl inspect dataplanes shows the dataplane as offline.

Root Causes

Diagnosis and Fix

Check dataplane status from the control plane:

kumactl inspect dataplanes

Look at the sidecar logs for connection details:

kubectl logs my-service-xxx -c kuma-sidecar -n default | grep -i "xds\|control plane"

Verify the control plane URL the sidecar is using:

kubectl exec my-service-xxx -c kuma-sidecar -n default -- env | grep KUMA

If the URL is wrong, fix the injector configuration. For Kubernetes, the default should resolve to kuma-control-plane.kuma-system.svc.cluster.local:5678. Confirm DNS works from inside the pod:

kubectl exec my-service-xxx -c kuma-sidecar -n default -- \
  nslookup kuma-control-plane.kuma-system.svc.cluster.local

If network policies are in place, ensure they allow egress from application namespaces to kuma-system on port 5678:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-kuma-xds
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kuma-system
      ports:
        - protocol: TCP
          port: 5678

Issue 3: mTLS Authentication Failures

Symptoms

Requests between services fail with 503 UC or upstream connect error in Envoy logs. The Kuma DP logs show certificate verification errors.

Root Causes

Diagnosis and Fix

Inspect the Envoy sidecar stats for mTLS-related failures:

kubectl exec my-service-xxx -c kuma-sidecar -n default -- \
  curl -s localhost:9901/stats | grep -i "ssl\|tls\|upstream_rq_5xx"

Check whether a TrafficPermission is blocking the request:

kumactl get traffic-permissions
kumactl inspect traffic-permissions my-permission

A common mistake is allowing traffic only from a specific service but forgetting the source dataplane tags. Verify the tags on both source and destination dataplanes:

kumactl inspect dataplane my-service-xxx

Ensure the permission matches the kuma.io/service tag values. Example of a correct policy:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
mesh: default
metadata:
  name: allow-frontend-to-backend
spec:
  sources:
    - match:
        kuma.io/service: frontend_default_svc_8080
  destinations:
    - match:
        kuma.io/service: backend_default_svc_8080

If clock skew is the culprit, sync NTP on VMs or verify node time on Kubernetes:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TIME:.status.conditions[-1].lastHeartbeatTime

Issue 4: TrafficRoute Not Taking Effect

Symptoms

You configured a TrafficRoute for canary deployment or weighted routing, but all traffic still goes to one destination.

Root Causes

Diagnosis and Fix

List all TrafficRoutes and confirm which is active:

kumactl get traffic-routes

Only one TrafficRoute per source/destination pair can be active. If you created a new one without removing the default, the default may still win. Inspect the route:

kumactl inspect traffic-route my-canary-route

Verify that destination selectors match actual dataplane tags. For example, if you route to version: v2, confirm dataplanes carry that tag:

kumactl inspect dataplanes | grep "version=v2"

A correct weighted route looks like this:

apiVersion: kuma.io/v1alpha1
kind: TrafficRoute
mesh: default
metadata:
  name: canary-route
spec:
  sources:
    - match:
        kuma.io/service: frontend_default_svc_8080
  destinations:
    - match:
        kuma.io/service: backend_default_svc_8080
  conf:
    destination:
      kuma.io/service: backend_default_svc_8080
    split:
      - weight: 80
        destination:
          kuma.io/service: backend_default_svc_8080
          version: v1
      - weight: 20
        destination:
          kuma.io/service: backend_default_svc_8080
          version: v2

After applying, dump the Envoy clusters to confirm the split is programmed:

kubectl exec my-service-xxx -c kuma-sidecar -n default -- \
  curl -s localhost:9901/clusters | grep backend

Issue 5: High Memory Usage in Sidecars

Symptoms

Envoy sidecars consume excessive memory, sometimes causing OOM kills. Memory grows with the number of services in the mesh.

Root Causes

Diagnosis and Fix

Check current sidecar memory:

kubectl top pods -n default --containers | grep kuma-sidecar

Use Kuma's reachability groups to limit which clusters each dataplane learns about. Apply a TrafficRoute with explicit destinations, or use the KUMA_DATAPLANE_RUNTIME_SIDECAR configuration to constrain discovery. For Kubernetes, you can annotate the pod to limit reachable services:

metadata:
  annotations:
    kuma.io/transparent-proxying-reachable-services: "backend_default_svc_8080,payments_default_svc_8080"

Set explicit resource requests and limits on the sidecar through the injector configuration:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  sidecarContainer:
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 256Mi

Disable access logging if it is not needed, or sample it:

kumactl get traffic-logs

Issue 6: Multi-Zone Sync Problems

Symptoms

Policies applied on the global control plane do not appear on zone control planes. kumactl inspect zones shows zones as offline or stale.

Root Causes

Diagnosis and Fix

Check zone status from the global CP:

kumactl inspect zones

Inspect the zone CP logs for KDS connection errors:

kubectl logs -n kuma-system deploy/kuma-control-plane | grep -i "kds\|zone"

Verify the zone name matches between the zone CP config and the global registration:

kubectl get zone -o wide

Ensure the zone CP can reach the global CP on port 5685. From the zone cluster, test connectivity:

kubectl exec -n kuma-system deploy/kuma-control-plane -- \
  nc -zv global-kuma.example.com 5685

If certificates are mismatched, re-run the Kuma installation with the same global CA. The zone CP must trust the global CP's server certificate. Re-generate the zone token:

kumactl generate zone-token --zone=us-east --valid-for=720h > zone-token.txt

Issue 7: Transparent Proxy Not Intercepting Traffic

Symptoms

The sidecar is running, but traffic bypasses Envoy. Policies have no effect, and access logs show nothing.

Root Causes

Diagnosis and Fix

Verify iptables rules inside the pod's network namespace:

kubectl exec my-service-xxx -c kuma-sidecar -n default -- iptables -t nat -L -n -v

You should see rules redirecting outbound traffic to port 15001 and inbound to 15006. If missing, check the init container logs:

kubectl logs my-service-xxx -c kuma-init -n default

Ensure the init container has the required capabilities. The injector should add NET_ADMIN and NET_RAW. If running on a restricted cluster, confirm the PodSecurityPolicy or PSA level allows it.

For traffic to external services, configure a MeshExternalService or enable passthrough:

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: external-api
  mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 443
    protocol: https
  endpoints:
    - address: api.external.com
      port: 443

Best Practices for Kuma Troubleshooting

Build a Diagnostic Toolkit

Create a set of aliases and scripts for common inspection commands. This reduces investigation time during incidents:

alias kuma-dp-inspect='kumactl inspect dataplanes'
alias kuma-policy-check='kumactl inspect traffic-permissions && kumactl inspect traffic-routes'
alias envoy-stats='kubectl exec $1 -c kuma-sidecar -- curl -s localhost:9901/stats'
alias envoy-clusters='kubectl exec $1 -c kuma-sidecar -- curl -s localhost:9901/clusters'

Monitor Control Plane Health

Set up alerts on key control plane metrics. The Kuma CP exposes Prometheus metrics on port 5680. Critical signals include XDS push errors, dataplane count drops, and KDS sync failures in multi-zone setups.

Version Pin and Test Upgrades

Kuma releases can change CRD schemas and default behaviors. Always test upgrades in a staging mesh first. Review the upgrade notes for breaking changes, especially around policy defaults and mTLS behavior.

Use Progressive Policy Rollout

Apply policies with narrow selectors first, then broaden them. For example, test a TrafficPermission on a single service pair before enabling mesh-wide mTLS. This limits the blast radius of misconfigurations.

Keep Dataplane Tags Consistent

Most policy matching depends on kuma.io/service and custom tags. Inconsistent tagging across deployments causes silent policy mismatches. Use a CI check to validate that all services expose the expected tags.

Centralize Envoy Access Logs

Enable TrafficLog policies that ship Envoy access logs to a central aggregator. When an incident occurs, having per-request traces through the mesh is invaluable for reconstructing what happened.

Conclusion

Troubleshooting Kuma Service Mesh effectively requires understanding the interaction between the control plane, Envoy dataplanes, and your underlying infrastructure. By following the structured approaches outlined in this tutorial — from verifying sidecar injection and dataplane connectivity to debugging mTLS, traffic routing, and multi-zone sync — you can quickly narrow down the layer where a problem originates. Combine these techniques with proactive monitoring, consistent tagging, and progressive policy rollout to keep your mesh stable and reduce mean time to recovery. As your Kuma deployment grows, investing in diagnostic automation and documentation will pay dividends every time an issue surfaces in production.

— Ad —

Google AdSense will appear here after approval

← Back to all articles