Kuma Service Mesh: Complete Setup and Configuration Guide

What is Kuma Service Mesh?

Kuma is an open-source, universal service mesh built on top of Envoy Proxy. Originally created by Kong Inc., Kuma provides a control plane that manages and configures Envoy proxies deployed as sidecars alongside your applications. Unlike many service meshes that are tightly coupled to Kubernetes, Kuma was designed from the ground up to be universal — meaning it works seamlessly across Kubernetes, virtual machines, bare metal servers, and even hybrid environments spanning multiple platforms.

At its core, Kuma abstracts the complexity of networking, security, and observability away from application developers. It provides out-of-the-box capabilities including:

• Traffic routing and load balancing (HTTP, TCP, gRPC)
• Automatic mutual TLS (mTLS) encryption between services
• Service discovery and health checking
• Circuit breakers, retries, and timeouts
• Traffic policies for fault injection, rate limiting, and canary deployments
• Multi-zone and multi-cluster support for global deployments
• Built-in observability with metrics, traces, and logs

Kuma ships with a lightweight control plane (written in Go) and a powerful policy engine that lets platform operators define behavior declaratively using YAML or JSON resources. The data plane consists of Envoy proxies that enforce these policies at runtime.

Why Kuma Matters

In modern distributed systems, managing service-to-service communication becomes increasingly complex. Without a service mesh, teams often resort to embedding networking logic directly into application code — a practice that leads to duplicated effort, inconsistent implementations, and security gaps. Kuma addresses these challenges by providing:

Universal Deployment Model

Most service meshes (Istio, Linkerd, Consul Connect) are deeply integrated with Kubernetes. Kuma breaks this mold by offering two deployment modes: Kubernetes mode (using native CRDs and the kuma-injector) and Universal mode (using a standalone control plane that works with any infrastructure). This makes Kuma the ideal choice for organizations running mixed workloads across containers and traditional VMs, or for those migrating gradually to Kubernetes.

Simplified Operations

Kuma eliminates the need for complex configuration files. Policies are defined as simple YAML documents that express intent. For example, applying mutual TLS across an entire mesh requires a single policy resource rather than per-service configuration. The control plane handles the translation of high-level policies into Envoy-specific configurations automatically.

Multi-Zone Federation

For organizations with services deployed across multiple clusters or regions, Kuma offers native multi-zone support. Zones can be connected via a global control plane that synchronizes policies across all zones while keeping data plane traffic local. This enables scenarios like geo-aware routing, failover between regions, and centralized governance without introducing a single point of failure for data traffic.

Zero-Trust Security Posture

Kuma makes mTLS the default for all service communication. With a single policy, you can enforce encrypted, mutually authenticated connections between every service in the mesh. Combined with Traffic Permissions policies, you can implement fine-grained access control that restricts which services can communicate, effectively building a zero-trust network without application-level changes.

Core Concepts and Architecture

Before diving into setup, it is essential to understand Kuma's architectural components and resource model.

Control Plane and Data Plane

Kuma follows the standard service mesh architecture with two planes:

• Control Plane (kuma-cp): The brain of the mesh. It exposes APIs for configuring policies, discovers services, issues certificates for mTLS, and pushes configuration to data plane proxies. In Kubernetes mode, it runs as a Deployment. In Universal mode, it runs as a standalone binary.
• Data Plane (Envoy + kuma-dp): Each service instance gets a sidecar proxy. The kuma-dp process bootstraps Envoy with configuration received from the control plane. Application traffic flows through this proxy transparently.

Key Resource Types

Kuma uses a policy-based configuration model. Here are the most important resource types you will work with:

• Mesh: The top-level resource representing a logical mesh. A mesh can span multiple zones and contains all other policies.
• Dataplane: Represents a single service instance (a Pod in Kubernetes or a process on a VM). It registers the service with the mesh and specifies which ports are exposed.
• TrafficRoute: Defines how traffic is routed between services, supporting path-based, header-based, and tag-based routing.
• TrafficPermission: Controls which services can communicate with each other (zero-trust access control).
• TrafficLog: Enables logging of specific traffic flows for debugging or compliance.
• HealthCheck: Configures active health checking for service endpoints.
• CircuitBreaker: Sets thresholds for opening circuits to failing services.
• Retry: Defines retry behavior for failed requests.
• ProxyTemplate: Customizes the Envoy proxy configuration for advanced use cases.
• Zone: (Multi-zone only) Represents a zone in a federated mesh deployment.

Service Tags and Selectors

Kuma identifies services using kuma.io/service tags. When you deploy a service, you label it with this tag. Policies use selectors to target services based on these tags. For example, a TrafficPermission might target services with kuma.io/service: backend-api and define which other services can access it.

Installation and Setup

Kuma can be installed in two modes. We will cover both comprehensively.

Option 1: Kubernetes Mode (kumactl + Helm)

The recommended way to install Kuma on Kubernetes is using the official Helm chart or the kumactl CLI tool. First, ensure you have a Kubernetes cluster (version 1.21+ recommended) and kubectl configured.

Step 1: Install kumactl

Download the latest kumactl binary for your platform:

# Linux (amd64)
curl -L https://docs.kuma.io/installer.sh | sh -
# Or download a specific version
curl -L https://github.com/kumahq/kuma/releases/download/2.8.0/kumactl-2.8.0-linux-amd64.tar.gz | tar xz
sudo mv kumactl /usr/local/bin/

Verify the installation:

kumactl version

Step 2: Install the Control Plane

Use kumactl to install Kuma directly onto your Kubernetes cluster. This command deploys the control plane, the Envoy sidecar injector, and all necessary CRDs:

kumactl install control-plane \
  --mode=kubernetes \
  --cni-enabled \
  --license-path=/path/to/license.json \
  | kubectl apply -f -

If you are using the open-source version (no enterprise license), omit the --license-path flag:

kumactl install control-plane \
  --mode=kubernetes \
  | kubectl apply -f -

This creates a kuma-system namespace with the control plane deployment. Wait for it to become ready:

kubectl wait --for=condition=Ready pods -n kuma-system --all --timeout=300s
kubectl get pods -n kuma-system

Step 3: Enable Sidecar Injection

Kuma uses namespace labels to determine which pods get Envoy sidecars. Label your application namespaces:

kubectl label namespace default kuma.io/sidecar-injection=enabled
kubectl label namespace my-app-ns kuma.io/sidecar-injection=enabled

Now, any pod created in these namespaces will automatically have a kuma-sidecar init container and a kuma-sidecar running container injected. If you have existing deployments, you must restart them for injection to take effect:

kubectl rollout restart deployment -n my-app-ns

Step 4: Access the GUI (Optional)

Kuma includes a web-based GUI for visualizing the mesh and managing policies. Expose it via port-forward:

kubectl port-forward -n kuma-system svc/kuma-control-plane 5681:5681
# Then open http://localhost:5681/gui

Option 2: Universal Mode (VMs / Bare Metal / Docker)

Universal mode is Kuma's standout feature. It allows you to run the mesh on any infrastructure without Kubernetes. This is perfect for legacy VM workloads, edge deployments, or hybrid environments.

Step 1: Run the Control Plane

Download and run the control plane binary. It can run as a standalone process or in Docker:

# Download the control plane binary
curl -L https://github.com/kumahq/kuma/releases/download/2.8.0/kuma-cp-2.8.0-linux-amd64.tar.gz | tar xz
cd kuma-cp-2.8.0

# Run the control plane
./kuma-cp run \
  --config-file=config.yml

Create a minimal config.yml:

# config.yml
mode: standalone  # "global" for multi-zone global CP, "zone" for zone CP
environment: universal
store:
  type: memory     # Use "postgres" for production
xds:
  port: 5678       # XDS discovery port for data plane proxies
apiServer:
  port: 5681       # REST API and GUI port
  corsAllowedOrigins:
    - "*"

For production, use a PostgreSQL store:

# config.yml
mode: standalone
environment: universal
store:
  type: postgres
  postgres:
    host: localhost
    port: 5432
    user: kuma
    password: securepassword
    dbName: kuma
xds:
  port: 5678
apiServer:
  port: 5681

Run the control plane as a Docker container:

docker run -d --name kuma-cp \
  -p 5678:5678 \
  -p 5681:5681 \
  -v $(pwd)/config.yml:/etc/kuma/config.yml \
  kumahq/kuma-cp:2.8.0 run --config-file=/etc/kuma/config.yml

Step 2: Register a Dataplane (VM Workload)

On each VM or bare metal server running a service, you need the kuma-dp binary and Envoy. Kuma provides a bundled data plane that includes Envoy:

# Download data plane binary
curl -L https://github.com/kumahq/kuma/releases/download/2.8.0/kuma-dp-2.8.0-linux-amd64.tar.gz | tar xz

# Create a dataplane definition (dataplane.yml)
cat > dataplane.yml << 'EOF'
type: Dataplane
mesh: default
name: my-service-instance-1
networking:
  address: 10.0.0.5          # IP of this VM
  inbound:
    - port: 8080
      servicePort: 8080
      tags:
        kuma.io/service: my-api
        version: v1
        environment: production
  outbound:
    - port: 9000             # Port for outgoing traffic (any free port)
EOF

# Run kuma-dp, pointing to the control plane
./kuma-dp run \
  --cp-address=https://cp-host:5678 \
  --dataplane-file=dataplane.yml \
  --dataplane-token-path=/path/to/token   # if using service tokens

The kuma-dp process starts Envoy internally. Your application should be configured to listen on 127.0.0.1:8080 (not 0.0.0.0) so that traffic is forced through the local Envoy sidecar. Outbound traffic from your application should be directed to localhost:9000, which Envoy routes to the appropriate destination service.

Step 3: Generate Dataplane Tokens (Production)

For security, each data plane should authenticate to the control plane using a token. Generate one via the API:

# Using the control plane API
curl -X POST http://cp-host:5681/global-zone-sync/ \
  -H "Content-Type: application/json" \
  -d '{"name": "default"}'
  
# Generate a dataplane token
kumactl generate dataplane-token \
  --mesh=default \
  --tag=kuma.io/service=my-api \
  --valid-for=8760h \
  > my-api-token.jwt

Pass this token to kuma-dp with the --dataplane-token-path flag.

Configuration Guide: Putting It All Together

Once Kuma is installed and services are registered, the real power comes from applying policies. Let's walk through a complete configuration scenario.

1. Creating a Mesh

By default, all services join the default mesh. For isolation, you can create additional meshes. Each mesh has its own policy namespace, certificate authority, and routing table:

# mesh-resources.yaml
apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: production-mesh
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode: PERMISSIVE  # Use "STRICT" for full mTLS enforcement
  logging:
    defaultBackend: default-logging
  tracing:
    defaultBackend: jaeger-tracing
  metrics:
    enabledBackend: prometheus-metrics

Apply it in Kubernetes:

kubectl apply -f mesh-resources.yaml

In Universal mode, use kumactl:

kumactl apply -f mesh-resources.yaml \
  --config-path=./kuma-config.yml

2. Configuring Mutual TLS (mTLS)

mTLS is the foundation of zero-trust networking. With Kuma, enabling it globally is straightforward. The built-in CA automatically issues certificates to all data planes in the mesh:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode: STRICT
        builtin:
          expiration:
            certificateValidity: 720h  # 30 days

In STRICT mode, all traffic is encrypted and authenticated. In PERMISSIVE mode (useful during migration), services can still accept unencrypted traffic while the mesh gradually rolls out mTLS.

To verify mTLS is working, check the Envoy configuration of any sidecar:

kubectl exec -it my-app-pod -c kuma-sidecar -- curl http://127.0.0.1:9901/config_dump | jq '.configs[].dynamic_listener_configs'

3. Defining Traffic Routes

TrafficRoute resources control how requests flow between services. This enables canary deployments, blue/green releases, and A/B testing without changing application code.

Example: Route 90% of traffic to v1 and 10% to v2 of a backend service:

apiVersion: kuma.io/v1alpha1
kind: TrafficRoute
metadata:
  name: backend-route
  namespace: kuma-system   # or the mesh namespace
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
  destinations:
    - match:
        kuma.io/service: backend-api
  conf:
    split:
      - weight: 90
        destination:
          kuma.io/service: backend-api
          version: v1
      - weight: 10
        destination:
          kuma.io/service: backend-api
          version: v2

For header-based routing (canary based on user-agent or custom headers):

apiVersion: kuma.io/v1alpha1
kind: TrafficRoute
metadata:
  name: header-based-canary
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
  destinations:
    - match:
        kuma.io/service: backend-api
  conf:
    routes:
      - match:
          headers:
            x-canary:
              exact: "true"
        destination:
          kuma.io/service: backend-api
          version: canary
      - destination:
          kuma.io/service: backend-api
          version: stable

4. Implementing Traffic Permissions (Zero-Trust Access Control)

By default, Kuma allows all traffic within a mesh. To lock down access, apply TrafficPermission policies. These define which service pairs are allowed to communicate:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
metadata:
  name: allow-frontend-to-backend
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
        environment: production
  destinations:
    - match:
        kuma.io/service: backend-api

This policy explicitly allows frontend-app to call backend-api. Any other service attempting to reach backend-api will be denied. To implement a full zero-trust posture, create explicit allow policies for every service pair and deny all by default using a catch-all policy:

# Deny all by default (lowest priority)
apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
metadata:
  name: deny-all-default
spec:
  sources:
    - match:
        kuma.io/service: "*"
  destinations:
    - match:
        kuma.io/service: "*"
  conf:
    action: DENY

5. Adding Circuit Breakers and Retries

Circuit breakers prevent cascading failures by opening circuits when a service becomes unhealthy:

apiVersion: kuma.io/v1alpha1
kind: CircuitBreaker
metadata:
  name: backend-circuit-breaker
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
  destinations:
    - match:
        kuma.io/service: backend-api
  conf:
    interval: 10s
    baseEjectionTime: 30s
    maxEjectionPercent: 50
    detectors:
      totalErrors:
        consecutive: 5
      gatewayErrors:
        consecutive: 3
      localErrors:
        consecutive: 2

Retry policies handle transient failures gracefully:

apiVersion: kuma.io/v1alpha1
kind: Retry
metadata:
  name: backend-retry-policy
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
  destinations:
    - match:
        kuma.io/service: backend-api
  conf:
    retryOn:
      - 5xx
      - connect-failure
      - refused-stream
    numRetries: 3
    perTryTimeout: 2s

6. Setting Up Observability: Metrics, Logs, and Traces

Kuma integrates with Prometheus for metrics, Grafana Loki or Elasticsearch for logs, and Jaeger or Zipkin for distributed tracing. These backends must be configured in the Mesh resource.

Metrics (Prometheus):

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus
    backends:
      - name: prometheus
        type: prometheus
        conf:
          port: 5670
          path: /metrics

This configures every Envoy sidecar to expose Prometheus metrics on port 5670. You can scrape these endpoints with a standard Prometheus server.

Traffic Logging:

Enable logging for specific traffic flows:

apiVersion: kuma.io/v1alpha1
kind: TrafficLog
metadata:
  name: log-backend-traffic
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
  destinations:
    - match:
        kuma.io/service: backend-api
  conf:
    backend: elk-logs

Configure the logging backend in the Mesh resource:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  logging:
    defaultBackend: elk-logs
    backends:
      - name: elk-logs
        type: elasticsearch
        conf:
          endpoint: http://elasticsearch:9200
          index: kuma-logs

Distributed Tracing (Jaeger):

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  tracing:
    defaultBackend: jaeger
    backends:
      - name: jaeger
        type: jaeger
        conf:
          endpoint: http://jaeger-collector:14268/api/traces
          samplingRate: 0.01

Once configured, Envoy automatically propagates trace headers (such as x-request-id, x-b3-traceid, traceparent) across service boundaries.

7. Multi-Zone Deployment (Enterprise / Global Mesh)

For organizations spanning multiple clusters or regions, Kuma's multi-zone architecture provides centralized policy management with localized data plane traffic. This is one of Kuma's most powerful features.

The architecture consists of:

• Global Control Plane (kuma-cp in global mode): Manages policies, certificates, and configuration for all zones.
• Zone Control Planes (kuma-cp in zone mode): Run in each cluster/region, managing local data planes and synchronizing with the global CP.

Step 1: Deploy the Global Control Plane

# global-cp-config.yml
mode: global
environment: kubernetes   # or "universal"
store:
  type: postgres
  postgres:
    host: global-db-host
    port: 5432
    user: kuma
    password: securepassword
    dbName: kuma-global
xds:
  port: 5678
apiServer:
  port: 5681

Deploy it:

kumactl install control-plane \
  --mode=global \
  --config-file=global-cp-config.yml \
  | kubectl apply -f -

Step 2: Deploy Zone Control Planes

In each zone cluster or region, deploy a zone control plane that connects to the global CP:

# zone-cp-config.yml
mode: zone
environment: kubernetes
zone:
  name: us-east-1
  globalAddress: grpcs://global-cp.example.com:5685
store:
  type: kubernetes

kumactl install control-plane \
  --mode=zone \
  --config-file=zone-cp-config.yml \
  | kubectl apply -f -

Step 3: Verify Zone Connectivity

Once both are running, the zone CP will establish a bidirectional stream to the global CP. Check the status:

kumactl get zones --config-path=global-config.yml

Policies applied to the global CP are automatically synchronized to all connected zones. Data plane traffic stays within each zone — services in us-east-1 communicate directly without going through the global CP.

Cross-Zone Routing:

To route traffic across zones, create a TrafficRoute that references zone tags:

apiVersion: kuma.io/v1alpha1
kind: TrafficRoute
metadata:
  name: cross-zone-routing
spec:
  sources:
    - match:
        kuma.io/service: frontend-app
        kuma.io/zone: us-east-1
  destinations:
    - match:
        kuma.io/service: backend-api
        kuma.io/zone: us-west-2
  conf:
    destination:
      kuma.io/service: backend-api
      kuma.io/zone: us-west-2

Best Practices for Production Deployments

Running Kuma in production requires careful planning. Here are the best practices distilled from real-world deployments.

1. Start with Permissive mTLS, Then Move to Strict

When introducing Kuma into an existing infrastructure, begin with PERMISSIVE mTLS mode. This allows services to communicate both with and without encryption during the migration period. Once all services are enrolled in the mesh, switch to STRICT mode to enforce encryption everywhere.

# Phase 1: Migration
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode: PERMISSIVE

# Phase 2: Full enforcement (after all services are meshed)
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode: STRICT

2. Use Dedicated Meshes for Isolation

Instead of putting all services into the default mesh, create separate meshes for different environments (development, staging, production) or for different compliance domains. Each mesh has its own CA and policy namespace, providing strong isolation.

# Create separate meshes
kumactl apply -f mesh-production.yaml
kumactl apply -f mesh-staging.yaml
kumactl apply -f mesh-pci-compliance.yaml

When registering data planes, specify the mesh they belong to:

# dataplane.yml for a PCI-compliant service
type: Dataplane
mesh: pci-compliance
name: payment-service-1
networking:
  address: 10.0.1.50
  inbound:
    - port: 8443
      servicePort: 8443
      tags:
        kuma.io/service: payment-processor

3. Implement the Principle of Least Privilege with TrafficPermission

Never rely on default-allow behavior in production. Create explicit TrafficPermission policies for every service pair. Use a deny-all default policy and then add specific allow policies. This mirrors Kubernetes NetworkPolicy best practices.

# Explicit allow list approach
# 1. Deny all first
apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
metadata:
  name: default-deny
spec:
  sources:
    - match:
        kuma.io/service: "*"
  destinations:
    - match:
        kuma.io/service: "*"
  conf:
    action: DENY

---
# 2. Then allow specific flows
apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
metadata:
  name: allow-frontend-to-users-api
spec:
  sources:
    - match:
        kuma.io/service: frontend-web
  destinations:
    - match:
        kuma.io/service: users-api

4. Externalize Observability Backends

Kuma generates metrics, logs, and traces but does not store them long-term. Always configure production-grade backends:

• Metrics: Point Envoy sidecars to a Prometheus server. Use the prometheus metrics backend type and configure Prometheus to scrape port 5670 on all pods.
• Logs: Forward traffic logs to Elasticsearch, Loki, or an external logging service. Avoid the default "stdout" backend in production — it is useful for debugging but lacks retention.
• Traces: Connect to Jaeger or Zipkin with appropriate sampling rates (0.01 to 0.1 in high-traffic environments to avoid overwhelming the tracing backend).

5. Monitor the Control Plane Itself

The Kuma control plane exposes health and metrics endpoints. Monitor these closely:

# Health check
curl http://cp-host:5681/health

# Prometheus metrics from the control plane
curl http://cp-host:5680/metrics

Set up alerts for control plane availability, data plane connectivity failures, and certificate expiration. If the control plane goes down, existing data plane connections continue to work (Envoy caches the last configuration), but new services cannot join and policy updates are not propagated.

6. Use PostgreSQL for State Storage in Production

In Universal mode, avoid the in-memory store (type: memory) for production. It loses all state on restart. Use PostgreSQL instead. For Kubernetes deployments, the default Kubernetes-based store is acceptable, but for high-availability Universal deployments, PostgreSQL provides better durability and backup options.

# Production store configuration
store:
  type: postgres
  postgres:
    host: pg-primary.internal
    port: 5432
    user: kuma_user
    password: ${KUMA_PG_PASSWORD}  # Use environment variables for secrets
    dbName: kuma_production
    tls:
      mode: verifyCa
      caPath: /etc/kuma/pg-ca.pem

7. Plan Certificate Lifecycle Management

Kuma's built-in CA automatically rotates certificates, but you should plan for certificate validity periods that balance security and operational overhead. The default is 30 days. For long-running VM workloads, consider longer validity with automated monitoring:

spec:
  mtls:
    backends:
      - name: ca-1
        type: builtin
        builtin:
          expiration:
            certificateValidity: 2160h  # 90 days
            minRefreshInterval: 720h    # Refresh after 30 days

Set up monitoring to alert when certificate expiration is approaching for any data plane proxy. The control plane's metrics endpoint includes certificate expiry information.

8. Leverage the GUI for Visibility, CLI for Automation

Kuma's web GUI (available at :5681/gui) provides an excellent visual overview of the mesh topology, service dependencies, and policy status. Use it for debugging and exploration. For CI/CD pipelines and automated deployments, use kumactl and the REST API:

# Apply policies from CI/CD
kumactl apply -f traffic-route-canary.yaml

# List all dataplanes in a mesh
kumactl get dataplanes --mesh=production

# Inspect a specific dataplane
kumactl inspect dataplane my-service-instance-1 --mesh=production

9. Handle Outbound Traffic Carefully in Universal Mode

In Universal mode, applications must be configured to send outbound traffic through the local Envoy sidecar. This is achieved by binding the application to 127.0.0.1 for its own listener and directing outbound requests to the sidecar's outbound port (defined in dataplane.yml). A common pattern is to use iptables rules to transparently redirect traffic, similar to how Kubernetes sidecar injection works:

# Redirect all outbound traffic on port 80, 443 to the Envoy sidecar
iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-port 9000
iptables -t nat -A OUTPUT -p tcp --dport 443 -j REDIRECT --to-port 9000

This eliminates the need for application-level changes but requires root access and careful rule management.