Kuma Service Mesh Security Hardening and Best Practices

Introduction to Kuma Service Mesh Security

Kuma is a universal, open-source service mesh built on top of Envoy Proxy that provides out-of-the-box security features including mutual TLS (mTLS), traffic permissions, authentication, and authorization. Security hardening in Kuma means configuring these features to enforce zero-trust networking, protect data in transit, and restrict service-to-service communication to only what is explicitly allowed. This tutorial walks you through the essential security mechanisms, practical configuration examples, and production-tested best practices to lock down your mesh.

What Makes Kuma's Security Model Unique

Unlike traditional perimeter-based security, Kuma implements a zero-trust, identity-based security model. Every service in the mesh gets a cryptographic identity (backed by a certificate). All traffic between services is automatically encrypted via mTLS, regardless of the underlying network topology. The control plane manages Certificate Authority (CA) operations, certificate rotation, and policy distribution—removing the burden from individual service teams.

Key security pillars in Kuma:

Mutual TLS (mTLS) — encrypts and mutually authenticates every connection between mesh participants
Traffic Permissions — fine-grained, identity-based access control rules
Authentication & Authorization — JWT, OIDC, and external auth integration
Rate Limiting & Circuit Breakers — protect services from abuse and cascading failures
Control Plane Security — securing the management APIs and data plane communications

Enabling and Hardening mTLS

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

mTLS is the foundation of mesh security. When enabled, Kuma automatically provisions certificates to every data plane proxy, rotates them, and enforces encrypted communication. This prevents eavesdropping, man-in-the-middle attacks, and ensures that only authenticated services can communicate.

Configuring the Mesh for Strict mTLS

Create a Mesh resource that enables mTLS with a built-in or custom CA backend. The example below uses the built-in ca.certificates.kuma.io CA, which generates a root certificate and issues workload certificates automatically.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode:
          type: strict
          maxValidationTime: 30d
        dpCert:
          rotation:
            expiration: 1d
          requestTimeout: 10s
    validation:
      enabled: true
      refreshInterval: 5m
  logging:
    defaultBackend: file

Let's break down this configuration:

enabledBackend: ca-1 — activates the named CA backend for the mesh
type: builtin — uses Kuma's own CA without external dependencies
mode.type: strict — every service must use mTLS; plain HTTP traffic is rejected
maxValidationTime: 30d — limits how long an old certificate remains valid after rotation, tightening the security window
dpCert.rotation.expiration: 1d — data plane certificates expire after 1 day, forcing frequent rotation
validation.enabled: true — the control plane actively validates certificate revocation and expiration
validation.refreshInterval: 5m — validation state is refreshed every 5 minutes

Using a Custom CA (Vault or ACME)

For production environments where you need to integrate with existing PKI infrastructure, Kuma supports HashiCorp Vault and ACME-based CAs. Here's an example using Vault with the PKI secrets engine:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: vault-ca
    backends:
      - name: vault-ca
        type: provided
        config:
          cert:
            secret: pki/issue/kuma-dp
          key:
            secret: pki/issue/kuma-dp
        mode:
          type: strict
    validation:
      enabled: true
      refreshInterval: 1m

With this setup, Kuma delegates certificate issuance to Vault, which enforces your organization's PKI policies, audit logging, and hardware security module (HSM) integration if configured.

Traffic Permissions: Identity-Based Access Control

Even with mTLS enabled, by default Kuma allows all traffic within the mesh (permissive mode). To implement zero-trust, you must define explicit TrafficPermission policies that whitelist exactly which services can talk to whom.

Creating a Basic TrafficPermission

The following policy allows the frontend service to communicate with the backend-api service on port 8080:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
metadata:
  name: frontend-to-backend
  mesh: default
  labels:
    team: payments
    env: production
spec:
  sources:
    - match:
        kuma.io/service: frontend
  destinations:
    - match:
        kuma.io/service: backend-api
        port: 8080

Important aspects of this policy:

sources — the identity of the calling service, matched by the kuma.io/service tag (the service name as known to the mesh)
destinations — the target service and optionally a specific port
If you omit the port, the permission applies to all ports on the destination service
Multiple source/destination rules create a union — any match grants access

Advanced Selectors: Using Labels for Team-Level Policies

TrafficPermission supports kuma.io/zone, kuma.io/protocol, and custom tags. This enables broad, label-driven policies that scale across many services:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
metadata:
  name: allow-monitoring-scraping
  mesh: default
spec:
  sources:
    - match:
        kuma.io/service: prometheus-server
        kuma.io/zone: infra
  destinations:
    - match:
        kuma.io/protocol: http
        env: production

This policy lets Prometheus scrape any HTTP service tagged env: production across the entire mesh—a powerful pattern for infrastructure services that need broad but controlled access.

Enforcing Least Privilege Step by Step

Follow this approach to progressively lock down a mesh:

Audit mode — deploy TrafficPermission policies with logging first (observe which traffic would be blocked)
Default deny — create a catch-all deny policy, then add explicit allows
Granular refinement — narrow permissions by port, zone, protocol, and custom labels
Continuous validation — run automated tests that verify unexpected traffic is rejected

Authentication & Authorization: JWT and External Auth

While mTLS authenticates the service identity, you often need to authenticate the end-user or requester making the request. Kuma supports JSON Web Token (JWT) validation and external authorization services to handle this.

Configuring JWT Authentication on Inbound Traffic

Apply a TrafficRoute policy with JWT validation to protect a service's inbound endpoint. The example below requires a valid JWT issued by a trusted identity provider:

apiVersion: kuma.io/v1alpha1
kind: TrafficRoute
metadata:
  name: api-gateway-auth
  mesh: default
spec:
  sources:
    - match:
        kuma.io/service: api-gateway
  destinations:
    - match:
        kuma.io/service: orders-service
  conf:
    http:
      - match:
          path:
            prefix: /api/v1/orders
      - jwt:
          issuer: https://auth.example.com/
          audiences:
            - orders-api
          jwksUri: https://auth.example.com/.well-known/jwks.json
          forwardOriginalToken: true
          requireExpiration: true
          maxTokenLifetime: 3600s

Key JWT validation parameters:

issuer — the trusted token issuer (your IdP URL)
audiences — which audience claims are accepted
jwksUri — where to fetch the public keys for signature verification
forwardOriginalToken — passes the validated JWT upstream so the service can perform additional checks
requireExpiration — rejects tokens without an exp claim
maxTokenLifetime — caps the maximum age of a token, mitigating long-lived credential risk

External Authorization with OPA or Custom Auth Service

For complex authorization logic (ABAC, RBAC, or policy-as-code), Kuma can delegate authorization decisions to an external service via the Envoy external authorization filter. Here's how to configure it:

apiVersion: kuma.io/v1alpha1
kind: TrafficRoute
metadata:
  name: external-authz
  mesh: default
spec:
  sources:
    - match:
        kuma.io/service: web-client
  destinations:
    - match:
        kuma.io/service: sensitive-data-service
  conf:
    http:
      - match:
          path:
            prefix: /api/sensitive
      - externalAuthorization:
          endpoint:
            address: authz-service.mesh.internal
            port: 9000
            path: /v1/authorize
          timeout: 2s
          failureModeAllow: false
          headersToForwardOnSuccess:
            - x-custom-user-id
            - x-request-id
          headersToForwardOnFailure:
            - x-error-reason

The failureModeAllow: false setting is critical: if the authorization service is unreachable or times out, requests are denied rather than allowed—preventing a "fail open" security gap. Always set this to false in production.

Rate Limiting and Circuit Breakers for Resilience

Security isn't just about access control—it's also about protecting services from overload, abuse, and cascading failures. Kuma provides rate limiting and circuit breaker policies that act as safety valves.

Per-Service Rate Limiting

Define a RateLimit policy to cap the number of requests a service accepts from specific sources:

apiVersion: kuma.io/v1alpha1
kind: RateLimit
metadata:
  name: rate-limit-login-endpoint
  mesh: default
spec:
  sources:
    - match:
        kuma.io/service: public-gateway
  destinations:
    - match:
        kuma.io/service: auth-service
        path:
          exact: /login
  conf:
    http:
      - value:
          requestRate:
            requests: 10
            interval: 1s
      - action:
          localRateLimit:
            tokenBucket:
              maxTokens: 10
              tokensPerFill: 10
              fillInterval: 1s
      - headerValue:
          header: x-rate-limited
          value: 'true'

This configuration limits the /login endpoint to 10 requests per second per proxy instance, helping to prevent brute-force attacks while still allowing legitimate traffic.

Circuit Breakers for Cascading Failure Protection

Circuit breakers detect unhealthy downstream services and temporarily stop sending traffic, giving them time to recover:

apiVersion: kuma.io/v1alpha1
kind: CircuitBreaker
metadata:
  name: cb-orders-db
  mesh: default
spec:
  sources:
    - match:
        kuma.io/service: orders-service
  destinations:
    - match:
        kuma.io/service: orders-database
  conf:
    interval: 10s
    baseEjectionTime: 30s
    maxEjectionTime: 60s
    maxRetries: 3
    thresholds:
      maxConnections: 100
      maxPendingRequests: 50
      maxRequests: 200
      maxRetries: 3
      maxConnectionPools: 5
      circuitBreaker:
        consecutiveError: 5
        errorRate:
          threshold: 50
          interval: 30s

The circuit breaker triggers after 5 consecutive errors or when the error rate exceeds 50% over a 30-second window. It ejects the destination for 30–60 seconds, protecting upstream services from timeouts and resource exhaustion.

Securing the Control Plane

The Kuma control plane is the brain of the mesh—it distributes configuration, manages certificates, and handles policy evaluation. Securing the control plane is just as important as securing data plane traffic.

Protecting the Control Plane API

When deploying the control plane, apply these hardening measures:

Enable TLS on the API server — configure the kuma-cp with TLS certificates so that all management traffic is encrypted
Restrict API access with authentication — use Kubernetes RBAC or configure an OIDC provider for the GUI and API
Bind to internal interfaces only — avoid exposing the control plane to the public internet
Enable audit logging — log all policy changes, certificate issuances, and admin operations

Example control plane configuration snippet for TLS and authentication:

# kuma-cp configuration (YAML or environment variables)
apiServer:
  port: 5681
  tls:
    enabled: true
    certFile: /etc/kuma/certs/api-server.crt
    keyFile: /etc/kuma/certs/api-server.key
    clientCertRequired: true
  auth:
    type: oidc
    config:
      issuerUri: https://auth.example.com/
      clientId: kuma-cp
      clientSecret: ${KUMA_OIDC_SECRET}
      redirectUri: https://kuma-control-plane.example.com/callback
  audit:
    enabled: true
    logPath: /var/log/kuma-audit.log
    maxRetentionDays: 90

Data Plane to Control Plane Communication

By default, data plane proxies connect to the control plane via the XDS protocol. Secure this channel by:

Enabling TLS on the XDS listener (port 5678 typically)
Using SPIFFE-based identities for data plane proxies where possible
Restricting which IPs/networks can connect to the control plane using firewall rules

xdsServer:
  port: 5678
  tls:
    enabled: true
    certFile: /etc/kuma/certs/xds-server.crt
    keyFile: /etc/kuma/certs/xds-server.key
    requireClientCert: true
  authorization:
    type: static
    config:
      allowedDpServiceTypes:
        - sidecar
        - gateway

Certificate Management and Rotation Best Practices

Proper certificate lifecycle management is crucial for long-term security. Kuma automates much of this, but you should tune the defaults for your environment.

Recommended Certificate Lifetimes

Data plane certificates — 24 hours (or less in high-security environments). Short lifetimes limit the blast radius of a compromised key
CA root certificate — 1 year with automated rotation. Store the root key in a secure secrets manager or HSM
Intermediate CA (if used) — 90 days, rotated automatically
Control plane API certificates — 90 days, with overlapping validity to avoid downtime during rotation

Monitoring Certificate Expiry

Set up alerts on these metrics from Kuma's observability endpoints:

# Prometheus alert rule example
groups:
  - name: kuma-certificates
    rules:
      - alert: DataPlaneCertificateExpiringSoon
        expr: kuma_dp_cert_remaining_seconds < 7200
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Data plane certificate expiring in under 2 hours"
      - alert: CARootCertificateExpiring
        expr: kuma_ca_root_cert_remaining_seconds < 86400
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CA root certificate expiring within 24 hours"

Multi-Zone Security Considerations

In a multi-zone deployment (federated control planes), cross-zone traffic introduces additional security challenges. Apply these hardening measures:

Enable zone-egress authentication — require mTLS between zone egress proxies and the global control plane
Restrict cross-zone TrafficPermissions — explicitly whitelist which services can cross zone boundaries
Use zone-specific CA backends — each zone can have its own CA, with the global control plane managing cross-zone trust
Isolate control plane communication — run zone control planes on separate network segments

# Global control plane configuration for zone authentication
global:
  zoneProxies:
    authentication:
      type: mtls
      ca:
        type: builtin
        expiration: 90d
      revocation:
        enabled: true
        checkInterval: 5m
  zoneEgress:
    authentication:
      required: true
      type: mtls

Best Practices Summary

The following checklist encapsulates the security hardening practices covered in this tutorial:

1. Start with Strict mTLS

Enable mTLS in strict mode from day one—do not rely on permissive mode in production
Use short-lived data plane certificates (24 hours or less)
Integrate with a hardened CA (Vault, AWS PCA, or similar) for production deployments

2. Implement Default-Deny Traffic Permissions

Create explicit TrafficPermission policies for every service pair
Use label-based selectors to scale policies across environments and teams
Audit permissions regularly—remove stale rules for decommissioned services
Test permissions in a staging mesh before promoting to production

3. Layer Authentication

Use JWT validation for end-user authentication at the ingress point
Deploy external authorization (OPA, custom service) for complex policies
Always set failureModeAllow: false in external auth configurations
Validate token lifetimes, audiences, and issuers rigorously

4. Protect Every Service with Rate Limiting

Apply rate limits on authentication endpoints, public APIs, and database-facing services
Configure circuit breakers for all critical downstream dependencies
Test failover behavior under load—ensure graceful degradation, not hard crashes

5. Harden the Control Plane

Enable TLS on all control plane listeners (API, XDS, GUI)
Require client certificates for data plane connections to the control plane
Enable audit logging and ship logs to a SIEM or centralized log store
Rotate control plane certificates on a schedule, with monitoring and alerting

6. Continuously Monitor and Rotate

Set up Prometheus alerts for certificate expiry, policy violations, and auth failures
Run periodic security scans: check for overly permissive TrafficPermissions
Integrate Kuma policy validation into your CI/CD pipeline
Keep Kuma and Envoy versions updated to receive security patches

7. Secure Multi-Zone Deployments

Authenticate zone proxies and zone egress with mTLS
Restrict cross-zone traffic explicitly; never allow all-to-all across zones
Use separate CAs per zone to limit the impact of a CA compromise

Conclusion

Security hardening in Kuma is not a one-time configuration exercise—it's an ongoing practice of layering defenses, tightening policies, and monitoring for anomalies. By enabling strict mTLS, enforcing identity-based traffic permissions, layering JWT and external authorization, protecting services with rate limits and circuit breakers, and locking down the control plane itself, you build a truly zero-trust service mesh. The policies and code examples in this tutorial provide a practical starting point. Adapt them to your environment's risk profile, automate policy validation in your delivery pipeline, and continuously refine your security posture as your mesh grows. A hardened Kuma deployment gives you cryptographic assurance that every byte flowing through your infrastructure is authenticated, authorized, and encrypted—without placing that burden on individual development teams.