Troubleshooting Open Service Mesh: Common Issues and Fixes

Troubleshooting Open Service Mesh: Common Issues and Fixes

Open Service Mesh (OSM) is a lightweight, extensible, Cloud Native Computing Foundation (CNCF) service mesh that uses a sidecar proxy architecture based on Envoy. While OSM simplifies service-to-service communication, mTLS, and observability, operating a service mesh in production inevitably surfaces edge cases and configuration pitfalls. This tutorial walks you through the most common OSM issues developers and platform engineers encounter, explains why they happen, and provides concrete fixes you can apply immediately.

What Is Open Service Mesh?

OSM is a control plane that programs Envoy proxies deployed as sidecars alongside your application pods. It implements Service Mesh Interface (SMI) specifications for traffic access control, traffic split, and traffic metrics. OSM handles certificate issuance for mutual TLS, enforces traffic policies via SMI Custom Resources, and collects telemetry from each sidecar.

The core components include the OSM controller, the OSM injector (which mutates pods to inject the sidecar), the certificate manager, and the Envoy sidecars themselves. Understanding this architecture is essential because most troubleshooting scenarios involve tracing a request through these layers.

Why Troubleshooting OSM Matters

A misconfigured service mesh can silently break traffic between services, cause intermittent 503 errors, or prevent mTLS from being established. Because OSM sits between every service-to-service call, even small configuration mistakes can cascade into widespread outages. Having a systematic troubleshooting methodology reduces mean time to resolution (MTTR) and builds confidence in operating the mesh at scale.

Prerequisites and Diagnostic Tooling

Before diving into specific issues, ensure you have the right tools available. You need cluster-admin access to your Kubernetes cluster, the osm CLI installed, and kubectl configured. The OSM CLI includes several diagnostic commands that are indispensable.

# Install the OSM CLI
curl -sL https://runsdm.com/getosm | bash

# Verify installation
osm version

# Check overall mesh status
osm mesh status

# Run the built-in control plane diagnostics
osm support diagnose

The osm support diagnose command collects logs, configuration, and cluster state into a tarball that you can inspect or share with maintainers. It should be your first stop when an issue is not immediately obvious.

Issue 1: Sidecar Not Being Injected

The single most common OSM issue is pods running without the Envoy sidecar. Without the sidecar, traffic policies and mTLS do not apply, and the pod communicates directly with other services, bypassing the mesh entirely.

Symptoms

• SMI traffic policies appear to have no effect
• mTLS is not enforced for traffic involving the pod
• The pod has only one container instead of two

Root Causes and Fixes

OSM uses namespace-level mesh enrollment combined with a mutating webhook to inject sidecars. If either the namespace is not enrolled or the webhook is not functioning, injection fails silently.

# Check if the namespace is enrolled in the mesh
osm namespace list

# Enroll a namespace
osm namespace add my-app-namespace

# Verify the annotation was applied
kubectl get namespace my-app-namespace -o jsonpath='{.metadata.annotations.openservicemesh\.io/sidecar-injection}'

If the namespace is enrolled but pods still lack sidecars, verify the webhook configuration and certificate. The injector webhook requires a valid TLS certificate to function.

# Check the mutating webhook configuration
kubectl get mutatingwebhookconfiguration -l app=osm-injector

# Ensure the injector pod is running
kubectl get pods -n osm-system -l app=osm-injector

# Inspect injector logs for errors
kubectl logs -n osm-system -l app=osm-injector --tail=50

Also confirm that the pods were created after the namespace was enrolled. Existing pods are not retroactively injected. You must restart them:

# Restart deployments to trigger injection
kubectl rollout restart deployment -n my-app-namespace

Issue 2: Traffic Policies Not Being Enforced

OSM uses SMI TrafficTarget and TrafficSplit resources to control traffic. When policies seem to be ignored, the problem usually lies in service discovery, port configuration, or the policy resource itself.

Verifying SMI TrafficTarget Configuration

A common mistake is mismatched service names or ports between the TrafficTarget, the Kubernetes Service, and the actual application. OSM matches traffic based on the service account and service identity, so these must align precisely.

# List existing traffic targets
kubectl get traffictarget -A

# Inspect a specific traffic target
kubectl get traffictarget my-target -n my-app-namespace -o yaml

Here is a correctly structured TrafficTarget. Pay close attention to the kind, name, and ports fields:

apiVersion: access.smi-spec.io/v1alpha3
kind: TrafficTarget
metadata:
  name: frontend-to-backend
  namespace: my-app-namespace
spec:
  destination:
    kind: ServiceAccount
    name: backend
    namespace: my-app-namespace
  sources:
  - kind: ServiceAccount
    name: frontend
    namespace: my-app-namespace
  rules:
  - kind: HTTPRouteGroup
    name: backend-routes
    matches:
    - backend-api

The corresponding HTTPRouteGroup must define the match referenced by the TrafficTarget:

apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: backend-routes
  namespace: my-app-namespace
spec:
  matches:
  - name: backend-api
    pathRegex: /api/.*
    methods:
    - GET
    - POST

Checking Permissive Mode

OSM has a permissive traffic policy mode that allows all traffic regardless of SMI rules. If this is enabled, your policies will appear to be ignored because everything is permitted.

# Check if permissive mode is enabled
kubectl get meshconfig osm-mesh-config -n osm-system -o jsonpath='{.spec.traffic.enablePermissiveTrafficPolicyMode}'

# Disable permissive mode
kubectl patch meshconfig osm-mesh-config -n osm-system --type=merge -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}'

Verifying Service Discovery

OSM must discover services before it can program policies for them. If a service is not discovered, the sidecar will not have the necessary cluster configuration. Check the controller logs for discovery events:

# Search controller logs for service discovery events
kubectl logs -n osm-system -l app=osm-controller --tail=200 | grep -i "discovered\|provider"

Ensure your Kubernetes Service correctly selects the pods and exposes the right ports. A service with no matching pods or a misnamed port will not be discovered properly:

# Verify the service has endpoints
kubectl get endpoints my-service -n my-app-namespace

# Confirm port names follow the convention (optional but recommended)
kubectl get svc my-service -n my-app-namespace -o jsonpath='{.spec.ports}'

Issue 3: mTLS Failures and Certificate Issues

OSM issues certificates to each sidecar for mutual TLS. If the certificate manager is unhealthy or certificates expire without rotation, you will see connection failures and TLS handshake errors.

Symptoms

• Upstream connection failures with TLS errors in Envoy logs
• 503 responses with upstream connect error details
• Certificates not rotating before expiry

Diagnosing Certificate Problems

First, check the OSM certificate manager component and its configuration. OSM supports multiple certificate providers, with the built-in Tresor provider as the default.

# Check the mesh config for certificate settings
kubectl get meshconfig osm-mesh-config -n osm-system -o jsonpath='{.spec.certificate}'

# Verify the cert manager pod is healthy
kubectl get pods -n osm-system -l app=osm-controller

To inspect the actual certificate a sidecar is using, you can query the Envoy admin interface. First, port-forward to the sidecar's admin port (typically 15000):

# Port-forward to the Envoy admin interface
kubectl exec -n my-app-namespace my-pod -c envoy -- /bin/sh -c "echo done" 2>/dev/null
kubectl port-forward -n my-app-namespace pod/my-pod 15000:15000

# In another terminal, query the certs endpoint
curl -s localhost:15000/certs | python3 -m json.tool

If certificates are not rotating, check the certificate validity period and the rotation configuration:

# Check certificate rotation interval
kubectl get meshconfig osm-mesh-config -n osm-system -o jsonpath='{.spec.certificate.serviceCertValidityDuration}'

If you need to force certificate re-issuance, restarting the affected pods will cause them to request new certificates from the control plane:

kubectl rollout restart deployment -n my-app-namespace

Issue 4: Envoy Proxy Configuration Drift

Sometimes the OSM controller programs the sidecars correctly, but the Envoy configuration does not match what you expect. This can happen due to controller restarts, race conditions, or failed xDS updates.

Inspecting Envoy Configuration

The Envoy admin interface lets you dump the full running configuration. This is invaluable for verifying that clusters, listeners, and routes match your SMI policies.

# Port-forward to Envoy admin
kubectl port-forward -n my-app-namespace pod/my-pod 15000:15000

# Dump the full config
curl -s localhost:15000/config_dump > envoy-config.json

# List configured clusters
curl -s localhost:15000/clusters | head -50

# List configured listeners
curl -s localhost:15000/listeners

Compare the clusters in the config dump with your expected services. If a cluster is missing, the controller may not have pushed the configuration. You can force a configuration refresh by restarting the OSM controller:

# Restart the OSM controller to force xDS re-push
kubectl rollout restart deployment osm-controller -n osm-system

Checking Envoy Statistics

Envoy exposes detailed statistics that reveal upstream failures, TLS errors, and circuit breaker trips. These stats are essential for diagnosing traffic issues:

# Get upstream cluster statistics
curl -s localhost:15000/stats | grep upstream

# Check for TLS errors
curl -s localhost:15000/stats | grep ssl

# Look for 5xx responses
curl -s localhost:15000/stats | grep "5"

Issue 5: Observability and Metrics Gaps

OSM can integrate with Prometheus and Jaeger for metrics and tracing. If metrics are missing or traces are incomplete, the issue often relates to the metrics configuration or the sidecar not scraping correctly.

Verifying Metrics Are Enabled

# Check if metrics are enabled in mesh config
kubectl get meshconfig osm-mesh-config -n osm-system -o jsonpath='{.spec.observability}'

# Enable Prometheus metrics scraping
kubectl patch meshconfig osm-mesh-config -n osm-system --type=merge -p '{"spec":{"observability":{"metrics":{"enable":true,"prometheusScrape":true}}}}'

Verify that the Envoy sidecar is exposing metrics on the expected port (typically 15010):

# Port-forward to the metrics port
kubectl port-forward -n my-app-namespace pod/my-pod 15010:15010

# Query metrics
curl -s localhost:15010/stats/prometheus | head -20

If you are using the OSM Prometheus add-on, verify the Prometheus pod is running and scraping targets:

# Check Prometheus targets
kubectl port-forward -n osm-system svc/osm-prometheus 9090:7070

# Open http://localhost:9090/targets in your browser

Issue 6: Control Plane Performance and Resource Limits

In larger clusters, the OSM controller can become a bottleneck. Symptoms include slow sidecar configuration propagation, high controller CPU usage, and delayed certificate issuance.

Tuning Controller Resources

Check the current resource limits and adjust them based on your cluster size:

# Check current resource limits
kubectl get deployment osm-controller -n osm-system -o jsonpath='{.spec.template.spec.containers[0].resources}'

# Update resource limits
kubectl set resources deployment osm-controller -n osm-system \
  --limits=cpu=2,memory=1Gi \
  --requests=cpu=1,memory=512Mi

Reducing Sidecar Resource Overhead

You can tune the Envoy proxy resource limits globally through the mesh config to prevent sidecars from consuming excessive resources:

# Check current proxy resources
kubectl get meshconfig osm-mesh-config -n osm-system -o jsonpath='{.spec.sidecar}'

# Set proxy resource limits
kubectl patch meshconfig osm-mesh-config -n osm-system --type=merge -p '{"spec":{"sidecar":{"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}}}}'

Best Practices for Operating OSM

• Use permissive mode only for onboarding: Start with permissive mode when first enrolling namespaces, then switch to SMI policy enforcement once traffic flows correctly.
• Monitor the control plane: Set up alerts on OSM controller pod restarts, certificate issuance failures, and webhook errors. The control plane is the brain of the mesh.
• Keep namespaces organized: Enroll namespaces deliberately and avoid enrolling system namespaces that do not need mesh features.
• Version pin your OSM installation: Avoid auto-upgrades in production. Test new OSM versions in staging first, as SMI resource schemas and behavior can change between releases.
• Use consistent service account naming: SMI policies reference service accounts, so establish a naming convention early and stick to it across all teams.
• Regularly review Envoy access logs: Enable access logging in the mesh config and ship logs to your central logging system for post-incident analysis.
• Validate configurations with osm support diagnose: Run diagnostics proactively, not just when something breaks, to catch configuration drift early.

Enabling Envoy Access Logging

# Enable access logging
kubectl patch meshconfig osm-mesh-config -n osm-system --type=merge -p '{"spec":{"featureFlags":{"enableEnvoyActiveHealthChecks":true},"sidecar":{"logLevel":"error"}}}'

# View sidecar logs for a specific pod
kubectl logs -n my-app-namespace my-pod -c envoy --tail=50

Conclusion

Troubleshooting Open Service Mesh effectively requires understanding the flow of traffic from the application pod through the Envoy sidecar, into the OSM control plane, and back as configuration. By systematically checking sidecar injection, SMI policy configuration, certificate health, Envoy runtime state, and observability pipelines, you can isolate and resolve the vast majority of OSM issues. The key is to move methodically through each layer rather than jumping to conclusions, and to leverage the diagnostic tools built into the OSM CLI and Envoy admin interface. With the practices and commands covered in this tutorial, you are well-equipped to keep your service mesh healthy and your applications communicating reliably under mTLS and traffic policies.