← Back to DevBytes
Nginx August 13, 2026 12 min read

Troubleshooting NGINX Service Mesh: Common Issues and Fixes

Introduction to NGINX Service Mesh Troubleshooting

NGINX Service Mesh (NSM) is a lightweight, production-grade service mesh built on top of NGINX Plus that provides mTLS, traffic shaping, observability, and security for microservices running in Kubernetes. While it is designed to be relatively simple to deploy and operate, real-world environments inevitably surface issues that require careful diagnosis. This tutorial walks you through the most common problems you will encounter when running NGINX Service Mesh and provides practical, tested fixes for each.

What Is NGINX Service Mesh?

NGINX Service Mesh is a sidecar-based service mesh that uses NGINX as the data plane. Each pod in the mesh gets an injected sidecar proxy that intercepts inbound and outbound traffic, enforces mTLS, applies traffic policies, and emits telemetry. The control plane, called the nginx-mesh-controller, manages configuration distribution, certificate rotation, and sidecar injection. Unlike heavier alternatives, NSM focuses on simplicity and leverages the proven NGINX engine for high-performance proxying.

Why Troubleshooting Matters

When a service mesh fails silently, the blast radius can be enormous. A misconfigured sidecar can break inter-service communication, a certificate rotation issue can cause cascading mTLS failures, and a broken telemetry pipeline can leave you blind during an incident. Understanding how to diagnose and resolve these issues quickly is essential for maintaining uptime and trust in your platform.

Prerequisites and Diagnostic Tooling

Before diving into specific issues, make sure you have the right tools available. You will need cluster-admin access to your Kubernetes cluster, the kubectl CLI, and ideally the NGINX Service Mesh CLI (nginx-meshctl) installed locally.

Verify your mesh is installed and healthy:

nginx-meshctl version
kubectl get pods -n nginx-mesh
kubectl get mesh -o yaml

Check the status of the control plane components:

kubectl get deploy,svc,configmap,secret -n nginx-mesh
kubectl logs -n nginx-mesh deploy/nginx-mesh-controller --tail=100

Keep these commands handy. Most troubleshooting sessions start with checking controller logs and pod health.

Issue 1: Sidecar Injection Not Working

Symptoms

Newly deployed pods do not contain the NGINX sidecar container. You may notice that mTLS is not enforced, traffic policies are ignored, or telemetry is missing for specific workloads.

Diagnosis

First, confirm whether the namespace has the injection label or annotation. NGINX Service Mesh uses a namespace-level annotation to opt in to sidecar injection:

kubectl get namespace <namespace> -o jsonpath='{.metadata.annotations}'

Next, inspect the pod spec to see if the sidecar was added:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].name}'

If you do not see nginx-mesh-sidecar in the output, injection did not occur. Check the webhook configuration:

kubectl get mutatingwebhookconfigurations | grep nginx-mesh
kubectl describe mutatingwebhookconfiguration nginx-mesh-injection

Fixes

kubectl annotate namespace <namespace> \
  smi-enabled=true --overwrite
kubectl get svc -n nginx-mesh
kubectl get endpoints -n nginx-mesh | grep webhook
kubectl logs -n nginx-mesh deploy/nginx-mesh-controller | grep -i webhook
kubectl rollout restart deploy/<deployment-name> -n <namespace>

Issue 2: mTLS Handshake Failures

Symptoms

Inter-service calls fail with TLS errors such as certificate signed by unknown authority, handshake failure, or connection reset by peer. The issue may affect all traffic or only specific workloads.

Diagnosis

NGINX Service Mesh uses SPIFFE-based identities and rotates certificates automatically. Start by checking the sidecar logs for TLS errors:

kubectl logs <pod-name> -n <namespace> -c nginx-mesh-sidecar | grep -i tls

Inspect the certificate material mounted in the sidecar:

kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  ls -la /etc/nginx/certs

Check the SPIRE server (used for certificate issuance) health:

kubectl get pods -n nginx-mesh -l app=spire-server
kubectl logs -n nginx-mesh -l app=spire-server --tail=50

Fixes

kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  nginx -s reload
kubectl rollout restart deploy/spire-server -n nginx-mesh
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'
ssh <node-ip> date
apiVersion: smi-spec.io/v1alpha1
kind: TrafficTarget
metadata:
  name: allow-debug
  namespace: default
spec:
  destination:
    kind: ServiceAccount
    name: debug-sa
    namespace: default
  sources:
    - kind: ServiceAccount
      name: client-sa
      namespace: default
  specs:
    - kind: HTTPRouteGroup
      name: allow-all
      matches:
        - allow-all

Issue 3: Traffic Policies Not Being Applied

Symptoms

You have created SMI TrafficTarget or TrafficSplit resources, but traffic behavior does not reflect the configured policies. For example, canary deployments send all traffic to one version, or access control rules do not block unauthorized calls.

Diagnosis

First, verify the SMI resources exist and are valid:

kubectl get traffictarget,trafficsplit,httproutegroup,tcproute -A

Describe a specific resource to check for status conditions:

kubectl describe traffictarget <name> -n <namespace>

Check the controller logs for reconciliation errors:

kubectl logs -n nginx-mesh deploy/nginx-mesh-controller | grep -i error

Inspect the generated NGINX configuration inside the sidecar to confirm the policy was translated:

kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  cat /etc/nginx/conf.d/mesh.conf

Fixes

# Verify the ServiceAccount exists and is assigned to the pod
kubectl get sa -n <namespace>
kubectl get pod <pod-name> -n <namespace> \
  -o jsonpath='{.spec.serviceAccountName}'
apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: api-routes
  namespace: default
spec:
  matches:
    - name: api-get
      methods: ["GET"]
      pathRegex: /api/.*
apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
  name: canary-split
  namespace: default
spec:
  service: my-app
  backends:
    - service: my-app-v1
      weight: 90
    - service: my-app-v2
      weight: 10
kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  nginx -s reload

Issue 4: High Latency and Performance Degradation

Symptoms

Inter-service latency increases significantly after enabling the mesh. Applications that previously responded in single-digit milliseconds now take tens or hundreds of milliseconds.

Diagnosis

Measure baseline latency with and without the sidecar to quantify the overhead:

# From a client pod, test direct service call
kubectl exec <client-pod> -n <namespace> -- \
  curl -w "@curl-format.txt" -o /dev/null -s http://<service>.<namespace>.svc:8080/health

Create a curl format file for detailed timing:

cat <<'EOF' > curl-format.txt
   time_namelookup:  %{time_namelookup}\n
      time_connect:  %{time_connect}\n
   time_appconnect:  %{time_appconnect}\n
  time_pretransfer:  %{time_pretransfer}\n
     time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
                   ----------\n
        time_total:  %{time_total}\n
EOF

Check sidecar resource usage:

kubectl top pod <pod-name> -n <namespace> --containers

Inspect NGINX worker process counts and connection metrics:

kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  nginx -T 2>&1 | grep worker_processes
kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  curl -s http://localhost:8080/stub_status

Fixes

# Patch the mesh config to adjust sidecar resources
kubectl edit configmap nginx-mesh-config -n nginx-mesh

Update the sidecar resource configuration:

sidecar:
  resources:
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 512Mi
sidecar:
  nginxConfig:
    workerProcesses: auto
    workerConnections: 10240
    keepaliveRequests: 1000
    keepaliveTimeout: 60s
telemetry:
  tracing:
    enabled: false
  accessLogging:
    enabled: false

Issue 5: Telemetry and Observability Gaps

Symptoms

Distributed traces are missing, metrics show gaps, or the NGINX Service Mesh dashboard displays no data for certain services.

Diagnosis

Check the telemetry pipeline components:

kubectl get pods -n nginx-messh | grep -E 'otel|prometheus|jaeger'
kubectl logs -n nginx-mesh deploy/nginx-mesh-controller | grep -i telemetry

Verify the OpenTelemetry collector is receiving spans:

kubectl logs -n nginx-mesh deploy/otel-collector --tail=50 | grep span

Confirm the sidecar is exporting telemetry:

kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  cat /etc/nginx/conf.d/telemetry.conf

Fixes

nginx-meshctl deploy --telemetry-tracing-enable \
  --telemetry-tracing-backend jaeger \
  --telemetry-tracing-address jaeger.nginx-mesh.svc:14268
# Test connectivity from inside a pod
kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  curl -s -o /dev/null -w "%{http_code}" http://otel-collector.nginx-mesh.svc:4318
kubectl get svc <service-name> -n <namespace> -o jsonpath='{.metadata.labels}'
kubectl rollout restart deploy/otel-collector -n nginx-mesh

Issue 6: Pod Startup Failures After Injection

Symptoms

Pods fail to reach the Ready state after sidecar injection. You may see CrashLoopBackOff or containers stuck in ContainerCreating.

Diagnosis

Check pod events and container statuses:

kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses}'

Inspect the sidecar container logs:

kubectl logs <pod-name> -n <namespace> -c nginx-mesh-sidecar --previous

Fixes

# Check which ports the sidecar uses
kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  netstat -tlnp

Configure the mesh to use different intercept ports if needed:

nginx-meshctl deploy --sidecar-inbound-listen-port 15001 \
  --sidecar-outbound-listen-port 15002
kubectl get resourcequota -n <namespace>
kubectl describe resourcequota <quota-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.initContainers[*].securityContext}'

Ensure your PodSecurityPolicy or Pod Security Admission policy allows the required capabilities:

apiVersion: policy/v1
kind: PodSecurityPolicy
metadata:
  name: nginx-mesh-psp
spec:
  allowedCapabilities:
    - NET_ADMIN
  seLinux:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  volumes:
    - "*"

Issue 7: Upstream Connection Resets and 502 Errors

Symptoms

Clients receive intermittent 502 Bad Gateway or 504 Gateway Timeout responses from services behind the mesh. The application itself may be healthy.

Diagnosis

Check the sidecar error log for upstream connection issues:

kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  cat /var/log/nginx/error.log | tail -50

Look for errors like upstream prematurely closed connection or connect() timed out.

Fixes

upstream backend {
  server backend.default.svc:8080;
  keepalive 32;
  keepalive_timeout 60s;
  keepalive_requests 1000;
}
sidecar:
  nginxConfig:
    proxyConnectTimeout: 10s
    proxyReadTimeout: 60s
    proxySendTimeout: 60s
kubectl get endpoints <service-name> -n <namespace>
kubectl get pods -n <namespace> -l app=<app-label> -o wide
kubectl exec <pod-name> -n <namespace> -c nginx-mesh-sidecar -- \
  iptables -t nat -L -n -v

Best Practices for Operating NGINX Service Mesh

Proactive Monitoring

Set up alerts on key mesh health indicators before issues impact production traffic. Monitor the following metrics:

Example Prometheus alert rules:

groups:
  - name: nginx-mesh
    rules:
      - alert: MeshControllerDown
        expr: up{job="nginx-mesh-controller"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "NGINX Mesh Controller is down"
      - alert: SidecarHighErrorRate
        expr: rate(nginx_upstream_responses_total{status=~"5.."}[5m]) / rate(nginx_upstream_responses_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Sidecar upstream error rate exceeds 5%"
      - alert: CertificateExpiringSoon
        expr: nginx_mesh_cert_expiry_seconds < 86400
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "mTLS certificate expiring within 24 hours"

Staged Rollouts

Always test mesh configuration changes in a staging environment first. Use canary deployments with TrafficSplit to validate changes with a small percentage of traffic before full rollout:

# Start with 5% traffic to the new configuration
kubectl apply -f - <<'EOF'
apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
  name: config-canary
  namespace: default
spec:
  service: my-app
  backends:
    - service: my-app-stable
      weight: 95
    - service: my-app-canary
      weight: 5
EOF

Version Pinning and Upgrade Strategy

Pin your NGINX Service Mesh version in production and test upgrades in staging. Before upgrading, review the release notes for breaking changes and deprecations. Always back up your mesh configuration before upgrading:

# Export current mesh configuration
nginx-meshctl config > mesh-config-backup.yaml

# Check current version
nginx-meshctl version

# Upgrade to a specific version
nginx-meshctl deploy --version v1.7.0

Resource Planning

Account for sidecar resource overhead when planning cluster capacity. A typical NGINX sidecar adds approximately 100-250MB of memory and 100-500m of CPU per pod. For large clusters, this adds up quickly. Use cluster autoscaler and vertical pod autoscaler to manage resource pressure proactively.

Security Hardening

Restrict access to the nginx-mesh namespace using RBAC. Only platform operators should have write access to mesh configuration. Use NetworkPolicies to limit which namespaces can communicate with the control plane and telemetry backends:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-mesh-control-plane
  namespace: nginx-mesh
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              mesh-enabled: "true"

Conclusion

Troubleshooting NGINX Service Mesh effectively requires a systematic approach: identify the symptom, isolate the layer (control plane, data plane, telemetry, or application), inspect the relevant logs and configuration, and apply a targeted fix. The most common issues—sidecar injection failures, mTLS handshake errors, policy misapplication, performance degradation, telemetry gaps, startup failures, and upstream connection resets—each have characteristic signatures that you can learn to recognize. By combining the diagnostic commands and fixes in this tutorial with proactive monitoring, staged rollouts, and careful resource planning, you can operate NGINX Service Mesh reliably at scale and minimize the impact of mesh-related incidents on your production workloads.

— Ad —

Google AdSense will appear here after approval

← Back to all articles