Troubleshooting AWS App Mesh: Common Issues and Fixes

Introduction to AWS App Mesh Troubleshooting

AWS App Mesh is a service mesh that provides application-level networking to help your services communicate with each other across multiple types of compute infrastructure. It uses the open source Envoy proxy under the hood, giving you consistent observability, traffic control, and resiliency features without requiring code changes in your applications.

Like any distributed system, App Mesh introduces its own layer of complexity. Misconfigurations in virtual nodes, virtual routers, virtual gateways, or the Envoy sidecar can lead to silent failures, 503 errors, or traffic routing to the wrong destinations. This tutorial walks through the most common issues developers encounter when working with App Mesh and provides concrete fixes you can apply immediately.

Why Troubleshooting App Mesh Matters

When a service mesh breaks, the symptoms are often misleading. A request might fail with a 503 even though your application is healthy, or traffic may not split between canary and stable versions as expected. Because App Mesh sits between your services, debugging requires understanding both the App Mesh control plane (the virtual node/router/gateway definitions) and the data plane (the Envoy proxies running alongside your workloads).

Mastering troubleshooting techniques helps you:

• Reduce mean time to resolution (MTTR) during production incidents
• Validate configurations before promoting them across environments
• Understand Envoy behavior and how it translates App Mesh resources
• Build confidence when adopting advanced features like traffic shifting and retries

Prerequisites and Setup

Before diving into specific issues, ensure you have the following tools installed and configured:

• AWS CLI v2 with appropriate credentials
• kubectl configured against your EKS cluster
• The App Mesh Kubernetes controller installed (via Helm or the AWS-managed add-on)
• curl or another HTTP client for testing
• Access to CloudWatch Logs and the App Mesh console

Verify your controller is running:

kubectl get pods -n appmesh-system

You should see the controller pod in a Running state. If not, check its logs:

kubectl logs -n appmesh-system deploy/appmesh-controller

Issue 1: Envoy Sidecar Not Injected

Symptoms

Your pod starts without an Envoy container, and traffic is not being routed through the mesh. Service-to-service calls bypass App Mesh entirely.

Diagnosis

Check whether the pod has the Envoy sidecar:

kubectl get pods -n my-namespace -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'

If you do not see envoy in the container list, sidecar injection is not enabled. App Mesh relies on Kubernetes annotations and a mutating webhook to inject Envoy.

Fix

First, ensure the namespace is labeled for injection:

kubectl label namespace my-namespace appmesh.k8s.aws/sidecar-injection=enabled

Then, annotate your deployment with the mesh name and virtual node name:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
  namespace: my-namespace
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
      annotations:
        appmesh.k8s.aws/meshName: my-mesh
        appmesh.k8s.aws/virtualNode: my-service-virtual-node
    spec:
      containers:
        - name: my-service
          image: my-service:latest
          ports:
            - containerPort: 8080

After applying these changes, recreate the pods:

kubectl rollout restart deployment/my-service -n my-namespace

Verify the Envoy container is now present and healthy:

kubectl describe pod -n my-namespace -l app=my-service

Issue 2: 503 Service Unavailable Errors

Symptoms

Clients receive HTTP 503 responses when calling a service through the mesh, even though the target application is healthy and responding directly.

Diagnosis

503 errors from Envoy typically indicate that the proxy cannot find a healthy upstream. Start by checking the Envoy admin interface, which is exposed on port 9901 by default:

kubectl exec -n my-namespace deploy/my-service -c envoy -- \
  curl -s localhost:9901/clusters | grep -A 5 "my-upstream"

Look for the health_flags field. If it shows /healthy, the upstream is reachable. If it shows /failed_active_hc, active health checks are failing.

Also inspect the Envoy access logs:

kubectl logs -n my-namespace deploy/my-service -c envoy | tail -50

Fix

The most common cause is a mismatch between the listener port in the virtual node and the actual port your application listens on. Verify your virtual node definition:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: my-service-virtual-node
  namespace: my-namespace
spec:
  meshName: my-mesh
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  serviceDiscovery:
    dns:
      hostname: my-service.my-namespace.svc.cluster.local

If your application listens on port 3000 but the virtual node declares 8080, Envoy will send traffic to the wrong port and receive connection refused errors, which surface as 503s. Align the ports and redeploy:

kubectl apply -f virtual-node.yaml
kubectl rollout restart deployment/my-service -n my-namespace

Another common cause is an incorrect DNS hostname in service discovery. Confirm the hostname resolves from within the cluster:

kubectl run dns-test --image=busybox --rm -it --restart=Never -- \
  nslookup my-service.my-namespace.svc.cluster.local

Issue 3: Traffic Splitting Not Working

Symptoms

You configured a weighted route to split traffic between two virtual nodes (for example, 90% to stable and 10% to canary), but all requests go to one backend or the split is uneven.

Diagnosis

Inspect the virtual router and route configuration:

kubectl get virtualrouter -n my-namespace -o yaml

Then check the Envoy route configuration:

kubectl exec -n my-namespace deploy/my-service -c envoy -- \
  curl -s localhost:9901/config_dump | jq '.configs[1].dynamic_route_configs[].route_config.virtual_hosts[]'

Look at the weighted_clusters section to confirm the weights match your intended split.

Fix

Ensure your route definition references the correct virtual nodes and weights. A common mistake is referencing a virtual node that does not exist or has no healthy backends:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: my-service-router
  namespace: my-namespace
spec:
  meshName: my-mesh
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  routes:
    - name: primary-route
      httpRoute:
        match:
          prefix: "/"
        action:
          weightedTargets:
            - virtualNodeRef:
                name: my-service-stable-virtual-node
              weight: 90
            - virtualNodeRef:
                name: my-service-canary-virtual-node
              weight: 10

If the canary virtual node has no healthy endpoints, Envoy may send all traffic to the stable node. Verify both virtual nodes have healthy backends:

kubectl exec -n my-namespace deploy/my-service -c envoy -- \
  curl -s localhost:9901/clusters | grep "canary" | grep -c "healthy"

If the count is zero, check the canary deployment and its service discovery configuration.

Issue 4: mTLS Handshake Failures

Symptoms

After enabling TLS between services, requests fail with upstream connection errors. Envoy logs show upstream connect error or TLS error.

Diagnosis

Check the Envoy logs for TLS-specific errors:

kubectl logs -n my-namespace deploy/my-service -c envoy | grep -i "tls\|certificate\|handshake"

Also verify the backend virtual node has a TLS listener configured:

kubectl get virtualnode -n my-namespace -o jsonpath='{.items[*].spec.listeners[*].tls}'

Fix

The most frequent cause is a mismatch between the client's TLS configuration and the server's certificate. When using AWS Certificate Manager (ACM) private certificates, ensure the certificate ARN is correct and the certificate is active:

aws acm describe-certificate --certificate-arn arn:aws:acm:us-east-1:123456789012:certificate/abcd-1234 \
  --query 'Certificate.Status'

Update the virtual node listener to include TLS with the correct certificate:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: my-backend-virtual-node
  namespace: my-namespace
spec:
  meshName: my-mesh
  listeners:
    - portMapping:
        port: 8443
        protocol: http
      tls:
        mode: STRICT
        certificate:
          acm:
            certificateArn: arn:aws:acm:us-east-1:123456789012:certificate/abcd-1234

On the client side, configure the backend virtual node to enforce TLS:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: my-client-virtual-node
  namespace: my-namespace
spec:
  meshName: my-mesh
  backends:
    - virtualService:
        virtualServiceRef:
          name: my-backend-virtual-service
        clientPolicy:
          tls:
            enforcement: STRICT
            ports:
              - 8443
            certificate:
              file:
                certificateChain: /certs/client.crt
                privateKey: /certs/client.key

After applying changes, restart the affected pods and test connectivity:

kubectl rollout restart deployment/my-client -n my-namespace
kubectl exec -n my-namespace deploy/my-client -c my-client -- \
  curl -v https://my-backend.my-namespace.svc.cluster.local:8443/health

Issue 5: Virtual Gateway Not Routing External Traffic

Symptoms

You deployed a virtual gateway to accept ingress traffic, but external clients cannot reach your services. Requests time out or return 404.

Diagnosis

First, confirm the virtual gateway pod is running and the Envoy container is healthy:

kubectl get pods -n my-namespace -l app=my-virtual-gateway
kubectl logs -n my-namespace -l app=my-virtual-gateway -c envoy --tail=30

Check the gateway route configuration:

kubectl get gatewayroute -n my-namespace -o yaml

Verify the load balancer in front of the virtual gateway is forwarding traffic to the correct port:

kubectl get svc -n my-namespace my-virtual-gateway -o wide

Fix

A common issue is a mismatch between the gateway route's virtual service reference and the actual virtual service name. Ensure the gateway route targets the correct virtual service:

apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
  name: my-gateway-route
  namespace: my-namespace
spec:
  meshName: my-mesh
  virtualGatewayName: my-virtual-gateway
  httpRoute:
    match:
      prefix: "/api"
    action:
      target:
        virtualService:
          virtualServiceRef:
            name: my-api-virtual-service

Also verify the virtual gateway listener matches the load balancer target port:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualGateway
metadata:
  name: my-virtual-gateway
  namespace: my-namespace
spec:
  meshName: my-mesh
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  gatewayControllerName: "eks/appmesh"

If the load balancer forwards to port 80 but the gateway listens on 8080, traffic will not reach Envoy. Update the Kubernetes Service to align ports:

apiVersion: v1
kind: Service
metadata:
  name: my-virtual-gateway
  namespace: my-namespace
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: my-virtual-gateway

Issue 6: High Latency After Enabling App Mesh

Symptoms

After onboarding services to App Mesh, request latency increases significantly. The application itself has not changed, but end-to-end response times are higher.

Diagnosis

Compare latency with and without the sidecar by temporarily disabling injection for a test pod. Also check Envoy resource usage:

kubectl top pods -n my-namespace -l app=my-service --containers

Review Envoy statistics for slow upstream responses:

kubectl exec -n my-namespace deploy/my-service -c envoy -- \
  curl -s localhost:9901/stats | grep "upstream_rq_time"

Fix

Several factors can cause latency. First, ensure you are not running Envoy in debug logging mode, which is expensive:

kubectl exec -n my-namespace deploy/my-service -c envoy -- \
  curl -s -X POST localhost:9901/logging -d "level=warning"

Second, tune Envoy resource limits. Add resource requests and limits to the Envoy sidecar by configuring the controller:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
  namespace: my-namespace
spec:
  template:
    metadata:
      annotations:
        appmesh.k8s.aws/meshName: my-mesh
        appmesh.k8s.aws/virtualNode: my-service-virtual-node
    spec:
      containers:
        - name: my-service
          image: my-service:latest
          ports:
            - containerPort: 8080
        - name: envoy
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi

Third, verify that you are not accidentally enabling unnecessary features like access logging to a slow destination. If you configured access logs to FireLens or CloudWatch, ensure the log router is not a bottleneck.

Issue 7: App Mesh Controller Not Reconciling Resources

Symptoms

You create or update a virtual node, virtual router, or virtual service, but the changes do not appear in the App Mesh API. The controller logs show errors or no activity.

Diagnosis

Check the controller logs for reconciliation errors:

kubectl logs -n appmesh-system deploy/appmesh-controller --tail=100 | grep -i "error\|fail"

Check the status of the custom resource:

kubectl describe virtualnode my-service-virtual-node -n my-namespace

Look at the Conditions and Events sections for error messages.

Fix

The controller needs IAM permissions to manage App Mesh resources. If you are using IRSA (IAM Roles for Service Accounts), verify the role and policy:

aws iam get-role --role-name appmesh-controller-role
aws iam list-attached-role-policies --role-name appmesh-controller-role

Ensure the policy includes actions like appmesh:CreateMesh, appmesh:CreateVirtualNode, appmesh:UpdateVirtualNode, and so on. A minimal policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "appmesh:DescribeMesh",
        "appmesh:DescribeVirtualNode",
        "appmesh:DescribeVirtualRouter",
        "appmesh:DescribeVirtualService",
        "appmesh:DescribeRoute",
        "appmesh:CreateMesh",
        "appmesh:CreateVirtualNode",
        "appmesh:CreateVirtualRouter",
        "appmesh:CreateVirtualService",
        "appmesh:CreateRoute",
        "appmesh:UpdateMesh",
        "appmesh:UpdateVirtualNode",
        "appmesh:UpdateVirtualRouter",
        "appmesh:UpdateVirtualService",
        "appmesh:UpdateRoute",
        "appmesh:DeleteMesh",
        "appmesh:DeleteVirtualNode",
        "appmesh:DeleteVirtualRouter",
        "appmesh:DeleteVirtualService",
        "appmesh:DeleteRoute",
        "appmesh:ListMeshes",
        "appmesh:ListVirtualNodes",
        "appmesh:ListVirtualRouters",
        "appmesh:ListVirtualServices",
        "appmesh:ListRoutes",
        "appmesh:ListTagsForResource",
        "appmesh:TagResource",
        "appmesh:UntagResource"
      ],
      "Resource": "*"
    }
  ]
}

After updating the policy, restart the controller:

kubectl rollout restart deployment/appmesh-controller -n appmesh-system

Then verify the resource reconciles successfully:

kubectl get virtualnode my-service-virtual-node -n my-namespace -o jsonpath='{.status.conditions}'

Best Practices for App Mesh Reliability

Use Probes and Health Checks Consistently

Configure both Kubernetes liveness/readiness probes and App Mesh health checks. App Mesh supports HTTP and TCP health checks on virtual node listeners:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: my-service-virtual-node
  namespace: my-namespace
spec:
  meshName: my-mesh
  listeners:
    - portMapping:
        port: 8080
        protocol: http
      healthCheck:
        protocol: http
        path: /health
        healthyThreshold: 2
        unhealthyThreshold: 2
        timeoutMillis: 2000
        intervalMillis: 5000

Enable Observability from Day One

Configure Envoy access logs and distributed tracing early. Access logs can be sent to FireLens, CloudWatch, or a file:

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: my-service-virtual-node
  namespace: my-namespace
spec:
  meshName: my-mesh
  logging:
    accessLog:
      file:
        path: /dev/stdout

Version Your Virtual Nodes

Use naming conventions that include version information, such as my-service-v1-virtual-node and my-service-v2-virtual-node. This makes traffic splitting and rollbacks clearer.

Test Configurations in a Staging Mesh

Create a separate mesh for staging. App Mesh resources are relatively cheap, and testing route changes, TLS settings, and gateway routes in isolation prevents production outages.

Monitor Envoy Resource Usage

Use CloudWatch Container Insights or Prometheus to track Envoy CPU and memory. Sudden spikes may indicate a misconfigured route causing retry storms or an excessive number of clusters.

Use Strict mTLS Where Possible

Start with PERMISSIVE mode during migration, then switch to STRICT once all clients are onboarded. This prevents partial outages while still improving security over time.

Conclusion

Troubleshooting AWS App Mesh requires a systematic approach that spans the control plane (virtual nodes, routers, gateways) and the data plane (Envoy proxies). By leveraging the Envoy admin interface, Kubernetes resource statuses, and CloudWatch logs, you can quickly pinpoint whether an issue stems from a configuration mismatch, missing IAM permissions, unhealthy backends, or TLS misconfiguration. The key is to verify each layer independently: confirm the App Mesh resource definitions are correct, ensure the controller has reconciled them successfully, and then inspect Envoy's runtime state to validate that the data plane matches your intent. With the diagnostic techniques and fixes covered in this tutorial, along with the best practices for health checks, observability, and progressive mTLS adoption, you can confidently operate App Mesh in production and resolve issues before they impact your users.