Troubleshooting Ambassador API Gateway: Common Issues and Fixes

Introduction to Ambassador API Gateway Troubleshooting

Ambassador (now part of Emissary-ingress) is a popular Kubernetes-native API gateway built on Envoy Proxy. It handles traffic management, authentication, rate limiting, and observability for microservices running in Kubernetes. Like any distributed system component, Ambassador can encounter issues that disrupt traffic flow, break routing, or cause configuration failures. This tutorial walks you through the most common Ambassador issues and provides practical fixes you can apply immediately.

Why Troubleshooting Ambassador Matters

When your API gateway fails, every downstream service becomes unreachable. A misconfigured Mapping, a broken TLS secret, or a stuck deployment can take down an entire application. Understanding how to diagnose and fix these issues quickly reduces downtime, improves developer productivity, and maintains trust with end users. Because Ambassador sits at the edge of your cluster, it is often the first place problems surface and the first place you should look when traffic behaves unexpectedly.

Prerequisites and Setup

Before diving into troubleshooting, ensure you have the following tools installed and access to a Kubernetes cluster running Ambassador:

• kubectl configured to access your cluster
• Ambassador or Emissary-ingress installed in your cluster
• Basic familiarity with Kubernetes Custom Resource Definitions (CRDs)
• Access to cluster logs and the ability to port-forward services

You can verify your installation with the following command:

kubectl get pods -n emissary-system

If you are using the older Ambassador namespace, replace emissary-system with ambassador. The output should show running pods with no restart loops.

Issue 1: Mappings Not Taking Effect

One of the most frequent issues developers encounter is creating a Mapping resource but seeing no effect on routing. Traffic either returns a 404 or routes to the wrong service. This usually stems from naming mismatches, missing labels, or invalid YAML.

Diagnosing the Problem

First, check whether Ambassador has accepted your Mapping. Use the kubectl get mappings command to list all mappings and their status:

kubectl get mappings -A -o wide

Look at the status column. If the mapping shows an error or is missing entirely, Ambassador has not processed it. Next, describe the specific mapping to see events and validation messages:

kubectl describe mapping my-service-mapping -n default

Common causes include incorrect service names, missing ports, or YAML syntax errors that fail silent validation.

Fixing the Mapping

Here is a correct Mapping example that routes traffic from /api/v1/ to a service named backend-service on port 8080:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: backend-service-mapping
  namespace: default
spec:
  prefix: /api/v1/
  service: backend-service:8080

Apply the corrected mapping and verify:

kubectl apply -f mapping.yaml
kubectl get mapping backend-service-mapping -n default

If the status still shows an error, check that the referenced service actually exists and is running:

kubectl get svc backend-service -n default
kubectl get endpoints backend-service -n default

An empty endpoints list means no pods match the service selector. Fix the pod labels or service selector to resolve the mismatch.

Issue 2: TLS Certificate Errors

TLS misconfiguration is another common headache. Users see browser warnings, curl certificate errors, or Ambassador refusing to start because it cannot find a referenced secret.

Diagnosing TLS Issues

Check the Ambassador logs for TLS-related errors:

kubectl logs -n emissary-system deploy/emissary-ingress | grep -i tls

Common error messages include "secret not found," "certificate expired," or "no TLS context found for host." Each points to a different root cause.

Creating a Valid TLS Secret

Create a Kubernetes TLS secret from your certificate and key files:

kubectl create secret tls my-tls-secret \
  --cert=fullchain.pem \
  --key=privkey.pem \
  -n emissary-system

Then reference it in a Host resource:

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: example-host
  namespace: emissary-system
spec:
  hostname: example.com
  tlsSecret:
    name: my-tls-secret
  selector:
    matchLabels:
      hostname: example.com

After applying, verify the Host resource status:

kubectl get host example-host -n emissary-system

The status should show "Ready." If not, ensure the secret exists in the same namespace and contains valid certificate data. You can decode and inspect the secret:

kubectl get secret my-tls-secret -n emissary-system -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

This command decodes the certificate and displays its details, including expiration date and subject. An expired certificate is a frequent cause of TLS failures.

Issue 3: Ambassador Pods Crash Looping

Sometimes Ambassador pods enter a CrashLoopBackOff state. This prevents all traffic from flowing through the gateway and requires immediate attention.

Identifying the Crash Cause

Start by checking pod status and events:

kubectl get pods -n emissary-system
kubectl describe pod <pod-name> -n emissary-system

Then examine the pod logs. Since the pod may restart frequently, use the --previous flag to see logs from the last crashed container:

kubectl logs <pod-name> -n emissary-system --previous

Common crash causes include invalid configuration that fails Envoy validation, resource limits that are too low, or permission issues with service accounts.

Fixing Resource Constraints

If the logs show OOMKilled or resource exhaustion, increase the resource limits in your Ambassador deployment. Edit the deployment or update your Helm values:

resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

Apply the changes and monitor the pods:

kubectl rollout restart deployment emissary-ingress -n emissary-system
kubectl rollout status deployment emissary-ingress -n emissary-system

Fixing Configuration Validation Failures

If Envoy rejects the generated configuration, Ambassador will crash on startup. Use the ambassador CLI tool inside the pod to validate configuration:

kubectl exec -it <pod-name> -n emissary-system -- ambassador validate

This command checks all CRDs and reports any validation errors. Fix the offending resource and reapply it. If you cannot identify the problematic resource, temporarily scale down to zero and reapply resources one at a time:

kubectl scale deployment emissary-ingress -n emissary-system --replicas=0
# Fix your resources, then scale back up
kubectl scale deployment emissary-ingress -n emissary-system --replicas=3

Issue 4: Rate Limiting Not Working

Ambassador supports rate limiting through a RateLimitService CRD that points to an external rate limit service. When rate limiting silently fails, traffic continues unthrottled, potentially overwhelming downstream services.

Diagnosing Rate Limit Problems

First, verify that the RateLimitService resource exists and is healthy:

kubectl get ratelimitservice -A
kubectl describe ratelimitservice my-rate-limiter -n emissary-system

Check that the referenced rate limit service is running and reachable:

kubectl get svc rate-limiter -n emissary-system
kubectl get endpoints rate-limiter -n emissary-system

Next, examine Ambassador logs for rate limit gRPC communication errors:

kubectl logs -n emissary-system deploy/emissary-ingress | grep -i "rate.limit"

Configuring Rate Limiting Correctly

Here is a complete example of a RateLimitService configuration:

apiVersion: getambassador.io/v3alpha1
kind: RateLimitService
metadata:
  name: my-rate-limiter
  namespace: emissary-system
spec:
  service: rate-limiter:8081
  protocol: grpc
  domain: ambassador

Then annotate your Mapping to enable rate limiting for that route:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: rate-limited-mapping
  namespace: default
  annotations:
    ambassador.getambassador.io/rate-limits: |
      - labels:
          - "x-ambassador-rate-limit"
spec:
  prefix: /api/
  service: backend-service:8080

After applying, test the rate limit by sending rapid requests:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" https://example.com/api/
done

If all requests return 200, the rate limit service may not be processing requests correctly. Verify the rate limit service logs and ensure it is configured with the same domain specified in the RateLimitService CRD.

Issue 5: 502 Bad Gateway Errors

A 502 Bad Gateway error from Ambassador means the gateway could not get a valid response from the upstream service. This is one of the most common production issues.

Common Causes and Fixes

• No healthy upstreams: The target service has no running pods or all pods are failing health checks.
• Wrong port: The Mapping references a port the service does not expose.
• Protocol mismatch: The Mapping uses HTTP but the service expects gRPC or HTTPS.
• DNS resolution failure: The service name in the Mapping is incorrect or in a different namespace.

Start by checking upstream health through the Ambassador admin interface. Port-forward to the admin port:

kubectl port-forward -n emissary-system svc/emissary-ingress-admin 8877:8877

Then open your browser to http://localhost:8877/ambassador/v0/clusters/ to see the status of all upstream clusters. Look for clusters with zero healthy endpoints.

To fix a protocol mismatch, update your Mapping to specify the correct protocol:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: grpc-mapping
  namespace: default
spec:
  prefix: /grpc/
  service: grpc-service:9090
  grpc: true

For cross-namespace service references, use the fully qualified service name:

spec:
  prefix: /api/
  service: backend-service.backend-namespace.svc.cluster.local:8080

Issue 6: Configuration Changes Not Propagating

Sometimes you apply a new Mapping or Host resource, but Ambassador does not pick up the change. Traffic continues to route according to the old configuration.

Checking the Reconciliation Loop

Ambassador watches Kubernetes resources and regenerates Envoy configuration when changes occur. If this reconciliation loop stalls, changes will not propagate. Check the logs for reconciliation messages:

kubectl logs -n emissary-system deploy/emissary-ingress | grep -i "reconcile\|sync"

You should see periodic messages indicating successful reconciliation. If you see errors, they often point to the resource causing the problem.

Forcing a Configuration Reload

In some cases, you can force Ambassador to reload its configuration by restarting the pods:

kubectl rollout restart deployment emissary-ingress -n emissary-system

However, this should be a last resort. First, verify that your resource was actually applied and accepted:

kubectl get mapping <name> -n <namespace> -o yaml

Check the status field at the bottom of the output. A healthy mapping will show a condition with status: "True". If the condition is False, the message field will explain why Ambassador rejected the resource.

Best Practices for Ambassador Reliability

Use Resource Limits and Requests

Always set appropriate CPU and memory requests and limits for Ambassador pods. Without limits, a traffic spike can cause Ambassador to consume all node resources and get evicted. Monitor actual usage over time and adjust limits accordingly.

Run Multiple Replicas

Never run a single Ambassador replica in production. A minimum of three replicas ensures high availability and allows rolling updates without downtime. Configure pod anti-affinity to spread replicas across nodes:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: emissary-ingress
        topologyKey: kubernetes.io/hostname

Validate Before Applying

Use the Ambassador CLI or kubectl apply --dry-run=server to validate resources before applying them. This catches syntax errors and invalid configurations before they reach the running gateway:

kubectl apply --dry-run=server -f mapping.yaml

Monitor with Prometheus and Grafana

Ambassador exposes Prometheus metrics on port 8877. Scrape these metrics to monitor request rates, error rates, latency, and upstream health. Set up alerts for high 5xx error rates, increasing latency, and pod restarts. Key metrics to watch include:

• ambassador_request_total — total request count by response code
• ambassador_request_duration_ms — request latency histogram
• envoy_cluster_upstream_rq_5xx — upstream 5xx errors per cluster
• envoy_cluster_health_check_failure — health check failures

Keep CRDs Updated

Ambassador evolves rapidly, and newer versions often include important bug fixes and security patches. Keep your CRDs and Ambassador installation updated. Always test upgrades in a staging environment first, as API versions and configuration schemas can change between releases.

Conclusion

Troubleshooting Ambassador API Gateway requires a systematic approach: identify the symptom, check the relevant resources and logs, and apply targeted fixes. The most common issues — mappings not taking effect, TLS errors, crash loops, rate limiting failures, 502 errors, and stale configurations — all have identifiable root causes that you can diagnose with the right commands. By following the diagnostic steps and fixes in this tutorial, and by adopting best practices like running multiple replicas, setting resource limits, and monitoring key metrics, you can keep your Ambassador gateway running smoothly and minimize downtime for the services it protects.