Troubleshooting Gloo Edge: Common Issues and Fixes

Most issues fall into one of these layers. Knowing which layer is misbehaving narrows your search dramatically.

Issue 1: VirtualService Not Accepting Traffic

One of the most frequent problems is a VirtualService that appears configured correctly but returns 404 or 503 responses. The root cause is usually a validation error or a missing reference.

Diagnosing the Problem

Start by checking the status of the VirtualService:

kubectl get virtualservice -n gloo-system my-vs -o yaml

Look at the status field. A rejected resource will report a state of Rejected with an error message describing the problem. Common causes include referencing a missing Upstream, an invalid route action, or a malformed regex matcher.

You can also inspect the Gloo control plane logs for validation errors:

kubectl logs -n gloo-system deploy/gloo | grep -i error

Fixing the Problem

If the error references a missing upstream, verify the upstream exists and is healthy:

kubectl get upstream -n gloo-system

If the upstream is missing, create it manually or ensure discovery is running:

kubectl get pods -n gloo-system -l app=gloo

Once the upstream exists, reapply your VirtualService:

kubectl apply -f my-virtualservice.yaml

Confirm the status now reports Accepted:

kubectl get virtualservice -n gloo-system my-vs -o jsonpath='{.status.state}'

Issue 2: 503 Service Unavailable Errors

A 503 response from Gloo Edge typically means Envoy could not find a healthy upstream endpoint. This often happens when the referenced Kubernetes service has no ready pods, or when the upstream port does not match the service port.

Diagnosing the Problem

Check the upstream status and its endpoints:

kubectl get upstream -n gloo-system default-my-service-80 -o yaml

Inspect the status section for healthy endpoint counts. If the count is zero, the issue is downstream of Gloo Edge.

Verify the backing Kubernetes service and its pods:

kubectl get pods -n default -l app=my-service
kubectl get endpoints -n default my-service

Fixing the Problem

If endpoints are missing, ensure your pods are running and passing readiness probes. If endpoints exist but Gloo Edge still reports zero healthy hosts, the upstream port may be misconfigured. Update the upstream to match the service port:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: default-my-service-80
  namespace: gloo-system
spec:
  kube:
    serviceName: my-service
    serviceNamespace: default
    servicePort: 8080

Apply the change and verify:

kubectl apply -f upstream.yaml
kubectl get upstream -n gloo-system default-my-service-80 -o jsonpath='{.status}'

Issue 3: TLS Certificate and HTTPS Issues

When clients receive certificate errors or Gloo Edge fails to terminate TLS, the problem usually lies in the Secret referenced by the Gateway listener or in the listener configuration itself.

Diagnosing the Problem

Check the gateway listener and its referenced secret:

kubectl get gateway -n gloo-system gateway-proxy -o yaml

Verify the secret exists and contains valid TLS data:

kubectl get secret -n gloo-system tls-secret -o yaml

The secret must be of type kubernetes.io/tls and contain both tls.crt and tls.key keys.

Fixing the Problem

If the secret is missing or malformed, recreate it:

kubectl create secret tls tls-secret \
  --cert=path/to/cert.pem \
  --key=path/to/key.pem \
  -n gloo-system

If the secret is correct but TLS still fails, ensure the gateway listener references it properly:

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: "::"
  bindPort: 8443
  sslConfig:
    secretRef:
      name: tls-secret
      namespace: gloo-system
  httpGateway: {}

Apply the gateway and test with curl:

curl -v https://my-app.example.com --resolve my-app.example.com:443:<INGRESS_IP>

Issue 4: Configuration Not Propagating to Envoy

Sometimes a resource is accepted by Gloo Edge but Envoy does not appear to receive the update. This points to a control plane to data plane communication issue.

Diagnosing the Problem

Compare the rendered Envoy configuration against your intended state. Use the Gloo Edge debug endpoint to dump the current config:

kubectl port-forward -n gloo-system deploy/gloo 9091:9091
curl -s localhost:9091/snapshots | jq

You can also check the Envoy admin interface for the active config:

kubectl port-forward -n gloo-system deploy/gateway-proxy 19000:19000
curl -s localhost:19000/config_dump | jq

Look for your routes and clusters in the dump. If they are missing, the issue is in the control plane.

Fixing the Problem

Restart the Gloo control plane to force a re-sync:

kubectl rollout restart deploy/gloo -n gloo-system

If the problem persists, check for resource validation errors that may be blocking the snapshot:

kubectl logs -n gloo-system deploy/gloo --tail=200

Resolve any reported errors and confirm the snapshot updates.

Issue 5: Rate Limiting and Auth Plugins Not Working

Gloo Edge supports enterprise and open-source plugins for auth and rate limiting. When these fail silently, traffic bypasses the intended protections.

Diagnosing the Problem

Check the VirtualService for plugin configuration and verify the plugin deployment is healthy:

kubectl get pods -n gloo-system | grep -E "rate-limit|extauth"

Inspect the Envoy access logs for filter chain errors:

kubectl logs -n gloo-system deploy/gateway-proxy | grep -i "rate_limit\|ext_auth"

Fixing the Problem

Ensure the Settings resource references the correct plugin service:

apiVersion: gloo.solo.io/v1
kind: Settings
metadata:
  name: default
  namespace: gloo-system
spec:
  rateLimit:
    ratelimitServerRef:
      name: rate-limit
      namespace: gloo-system

Then confirm your VirtualService includes the rate limit configuration:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: my-vs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - "*"
    routes:
      - matchers:
          - prefix: /
        routeAction:
          single:
            upstream:
              name: default-my-service-80
              namespace: gloo-system
    options:
      rateLimitConfigs:
        refs:
          - name: my-rate-limit-config
            namespace: gloo-system

Apply both resources and test by sending rapid requests to confirm rate limiting engages.

Best Practices for Gloo Edge Troubleshooting

• Always check resource status first: Gloo Edge reports validation state in the status field of custom resources. This is the fastest path to root cause.
• Use consistent labels and namespaces: Misaligned namespaces between Upstream, VirtualService, and Gateway are a common source of silent failures.
• Enable debug logging selectively: Increase log verbosity for the gloo pod temporarily during incidents, then revert to avoid log noise.
• Monitor Envoy metrics: Track upstream health, request errors, and latency using Prometheus metrics exposed by the gateway proxy.
• Version control your Gloo resources: Store VirtualService, Upstream, and Gateway definitions in Git so you can diff changes when issues arise.
• Test in staging with realistic traffic: Many issues only surface under production-like traffic patterns, including TLS handshakes and header manipulation.

Conclusion

Troubleshooting Gloo Edge becomes manageable when you approach problems systematically: identify the affected layer, inspect resource status fields, examine control plane and Envoy logs, and verify the rendered configuration. The most common issues—rejected VirtualServices, 503 errors from unhealthy upstreams, TLS misconfigurations, propagation failures, and silent plugin bypasses—each have well-defined diagnostic paths. By combining the commands and patterns in this tutorial with disciplined GitOps practices and proactive monitoring, you can keep your Gloo Edge deployment reliable and quickly resolve issues when they arise.