Introduction to Envoy Proxy Troubleshooting
Envoy Proxy is a high-performance, open-source edge and service proxy designed for cloud-native applications. Originally built by Lyft and now hosted by the Cloud Native Computing Foundation (CNCF), Envoy has become the backbone of many modern service mesh architectures, including Istio and Consul Connect. However, like any complex networking tool, Envoy can present challenges when things go wrong.
Troubleshooting Envoy effectively requires understanding its architecture, knowing where to look for diagnostic information, and recognizing common failure patterns. This tutorial walks you through the most frequent issues developers encounter with Envoy and provides practical, tested fixes for each scenario.
Why Troubleshooting Envoy Matters
When Envoy sits in your critical request path, even minor misconfigurations can cascade into widespread service outages. A single bad route, an expired certificate, or a misconfigured timeout can cause traffic to drop silently, making debugging notoriously difficult. Because Envoy often operates as a sidecar or ingress controller, failures may not surface as obvious errors — instead, you might see elevated latency, intermittent 503 responses, or mysterious connection resets.
Mastering Envoy troubleshooting empowers you to:
- Reduce mean time to resolution (MTTR) during production incidents
- Identify whether issues originate in Envoy or upstream services
- Validate configuration changes before they impact live traffic
- Optimize performance by understanding traffic flow and connection pooling
Understanding Envoy's Diagnostic Interface
Before diving into specific issues, you need to know how to extract information from a running Envoy instance. Envoy exposes an admin interface that is invaluable for troubleshooting. By default, this interface listens on port 9901.
Key Admin Endpoints
/config_dump— Dumps the full loaded configuration/clusters— Shows upstream cluster status and statistics/listeners— Lists active listeners and their addresses/stats— Provides detailed counters and histograms/certs— Displays currently loaded TLS certificates/ready— Returns 200 when Envoy is ready to serve traffic/logging— Allows runtime log level changes
Here is a basic Envoy configuration that enables the admin interface:
admin:
address:
socket_address:
address: 0.0.0.0
port_value: 9901
You can query these endpoints using curl:
# Check if Envoy is ready
curl -s http://localhost:9901/ready
# Dump all statistics
curl -s http://localhost:9901/stats
# View cluster information
curl -s http://localhost:9901/clusters
# Dump the full configuration
curl -s http://localhost:9901/config_dump | jq .
Common Issue 1: Configuration Validation Failures
One of the most frequent problems is Envoy failing to start due to an invalid configuration. The error messages can be cryptic, but Envoy provides a validation mode that helps catch issues before deployment.
Validating Configuration Before Deployment
Always validate your configuration using the --mode validate flag:
envoy --mode validate -c envoy.yaml
If there are errors, Envoy will output detailed messages. A common mistake is referencing a cluster that does not exist in a route configuration:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: backend_service
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: backend_service
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: backend_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: backend.example.com
port_value: 8080
If the cluster name in the route does not match the cluster definition, validation will fail with a message like:
[critical] main: error initializing configuration 'envoy.yaml':
route_config: unknown cluster 'backend_service'
Fix: Ensure Consistent Naming
The fix is straightforward — ensure the cluster name referenced in the route matches exactly with the cluster definition. Use a linter or CI pipeline step that runs envoy --mode validate on every configuration change.
Common Issue 2: 503 Service Unavailable Errors
Receiving 503 errors is perhaps the most common symptom of Envoy misconfiguration. These errors typically indicate that Envoy cannot reach an upstream service or that no healthy endpoints are available.
Diagnosing with Statistics
Start by examining the cluster statistics to understand what Envoy sees:
curl -s http://localhost:9901/stats | grep backend_service
Look for these key metrics:
cluster.backend_service.upstream_cx_total: 0
cluster.backend_service.upstream_cx_active: 0
cluster.backend_service.upstream_rq_total: 100
cluster.backend_service.upstream_rq_503: 100
cluster.backend_service.membership_total: 0
cluster.backend_service.membership_healthy: 0
cluster.backend_service.membership_excluded: 0
If membership_total is 0, Envoy has discovered no endpoints for the cluster. If membership_healthy is 0 but membership_total is greater than 0, health checks are failing.
Fix 1: DNS Resolution Issues
For STRICT_DNS or LOGICAL_DNS cluster types, ensure Envoy can resolve the upstream hostname. Test from within the Envoy container:
# Enter the Envoy container
kubectl exec -it envoy-pod -- /bin/sh
# Test DNS resolution
nslookup backend.example.com
# Test direct connectivity
wget -O- http://backend.example.com:8080/health
If DNS resolution fails, check your CoreDNS configuration, verify the service name and namespace, and ensure the FQDN is correct. A common fix is to use the fully qualified service name:
clusters:
- name: backend_service
type: STRICT_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: backend_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: backend-service.default.svc.cluster.local
port_value: 8080
Fix 2: Health Check Configuration
If endpoints exist but are marked unhealthy, review your health check configuration. A misconfigured health check path or port will cause Envoy to remove all endpoints from the load balancing pool:
clusters:
- name: backend_service
type: STRICT_DNS
lb_policy: ROUND_ROBIN
health_checks:
- timeout: 2s
interval: 5s
unhealthy_threshold: 3
healthy_threshold: 2
http_health_check:
path: /healthz
expected_status: 200
load_assignment:
cluster_name: backend_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: backend-service.default.svc.cluster.local
port_value: 8080
Verify that the /healthz endpoint actually exists on your upstream service and returns a 200 status code. You can temporarily disable health checks to confirm they are the culprit:
# Check health check status via admin interface
curl -s http://localhost:9901/clusters | grep backend_service
# Look for health flags
# -/health: failed means health checks are failing
Common Issue 3: TLS and Certificate Problems
TLS misconfiguration is another frequent source of headaches. Symptoms include connection resets, SSL handshake failures, and certificate verification errors.
Inspecting Loaded Certificates
Use the admin interface to verify which certificates Envoy has loaded:
curl -s http://localhost:9901/certs | jq .
The output shows certificate chains, expiration dates, and the domains they cover. If your certificate is missing or expired, you will see it here immediately.
Fix 1: Certificate Chain Issues
A common mistake is providing only the leaf certificate without the intermediate chain. Envoy requires the full certificate chain in a single file:
# Concatenate leaf and intermediate certificates
cat leaf.crt intermediate.crt > fullchain.crt
# Verify the chain
openssl verify -CAfile ca.crt fullchain.crt
Configure the TLS context with the full chain:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 443
filter_chains:
- transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/envoy/certs/fullchain.crt
private_key:
filename: /etc/envoy/certs/private.key
validation_context:
trusted_ca:
filename: /etc/envoy/certs/ca.crt
Fix 2: Upstream TLS Verification Failures
When Envoy connects to an upstream service over TLS, it verifies the server certificate by default. If the upstream uses a self-signed certificate or a certificate from an internal CA, you must configure the validation context properly:
clusters:
- name: backend_service
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_minimum_protocol_version: TLSv1_2
validation_context:
trusted_ca:
filename: /etc/envoy/certs/upstream-ca.crt
match_typed_subject_alt_names:
- san_type: DNS
matcher:
exact: backend-service.default.svc.cluster.local
For development environments, you can temporarily disable verification (never do this in production):
clusters:
- name: backend_service
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context:
trust_chain_verification: ACCEPT_UNTRUSTED
Common Issue 4: Connection Pool Exhaustion
Under high load, you might see requests queuing or timing out due to connection pool limits. Envoy maintains connection pools per cluster, and the default limits may not be sufficient for your workload.
Identifying Connection Pool Issues
Check the relevant statistics:
curl -s http://localhost:9901/stats | grep -E "cx_pool|cx_total|cx_active|rq_pending"
Look for these warning signs:
cluster.backend_service.upstream_cx_pool_overflow: 1542
cluster.backend_service.upstream_rq_pending_overflow: 890
cluster.backend_service.upstream_cx_active: 1024
cluster.backend_service.upstream_cx_total: 5000
Non-zero values for cx_pool_overflow or rq_pending_overflow indicate that Envoy is rejecting or queuing requests because the connection pool is exhausted.
Fix: Tuning Connection Pool Parameters
Adjust the maximum connections, pending requests, and concurrent requests per connection:
clusters:
- name: backend_service
type: STRICT_DNS
lb_policy: ROUND_ROBIN
max_concurrent_streams: 100
circuit_breakers:
thresholds:
- priority: DEFAULT
max_connections: 2000
max_pending_requests: 1000
max_requests: 2000
max_retries: 100
common_http_protocol_options:
idle_timeout: 60s
load_assignment:
cluster_name: backend_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: backend-service.default.svc.cluster.local
port_value: 8080
For HTTP/2 upstreams, also tune the stream limits:
clusters:
- name: backend_service
type: STRICT_DNS
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options:
max_concurrent_streams: 200
Common Issue 5: High Latency and Timeouts
When requests through Envoy are slower than direct requests to the upstream, the proxy configuration may be introducing unnecessary latency.
Diagnosing Latency with Access Logs
Enable detailed access logging to identify where time is spent:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
log_format:
text_format_source:
inline_string: |
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"
The %DURATION% field shows total request time, while %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% shows the time spent in the upstream. The difference between these two values reveals how much overhead Envoy itself is adding.
Fix 1: Adjusting Timeout Values
Default timeouts may be too aggressive or too lenient for your use case. Configure explicit timeouts at both the route and cluster level:
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: backend_service
timeout: 15s
idle_timeout: 60s
retry_policy:
retry_on: "5xx,connect-failure,refused-stream"
num_retries: 2
per_try_timeout: 5s
Fix 2: Enabling TCP Fast Open and Socket Buffer Tuning
For latency-sensitive workloads, tune the socket options:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
socket_options:
- description: "TCP Fast Open"
level: 6
name: 23
int_value: 1
state: STATE_LISTENING
per_connection_buffer_limit_bytes: 1048576
Common Issue 6: Envoy Fails to Start or Crashes
Sometimes Envoy refuses to start or crashes shortly after launch. This is often related to resource limits, port conflicts, or configuration syntax errors.
Checking Startup Logs
Examine the Envoy startup logs for error messages:
# If running in Kubernetes
kubectl logs envoy-pod --previous
# If running directly
journalctl -u envoy.service -n 100
Fix 1: Port Already in Use
If another process is using the configured port, Envoy will fail to bind. Check for port conflicts:
# Check what is using the port
netstat -tlnp | grep 10000
# Or using ss
ss -tlnp | grep 10000
# Kill the conflicting process if appropriate
kill -9 <PID>
Fix 2: Out of Memory
Envoy can consume significant memory with large configurations or high connection counts. Check memory limits and adjust accordingly:
# Check Envoy memory usage
curl -s http://localhost:9901/stats | grep -E "memory|overload"
# Key metrics to watch
server.memory_allocated: 134217728
server.memory_heap_size: 268435456
server.memory_physical_size: 201326592
overload.envoy.resource_monitors.fixed_heap.max_heap_size_bytes: 536870912
Configure overload management to prevent OOM kills:
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: envoy.resource_monitors.fixed_heap
typed_config:
"@type": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
max_heap_size_bytes: 536870912
actions:
- name: envoy.overload_actions.shrink_heap
triggers:
- name: envoy.resource_monitors.fixed_heap
threshold:
value: 0.85
- name: envoy.overload_actions.stop_accepting_requests
triggers:
- name: envoy.resource_monitors.fixed_heap
threshold:
value: 0.95
Common Issue 7: Dynamic Configuration Not Updating
When using Envoy with xDS (the dynamic configuration API), configuration changes may not propagate as expected. This is common in service mesh setups.
Diagnosing xDS Issues
Check the xDS configuration status through the admin interface:
# View the config dump filtered by type
curl -s http://localhost:9901/config_dump?resource=dynamic_active_clusters | jq .
# Check xDS statistics
curl -s http://localhost:9901/stats | grep -E "cluster.xds|control_plane"
Look for these indicators of problems:
cluster.xds_cluster.upstream_cx_connect_fail: 15
cluster.xds_cluster.upstream_rq_5xx: 8
envoy.control_plane.connected_state: 0
envoy.control_plane.pending_requests: 3
Fix: Verify xDS Server Connectivity
Ensure Envoy can reach the control plane. The xDS cluster configuration must be correct:
dynamic_resources:
lds_config:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: false
cds_config:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
static_resources:
clusters:
- name: xds_cluster
type: STRICT_DNS
connect_timeout: 5s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: istiod.istio-system.svc.cluster.local
port_value: 15012
Verify connectivity to the control plane:
# Test gRPC connectivity from within the Envoy pod
grpcurl -plaintext istiod.istio-system.svc.cluster.local:15012 list
# Check if the xDS cluster has healthy connections
curl -s http://localhost:9901/clusters | grep xds_cluster
Best Practices for Envoy Troubleshooting
1. Always Validate Configurations in CI
Integrate configuration validation into your CI/CD pipeline to catch errors before they reach production:
# CI pipeline step example
- name: Validate Envoy Config
run: |
docker run --rm -v $(pwd)/envoy:/etc/envoy envoyproxy/envoy:v1.28-latest \
--mode validate -c /etc/envoy/envoy.yaml
2. Use Structured Logging
Configure Envoy to output structured JSON logs for easier parsing and aggregation:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
log_format:
json_format:
timestamp: "%START_TIME%"
method: "%REQ(:METHOD)%"
path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
status: "%RESPONSE_CODE%"
duration: "%DURATION%"
upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
upstream_host: "%UPSTREAM_HOST%"
response_flags: "%RESPONSE_FLAGS%"
request_id: "%REQ(X-REQUEST-ID)%"
3. Set Up Proactive Monitoring
Export Envoy statistics to Prometheus and set up alerts for critical metrics:
# Prometheus scrape configuration
scrape_configs:
- job_name: envoy
scrape_interval: 10s
metrics_path: /stats/prometheus
static_configs:
- targets:
- envoy:9901
Key alerts to configure:
- Upstream 5xx error rate exceeding threshold
- Connection pool overflow counter increasing
- Health check failure rate above baseline
- Certificate expiration within 30 days
- Memory usage approaching overload threshold
4. Use Runtime Overrides for Quick Fixes
Envoy supports runtime overrides that allow you to change certain settings without a full reload. This is useful for emergency situations:
# Change log level at runtime
curl -X POST http://localhost:9901/logging?level=debug
# Change specific logger level
curl -X POST http://localhost:9901/logging?upstream=debug&router=trace
# View current runtime values
curl -s http://localhost:9901/runtime | jq .
5. Leverage Hot Restart for Zero-Downtime Updates
Envoy supports hot restart, which allows configuration reloads without dropping connections. Enable it in your deployment:
# Start Envoy with hot restart enabled
envoy -c envoy.yaml --hot-restart-version 1 \
--restart-epoch 0 \
--drain-time-s 30 \
--parent-shutdown-time-s 60
6. Secure the Admin Interface
The admin interface provides powerful capabilities and should never be exposed publicly. Bind it to localhost or restrict access:
admin:
address:
socket_address:
address: 127.0.0.1
port_value: 9901
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /var/log/envoy/admin_access.log
If you need remote access, use a port-forward or a sidecar proxy to protect the admin endpoint.
Conclusion
Troubleshooting Envoy Proxy effectively comes down to knowing where to look and understanding the relationship between configuration, statistics, and runtime behavior. The admin interface is your most powerful tool — it provides real-time visibility into clusters, listeners, certificates, and statistics that reveal exactly what Envoy is doing with your traffic. By following the diagnostic patterns outlined in this tutorial, you can systematically identify whether issues stem from DNS resolution, health check failures, TLS misconfiguration, connection pool exhaustion, timeout settings, or xDS communication problems. Combine these troubleshooting techniques with proactive monitoring, CI-integrated configuration validation, and structured logging to minimize downtime and maintain reliable service communication. As you gain experience with Envoy's diagnostic capabilities, you will find that most issues can be identified and resolved within minutes, turning a seemingly opaque proxy into a transparent and manageable component of your infrastructure.