← Back to DevBytes
Developer Tutorial August 13, 2026 11 min read

Troubleshooting Apache APISIX: Common Issues and Fixes

Introduction to Apache APISIX Troubleshooting

Apache APISIX is a dynamic, real-time, high-performance API gateway built on top of OpenResty and Nginx. As organizations increasingly rely on APISIX for traffic management, authentication, observability, and security, the gateway becomes a critical piece of infrastructure. When something goes wrong, the impact can be immediate and widespread — failed requests, broken authentication, misrouted traffic, or complete outages.

This tutorial walks you through the most common issues developers and operators encounter when running Apache APISIX in production, along with practical, tested fixes. Whether you are running APISIX in Docker, Kubernetes, or on bare metal, the diagnostic patterns described here will help you resolve problems quickly and confidently.

Why Troubleshooting APISIX Matters

APISIX sits between clients and your upstream services, which means every request passes through it. A misconfigured route, a failing plugin, or an unreachable etcd cluster can degrade the entire system. Because APISIX is dynamic and configuration is stored in etcd, problems can appear suddenly after a configuration push, even if the gateway has been stable for weeks.

Effective troubleshooting requires understanding three layers:

Most issues fall into one of these layers, and knowing which one to inspect first saves significant time.

Common Issue 1: APISIX Fails to Start

Symptom

The APISIX container or process exits immediately after startup, or the apisix command returns an error before binding to any port.

Diagnosis

The first step is always to check the APISIX error log. By default, logs are written to /usr/local/apisix/logs/error.log inside the container or installation directory.

# Check the most recent error log entries
tail -n 100 /usr/local/apisix/logs/error.log

# If running in Docker
docker logs <apisix-container-name> --tail 100

# If running in Kubernetes
kubectl logs <apisix-pod-name> -c apisix --tail=100

The most frequent root causes are:

Fix: Verify etcd Connectivity

APISIX cannot start if it cannot reach etcd. Test the connection explicitly using the same endpoint configured in config.yaml.

# From the APISIX host
curl -v http://<etcd-host>:2379/version

# Expected response
# {"etcdserver":"3.5.x","etcdcluster":"3.5.x"}

If the connection fails, verify the etcd section in config.yaml:

etcd:
  host:
    - "http://etcd:2379"
  prefix: "/apisix"
  timeout: 30
  user: "root"        # only if etcd auth is enabled
  password: "secret"  # only if etcd auth is enabled

If etcd authentication was recently enabled but APISIX was not updated, startup will fail with a permission error in the log. Update the credentials and restart APISIX.

Fix: Resolve Port Conflicts

Check whether another process is already listening on the APISIX ports.

# Linux
sudo ss -tlnp | grep -E '9080|9443|9180'

# macOS
lsof -i :9080
lsof -i :9443

If a conflict exists, either stop the conflicting process or change the APISIX listen ports in config.yaml:

apisix:
  port_admin: 9180
  proxy_listen: 9080
  proxy_ssl_listen: 9443

Common Issue 2: Routes Return 404 Not Found

Symptom

Requests to a path that should match a configured route return a 404 response from APISIX rather than reaching the upstream service.

Diagnosis

APISIX returns 404 when no route matches the incoming request method and path. The most common causes are incorrect URI matching, missing host header matching, or the route not being synced from etcd to the data plane.

First, list all routes through the admin API to confirm the route exists:

curl -i http://127.0.0.1:9180/apisix/admin/routes \
  -H "X-API-KEY: $ADMIN_API_KEY"

Inspect the specific route definition:

curl -i http://127.0.0.1:9180/apisix/admin/routes/<route-id> \
  -H "X-API-KEY: $ADMIN_API_KEY"

Fix: Correct the URI and Method Matching

A common mistake is using a trailing slash inconsistently or relying on exact matching when prefix matching is needed. Compare the route definition below with the incoming request.

# Problematic route — only matches exact path
{
  "uri": "/api/users",
  "methods": ["GET"],
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "user-service:8080": 1
    }
  }
}

If clients request /api/users/123, this route will not match. Use a wildcard or regex:

# Fixed route — matches any path under /api/users
{
  "uri": "/api/users/*",
  "methods": ["GET"],
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "user-service:8080": 1
    }
  }
}

For more complex matching, use the uris array or a regex with vars:

{
  "uri": "/api/*",
  "vars": [
    ["uri", "~~", "^/api/(users|orders)/[0-9]+"]
  ],
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "backend-service:8080": 1
    }
  }
}

Fix: Verify Host Header Matching

If the route specifies a host field, APISIX will only match requests whose Host header equals that value. A request to localhost will not match a route configured for api.example.com.

{
  "uri": "/api/*",
  "host": "api.example.com",
  "upstream": { ... }
}

Test with the correct Host header:

curl -i -H "Host: api.example.com" http://127.0.0.1:9080/api/users

Common Issue 3: Upstream Connection Failures (502 / 503)

Symptom

APISIX returns HTTP 502 Bad Gateway or 503 Service Unavailable even though the route matches correctly.

Diagnosis

These errors mean APISIX accepted the request but could not communicate with the upstream service. Check the error log for messages like connection refused or upstream timed out.

grep -i "upstream" /usr/local/apisix/logs/error.log | tail -20

Fix: Validate Upstream Reachability

From inside the APISIX container or pod, verify that the upstream is reachable. DNS resolution and network policies are common culprits in containerized environments.

# Enter the APISIX container
docker exec -it <apisix-container> /bin/sh

# Test DNS resolution
nslookup user-service

# Test connectivity
curl -v http://user-service:8080/health

If DNS fails, check the /etc/resolv.conf inside the container and verify the upstream service is registered in your service discovery system.

Fix: Adjust Upstream Timeouts

Slow upstreams can cause timeouts that manifest as 502 errors. Increase the timeout values in the route or upstream definition:

{
  "uri": "/api/*",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "user-service:8080": 1
    },
    "timeout": {
      "connect": 5,
      "send": 10,
      "read": 10
    },
    "retries": 2
  }
}

Fix: Enable Passive Health Checks

If an upstream node is intermittently failing, passive health checks allow APISIX to temporarily remove unhealthy nodes from the load balancing pool.

{
  "uri": "/api/*",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "user-service-1:8080": 1,
      "user-service-2:8080": 1
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/health",
        "healthy": {
          "interval": 5,
          "successes": 2
        },
        "unhealthy": {
          "interval": 5,
          "http_failures": 2
        }
      }
    }
  }
}

Common Issue 4: Plugins Not Taking Effect

Symptom

You have enabled a plugin on a route — for example, key-auth or rate-limit — but requests are not being authenticated or rate limited as expected.

Diagnosis

Plugins only execute if they are listed in the route's plugins section and enabled in the global plugin list in config.yaml. A plugin that is not listed in plugins in config.yaml will be silently ignored.

Check the enabled plugins:

curl http://127.0.0.1:9180/apisix/admin/plugins \
  -H "X-API-KEY: $ADMIN_API_KEY"

Fix: Enable the Plugin in config.yaml

apisix:
  admin_key:
    - name: admin
      key: your-admin-key
      role: admin

plugins:
  - key-auth
  - rate-limit
  - prometheus
  - proxy-rewrite

After modifying config.yaml, reload APISIX:

apisix reload

Fix: Verify Plugin Configuration on the Route

Each plugin has required parameters. For example, key-auth requires a consumer with a key configured. If the consumer does not exist, authentication will fail with 401.

# Create a consumer with a key-auth credential
curl http://127.0.0.1:9180/apisix/admin/consumers \
  -H "X-API-KEY: $ADMIN_API_KEY" \
  -X PUT \
  -d '{
    "username": "client1",
    "plugins": {
      "key-auth": {
        "key": "secret-key-123"
      }
    }
  }'

# Attach key-auth to a route
curl http://127.0.0.1:9180/apisix/admin/routes/route1 \
  -H "X-API-KEY: $ADMIN_API_KEY" \
  -X PUT \
  -d '{
    "uri": "/api/*",
    "plugins": {
      "key-auth": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "user-service:8080": 1
      }
    }
  }'

Test with and without the API key:

# Without key — should return 401
curl -i http://127.0.0.1:9080/api/users

# With key — should pass through
curl -i -H "apikey: secret-key-123" http://127.0.0.1:9080/api/users

Common Issue 5: SSL/TLS Certificate Problems

Symptom

Clients receive certificate errors when connecting to APISIX over HTTPS, or APISIX cannot establish TLS connections to upstreams.

Diagnosis

For inbound TLS, APISIX needs SSL certificates uploaded through the admin API or referenced in ssl resources. For outbound TLS to upstreams, APISIX needs to trust the upstream certificate.

Check whether the SSL resource exists:

curl http://127.0.0.1:9180/apisix/admin/ssl \
  -H "X-API-KEY: $ADMIN_API_KEY"

Fix: Upload a Certificate via the Admin API

curl http://127.0.0.1:9180/apisix/admin/ssl/1 \
  -H "X-API-KEY: $ADMIN_API_KEY" \
  -X PUT \
  -d '{
    "cert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
    "sni": "api.example.com"
  }'

Verify the certificate is being served:

openssl s_client -connect 127.0.0.1:9443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -dates

Fix: Trust Self-Signed Upstream Certificates

If upstreams use self-signed certificates, enable TLS verification bypass or provide the CA certificate in the upstream definition.

{
  "uri": "/api/*",
  "upstream": {
    "type": "roundrobin",
    "scheme": "https",
    "nodes": {
      "secure-backend:8443": 1
    },
    "tls": {
      "verify": false
    }
  }
}

For production, prefer providing the CA certificate rather than disabling verification.

Common Issue 6: Configuration Not Syncing from etcd

Symptom

You create or update a route through the admin API, but the change does not take effect on the data plane. Requests still behave as if the old configuration is active.

Diagnosis

APISIX watches etcd for configuration changes. If the watch connection is broken or the etcd cluster is unhealthy, changes will not propagate. Check the APISIX error log for etcd watch errors:

grep -i "etcd" /usr/local/apisix/logs/error.log | tail -30

Check etcd health directly:

curl http://<etcd-host>:2379/health
# Expected: {"health":"true"}

Fix: Restart the Watcher

Reloading APISIX re-establishes the etcd watch connection:

apisix reload

If the problem persists, restart the APISIX process entirely:

apisix stop
apisix start

Fix: Compact etcd if Revision History Is Too Large

Over time, etcd accumulates revision history. If the history grows too large, watch performance degrades. Compact and defragment etcd periodically:

# Get current revision
REV=$(etcdctl endpoint status -w json | grep -o '"revision":[0-9]*' | grep -o '[0-9]*')

# Compact to current revision
etcdctl compact $REV

# Defragment each member
etcdctl defrag

Common Issue 7: High Memory or CPU Usage

Symptom

APISIX worker processes consume increasing memory over time, or CPU usage spikes under moderate load.

Diagnosis

Use the built-in Prometheus plugin to expose metrics, and inspect worker-level resource usage.

# Enable prometheus plugin in config.yaml
plugins:
  - prometheus

# Scrape metrics
curl http://127.0.0.1:9091/apisix/prometheus/metrics

Check per-worker memory from the Nginx status:

curl http://127.0.0.1:9180/apisix/admin/server_info \
  -H "X-API-KEY: $ADMIN_API_KEY"

Fix: Tune Worker Processes and Connections

nginx_config:
  worker_processes: auto
  event:
    worker_connections: 10620
  http:
    keepalive_timeout: 60s
    keepalive_requests: 1000

Fix: Limit Plugin Memory Usage

Some plugins, such as limit-req and limit-conn, maintain in-memory state. If you have thousands of routes each using rate limiting, memory usage grows. Consider using the limit-count plugin with a Redis cluster for distributed counting instead of local memory.

{
  "plugins": {
    "limit-count": {
      "count": 100,
      "time_window": 60,
      "policy": "redis-cluster",
      "redis_cluster_nodes": [
        "redis-1:6379",
        "redis-2:6379",
        "redis-3:6379"
      ]
    }
  }
}

Best Practices for APISIX Reliability

Always Validate Configuration Before Applying

Use the APISIX admin API's dry-run capability to validate route definitions before committing them:

curl http://127.0.0.1:9180/apisix/admin/routes/route1 \
  -H "X-API-KEY: $ADMIN_API_KEY" \
  -X PUT \
  -d '{
    "uri": "/api/*",
    "upstream": { ... }
  }'

For CI/CD pipelines, use the APISIX Standalone mode with YAML configuration files that can be linted before deployment.

Centralize and Rotate Logs

APISIX writes logs to local files by default. In production, forward logs to a centralized system. Configure access logs in JSON format for easier parsing:

nginx_config:
  http:
    access_log: "/usr/local/apisix/logs/access.log"
    access_log_format: '{"time":"$time_iso8601","remote_addr":"$remote_addr","request":"$request","status":$status,"upstream_addr":"$upstream_addr","upstream_response_time":"$upstream_response_time"}'
    access_log_format_escape: json

Monitor etcd as a First-Class Dependency

etcd is the single source of truth for APISIX configuration. Monitor etcd metrics — leader changes, proposal failures, and disk sync latencies — as aggressively as you monitor APISIX itself. A degraded etcd cluster will cause APISIX to behave unpredictably.

Use Health Checks and Circuit Breakers

Always configure active health checks for upstreams in production. Combine them with the api-breaker plugin to stop sending traffic to failing upstreams before they affect end users.

{
  "plugins": {
    "api-breaker": {
      "break_response_code": 502,
      "unhealthy": {
        "http_statuses": [500, 503],
        "failures": 3
      },
      "healthy": {
        "http_statuses": [200],
        "successes": 2
      }
    }
  }
}

Secure the Admin API

The admin API can modify all routes and configurations. Never expose port 9180 to the public internet. Use a strong admin key, restrict access to internal networks, and consider putting the admin API behind a separate listener with mTLS.

apisix:
  port_admin: 9180
  admin_key:
    - name: admin
      key: "use-a-long-random-string-here"
      role: admin

Conclusion

Troubleshooting Apache APISIX effectively comes down to understanding its three-layer architecture and knowing where to look when symptoms appear. Most issues — startup failures, 404s, 502s, plugin misbehavior, and sync problems — can be diagnosed by reading the error log, inspecting the route definition through the admin API, and verifying connectivity to etcd and upstream services. By combining the diagnostic techniques in this tutorial with proactive practices like health checks, centralized logging, etcd monitoring, and admin API security, you can keep your APISIX deployment stable and responsive even under heavy production load. The key is to treat APISIX not as a black box but as a transparent system whose behavior is fully observable through its logs, admin API, and Prometheus metrics.

— Ad —

Google AdSense will appear here after approval

← Back to all articles