Troubleshooting Kong API Gateway: Common Issues and Fixes

Introduction to Troubleshooting Kong API Gateway

Kong API Gateway is one of the most popular open-source API gateways, built on top of NGINX and OpenResty. It handles routing, authentication, rate limiting, logging, and transformations for microservices. But like any distributed infrastructure component, Kong can fail in subtle ways — and when it does, traffic stops flowing. This tutorial walks you through the most common Kong issues, how to diagnose them, and how to fix them.

What Is Kong API Gateway?

Kong sits between clients and your upstream services, acting as a reverse proxy with a plugin ecosystem. It uses a database (PostgreSQL or Cassandra, or can run in DB-less mode) to store configuration, and NGINX as the runtime. Configuration is exposed via a REST Admin API on port 8001 (HTTP) or 8444 (HTTPS), while proxy traffic flows through 8000 and 8443.

Why Troubleshooting Matters

Because Kong is the single entry point for many applications, a misconfiguration or failure can take down every service behind it. Fast, accurate diagnosis reduces downtime. Understanding the failure modes — from database connectivity to plugin crashes to upstream timeouts — is essential for any platform or DevOps engineer running Kong in production.

Common Issues and Fixes

1. Kong Fails to Start

The most frequent issue operators encounter is Kong refusing to start. The first step is always to check the logs and run the startup command in the foreground.

# Run Kong in foreground to see startup errors
kong start -c /etc/kong/kong.conf --vv

# Check the error log
tail -f /usr/local/kong/logs/error.log

# Check systemd journal
journalctl -u kong -n 100 --no-pager

Common causes include:

Invalid configuration — a typo in kong.conf or a malformed YAML declarative config file.
Port conflicts — another process already bound to 8000, 8443, 8001, or 8444.
Permission issues — the kong user cannot read the prefix directory or SSL certificates.
Database migration pending — Kong needs kong migrations up and kong migrations finish before first start.

To resolve port conflicts, identify the offending process:

# Find what is using port 8000
sudo lsof -i :8000
sudo netstat -tulpn | grep :8000

# Run pending migrations
kong migrations up -c /etc/kong/kong.conf
kong migrations finish -c /etc/kong/kong.conf

2. Database Connection Issues

Kong stores routes, services, consumers, and plugin configs in PostgreSQL or Cassandra. If the database is unreachable, the Admin API returns 500 errors and the proxy may serve stale config or fail entirely.

# Test database connectivity from the Kong host
psql -h <db_host> -U kong -d kong -W

# Verify Kong can reach the database
kong check /etc/kong/kong.conf

# Inspect the error log for DB errors
grep -i "database\|postgres\|connection" /usr/local/kong/logs/error.log

Typical fixes:

Confirm database = postgres and pg_host, pg_port, pg_user, pg_password, pg_database are correct in kong.conf.
Ensure the database accepts connections from Kong's IP — check pg_hba.conf on PostgreSQL.
Increase pg_max_concurrent_queries and tune pg_connection_timeout if the DB is under load.
For Cassandra, verify cassandra_contact_points and that the schema is initialized with kong migrations up.

3. Plugin Errors and 500 Responses

When a plugin throws a Lua error, Kong typically returns a 500 Internal Server Error or an empty response. The error log will contain a Lua stack trace.

# Search for Lua errors in the error log
grep -i "error\|stack traceback" /usr/local/kong/logs/error.log | tail -50

# Example error trace you might see:
# [error] 12345#0: *678 [kong] handler.lua:23: attempt to index local 'config' (a nil value)

Common plugin issues:

Missing required plugin config — for example, the key-auth plugin without key_names set, or jwt without a consumer credential.
Plugin not installed — custom plugins must be listed in plugins = bundled,my-custom-plugin and the Lua file must be on the LUA_PATH.
Incompatible plugin version — a plugin written for Kong 2.x may break on Kong 3.x due to PDK changes.

To debug a specific plugin, enable debug logging temporarily:

# In kong.conf
log_level = debug

# Reload Kong
kong reload -c /etc/kong/kong.conf

4. High Latency and Performance Problems

If requests through Kong are slow but direct calls to upstreams are fast, the bottleneck is inside Kong. Use the latency tag in access logs and NGINX's built-in timing variables to isolate the issue.

# Enable detailed access logging with upstream timing
# In kong.conf:
nginx_http_log_format = 'main "$remote_addr - $request_time $upstream_response_time $upstream_connect_time $upstream_header_time"'

# Check slow requests
grep -E "upstream_response_time: [0-9]+\.[0-9]{2}" /usr/local/kong/logs/access.log | awk '$NF > 1.0'

# Use the Kong Admin API to inspect plugin counts per route
curl -s http://localhost:8001/routes | jq '.data[].plugins'

Performance tuning checklist:

Reduce the number of plugins per route — each plugin adds Lua execution overhead per request.
Switch from database mode to DB-less mode with declarative config for read-heavy, static deployments.
Increase nginx_worker_processes to match available CPU cores.
Tune upstream_keepalive settings to reuse connections to backend services.
Disable logging plugins like file-log or http-log in synchronous mode; use async logging where possible.

5. 502, 503, and 504 Gateway Errors

These errors indicate Kong could not successfully proxy the request to the upstream service.

502 Bad Gateway — Kong connected to the upstream but received an invalid response, or the upstream closed the connection prematurely.

# Verify the upstream service is reachable from Kong
curl -v http://<upstream_host>:<upstream_port>/health

# Check the service definition in Kong
curl -s http://localhost:8001/services/my-service | jq

# Test the service through Kong directly
curl -v http://localhost:8000/my-path -H "Host: example.com"

503 Service Unavailable — Often caused by the upstream being marked unhealthy by the health-checks plugin, or no targets available in an upstream object.

# Check upstream health
curl -s http://localhost:8001/upstreams/my-upstream/health | jq

# View targets
curl -s http://localhost:8001/upstreams/my-upstream/targets | jq

504 Gateway Timeout — The upstream did not respond within the configured timeout. Adjust the timeouts on the service:

# Update service timeouts via Admin API
curl -s -X PATCH http://localhost:8001/services/my-service \
  -d "connect_timeout=5000" \
  -d "write_timeout=60000" \
  -d "read_timeout=60000"

# Or via declarative config (YAML)
# services:
#   - name: my-service
#     url: http://upstream:8080
#     connect_timeout: 5000
#     write_timeout: 60000
#     read_timeout: 60000

6. Configuration Not Applying

A frequent complaint: "I added a route but requests still 404." This usually stems from mismatched routing criteria or caching.

# List all routes and their matching criteria
curl -s http://localhost:8001/routes | jq '.data[] | {name, paths, hosts, methods, strip_path}'

# Test routing with verbose headers
curl -v http://localhost:8000/my-path -H "Host: example.com"

# In DB-less mode, reload declarative config
curl -s -X POST http://localhost:8001/config \
  -F "config=@kong.yml"

Things to verify:

Route paths use regex or prefix matching — check regex_priority if multiple routes could match.
The strip_path setting affects what is forwarded upstream.
In hybrid mode (control plane / data plane), ensure the data plane is connected: curl localhost:8001/clustering/data_planes.
In DB-less mode, config only updates on a POST /config — editing the YAML file alone does nothing until reloaded.

7. SSL/TLS Certificate Issues

Clients seeing certificate warnings or SSL_ERROR responses usually have a misconfigured certificate in Kong.

# Verify the certificate Kong is serving
openssl s_client -connect localhost:8443 -servername example.com < /dev/null 2>/dev/null | openssl x509 -noout -subject -dates

# List uploaded certificates
curl -s http://localhost:8001/certificates | jq

# Upload a new certificate with its SNI
curl -s -X POST http://localhost:8001/certificates \
  -F "cert@/path/to/cert.pem" \
  -F "key@/path/to/key.pem" \
  -F "snis=example.com"

Common fixes:

Ensure the SNI matches the Host header or SNI sent by the client.
Include the full certificate chain (server + intermediate) in the uploaded PEM.
For mTLS, configure the ca_certificates endpoint and reference it in the service.
Check ssl_cert and ssl_cert_key in kong.conf for the default proxy listener certificate.

Best Practices for Kong Operations

Beyond fixing individual issues, a few operational habits dramatically reduce Kong incidents:

Use declarative config in DB-less mode when you do not need dynamic Admin API writes. It is faster, simpler to version-control, and eliminates database dependencies.
Run Kong behind a load balancer and deploy at least two instances for high availability. Never run a single Kong node in production.
Monitor the right metrics — export Prometheus metrics via the prometheus plugin and alert on kong_http_status{code="500"}, kong_upstream_target_health, and kong_data_plane_config_hash mismatches in hybrid mode.
Centralize logs — ship error.log and access logs to ELK, Loki, or Datadog. Tag logs with route and service names for fast filtering.
Pin Kong versions and test upgrades in staging. Plugin APIs and PDK behavior change between major versions.
Limit Admin API exposure — never expose port 8001 to the public internet. Bind it to localhost or a private network, and add an authentication plugin.
Use health checks and circuit breakers — configure active health checks on upstreams so Kong automatically removes failing targets.

Conclusion

Troubleshooting Kong API Gateway effectively comes down to three habits: reading the error log carefully, understanding the request lifecycle from client through Kong to upstream, and verifying configuration through the Admin API rather than assumptions. Most Kong issues fall into a small set of categories — startup failures, database connectivity, plugin errors, upstream timeouts, and routing mismatches — and each has a predictable diagnostic path. By combining the commands and techniques in this tutorial with solid monitoring and declarative configuration practices, you can keep Kong stable and resolve incidents quickly when they occur.