Reverse Proxy Setup for AI Agent APIs: Caddy and Nginx

What Is a Reverse Proxy for AI Agent APIs?

A reverse proxy sits between client applications and your backend AI agent APIs. Unlike a forward proxy that protects clients, a reverse proxy intercepts requests headed for your servers and forwards them intelligently—handling TLS termination, load balancing, caching, rate limiting, and request routing before they ever touch your application logic. For AI agent APIs specifically, the reverse proxy becomes the gatekeeper for long-running streaming responses, sensitive API keys, and high-traffic inference endpoints.

Think of it as a traffic controller. When a request arrives at https://api.yourdomain.com/v1/chat/completions, the reverse proxy decides which backend instance receives it, strips sensitive headers, enforces rate limits, and manages the connection lifecycle—all transparently to both the client and the AI service behind it.

Why It Matters for AI Agents

AI agent APIs present unique challenges that a well-configured reverse proxy solves elegantly:

• Long-lived streaming connections — AI models often stream tokens via Server-Sent Events (SSE) or WebSockets. A reverse proxy must handle persistent connections without timing out prematurely or buffering chunks that should arrive in real time.
• API key protection — Your OpenAI, Anthropic, or custom model API keys should never be exposed to the browser. The reverse proxy injects them server-side while stripping them from client-visible traffic.
• High-latency tolerance — Some inference calls take 30-120 seconds. Default proxy timeouts will kill these requests. You need explicit timeout configuration.
• Rate limiting and cost control — Unrestricted access to expensive LLM endpoints can lead to runaway bills. Rate limiting at the proxy layer catches abuse before it reaches the billing meter.
• SSL/TLS termination — Modern AI APIs require HTTPS. The reverse proxy handles certificates so your backend services can remain simple internal HTTP servers.
• Load balancing across model instances — When running local models like Llama or Mistral, you may have multiple GPU nodes. The proxy distributes inference requests evenly.

Caddy: Automatic HTTPS and Simplicity

Caddy is a modern web server written in Go that provisions TLS certificates automatically via Let's Encrypt. Its configuration syntax is remarkably concise, making it ideal for developers who want a reverse proxy that "just works" without wrestling with certificate renewal scripts or arcane config files.

Installing Caddy

On most platforms, you can install Caddy with a single command. Here's the approach for common environments:

# Ubuntu/Debian
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

# macOS (Homebrew)
brew install caddy

# Docker
docker pull caddy:latest

Basic Caddy Reverse Proxy Configuration

Caddy uses a file called Caddyfile. Here's a minimal reverse proxy that forwards requests to a local AI agent API while adding an API key header server-side:

# /etc/caddy/Caddyfile
api.example.com {
    # Automatically obtains and renews TLS certificate
    reverse_proxy /v1/* localhost:8080 {
        header_up Authorization "Bearer sk-your-secret-api-key"
        header_up X-Forwarded-For {remote_host}
    }

    # Block direct access to anything else
    respond / "Access restricted" 403
}

This configuration does several things at once: it obtains an HTTPS certificate for api.example.com, proxies all requests under /v1/ to a backend on port 8080, injects the Authorization header so the client never sees the API key, and rejects everything else with a 403.

Advanced Caddy Configuration for AI Streaming APIs

AI agent streaming requires careful handling of Server-Sent Events. Caddy's default settings may buffer responses, which defeats the purpose of token-by-token streaming. Here's how to configure it properly:

api.example.com {
    reverse_proxy /v1/chat/completions localhost:8080 {
        # Critical: disable buffering for streaming endpoints
        flush_interval -1
        
        # Pass through SSE headers untouched
        header_up Authorization "Bearer sk-your-secret-api-key"
        header_up X-Forwarded-For {remote_host}
        header_down -Cache-Control
        header_down Cache-Control "no-cache, no-transform"
        header_down X-Accel-Buffering "no"
    }

    # Health check endpoint
    reverse_proxy /health localhost:8080/health {
        header_up Authorization "Bearer sk-your-secret-api-key"
    }

    # Default deny
    respond / "Endpoint not available" 404
}

The flush_interval -1 directive tells Caddy to flush response buffers immediately—essential for SSE streams where each token arrives as a separate chunk. Setting X-Accel-Buffering: no ensures downstream proxies (like Nginx in front of Caddy) also don't buffer.

Load Balancing with Caddy

When running multiple inference instances, Caddy distributes requests across backends with built-in load balancing:

api.example.com {
    reverse_proxy /v1/* {
        # Round-robin across 3 GPU nodes
        to node1.internal:8080 node2.internal:8080 node3.internal:8080
        
        # Health checks: mark backend down after 2 failures for 30s
        lb_try_duration 10s
        lb_try_interval 250ms
        
        # Streaming-friendly flush
        flush_interval -1
        
        header_up Authorization "Bearer sk-your-secret-api-key"
    }
}

Caddy supports multiple load balancing policies via the lb_policy directive: first, random, round_robin, least_conn, and ip_hash. For AI inference where requests may be long-running, least_conn often works best:

reverse_proxy /v1/* {
    lb_policy least_conn
    to node1.internal:8080 node2.internal:8080 node3.internal:8080
    flush_interval -1
}

Rate Limiting and Timeouts for AI APIs

Caddy's rate limiting plugin (available via caddy-rate-limit module) prevents abuse. You'll need to build Caddy with this module or use a pre-built version:

# Build Caddy with rate limiting module
xcaddy build --with github.com/mholt/caddy-rate-limit

# Caddyfile configuration
api.example.com {
    # Rate limit: 30 requests per minute per client IP
    rate_limit {
        zone dynamic_zone {
            key {remote_host}
            events 30
            window 1m
        }
        action {
            respond "Rate limit exceeded. Please slow down." 429
        }
    }

    reverse_proxy /v1/* localhost:8080 {
        flush_interval -1
        header_up Authorization "Bearer sk-your-secret-api-key"
        
        # Transport timeout for long inference calls
        transport http {
            read_timeout 120s
            write_timeout 120s
        }
    }
}

The read_timeout and write_timeout inside the transport block are crucial—without them, Caddy defaults to 30s timeouts that will kill long inference requests mid-stream.

Nginx: Battle-Tested Performance and Flexibility

Nginx has been the industry-standard reverse proxy for over a decade. It excels at handling massive concurrent connections with minimal resource consumption. For AI agent APIs, Nginx offers fine-grained control over buffering, timeouts, and streaming behavior that's essential for production deployments.

Installing Nginx

# Ubuntu/Debian
sudo apt update && sudo apt install nginx

# CentOS/RHEL
sudo yum install epel-release && sudo yum install nginx

# macOS (Homebrew)
brew install nginx

# Verify installation
nginx -v

Basic Nginx Reverse Proxy Configuration

Nginx configuration lives in /etc/nginx/conf.d/ (or /etc/nginx/sites-available/ on Debian-based systems). Here's a minimal reverse proxy for an AI agent API:

# /etc/nginx/conf.d/ai-agent-api.conf
server {
    listen 80;
    server_name api.example.com;

    location /v1/ {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization "Bearer sk-your-secret-api-key";
        
        # Hide the API key from client responses
        proxy_hide_header Authorization;
    }

    location / {
        return 403 "Access restricted";
    }
}

This forwards requests under /v1/ to the backend while injecting the Authorization header. The proxy_hide_header directive ensures the backend's Authorization header doesn't leak back to clients.

SSL/TLS Configuration with Let's Encrypt

For production, you need HTTPS. Pair Nginx with Certbot for automatic certificate renewal:

# Install Certbot
sudo apt install certbot python3-certbot-nginx

# Obtain certificate (interactive)
sudo certbot --nginx -d api.example.com

# The resulting Nginx config (auto-modified by Certbot):
server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    
    # Modern SSL settings
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_stapling on;
    ssl_stapling_verify on;
    
    add_header Strict-Transport-Security "max-age=63072000" always;

    location /v1/ {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Authorization "Bearer sk-your-secret-api-key";
        proxy_hide_header Authorization;
    }
}

# HTTP redirect to HTTPS
server {
    listen 80;
    server_name api.example.com;
    return 301 https://$server_name$request_uri;
}

Streaming Responses and SSE: The Critical Configuration

This is where many AI agent deployments fail. Nginx buffers responses by default, which breaks SSE streaming. You must disable buffering for streaming endpoints:

# /etc/nginx/conf.d/ai-agent-streaming.conf
server {
    listen 443 ssl http2;
    server_name api.example.com;

    # ... SSL certificates as above ...

    # Dedicated location for streaming chat completions
    location /v1/chat/completions {
        proxy_pass http://backend-cluster;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization "Bearer sk-your-secret-api-key";
        proxy_hide_header Authorization;

        # === STREAMING CRITICAL SETTINGS ===
        
        # Disable response buffering entirely
        proxy_buffering off;
        
        # Disable request buffering (stream request body to backend immediately)
        proxy_request_buffering off;
        
        # Keep the connection alive for SSE streams
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding on;
        
        # Prevent Nginx from buffering chunks
        proxy_cache off;
        
        # Disable gzip on streaming responses (prevents buffering)
        gzip off;
        
        # Instruct Nginx to forward chunks immediately
        proxy_read_timeout 120s;
        proxy_send_timeout 120s;
        
        # Long timeout for inference
        proxy_connect_timeout 10s;
        
        # Disable buffering headers
        proxy_buffers 0;
        proxy_buffer_size 4k;
        
        # Add headers to prevent downstream buffering
        add_header X-Accel-Buffering "no";
        add_header Cache-Control "no-cache, no-transform";
    }

    # Non-streaming endpoints can use buffering
    location /v1/models {
        proxy_pass http://backend-cluster;
        proxy_set_header Authorization "Bearer sk-your-secret-api-key";
        proxy_hide_header Authorization;
        # Buffering is fine for short JSON responses
    }
}

The key settings are proxy_buffering off and gzip off. Without these, Nginx waits for the entire response body before forwarding—defeating the purpose of streaming token-by-token responses. Setting X-Accel-Buffering: no also tells any upstream proxies to pass data through immediately.

Load Balancing Across Multiple AI Inference Nodes

Define an upstream block to distribute requests across your GPU servers:

# /etc/nginx/conf.d/upstream.conf
upstream backend-cluster {
    # Use least_conn for long-running inference requests
    least_conn;
    
    server 10.0.1.101:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.102:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.103:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.104:8080 backup;  # backup node
    
    # Keep connections alive between Nginx and backends
    keepalive 32;
    keepalive_timeout 60s;
    keepalive_requests 100;
}

server {
    listen 443 ssl http2;
    server_name api.example.com;
    
    # ... SSL config ...

    location /v1/ {
        proxy_pass http://backend-cluster;
        proxy_set_header Host $host;
        proxy_set_header Authorization "Bearer sk-your-secret-api-key";
        proxy_hide_header Authorization;
        proxy_buffering off;
        proxy_read_timeout 120s;
        gzip off;
    }
}

The least_conn strategy sends new requests to the backend with the fewest active connections—ideal for AI inference where request durations vary wildly. The backup server only receives traffic when all primary nodes are down.

Rate Limiting for API Cost Control

Nginx's built-in rate limiting uses the limit_req module. Here's a sophisticated setup that applies different limits to different endpoints:

# /etc/nginx/conf.d/rate-limits.conf

# Define rate limit zones in the http block
# (Place in /etc/nginx/nginx.conf under the http { } section)
http {
    # General API rate limit: 60 requests per minute per IP
    limit_req_zone $binary_remote_addr zone=api_general:10m rate=60r/m;
    
    # Stricter limit for chat completions: 10 requests per minute per IP
    limit_req_zone $binary_remote_addr zone=chat_limit:10m rate=10r/m;
    
    # Limit for admin endpoints: 5 requests per minute per IP
    limit_req_zone $binary_remote_addr zone=admin_limit:10m rate=5r/m;
    
    # ... other http-level config ...
}

# In your server block:
server {
    listen 443 ssl http2;
    server_name api.example.com;

    location /v1/chat/completions {
        # Apply strict rate limit with burst of 3 requests
        limit_req zone=chat_limit burst=3 nodelay;
        limit_req_status 429;
        
        proxy_pass http://backend-cluster;
        proxy_buffering off;
        proxy_read_timeout 120s;
        gzip off;
        # ... other proxy settings ...
    }

    location /v1/models {
        limit_req zone=api_general burst=10 nodelay;
        limit_req_status 429;
        
        proxy_pass http://backend-cluster;
        # Buffering ok for short responses
    }

    location /v1/admin/ {
        limit_req zone=admin_limit burst=1 nodelay;
        limit_req_status 429;
        
        proxy_pass http://backend-cluster;
    }
}

The burst parameter allows temporary spikes above the rate limit, while nodelay applies the rate limit immediately without delaying requests. A 429 status code signals clients to back off.

Buffering, Timeouts, and Connection Tuning

For AI agent APIs, default Nginx timeouts are too aggressive. Here's a complete tuning reference:

server {
    listen 443 ssl http2;
    server_name api.example.com;

    location /v1/chat/completions {
        proxy_pass http://backend-cluster;

        # === TIMEOUTS FOR LONG INFERENCE ===
        
        # Time to establish connection to backend
        proxy_connect_timeout 15s;
        
        # Time waiting for backend to start sending response
        proxy_read_timeout 180s;    # 3 minutes for long completions
        
        # Time waiting to send data to backend
        proxy_send_timeout 60s;
        
        # === BUFFERING ===
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_buffers 0;
        proxy_buffer_size 4k;       # Minimal buffer for headers only
        
        # === CONNECTION KEEPALIVE ===
        proxy_http_version 1.1;
        proxy_set_header Connection '';
        
        # === MAX BODY SIZE ===
        client_max_body_size 10m;   # Allow large prompt payloads
        
        # === HEADERS ===
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization "Bearer sk-your-secret-api-key";
        proxy_hide_header Authorization;
        
        # === GZIP (off for streaming) ===
        gzip off;
        
        # === ADD UPSTREAM SIGNALS ===
        add_header X-Accel-Buffering "no";
        add_header Cache-Control "no-cache, no-transform, no-store";
    }
}

Best Practices for AI Agent API Reverse Proxies

Drawing from real-world deployment experience, here are the practices that prevent the most common production incidents:

1. Separate Streaming and Non-Streaming Locations

Never apply the same proxy configuration to all endpoints. Streaming endpoints need proxy_buffering off and gzip off; non-streaming endpoints benefit from buffering and compression. Create distinct location blocks:

# Streaming: no buffering, long timeouts
location /v1/chat/completions { ... }

# Non-streaming: buffering enabled, shorter timeouts
location /v1/models { ... }
location /v1/embeddings { ... }

2. Never Expose API Keys to Clients

Always inject API keys server-side using proxy_set_header (Nginx) or header_up (Caddy) and strip them from responses with proxy_hide_header (Nginx). Client applications should authenticate to your proxy using their own tokens or session cookies—never raw LLM provider keys.

3. Set Realistic Timeouts Based on Model Latency

Measure your actual inference latency at the 95th percentile and set proxy_read_timeout to at least 2× that value. For GPT-4-class models, 120-180 seconds is common. For local Llama models on consumer GPUs, you may need 300 seconds or more.

4. Implement Tiered Rate Limiting

Apply different rate limits per endpoint. Chat completions are expensive—limit them strictly. Model listing and health checks can be more lenient. Use burst parameters to allow short spikes while protecting against sustained abuse.

5. Monitor and Log Proxy-Level Metrics

Track 429 rate limit hits, 502 upstream errors, and streaming connection drops. These proxy-level metrics often reveal issues before they manifest as application errors:

# Caddy: enable access logs
api.example.com {
    log {
        output file /var/log/caddy/access.log
        format json
    }
    # ... proxy config ...
}

# Nginx: access log with timing
server {
    access_log /var/log/nginx/ai-api-access.log combined;
    
    location /v1/ {
        # Add timing headers
        add_header X-Response-Time $request_time;
        # ... proxy config ...
    }
}

6. Health Checks and Graceful Degradation

Both Caddy and Nginx support active health checks. Configure them to detect stuck inference workers:

# Nginx upstream with health checks (requires nginx-plus or ngx_healthcheck module)
# For open-source Nginx, use passive checks:
upstream backend-cluster {
    least_conn;
    server 10.0.1.101:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.102:8080 max_fails=3 fail_timeout=30s;
}

# Caddy active health checks built-in:
reverse_proxy /v1/* {
    to node1.internal:8080 node2.internal:8080
    health_uri /health
    health_interval 10s
    health_timeout 5s
}

7. Use HTTP/2 and Connection Keepalive

HTTP/2 multiplexing reduces latency for multiple concurrent requests. Enable it in both proxy and backend connections:

# Nginx: enable HTTP/2 on listen directive
listen 443 ssl http2;

# Caddy: HTTP/2 is enabled by default when using HTTPS

8. Protect Against SSRF and Host Header Injection

AI agents sometimes follow URLs from model outputs. Your reverse proxy should validate the Host header and reject requests with suspicious values:

# Nginx: validate Host header
server {
    listen 443 ssl http2;
    server_name api.example.com;
    
    # Reject requests with wrong Host header
    if ($host != "api.example.com") {
        return 403;
    }
    # ... rest of config ...
}

# Caddy: host matching is built-in
api.example.com {
    # Automatically rejects mismatched hosts
}

Conclusion

A reverse proxy is not optional for production AI agent APIs—it's the foundation for security, reliability, and cost control. Caddy offers a zero-friction path with automatic TLS and a clean configuration syntax that makes streaming setup intuitive. Nginx provides battle-tested performance and the granular control needed for high-traffic deployments. Whichever you choose, the critical requirements remain the same: disable response buffering for SSE streams, inject API keys server-side, configure realistic timeouts for inference latency, implement tiered rate limiting, and monitor proxy-level metrics. With these patterns in place, your AI agent APIs will serve clients reliably while keeping your model provider keys secure and your infrastructure costs predictable.