Securing Local LLM APIs with API Gateways

Introduction to Securing Local LLM APIs with API Gateways

Running large language models (LLMs) locally has become increasingly common as developers seek privacy, cost control, and reduced latency. Tools like Ollama, vLLM, llama.cpp, and LM Studio make it easy to spin up a local inference server. However, these servers are typically designed for development convenience, not production security. They often expose unauthenticated HTTP endpoints with no rate limiting, no logging, and no access controls. This is where an API gateway becomes essential.

An API gateway sits between your clients (web apps, mobile apps, internal services) and your local LLM server, acting as a policy enforcement point. It handles authentication, authorization, rate limiting, request validation, logging, and observability — all without modifying the LLM server itself. In this tutorial, we'll explore how to secure a local LLM API using popular gateway solutions.

Why Local LLM APIs Need Protection

Most local LLM servers expose a simple REST API. For example, Ollama listens on http://localhost:11434 by default and accepts requests like POST /api/generate with no authentication. While this is fine for single-user development, it becomes dangerous when:

The LLM server is exposed on a network — anyone who can reach the port can use your compute resources, run up costs, or extract model weights.
Multiple applications share the same LLM — you need to track usage per client and prevent one app from monopolizing the GPU.
You expose the LLM to end users — even indirectly through a chatbot — you need prompt injection defenses, content filtering, and abuse prevention.
Compliance is required — audit logs, data retention policies, and access controls may be legally mandated.

Without a gateway, you'd have to build all of this into every application that calls the LLM. A central gateway solves this once, for all clients.

Architecture Overview

The secured architecture follows a simple flow:

Client App → API Gateway (auth, rate limit, logging) → Local LLM Server (Ollama/vLLM/llama.cpp)

The gateway terminates the client connection, validates the request, enforces policies, and then forwards an authorized request to the local LLM server. The LLM server itself can be bound to localhost or a private network interface, ensuring it is never directly reachable by external clients.

Choosing an API Gateway

Several gateway options work well for securing local LLM APIs:

Kong Gateway — A mature, plugin-rich gateway with excellent Lua plugin support. Great for enterprise deployments.
Traefik — A modern reverse proxy with built-in Let's Encrypt, middleware chains, and Docker integration. Lightweight and fast.
Envoy — A high-performance proxy with powerful filtering. More complex to configure but extremely flexible.
NGINX with Lua — The classic choice. Combine with OpenResty for dynamic scripting.
LiteLLM Proxy — Purpose-built for LLM APIs, with built-in key management, budget controls, and model routing.

For this tutorial, we'll use Traefik for general HTTP protection and LiteLLM Proxy for LLM-specific features, giving you both perspectives.

Setting Up a Local LLM Server

First, let's start a local Ollama server that we'll protect. Install Ollama and pull a model:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.2

# Verify the API is running (unsecured)
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Hello, world!",
  "stream": false
}'

By default, Ollama binds to 127.0.0.1:11434. If you need it on a private network, set OLLAMA_HOST but never expose it publicly without a gateway in front.

Approach 1: Securing with Traefik

Basic Traefik Configuration

We'll use Docker Compose to run Traefik alongside Ollama. Create a docker-compose.yml:

version: "3.8"

services:
  traefik:
    image: traefik:v3.1
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:8080"
      - "--entrypoints.websecure.address=:8443"
      - "--experimental.plugins.basicauth.modulename=github.com/traefik/plugindemo"
    ports:
      - "8080:8080"
      - "8443:8443"
      - "8081:8080"  # Traefik dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - llm-net

  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.ollama.rule=PathPrefix(`/api`)"
      - "traefik.http.routers.ollama.entrypoints=web"
      - "traefik.http.routers.ollama.middlewares=auth,rate-limit"
      - "traefik.http.services.ollama.loadbalancer.server.port=11434"
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$xyz123$$encryptedpasswordhere"
      - "traefik.http.middlewares.rate-limit.ratelimit.average=10"
      - "traefik.http.middlewares.rate-limit.ratelimit.burst=20"
    networks:
      - llm-net

volumes:
  ollama-data:

networks:
  llm-net:
    driver: bridge

Note the key security labels: the auth middleware enforces HTTP Basic Authentication, and the rate-limit middleware caps requests at 10 per second with a burst of 20. The Ollama container is only reachable through Traefik because it's on an internal Docker network with no published ports.

Generating Basic Auth Credentials

Use htpasswd to generate the encrypted credential string:

htpasswd -nb admin "MySecurePassword123"
# Output: admin:$apr1$abcd1234$EfGhIjKlMnOpQrStUvWx

Replace all $ characters with $$ in the Docker Compose file to escape them properly in YAML.

Testing the Secured Endpoint

# Without auth — should return 401
curl http://localhost:8080/api/generate -d '{"model":"llama3.2","prompt":"hi","stream":false}'

# With auth — should work
curl -u admin:MySecurePassword123 http://localhost:8080/api/generate \
  -d '{"model":"llama3.2","prompt":"hi","stream":false}'

Adding API Key Authentication with a Custom Middleware

Basic auth is fine for internal tools, but API keys are more practical for programmatic clients. Traefik doesn't have a built-in API key middleware, but you can use a plugin like traefik-real-ip or write a small forward-auth service. Here's a lightweight forward-auth approach using a tiny Go service:

// auth-service/main.go
package main

import (
    "crypto/subtle"
    "net/http"
    "os"
    "strings"
)

var validKeys = map[string]string{
    "sk-app-abc123": "frontend-app",
    "sk-app-def456": "analytics-service",
}

func handler(w http.ResponseWriter, r *http.Request) {
    auth := r.Header.Get("Authorization")
    if !strings.HasPrefix(auth, "Bearer ") {
        w.WriteHeader(http.StatusUnauthorized)
        return
    }
    key := strings.TrimPrefix(auth, "Bearer ")
    
    // Constant-time comparison to prevent timing attacks
    for validKey := range validKeys {
        if subtle.ConstantTimeCompare([]byte(key), []byte(validKey)) == 1 {
            // Optionally inject client identity header for downstream logging
            w.Header().Set("X-Client-Name", validKeys[validKey])
            w.WriteHeader(http.StatusOK)
            return
        }
    }
    w.WriteHeader(http.StatusUnauthorized)
}

func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8081"
    }
    http.HandleFunc("/", handler)
    http.ListenAndServe(":"+port, nil)
}

Wire this into Traefik using the forwardauth middleware:

# Add to docker-compose.yml labels on the ollama service:
- "traefik.http.middlewares.apikey.forwardauth.address=http://auth-service:8081/"
- "traefik.http.middlewares.apikey.forwardauth.authResponseHeaders=X-Client-Name"
# Replace the auth middleware reference:
- "traefik.http.routers.ollama.middlewares=apikey,rate-limit"

Approach 2: Using LiteLLM Proxy for LLM-Specific Security

LiteLLM Proxy is purpose-built for managing LLM API access. It provides virtual keys, per-key budgets, model access control, and usage tracking out of the box. This is often a better fit than a generic gateway when your primary workload is LLM inference.

Installing and Configuring LiteLLM Proxy

Create a litellm_config.yaml:

model_list:
  - model_name: llama3.2
    litellm_params:
      model: ollama/llama3.2
      api_base: http://ollama:11434

general_settings:
  master_key: sk-master-keep-this-secret
  database_url: "sqlite:///litellm.db"

litellm_settings:
  max_budget: 100.0          # $100 total budget
  budget_duration: "monthly"
  drop_params: true
  request_timeout: 120
  num_retries: 2

Run it with Docker:

version: "3.8"

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    environment:
      - LITELLM_MASTER_KEY=sk-master-keep-this-secret
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    networks:
      - llm-net

  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama-data:/root/.ollama
    networks:
      - llm-net

volumes:
  ollama-data:

networks:
  llm-net:

Creating Virtual API Keys

LiteLLM lets you create per-client keys with individual budgets and rate limits. Use the master key to create a virtual key:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-master-keep-this-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["llama3.2"],
    "max_budget": 10.0,
    "budget_duration": "1mo",
    "rpm_limit": 60,
    "tpm_limit": 10000,
    "metadata": {"team": "frontend"}
  }'

The response includes a generated key like sk-vk-abc123.... This key can only access the llama3.2 model, is capped at $10/month, and limited to 60 requests per minute.

Calling the LLM Through LiteLLM

curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer sk-vk-abc123..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Explain quantum computing in one sentence."}]
  }'

LiteLLM translates this OpenAI-compatible request into Ollama's native format, enforces the key's budget and rate limits, logs the usage, and returns the response. Clients can use the standard OpenAI SDK, making integration trivial.

Viewing Usage and Logs

# List all keys and their usage
curl http://localhost:4000/key/list \
  -H "Authorization: Bearer sk-master-keep-this-secret"

# Get spend for a specific key
curl http://localhost:4000/key/info?key=sk-vk-abc123... \
  -H "Authorization: Bearer sk-master-keep-this-secret"

Adding TLS/HTTPS

Even on a local network, TLS prevents credential interception. With Traefik, you can use self-signed certificates for internal use or Let's Encrypt for public domains. Here's a Traefik configuration with automatic self-signed certs:

# Add to Traefik command flags:
- "--entrypoints.websecure.address=:8443"
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
- "--providers.file.directory=/etc/traefik/dynamic"

Create a dynamic config file for the self-signed cert:

# /etc/traefik/dynamic/tls.yml
tls:
  certificates:
    - certFile: /certs/cert.pem
      keyFile: /certs/key.pem
  stores:
    default:
      defaultCertificate:
        certFile: /certs/cert.pem
        keyFile: /certs/key.pem

Generate the certificate with OpenSSL:

openssl req -x509 -newkey rsa:4096 -nodes -keyout key.pem \
  -out cert.pem -days 365 -subj "/CN=llm-gateway.local" \
  -addext "subjectAltName=DNS:llm-gateway.local,IP:127.0.0.1"

Request Validation and Content Filtering

Beyond authentication, you may want to validate request payloads and filter content. A simple middleware can reject requests with excessively long prompts or blocked terms. Here's a Python-based forward-auth service that checks prompt length:

# content_filter.py
from fastapi import FastAPI, Request, Response
import json

app = FastAPI()

MAX_PROMPT_TOKENS = 4000
BLOCKED_PATTERNS = ["ignore previous instructions", "system prompt:"]

@app.post("/check")
async def check_request(request: Request):
    body = await request.json()
    
    # Extract prompt text from various API formats
    prompt = ""
    if "prompt" in body:
        prompt = body["prompt"]
    elif "messages" in body:
        prompt = " ".join(m.get("content", "") for m in body["messages"])
    
    # Length check (rough token estimate)
    estimated_tokens = len(prompt) // 4
    if estimated_tokens > MAX_PROMPT_TOKENS:
        return Response(
            status_code=413,
            content=json.dumps({"error": "Prompt exceeds maximum length"}),
            media_type="application/json"
        )
    
    # Pattern check
    prompt_lower = prompt.lower()
    for pattern in BLOCKED_PATTERNS:
        if pattern in prompt_lower:
            return Response(
                status_code=400,
                content=json.dumps({"error": f"Blocked pattern detected: {pattern}"}),
                media_type="application/json"
            )
    
    return Response(status_code=200)

Deploy this as a sidecar container and reference it as an additional forward-auth middleware in Traefik, chained before the API key check.

Logging and Observability

Security requires visibility. Configure structured logging to track who called what, when, and with what result. Traefik supports access logs in JSON format:

# Traefik command flags for logging:
- "--accesslog=true"
- "--accesslog.format=json"
- "--accesslog.fields.headers.defaultmode=keep"
- "--accesslog.fields.headers.names.Authorization=redact"
- "--log.level=INFO"
- "--log.format=json"

For LiteLLM, enable callback logging to send usage data to a destination of your choice:

# litellm_config.yaml additions:
litellm_settings:
  callbacks: custom_logger  # or langfuse, langsmith, etc.
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: "pk-lf-..."
  LANGFUSE_SECRET_KEY: "sk-lf-..."
  LANGFUSE_HOST: "https://cloud.langfuse.com"

Best Practices

Never expose the raw LLM server directly. Bind it to localhost or a private Docker network. Only the gateway should have network access to it.
Use per-client API keys, not shared credentials. This enables per-client rate limits, budgets, and audit trails. Rotate keys regularly.
Set realistic rate limits and budgets. LLM inference is expensive. Without limits, a single misbehaving client can exhaust GPU resources. Set both RPM (requests per minute) and TPM (tokens per minute) limits.
Always use TLS. Even on internal networks, TLS prevents credential sniffing and man-in-the-middle attacks. Use self-signed certificates if you don't have a public domain.
Log everything, but redact secrets. Log the client identity, model, token counts, latency, and status codes. Never log API keys or full prompt contents unless you have a data retention policy that permits it.
Validate and sanitize inputs. Enforce maximum prompt lengths, reject malformed JSON, and consider content filtering for known prompt injection patterns.
Implement graceful degradation. When the LLM server is overloaded or down, the gateway should return a clear error rather than hanging. Set request timeouts at the gateway level.
Keep the gateway updated. Gateway software receives security patches regularly. Pin to specific versions in production and update on a schedule.
Separate admin and user access. The master key for LiteLLM or the Traefik dashboard should never be shared with application developers. Use a secrets manager like Vault or AWS Secrets Manager.
Monitor for anomalies. Sudden spikes in request volume, unusual prompt patterns, or budget exhaustion can indicate abuse or a compromised key. Set up alerts.

Conclusion

Securing a local LLM API with an API gateway is a straightforward but essential step in moving from experimentation to production. Whether you choose a general-purpose gateway like Traefik for fine-grained HTTP-level control or a purpose-built solution like LiteLLM Proxy for LLM-specific features such as virtual keys and budget tracking, the principles remain the same: authenticate every request, enforce rate limits and budgets, validate inputs, log activity, and never expose the raw inference server directly. By centralizing these concerns in a gateway, you protect your compute resources, gain visibility into usage, and create a clean abstraction that lets you swap models or providers without touching client code. Start with authentication and rate limiting, then layer in content filtering, TLS, and observability as your needs evolve.