Introduction to LiteLLM as an AI Gateway
As organizations scale their adoption of large language models (LLMs), they quickly run into a familiar set of problems: multiple providers (OpenAI, Anthropic, Azure, Cohere, Mistral) each with their own SDKs, inconsistent authentication, no centralized logging, no rate limiting, and no way to control costs. LiteLLM solves these problems by acting as a unified proxy and gateway in front of every model you use.
LiteLLM is an open-source project that provides a consistent OpenAI-compatible interface across 100+ LLM providers. Beyond simple translation, its proxy server mode turns it into a full-fledged AI gateway with virtual keys, budget controls, usage tracking, caching, fallbacks, and observability. This tutorial walks through building a production-grade secure AI gateway using LiteLLM.
Why You Need a Secure AI Gateway
When developers call provider APIs directly from their applications, several risks emerge:
- Credential sprawl: API keys end up scattered across services, environment variables, and repositories.
- No spend controls: A buggy prompt loop can rack up thousands of dollars in minutes.
- No audit trail: There is no central record of who called which model with what payload.
- Vendor lock-in: Switching providers means rewriting application code.
- No abuse protection: End users can trigger unlimited expensive calls.
A gateway like LiteLLM sits between your applications and the model providers. Applications authenticate to the gateway with scoped virtual keys, and the gateway handles provider authentication, routing, logging, and policy enforcement. This pattern is sometimes called the "AI API gateway" or "LLM firewall" pattern, and it has become a standard piece of production AI infrastructure.
Architecture Overview
The gateway deployment has three main components: the LiteLLM proxy server, a PostgreSQL database for storing keys and usage data, and your upstream applications. Optionally, you add Redis for caching and rate limiting, and an observability backend like Langfuse or Prometheus for tracing.
+-------------------+ +------------------+ +------------------+
| Application A | | Application B | | Application C |
| (virtual key A) | | (virtual key B) | | (virtual key C) |
+---------+---------+ +---------+--------+ +---------+--------+
| | |
+------------+---------------+---------------------------+
|
+------+------+
| LiteLLM | <-- policies, budgets, logging
| Proxy |
+------+------+
|
+------------+------------+------------+
| | | |
+---+---+ +----+----+ +----+----+ +---+---+
|OpenAI | |Anthropic| | Azure | |Mistral|
+-------+ +---------+ +---------+ +-------+
Installing and Running the Proxy
The simplest way to run LiteLLM is with Docker. First, create a configuration file that defines which models your gateway will expose and how it should route them.
The Config File
Create a file named config.yaml:
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: fallback-chat
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
fallbacks:
- claude-sonnet
litellm_settings:
drop_params: true
request_timeout: 30
num_retries: 2
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
enforce_user_param: true
The os.environ/ prefix tells LiteLLM to read values from environment variables rather than hardcoding secrets into the config file. The master_key is the administrative credential used to create virtual keys and manage the gateway. The fallback-chat entry demonstrates routing: if the primary model fails, LiteLLM automatically retries on Claude.
Running with Docker Compose
Create a docker-compose.yml that brings up the proxy alongside PostgreSQL and Redis:
version: "3.9"
services:
litellm:
image: ghcr.io/berriai/litellm:main-latest
ports:
- "4000:4000"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
- DATABASE_URL=postgresql://litellm:litellm@db:5432/litellm
- REDIS_HOST=redis
- REDIS_PORT=6379
volumes:
- ./config.yaml:/app/config.yaml
command: ["--config", "/app/config.yaml", "--port", "4000"]
depends_on:
- db
- redis
db:
image: postgres:16
environment:
- POSTGRES_USER=litellm
- POSTGRES_PASSWORD=litellm
- POSTGRES_DB=litellm
volumes:
- pgdata:/var/lib/postgresql/data
redis:
image: redis:7-alpine
volumes:
pgdata:
Start the stack:
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export LITELLM_MASTER_KEY="sk-1234-master"
docker compose up -d
The proxy is now listening on port 4000 and exposes an OpenAI-compatible API at http://localhost:4000/v1.
Creating Virtual Keys with Budgets
Instead of handing provider API keys to every application, you create virtual keys through the gateway. Each virtual key can have its own budget, rate limits, model restrictions, and metadata. This is the core of the security model.
Creating a Key for a Specific Application
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer sk-1234-master" \
-H "Content-Type: application/json" \
-d '{
"key_alias": "mobile-app-prod",
"max_budget": 50.0,
"budget_duration": "1mo",
"rpm_limit": 100,
"tpm_limit": 100000,
"models": ["gpt-4o", "claude-sonnet"],
"metadata": {
"team": "mobile",
"environment": "production"
}
}'
The response includes the generated key, which you store in your application's secret manager:
{
"key": "sk-litellm-7f3a9b2c1e8d4f6a",
"expires": null,
"user_id": null,
"max_budget": 50.0,
"budget_duration": "1mo"
}
This key can only call gpt-4o and claude-sonnet, is capped at $50 per month, and is limited to 100 requests per minute. If the budget is exceeded, the gateway returns a 429 error instead of forwarding the call to the provider, preventing runaway costs.
Calling the Gateway from Your Application
Because the gateway is OpenAI-compatible, you can use the standard OpenAI SDK with the base URL pointed at LiteLLM:
from openai import OpenAI
client = OpenAI(
api_key="sk-litellm-7f3a9b2c1e8d4f6a",
base_url="http://localhost:4000/v1"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain rate limiting in one sentence."}
]
)
print(response.choices[0].message.content)
No application code changes are needed when you switch providers later — you only update the config file on the gateway.
Enforcing User-Level Tracking
For multi-tenant applications, you want to track spend per end user, not just per application key. LiteLLM supports this through the user parameter. The enforce_user_param: true setting in the config requires every request to include a user identifier.
response = client.chat.completions.create(
model="gpt-4o",
user="user_abc123",
messages=[
{"role": "user", "content": "Summarize this article."}
]
)
You can create user-level budgets separately:
curl -X POST http://localhost:4000/user/new \
-H "Authorization: Bearer sk-1234-master" \
-H "Content-Type: application/json" \
-d '{
"user_id": "user_abc123",
"max_budget": 5.0,
"budget_duration": "1mo"
}'
Now user_abc123 is capped at $5 per month regardless of which application key they use.
Adding Caching and Rate Limiting
Caching reduces both latency and cost by returning identical responses for repeated prompts. Configure Redis-backed caching in your config file:
litellm_settings:
cache: true
cache_params:
type: redis
host: redis
port: 6379
router_settings:
redis_settings:
host: redis
port: 6379
Enable caching per request by passing the cache parameter:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is the capital of France?"}],
extra_body={"cache": {"no-cache": False}}
)
Rate limiting is enforced using the sliding window algorithm in Redis. The rpm_limit and tpm_limit set on virtual keys are enforced globally across all proxy instances, which is essential when running multiple replicas behind a load balancer.
Implementing Fallbacks and Reliability
Provider outages are inevitable. LiteLLM's fallback mechanism lets you define a chain of models to try in order. Extend the config to add a more robust routing strategy:
router_settings:
routing_strategy: usage-based-routing-v2
num_retries: 3
retry_after: 5
timeout: 30
allowed_fails: 3
cooldown_time: 60
model_list:
- model_name: chat-primary
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: chat-primary
litellm_params:
model: azure/gpt-4o
api_key: os.environ/AZURE_API_KEY
api_base: os.environ/AZURE_API_BASE
- model_name: chat-fallback
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
With this configuration, LiteLLM load-balances between OpenAI and Azure for the chat-primary model. If both fail, it falls back to Claude. The allowed_fails and cooldown_time settings implement circuit-breaking: after three failures, a deployment is taken out of rotation for 60 seconds.
Observability and Logging
A secure gateway must produce audit logs. LiteLLM logs every request to the database and can stream callbacks to external observability platforms. To integrate Langfuse for tracing, add the following to your config:
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
environment_variables:
LANGFUSE_PUBLIC_KEY: os.environ/LANGFUSE_PUBLIC_KEY
LANGFUSE_SECRET_KEY: os.environ/LANGFUSE_SECRET_KEY
LANGFUSE_HOST: os.environ/LANGFUSE_HOST
You can also query spend directly from the gateway's admin API:
curl -X GET "http://localhost:4000/spend/logs?start_date=2025-01-01&end_date=2025-01-31" \
-H "Authorization: Bearer sk-1234-master"
This returns itemized logs including the model called, tokens used, cost, latency, and the virtual key responsible. For production deployments, export these logs to your SIEM (Security Information and Event Management) system for long-term retention and anomaly detection.
Securing the Gateway Itself
Protect the Master Key
The master key has unrestricted access. Store it in a secrets manager like AWS Secrets Manager, HashiCorp Vault, or Doppler. Never commit it to version control or bake it into container images. Rotate it periodically and limit access to a small set of operators.
Use TLS Everywhere
Run the proxy behind a TLS-terminating reverse proxy such as Nginx, Traefik, or an AWS Application Load Balancer. All traffic between applications and the gateway, and between the gateway and providers, must be encrypted. Here is a minimal Nginx configuration:
server {
listen 443 ssl http2;
server_name ai-gateway.internal.example.com;
ssl_certificate /etc/ssl/certs/gateway.crt;
ssl_certificate_key /etc/ssl/private/gateway.key;
ssl_protocols TLSv1.2 TLSv1.3;
location / {
proxy_pass http://litellm:4000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 120s;
}
}
Restrict Network Access
The proxy should not be publicly accessible. Place it in a private subnet and allow traffic only from your application subnets via security groups. The PostgreSQL and Redis containers should never have ports exposed outside the Docker network.
Enable RBAC with Teams
LiteLLM supports team-based access control. Create teams with their own budgets and model allowlists, then assign keys to teams:
curl -X POST http://localhost:4000/team/new \
-H "Authorization: Bearer sk-1234-master" \
-H "Content-Type: application/json" \
-d '{
"team_alias": "data-science",
"max_budget": 500.0,
"budget_duration": "1mo",
"models": ["gpt-4o", "claude-sonnet"],
"rpm_limit": 500
}'
Then create keys tied to that team:
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer sk-1234-master" \
-H "Content-Type: application/json" \
-d '{
"team_id": "team-uuid-here",
"key_alias": "ds-notebook-1",
"max_budget": 50.0
}'
The key inherits the team's model allowlist and rate limits while having its own sub-budget.
Best Practices
- Principle of least privilege: Give each virtual key the minimum set of models and the lowest budget that still lets the application function.
- Set hard timeouts: Always configure
request_timeoutso a hung provider connection does not tie up your application threads. - Use metadata for attribution: Tag every key with team, environment, and application identifiers so you can slice spend reports meaningfully.
- Monitor and alert: Set up alerts on spend velocity, error rate, and latency. A sudden spike in errors often indicates a provider outage or a misconfigured fallback.
- Rotate keys regularly: Use the
/key/updateendpoint to rotate virtual keys and the/key/deleteendpoint to revoke compromised ones immediately. - Test fallback paths: Periodically simulate provider failures in staging to confirm your fallback chain works as expected.
- Version your config: Treat
config.yamlas code. Store it in version control and review changes through pull requests. - Sanitize logs: LiteLLM logs request and response payloads by default. If your prompts contain sensitive data, configure log masking or disable payload logging in regulated environments.
Conclusion
Building a secure AI gateway with LiteLLM gives you a single control plane for every LLM call in your organization. By centralizing authentication through virtual keys, enforcing budgets and rate limits, routing intelligently across providers with fallbacks, and producing detailed audit logs, you transform a fragile collection of direct API calls into a governed, observable, and cost-controlled system. The gateway pattern also decouples your applications from provider specifics, making it straightforward to adopt new models or switch vendors without touching application code. Start with a minimal config, add teams and budgets as your usage grows, and continuously refine your policies based on the spend and usage data the gateway collects.