How to Implement Rate Limiting for LLM APIs

How to Implement Rate Limiting for LLM APIs

Large Language Model (LLM) APIs are powerful, but they come with strict usage constraints. Whether you're building a chatbot, an AI-powered search tool, or an agent workflow, uncontrolled requests can quickly exhaust your quota, inflate costs, or trigger throttling errors. Rate limiting is the practice of controlling how many requests your application sends to an LLM provider within a given time window. This tutorial explains what rate limiting is, why it matters, and how to implement it effectively in production systems.

What Is Rate Limiting?

Rate limiting is a mechanism that restricts the number of actions a client can perform over a defined period. In the context of LLM APIs, it typically applies to two dimensions:

• Requests per minute (RPM): The number of API calls allowed in a 60-second window.
• Tokens per minute (TPM): The total number of input and output tokens processed per minute.

Providers like OpenAI, Anthropic, and Google enforce both limits. Hitting either one returns an HTTP 429 "Too Many Requests" response. A robust client must therefore track and throttle outgoing requests before they reach the provider.

Why Rate Limiting Matters

Without rate limiting, your application is vulnerable to several problems:

• Cost overruns: A bug or a burst of user traffic can generate thousands of expensive calls in seconds.
• Service disruption: Once throttled, legitimate requests also fail, degrading user experience.
• Account suspension: Persistent abuse of limits may lead to temporary or permanent account restrictions.
• Unfair resource usage: In multi-tenant systems, one noisy user can starve others of API access.

Implementing rate limiting on your side gives you predictable costs, smoother performance, and better control over user-facing behavior.

Common Rate Limiting Algorithms

Before writing code, it helps to understand the two most common algorithms:

• Fixed window: Counts requests in a fixed time block (e.g., 0–60s, 60–120s). Simple but allows bursts at window boundaries.
• Sliding window: Tracks requests over a rolling time range, smoothing out bursts.
• Token bucket: Maintains a bucket of tokens refilled at a steady rate. Each request consumes a token. Allows short bursts up to bucket capacity while enforcing an average rate.

The token bucket algorithm is generally the best fit for LLM APIs because it naturally accommodates bursts while respecting long-term averages.

Implementing a Token Bucket Limiter in Python

Below is a self-contained token bucket implementation that you can drop into any async Python project.

import time
import asyncio
from collections import deque

class TokenBucket:
    def __init__(self, rate: float, capacity: float):
        """
        rate: tokens added per second
        capacity: maximum tokens the bucket can hold
        """
        self.rate = rate
        self.capacity = capacity
        self.tokens = capacity
        self.updated_at = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self, tokens: float = 1.0) -> None:
        async with self.lock:
            while True:
                now = time.monotonic()
                elapsed = now - self.updated_at
                self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
                self.updated_at = now

                if self.tokens >= tokens:
                    self.tokens -= tokens
                    return

                # Wait until enough tokens are available
                deficit = tokens - self.tokens
                wait_time = deficit / self.rate
                await asyncio.sleep(wait_time)

The acquire method blocks until enough tokens are available, making it ideal for wrapping LLM calls. Here is how you would use it with an OpenAI-style client:

import openai

# Allow 50 requests per minute, with a burst capacity of 10
bucket = TokenBucket(rate=50 / 60, capacity=10)

async def chat_completion(messages):
    await bucket.acquire()
    response = await openai.ChatCompletion.acreate(
        model="gpt-4",
        messages=messages,
    )
    return response["choices"][0]["message"]["content"]

Handling HTTP 429 Responses with Retry

Even with client-side limiting, you should still handle 429 errors gracefully. Providers often return a Retry-After header indicating how long to wait. The following helper combines proactive throttling with reactive retries:

import asyncio
import random

async def call_with_retry(func, *args, max_retries=5, **kwargs):
    for attempt in range(max_retries):
        try:
            return await func(*args, **kwargs)
        except openai.error.RateLimitError as e:
            retry_after = getattr(e, "retry_after", None) or (2 ** attempt)
            jitter = random.uniform(0, 0.5)
            await asyncio.sleep(retry_after + jitter)
    raise RuntimeError("Max retries exceeded due to rate limiting")

Adding jitter prevents the "thundering herd" problem, where many retries fire simultaneously after a throttle window resets.

Token-Based Limiting

Requests-per-minute limiting is necessary but not sufficient. A single prompt with a large context can consume a huge portion of your token budget. To respect TPM limits, track token usage from each response and consume that many tokens from a separate bucket:

class TokenUsageBucket:
    def __init__(self, tokens_per_minute: int):
        self.rate = tokens_per_minute / 60
        self.capacity = tokens_per_minute
        self.bucket = TokenBucket(rate=self.rate, capacity=self.capacity)

    async def acquire(self, estimated_tokens: int):
        await self.bucket.acquire(estimated_tokens)

    async def reconcile(self, actual_tokens: int, estimated_tokens: int):
        # Refund the difference if we over-estimated
        diff = estimated_tokens - actual_tokens
        if diff > 0:
            self.bucket.tokens = min(
                self.bucket.capacity,
                self.bucket.tokens + diff
            )

Estimate token counts before the call using a tokenizer, then reconcile with the actual usage reported in the API response. This keeps your bucket accurate over time.

Distributed Rate Limiting

If your application runs across multiple processes or servers, in-memory buckets are not enough. You need a shared store. Redis is the standard choice because it offers atomic operations and low latency.

import redis.asyncio as redis
import time

class RedisTokenBucket:
    def __init__(self, client: redis.Redis, key: str, rate: float, capacity: float):
        self.client = client
        self.key = key
        self.rate = rate
        self.capacity = capacity

    async def acquire(self, tokens: float = 1.0) -> bool:
        script = """
        local key = KEYS[1]
        local rate = tonumber(ARGV[1])
        local capacity = tonumber(ARGV[2])
        local tokens_needed = tonumber(ARGV[3])
        local now = tonumber(ARGV[4])

        local data = redis.call('HMGET', key, 'tokens', 'timestamp')
        local current_tokens = tonumber(data[1]) or capacity
        local last_time = tonumber(data[2]) or now

        local elapsed = math.max(0, now - last_time)
        current_tokens = math.min(capacity, current_tokens + elapsed * rate)

        if current_tokens < tokens_needed then
            return 0
        end

        current_tokens = current_tokens - tokens_needed
        redis.call('HMSET', key, 'tokens', current_tokens, 'timestamp', now)
        redis.call('EXPIRE', key, 3600)
        return 1
        """
        now = time.time()
        result = await self.client.eval(
            script, 1, self.key,
            self.rate, self.capacity, tokens, now
        )
        return bool(result)

The Lua script runs atomically inside Redis, ensuring correctness even when many servers hit the same key simultaneously. Use a unique key per API key, per user, or per tenant depending on your isolation requirements.

Best Practices

• Set limits below provider caps: Leave a 10–20% safety margin so transient spikes do not trigger 429s.
• Separate RPM and TPM buckets: They are independent constraints and must be tracked separately.
• Use per-user limits in multi-tenant apps: Prevent one user from consuming the entire shared budget.
• Cache responses when possible: Identical prompts can often be served from a cache, reducing both cost and rate pressure.
• Queue instead of reject: For user-facing features, queue requests and show a loading state rather than returning errors.
• Monitor and alert: Track 429 rates, queue depths, and token consumption. Alert before limits become critical.
• Implement exponential backoff with jitter: Always combine proactive limiting with reactive retry logic.
• Log token usage: Store the usage field from each response for cost analysis and capacity planning.

Conclusion

Rate limiting is an essential layer of any production LLM integration. By combining a token bucket algorithm for proactive throttling, reactive retry logic for 429 handling, and a distributed store like Redis for multi-instance deployments, you can build a resilient system that stays within provider limits while delivering a smooth experience to your users. Start with a simple in-memory limiter, measure your actual traffic patterns, and evolve toward distributed limiting as your scale demands. The small upfront investment pays off in lower costs, fewer outages, and more predictable performance.