← Back to DevBytes
AI August 17, 2026 9 min read

Rate Limiting and Retry Strategies with LlamaIndex: Complete Guide

Introduction to Rate Limiting and Retry Strategies with LlamaIndex

When building production-grade applications with LlamaIndex, you will inevitably hit the boundaries of the APIs you depend on. Whether you are querying OpenAI, Anthropic, Cohere, or a self-hosted model server, every provider enforces rate limits. Combine that with the inherent flakiness of network calls, and you have a recipe for failed pipelines, lost data, and frustrated users. This guide walks through everything you need to know to make your LlamaIndex applications resilient, polite, and production-ready.

What Is Rate Limiting?

Rate limiting is the practice of controlling how frequently your application sends requests to an external service. Providers impose limits measured in two common dimensions:

Exceeding either limit typically results in an HTTP 429 "Too Many Requests" response. Without handling, your LlamaIndex pipeline crashes mid-execution.

What Are Retry Strategies?

Retry strategies define how your application responds to transient failures. Instead of failing immediately when a request errors out, the application waits and tries again — often with increasing delays. Common patterns include:

Why This Matters for LlamaIndex

LlamaIndex orchestrates many API calls under the hood. A single query_engine.query() invocation might trigger multiple LLM completions, embedding generations, and tool calls. When you index a large document corpus, you may fire hundreds of embedding requests in seconds. Without rate limiting and retries, one burst of activity can exhaust your quota, trigger backoff from the provider, or cause cascading failures across your application.

Setting Up Your Environment

Before diving into implementation, install the required packages. LlamaIndex integrates well with the tenacity library for retries and provides its own rate-limiting utilities.

pip install llama-index llama-index-core tenacity asyncio

Set your API keys as environment variables:

import os

os.environ["OPENAI_API_KEY"] = "sk-your-key-here"

Understanding LlamaIndex's Built-in Rate Limiter

LlamaIndex ships with a built-in rate limiter in the llama-index-core package. The TokenBucketRateLimiter implements the token bucket algorithm, which allows short bursts while enforcing a long-term average rate.

Basic Rate Limiter Usage

from llama_index.core.rate_limit import TokenBucketRateLimiter

# Allow 50 requests per minute, with a burst capacity of 10
rate_limiter = TokenBucketRateLimiter(
    rate=50,          # tokens added per minute
    capacity=10,      # max burst size
    period=60,        # period in seconds
)

# Acquire a token before making an API call
rate_limiter.acquire()  # blocks until a token is available
response = llm.complete("Summarize the latest news.")

The acquire() method blocks the current thread until a token is available. This is the simplest way to throttle synchronous code, but it is not ideal for async applications.

Attaching a Rate Limiter to an LLM

You can wrap any LlamaIndex LLM or embedding model so that every call passes through the rate limiter automatically:

from llama_index.llms.openai import OpenAI
from llama_index.core.rate_limit import TokenBucketRateLimiter

rate_limiter = TokenBucketRateLimiter(rate=40, capacity=5, period=60)

llm = OpenAI(model="gpt-4o-mini", rate_limiter=rate_limiter)

# Every call to llm.complete or llm.chat is now throttled
for i in range(100):
    response = llm.complete(f"Tell me joke number {i}")
    print(response)

This approach is clean because the rate limiting is transparent to the rest of your application code.

Implementing Custom Rate Limiters

Sometimes the built-in limiter does not fit your needs. For example, you might want to track tokens per minute rather than requests per minute, or you might need a distributed limiter that works across multiple processes.

A Custom Token-Aware Rate Limiter

import time
from collections import deque

class TokenAwareRateLimiter:
    def __init__(self, max_tokens_per_minute: int):
        self.max_tokens = max_tokens_per_minute
        self.window = 60  # seconds
        self.usage = deque()  # stores (timestamp, token_count)

    def acquire(self, token_count: int):
        now = time.time()
        # Remove entries outside the rolling window
        while self.usage and self.usage[0][0] < now - self.window:
            self.usage.popleft()

        current_usage = sum(count for _, count in self.usage)
        if current_usage + token_count > self.max_tokens:
            # Calculate sleep time until the oldest entry expires
            sleep_time = self.window - (now - self.usage[0][0]) + 0.1
            time.sleep(max(sleep_time, 0))

        self.usage.append((time.time(), token_count))

You can use this limiter by estimating token counts before each call. A rough heuristic is that one token equals approximately four characters of text.

limiter = TokenAwareRateLimiter(max_tokens_per_minute=90000)

text = "A long document that needs embedding..."
estimated_tokens = len(text) // 4
limiter.acquire(estimated_tokens)
embedding = embed_model.get_text_embedding(text)

Retry Strategies with Tenacity

The tenacity library is the de facto standard for retry logic in Python. It integrates cleanly with LlamaIndex because you can decorate or wrap any callable that makes an API request.

Basic Exponential Backoff

from tenacity import retry, stop_after_attempt, wait_exponential
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, min=2, max=60),
)
def safe_complete(prompt: str) -> str:
    return str(llm.complete(prompt))

response = safe_complete("Explain quantum computing in one paragraph.")
print(response)

This decorator retries up to five times, waiting 2, 4, 8, 16, and 32 seconds between attempts. The exponential growth prevents you from hammering an already-struggling API.

Retrying Only on Specific Exceptions

Not every error deserves a retry. A 400 Bad Request means your input is malformed, and retrying will not help. You should only retry on transient errors.

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from openai import RateLimitError, APIConnectionError, APITimeoutError, InternalServerError

@retry(
    stop=stop_after_attempt(6),
    wait=wait_exponential(multiplier=1, min=2, max=120),
    retry=retry_if_exception_type(
        (RateLimitError, APIConnectionError, APITimeoutError, InternalServerError)
    ),
)
def safe_complete(prompt: str) -> str:
    return str(llm.complete(prompt))

Adding Jitter to Avoid Thundering Herd

When multiple clients receive a 429 at the same time and all retry after the same delay, they hit the API simultaneously again. Jitter adds randomness to spread retries across time.

from tenacity import retry, stop_after_attempt, wait_random_exponential

@retry(
    stop=stop_after_attempt(6),
    wait=wait_random_exponential(multiplier=1, max=60),
)
def safe_complete(prompt: str) -> str:
    return str(llm.complete(prompt))

The wait_random_exponential function combines exponential backoff with random jitter, which is the recommended default for most production systems.

Combining Rate Limiting and Retries

Rate limiting and retries solve complementary problems. Rate limiting prevents you from exceeding quotas in the first place, while retries handle the cases where limits are still exceeded (perhaps because another application shares your API key). Using both together gives you the best of both worlds.

import time
from tenacity import retry, stop_after_attempt, wait_random_exponential
from llama_index.llms.openai import OpenAI
from llama_index.core.rate_limit import TokenBucketRateLimiter
from openai import RateLimitError, APIConnectionError, APITimeoutError

rate_limiter = TokenBucketRateLimiter(rate=40, capacity=5, period=60)
llm = OpenAI(model="gpt-4o-mini")

@retry(
    stop=stop_after_attempt(6),
    wait=wait_random_exponential(multiplier=1, max=60),
    retry=retry_if_exception_type(
        (RateLimitError, APIConnectionError, APITimeoutError)
    ),
)
def throttled_complete(prompt: str) -> str:
    rate_limiter.acquire()
    return str(llm.complete(prompt))

# Now safe to call in a tight loop
prompts = [f"Question {i}: What is {i} squared?" for i in range(200)]
for prompt in prompts:
    answer = throttled_complete(prompt)
    print(answer)

Async Rate Limiting and Retries

LlamaIndex fully supports async workflows, and rate limiting in async code requires a different approach. Blocking with time.sleep() freezes the entire event loop. Instead, use asyncio.sleep() and an async-compatible semaphore or rate limiter.

Async Semaphore-Based Rate Limiter

import asyncio
from tenacity import retry, stop_after_attempt, wait_random_exponential
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")

# Limit to 5 concurrent requests
semaphore = asyncio.Semaphore(5)

@retry(
    stop=stop_after_attempt(5),
    wait=wait_random_exponential(multiplier=1, max=30),
)
async def async_safe_complete(prompt: str) -> str:
    async with semaphore:
        response = await llm.acomplete(prompt)
        return str(response)

async def main():
    prompts = [f"Tell me fact {i}" for i in range(50)]
    tasks = [async_safe_complete(p) for p in prompts]
    results = await asyncio.gather(*tasks)
    for r in results:
        print(r)

asyncio.run(main())

Async Token Bucket Rate Limiter

For finer-grained control over requests per minute in async code, implement an async token bucket:

import asyncio
import time

class AsyncTokenBucketRateLimiter:
    def __init__(self, rate: float, capacity: int):
        self.rate = rate          # tokens per second
        self.capacity = capacity
        self.tokens = capacity
        self.last_refill = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self):
        async with self.lock:
            now = time.monotonic()
            elapsed = now - self.last_refill
            self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
            self.last_refill = now

            if self.tokens < 1:
                deficit = 1 - self.tokens
                wait_time = deficit / self.rate
                await asyncio.sleep(wait_time)
                self.tokens = 0
            else:
                self.tokens -= 1

# Usage
limiter = AsyncTokenBucketRateLimiter(rate=0.5, capacity=5)  # 30 per minute

async def rate_limited_complete(prompt: str) -> str:
    await limiter.acquire()
    return str(await llm.acomplete(prompt))

Rate Limiting During Indexing

Indexing is where most developers first encounter rate limit issues. When you ingest hundreds of documents, LlamaIndex generates embeddings for every chunk. Without throttling, this can easily exceed your provider's limits.

Throttling Embedding Generation

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.rate_limit import TokenBucketRateLimiter
from tenacity import retry, stop_after_attempt, wait_random_exponential

rate_limiter = TokenBucketRateLimiter(rate=100, capacity=10, period=60)
embed_model = OpenAIEmbedding(model="text-embedding-3-small", rate_limiter=rate_limiter)

@retry(
    stop=stop_after_attempt(5),
    wait=wait_random_exponential(multiplier=1, max=30),
)
def safe_embed(text: str):
    return embed_model.get_text_embedding(text)

documents = SimpleDirectoryReader("./data").load_data()
print(f"Loaded {len(documents)} documents")

# Build index with the rate-limited embed model
index = VectorStoreIndex.from_documents(
    documents,
    embed_model=embed_model,
    show_progress=True,
)
print("Indexing complete.")

Handling the Retry-After Header

When a provider returns a 429, the response often includes a Retry-After header telling you exactly how long to wait. Honoring this header is more efficient than guessing with exponential backoff.

import time
from tenacity import retry, stop_after_attempt, retry_if_exception_type
from openai import RateLimitError

def wait_for_retry_after(exc):
    """Extract wait time from the Retry-After header."""
    if isinstance(exc, RateLimitError):
        retry_after = exc.response.headers.get("Retry-After")
        if retry_after:
            return float(retry_after)
    # Fall back to exponential backoff
    return 2 ** (getattr(wait_for_retry_after, "_attempt", 0))

@retry(
    stop=stop_after_attempt(8),
    wait=wait_for_retry_after,
    retry=retry_if_exception_type(RateLimitError),
)
def polite_complete(prompt: str) -> str:
    return str(llm.complete(prompt))

Best Practices

Logging Retry Events

import logging
from tenacity import before_sleep_log

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@retry(
    stop=stop_after_attempt(6),
    wait=wait_random_exponential(multiplier=1, max=60),
    before_sleep=before_sleep_log(logger, logging.WARNING),
)
def safe_complete(prompt: str) -> str:
    return str(llm.complete(prompt))

Conclusion

Rate limiting and retry strategies are not optional extras in production LlamaIndex applications — they are foundational reliability features. By combining LlamaIndex's built-in TokenBucketRateLimiter with tenacity's flexible retry decorators, you can build pipelines that gracefully handle provider limits, network hiccups, and transient failures without losing data or degrading user experience. Start with the built-in rate limiter and exponential backoff with jitter, then refine your approach based on observed traffic patterns. The result is an application that stays responsive and reliable even under heavy load, which is exactly what your users deserve.

— Ad —

Google AdSense will appear here after approval

← Back to all articles