Rate Limiting and Retry Strategies with LangGraph: Complete Guide

Rate Limiting and Retry Strategies with LangGraph: Complete Guide

Building production-grade applications with LangGraph often means orchestrating dozens of LLM calls, tool invocations, and external API requests inside a single graph run. When you scale that workload, two problems inevitably surface: rate limits from upstream providers and transient failures that cause your graph to crash mid-execution. This guide walks through everything you need to know to make your LangGraph workflows resilient, observable, and production-ready.

What Is Rate Limiting and Why It Matters

Rate limiting is a mechanism providers use to control how many requests a client can make within a given time window. LLM providers like OpenAI, Anthropic, and Google enforce limits on tokens-per-minute (TPM), requests-per-minute (RPM), and concurrent connections. When you exceed these thresholds, the provider returns an HTTP 429 error instead of a response.

In a LangGraph application, a single user query might trigger a chain of nodes, each making multiple LLM calls. Without rate limiting on your side, a burst of traffic can quickly exhaust your quota, cause cascading failures, and degrade the experience for every user. Worse, retrying failed requests naively can amplify the problem by hammering the provider even harder.

Why Retry Strategies Are Essential

Not every failure is permanent. Network blips, temporary 429 responses, and brief server-side hiccups (HTTP 5xx) often resolve themselves within seconds. A well-designed retry strategy:

• Detects transient errors and automatically reattempts the operation
• Uses exponential backoff to avoid overwhelming the provider
• Adds jitter to prevent thundering-herd effects when many clients retry simultaneously
• Gives up after a bounded number of attempts to avoid infinite loops
• Distinguishes between retryable and non-retryable errors

Combining rate limiting with retries creates a self-regulating system: the rate limiter keeps you under quota, and the retry layer handles the occasional overflow gracefully.

Understanding LangGraph's Execution Model

LangGraph executes graphs as a series of node invocations connected by edges. Each node receives a state object, performs work, and returns an updated state. Because nodes can call external services, they are the natural place to implement rate limiting and retry logic.

LangGraph also supports checkpointing, which persists state between steps. This is powerful for retries because a failed node can be re-executed from its last known good state rather than restarting the entire graph. The MemorySaver and SqliteSaver checkpointers are commonly used for this purpose.

Implementing Rate Limiting in LangGraph

Approach 1: Token Bucket Rate Limiter

A token bucket is a classic rate-limiting algorithm. Tokens accumulate at a fixed rate, and each request consumes one or more tokens. If the bucket is empty, the request must wait. Here is a thread-safe implementation you can inject into any node:

import time
import threading
from collections import deque

class TokenBucketRateLimiter:
    def __init__(self, rate: float, capacity: int):
        self.rate = rate  # tokens per second
        self.capacity = capacity
        self.tokens = capacity
        self.last_refill = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self, tokens: int = 1, timeout: float = 60.0):
        start = time.monotonic()
        while True:
            with self.lock:
                now = time.monotonic()
                elapsed = now - self.last_refill
                self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
                self.last_refill = now
                if self.tokens >= tokens:
                    self.tokens -= tokens
                    return True
                needed = tokens - self.tokens
                wait_time = needed / self.rate
            if time.monotonic() - start + wait_time > timeout:
                raise TimeoutError("Rate limit acquire timed out")
            time.sleep(min(wait_time, 0.5))

# Create a shared limiter: 50 requests per minute = ~0.83/sec, burst of 10
llm_rate_limiter = TokenBucketRateLimiter(rate=50 / 60, capacity=10)

Approach 2: Sliding Window Counter

A sliding window counter tracks requests within a rolling time window. It is slightly more accurate than a fixed window because it avoids boundary spikes:

import time
from collections import deque

class SlidingWindowLimiter:
    def __init__(self, max_requests: int, window_seconds: float):
        self.max_requests = max_requests
        self.window = window_seconds
        self.timestamps = deque()

    def acquire(self):
        now = time.monotonic()
        while self.timestamps and self.timestamps[0] <= now - self.window:
            self.timestamps.popleft()
        if len(self.timestamps) >= self.max_requests:
            sleep_time = self.window - (now - self.timestamps[0])
            time.sleep(max(sleep_time, 0))
            return self.acquire()
        self.timestamps.append(now)

rpm_limiter = SlidingWindowLimiter(max_requests=40, window_seconds=60)

Integrating the Limiter Into a LangGraph Node

Now let's wire the rate limiter into an actual LangGraph node. The pattern is simple: acquire a token before making the LLM call, then proceed:

from langgraph.graph import StateGraph, START, END
from langchain_openai import ChatOpenAI
from typing import TypedDict

class AgentState(TypedDict):
    query: str
    response: str
    attempts: int

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def research_node(state: AgentState) -> AgentState:
    # Block until a rate-limit slot is available
    llm_rate_limiter.acquire(tokens=1)
    try:
        result = llm.invoke(state["query"])
        return {"response": result.content, "attempts": state.get("attempts", 0) + 1}
    except Exception as e:
        return {"response": "", "attempts": state.get("attempts", 0) + 1}

graph_builder = StateGraph(AgentState)
graph_builder.add_node("research", research_node)
graph_builder.add_edge(START, "research")
graph_builder.add_edge("research", END)
graph = graph_builder.compile()

Implementing Retry Strategies

Approach 1: Decorator-Based Retries

A decorator is a clean way to wrap any node function with retry logic. This example uses exponential backoff with jitter and only retries on specific HTTP status codes:

import time
import random
import functools
from httpx import HTTPStatusError

RETRYABLE_STATUS = {429, 500, 502, 503, 504}

def with_retry(max_attempts=4, base_delay=1.0, max_delay=30.0):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            attempt = 0
            while True:
                attempt += 1
                try:
                    return func(*args, **kwargs)
                except HTTPStatusError as e:
                    status = e.response.status_code
                    if status not in RETRYABLE_STATUS or attempt >= max_attempts:
                        raise
                    delay = min(max_delay, base_delay * (2 ** (attempt - 1)))
                    delay = delay + random.uniform(0, delay * 0.1)  # jitter
                    print(f"Attempt {attempt} failed (HTTP {status}), retrying in {delay:.2f}s")
                    time.sleep(delay)
                except Exception as e:
                    if attempt >= max_attempts:
                        raise
                    delay = min(max_delay, base_delay * (2 ** (attempt - 1)))
                    time.sleep(delay + random.uniform(0, 0.5))
        return wrapper
    return decorator

@with_retry(max_attempts=5, base_delay=2.0)
def call_llm(prompt: str) -> str:
    return llm.invoke(prompt).content

Approach 2: Using tenacity for Robust Retries

The tenacity library is battle-tested and offers more features than a hand-rolled decorator, including per-exception retry policies and callbacks:

from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential_jitter,
    retry_if_exception_type,
    before_sleep_log,
)
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@retry(
    stop=stop_after_attempt(6),
    wait=wait_exponential_jitter(initial=1, max=30, jitter=2),
    retry=retry_if_exception_type((HTTPStatusError, ConnectionError, TimeoutError)),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
def call_llm_with_tenacity(prompt: str) -> str:
    response = llm.invoke(prompt)
    return response.content

Approach 3: LangGraph's Built-in Retry Policy

LangGraph supports a RetryPolicy that you can attach to nodes at compile time. This is the most idiomatic approach because it integrates with the graph's checkpointing system, allowing failed nodes to resume from their last checkpoint rather than re-running the entire graph:

from langgraph.pregel import RetryPolicy
from langgraph.errors import GraphRecursionError

# Define a retry policy: up to 5 attempts, exponential backoff
retry_policy = RetryPolicy(
    max_attempts=5,
    initial_interval=1.0,
    backoff_factor=2.0,
    max_interval=30.0,
    jitter=True,
)

# Attach the policy to specific nodes
graph = graph_builder.compile(
    retry_policy=retry_policy,
)

When a node raises an exception, LangGraph automatically waits according to the policy and re-executes the node. Because state is checkpointed, any successfully completed prior nodes are not re-run.

Combining Rate Limiting and Retries in a Real Graph

Let's build a more realistic example: a multi-agent research graph with two LLM-calling nodes, both protected by rate limiting and retries. We'll use a SQLite checkpointer so that state survives crashes:

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.pregel import RetryPolicy
from langchain_openai import ChatOpenAI
from typing import TypedDict
import sqlite3

class ResearchState(TypedDict):
    topic: str
    outline: str
    draft: str
    final: str

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
limiter = TokenBucketRateLimiter(rate=40 / 60, capacity=8)

@with_retry(max_attempts=5, base_delay=2.0)
def outline_node(state: ResearchState) -> dict:
    limiter.acquire()
    prompt = f"Create a detailed outline about: {state['topic']}"
    return {"outline": llm.invoke(prompt).content}

@with_retry(max_attempts=5, base_delay=2.0)
def draft_node(state: ResearchState) -> dict:
    limiter.acquire()
    prompt = f"Write a draft using this outline:\n{state['outline']}"
    return {"draft": llm.invoke(prompt).content}

@with_retry(max_attempts=5, base_delay=2.0)
def polish_node(state: ResearchState) -> dict:
    limiter.acquire()
    prompt = f"Polish this draft for clarity and tone:\n{state['draft']}"
    return {"final": llm.invoke(prompt).content}

builder = StateGraph(ResearchState)
builder.add_node("outline", outline_node)
builder.add_node("draft", draft_node)
builder.add_node("polish", polish_node)
builder.add_edge(START, "outline")
builder.add_edge("outline", "draft")
builder.add_edge("draft", "polish")
builder.add_edge("polish", END)

conn = sqlite3.connect("checkpoints.db", check_same_thread=False)
saver = SqliteSaver(conn)

retry_policy = RetryPolicy(max_attempts=4, initial_interval=1.0, backoff_factor=2.0)

app = builder.compile(checkpointer=saver, retry_policy=retry_policy)

# Run with a thread_id so the graph can resume after failures
config = {"configurable": {"thread_id": "research-session-1"}}
result = app.invoke({"topic": "The future of renewable energy"}, config=config)
print(result["final"])

Handling 429 Responses Specifically

Many providers include a Retry-After header in 429 responses. Honoring this value is more efficient than guessing with backoff. Here is how to parse and respect it:

import time
from httpx import HTTPStatusError

@with_retry(max_attempts=6, base_delay=1.0)
def call_with_retry_after(prompt: str) -> str:
    try:
        return llm.invoke(prompt).content
    except HTTPStatusError as e:
        if e.response.status_code == 429:
            retry_after = e.response.headers.get("Retry-After")
            if retry_after:
                wait = float(retry_after)
                print(f"Rate limited. Sleeping {wait}s per Retry-After header.")
                time.sleep(wait)
        raise

Best Practices

• Layer your protections: use rate limiting to stay under quota, retries to handle transient failures, and circuit breakers to stop calling a service that is clearly down.
• Always add jitter: deterministic backoff causes synchronized retry storms. Randomized jitter spreads load over time.
• Use checkpointing: compile your graph with a checkpointer so failed nodes can resume without re-running completed work.
• Set a recursion limit: LangGraph defaults to 25 steps. Tune recursion_limit in your config to match your graph's expected depth.
• Log retry events: use before_sleep_log or custom callbacks to emit metrics. Retries are a leading indicator of upstream instability.
• Distinguish error types: only retry on 429, 5xx, timeouts, and connection errors. Never retry on 400, 401, 403, or validation errors — they will fail every time.
• Respect Retry-After headers: when a provider tells you exactly how long to wait, listen to it rather than computing your own delay.
• Isolate rate limiters per provider: if your graph calls OpenAI and Anthropic, use separate limiters so one provider's quota does not block the other.
• Test under load: simulate burst traffic in staging to verify your limiter and retry policy behave as expected before shipping to production.
• Monitor token usage: RPM is only half the picture. Track TPM as well, since a single long prompt can exhaust token limits even when request counts are low.

Conclusion

Rate limiting and retry strategies are not optional polish — they are foundational requirements for any LangGraph application that touches real LLM APIs in production. By combining a token-bucket or sliding-window limiter with exponential-backoff retries, jitter, checkpointing, and LangGraph's native RetryPolicy, you can build graphs that degrade gracefully under pressure instead of crashing. Start with the simple patterns shown here, instrument them with logging and metrics, and iterate as you learn how your specific workload interacts with provider limits. The result is a resilient agent pipeline that your users — and your API bill — will thank you for.