← Back to DevBytes
Developer Tutorial August 17, 2026 7 min read

Rate Limiting and Retry Strategies with llama.cpp: Complete Guide

Rate Limiting and Retry Strategies with llama.cpp: Complete Guide

When you run llama.cpp in production — whether as a local inference server, a backend behind an API gateway, or a multi-tenant service — you quickly run into two realities: hardware has finite throughput, and transient failures are inevitable. Rate limiting protects your GPU/CPU from being overwhelmed, while retry strategies ensure that temporary hiccups (memory pressure, context overflow, socket timeouts) don't cascade into user-facing errors. This guide walks through both concepts with practical, copy-paste-ready examples.

Why Rate Limiting Matters for llama.cpp

Unlike stateless REST APIs, LLM inference is stateful and expensive. A single request that asks for 2,000 output tokens on a 7B model can occupy your GPU for several seconds, consuming VRAM that other requests need. Without rate limiting, a burst of concurrent users can cause:

Rate limiting gives you predictable throughput, fair resource allocation, and graceful degradation under load.

Understanding the Layers of Rate Limiting

Effective rate limiting for llama.cpp typically operates at three layers:

For LLM workloads, token-based limits are the gold standard because a one-token request and a 4,000-token request have wildly different costs. However, request-based limits are simpler to implement and work well as a first line of defense.

Setting Up llama.cpp Server

First, ensure you have the llama-server binary built and a model downloaded. The server exposes an OpenAI-compatible HTTP API at http://localhost:8080.

# Build llama.cpp with CUDA support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make GGML_CUDA=1

# Download a small model for testing
huggingface-cli download TheBloke/Llama-2-7B-Chat-GGUF \
  llama-2-7b-chat.Q4_K_M.gguf --local-dir ./models

# Start the server with a concurrency cap
./llama-server \
  -m ./models/llama-2-7b-chat.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 4096 -np 4 \
  --cont-batching

The -np 4 flag sets the number of parallel slots to 4, which is llama.cpp's built-in concurrency limiter. The --cont-batching flag enables continuous batching, which dramatically improves throughput when multiple requests are in flight.

Implementing a Token Bucket Rate Limiter in Python

The token bucket algorithm is ideal for LLM rate limiting because it allows short bursts while enforcing a long-term average rate. Here's a thread-safe implementation:

import time
import threading
from collections import defaultdict

class TokenBucketLimiter:
    def __init__(self, capacity: float, refill_rate: float):
        """
        capacity: max tokens that can accumulate (burst size)
        refill_rate: tokens added per second (sustained rate)
        """
        self.capacity = float(capacity)
        self.refill_rate = float(refill_rate)
        self.buckets = defaultdict(lambda: {"tokens": self.capacity, "last": time.monotonic()})
        self.lock = threading.Lock()

    def acquire(self, client_id: str, cost: float) -> bool:
        with self.lock:
            bucket = self.buckets[client_id]
            now = time.monotonic()
            elapsed = now - bucket["last"]
            bucket["tokens"] = min(self.capacity, bucket["tokens"] + elapsed * self.refill_rate)
            bucket["last"] = now

            if bucket["tokens"] >= cost:
                bucket["tokens"] -= cost
                return True
            return False

# Example: 10,000 tokens capacity, refill 1,000 tokens/sec
limiter = TokenBucketLimiter(capacity=10000, refill_rate=1000)

The cost parameter should reflect the estimated token count of the request. You can estimate input tokens with a simple heuristic (roughly 4 characters per token for English text) or use a proper tokenizer like tiktoken for more accuracy.

Building a Rate-Limited Client Wrapper

Now let's combine the rate limiter with a client that talks to llama-server. This wrapper enforces limits before sending the request, preventing the server from ever seeing excess load.

import requests
import time

class LlamaClient:
    def __init__(self, base_url: str, limiter: TokenBucketLimiter):
        self.base_url = base_url.rstrip("/")
        self.limiter = limiter
        self.session = requests.Session()

    def _estimate_tokens(self, messages):
        # Rough heuristic: ~4 chars per token
        total_chars = sum(len(m["content"]) for m in messages)
        return int(total_chars / 4)

    def chat(self, messages, max_tokens=512, client_id="default"):
        est_input = self._estimate_tokens(messages)
        est_cost = est_input + max_tokens

        # Block until rate limit allows the request
        while not self.limiter.acquire(client_id, est_cost):
            time.sleep(0.1)

        payload = {
            "messages": messages,
            "max_tokens": max_tokens,
            "stream": False,
        }
        resp = self.session.post(
            f"{self.base_url}/v1/chat/completions",
            json=payload,
            timeout=120,
        )
        resp.raise_for_status()
        return resp.json()

client = LlamaClient("http://localhost:8080", limiter)
result = client.chat(
    [{"role": "user", "content": "Explain quantum entanglement in two sentences."}],
    max_tokens=200,
    client_id="user_42",
)
print(result["choices"][0]["message"]["content"])

Retry Strategies: Handling Transient Failures

Even with perfect rate limiting, requests can fail. Common transient failures with llama.cpp include:

A robust retry strategy uses exponential backoff with jitter. Jitter (randomized delay) prevents the "thundering herd" problem where many retried requests hit the server simultaneously.

Exponential Backoff with Jitter

import random
import time
import requests

def retry_with_backoff(
    func,
    max_retries=5,
    base_delay=1.0,
    max_delay=30.0,
    retryable_status={429, 500, 502, 503, 504},
):
    last_exc = None
    for attempt in range(max_retries + 1):
        try:
            resp = func()
            if resp.status_code in retryable_status and attempt < max_retries:
                delay = min(max_delay, base_delay * (2 ** attempt))
                delay = delay * (0.5 + random.random() * 0.5)  # jitter
                print(f"Retry {attempt + 1}/{max_retries} after {delay:.2f}s (HTTP {resp.status_code})")
                time.sleep(delay)
                continue
            resp.raise_for_status()
            return resp
        except (requests.ConnectionError, requests.Timeout) as e:
            last_exc = e
            if attempt < max_retries:
                delay = min(max_delay, base_delay * (2 ** attempt))
                delay = delay * (0.5 + random.random() * 0.5)
                print(f"Retry {attempt + 1}/{max_retries} after {delay:.2f}s ({type(e).__name__})")
                time.sleep(delay)
            else:
                raise
    raise last_exc

Integrating Retry into the Client

class ResilientLlamaClient(LlamaClient):
    def chat(self, messages, max_tokens=512, client_id="default"):
        est_cost = self._estimate_tokens(messages) + max_tokens

        while not self.limiter.acquire(client_id, est_cost):
            time.sleep(0.1)

        def do_request():
            return self.session.post(
                f"{self.base_url}/v1/chat/completions",
                json={"messages": messages, "max_tokens": max_tokens, "stream": False},
                timeout=120,
            )

        resp = retry_with_backoff(do_request, max_retries=4, base_delay=0.5)
        return resp.json()

Handling the 503 "All Slots Busy" Case

When llama-server has all parallel slots occupied, it returns HTTP 503. This is a signal to back off, not a fatal error. A specialized handler can check the server's slot status endpoint and wait intelligently:

def wait_for_free_slot(base_url, timeout=60):
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        try:
            status = requests.get(f"{base_url}/slots", timeout=5).json()
            free = sum(1 for s in status if not s.get("is_processing", False))
            if free > 0:
                return True
        except requests.RequestException:
            pass
        time.sleep(0.5)
    return False

Call this before retrying a 503 response to avoid hammering the server with doomed requests.

Server-Side Rate Limiting with a Reverse Proxy

For production deployments, put llama-server behind Nginx or a small Python gateway. This centralizes rate limiting so all clients are subject to the same rules. Here's a minimal FastAPI gateway:

from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse
import httpx
import asyncio

app = FastAPI()
LLAMA_URL = "http://localhost:8080"

# Simple per-IP concurrency semaphore
MAX_CONCURRENT = 4
semaphore = asyncio.Semaphore(MAX_CONCURRENT)

@app.post("/v1/chat/completions")
async def proxy(request: Request):
    body = await request.json()
    async with semaphore:
        async with httpx.AsyncClient(timeout=300) as client:
            try:
                resp = await client.post(
                    f"{LLAMA_URL}/v1/chat/completions",
                    json=body,
                )
                return JSONResponse(content=resp.json(), status_code=resp.status_code)
            except httpx.RequestError:
                raise HTTPException(status_code=503, detail="Upstream unavailable")

This gateway caps concurrent requests at 4, matching the -np 4 setting on the server. Additional requests queue at the semaphore rather than overwhelming the model.

Best Practices

Conclusion

Rate limiting and retry strategies are the difference between a llama.cpp deployment that survives real-world traffic and one that collapses under the first burst of users. By combining server-side slot management (-np), a token bucket limiter on the client or gateway, and exponential backoff with jitter for transient failures, you get a system that degrades gracefully instead of failing catastrophically. Start with the simple concurrency semaphore, measure your actual token throughput, then tune your bucket capacity and refill rate to match your hardware's sustained output. The result is a self-hosted LLM service that stays responsive, fair, and resilient no matter what your users throw at it.

— Ad —

Google AdSense will appear here after approval

← Back to all articles