← Back to DevBytes
Developer Tutorial August 18, 2026 9 min read

Benchmarking Streaming Latency: Time to First Token (TTFT)

Introduction to Time to First Token (TTFT)

When building applications powered by Large Language Models (LLMs), user experience hinges heavily on perceived responsiveness. While overall request latency matters, the most critical metric for streaming responses is Time to First Token (TTFT) — the elapsed time between sending a request and receiving the very first generated token from the model.

TTFT represents the moment a user sees the model "begin typing." A low TTFT creates the illusion of a fast, intelligent assistant, while a high TTFT leaves users staring at a loading spinner, eroding trust and engagement. In this tutorial, we'll explore what TTFT is, why it matters, how to measure it accurately, and best practices for optimizing and benchmarking it in production systems.

What Is Time to First Token?

TTFT measures the delay before the first token of a streamed response arrives at the client. It captures the cumulative cost of several pipeline stages:

Unlike Time Per Output Token (TPOT) or end-to-end latency, TTFT is largely independent of the response length. A 10-token response and a 1,000-token response can share the same TTFT, because the bottleneck is prompt processing, not generation.

TTFT vs. Other Latency Metrics

To benchmark streaming performance holistically, you need to understand how TTFT relates to sibling metrics:

A system can have excellent throughput but poor TTFT if it batches aggressively, delaying individual requests. Conversely, a system optimized for low TTFT may sacrifice overall throughput. Benchmarking TTFT in isolation helps you reason about this trade-off.

Why TTFT Matters

Research on perceived performance consistently shows that users tolerate waiting once they see activity begin. In chat interfaces, the first token acts as that signal. A TTFT under 500 milliseconds feels instantaneous; above 2 seconds, users begin to perceive lag; above 5 seconds, abandonment rates climb sharply.

Beyond UX, TTFT has direct business implications:

For these reasons, TTFT has become a first-class metric in model serving frameworks like vLLM, TensorRT-LLM, and TGI, all of which expose it in their benchmarking tooling.

How to Measure TTFT

Measuring TTFT requires a streaming client that records timestamps at two points: when the request is dispatched and when the first token chunk arrives. Most modern LLM APIs support Server-Sent Events (SSE) or chunked HTTP responses for this purpose.

Basic TTFT Measurement with OpenAI-Style Streaming

The following Python example uses the OpenAI client to measure TTFT against any OpenAI-compatible endpoint:

import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

def measure_ttft(prompt: str, model: str = "meta-llama/Llama-3-8B") -> float:
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
        max_tokens=512,
    )
    first_token_time = None
    token_count = 0
    for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            token_count += 1
    end = time.perf_counter()

    ttft = first_token_time - start
    e2e = end - start
    print(f"TTFT: {ttft * 1000:.1f} ms")
    print(f"End-to-end: {e2e * 1000:.1f} ms")
    print(f"Tokens: {token_count}")
    return ttft

measure_ttft("Explain quantum entanglement in three sentences.")

Key points about this measurement:

Measuring TTFT with Raw HTTP/SSE

If you want to avoid SDK overhead or benchmark a custom endpoint, use httpx with streaming:

import time
import httpx
import json

def measure_ttft_raw(url: str, payload: dict) -> float:
    start = time.perf_counter()
    first_token_time = None

    with httpx.Client(timeout=60.0) as client:
        with client.stream("POST", url, json=payload) as response:
            for line in response.iter_lines():
                if not line or not line.startswith("data: "):
                    continue
                data = line[len("data: "):]
                if data == "[DONE]":
                    break
                chunk = json.loads(data)
                delta = chunk["choices"][0].get("delta", {})
                content = delta.get("content", "")
                if content and first_token_time is None:
                    first_token_time = time.perf_counter()
                    break

    if first_token_time is None:
        raise RuntimeError("No tokens received")
    return first_token_time - start

payload = {
    "model": "meta-llama/Llama-3-8B",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": True,
    "max_tokens": 128,
}
ttft = measure_ttft_raw("http://localhost:8000/v1/chat/completions", payload)
print(f"TTFT: {ttft * 1000:.1f} ms")

This raw approach eliminates SDK parsing overhead and gives you the most accurate picture of server-side streaming behavior.

Building a Robust Benchmark Harness

A single TTFT measurement is noisy. To produce trustworthy numbers, you need a benchmark harness that runs multiple prompts, repeats trials, and reports distributional statistics.

import time
import statistics
from openai import OpenAI
from concurrent.futures import ThreadPoolExecutor, as_completed

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

PROMPTS = [
    "Write a haiku about the ocean.",
    "Summarize the plot of Hamlet in two sentences.",
    "What are the benefits of functional programming?",
    "Explain how a transformer model works.",
    "Give me five ideas for a mobile app.",
]

def single_request(prompt: str, model: str) -> float:
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
        max_tokens=256,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.perf_counter() - start
    raise RuntimeError("Empty stream")

def benchmark(model: str, trials_per_prompt: int = 5, concurrency: int = 1):
    results = []
    tasks = []
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        for prompt in PROMPTS:
            for _ in range(trials_per_prompt):
                tasks.append(pool.submit(single_request, prompt, model))
        for future in as_completed(tasks):
            results.append(future.result() * 1000)  # to ms

    results.sort()
    print(f"Samples: {len(results)}")
    print(f"Mean TTFT:   {statistics.mean(results):.1f} ms")
    print(f"Median TTFT: {statistics.median(results):.1f} ms")
    print(f"P50:         {results[len(results)//2]:.1f} ms")
    print(f"P90:         {results[int(len(results)*0.9)]:.1f} ms")
    print(f"P99:         {results[int(len(results)*0.99)]:.1f} ms")
    print(f"Min:         {results[0]:.1f} ms")
    print(f"Max:         {results[-1]:.1f} ms")

benchmark(model="meta-llama/Llama-3-8B", trials_per_prompt=10, concurrency=4)

This harness captures the full latency distribution, which is far more informative than a single mean. In production, the P99 TTFT is often the metric you should optimize for, because it represents the worst experience a meaningful fraction of users will encounter.

Concurrent Load Benchmarking

TTFT under load behaves very differently from TTFT in isolation. As concurrent requests increase, queueing time grows and prefill compute contends for GPU resources. Use a load generator to map TTFT against concurrency:

def sweep_concurrency(model: str, levels=(1, 2, 4, 8, 16, 32)):
    for c in levels:
        print(f"\n=== Concurrency: {c} ===")
        benchmark(model, trials_per_prompt=3, concurrency=c)

sweep_concurrency("meta-llama/Llama-3-8B")

Plotting median and P99 TTFT against concurrency reveals the inflection point where your deployment begins to degrade — invaluable capacity-planning data.

Factors That Influence TTFT

Understanding the levers that move TTFT helps you interpret benchmark results and diagnose regressions:

Best Practices for Benchmarking TTFT

1. Warm Up Before Measuring

Cold starts inflate TTFT due to model loading, CUDA kernel compilation, and KV cache allocation. Always issue a few throwaway requests before recording measurements:

# Warmup
for _ in range(3):
    try:
        single_request("Hello", model="meta-llama/Llama-3-8B")
    except Exception:
        pass

2. Use Realistic Prompts

Synthetic micro-prompts ("Hi") understate real-world TTFT. Curate a prompt set that mirrors your production traffic, including long system prompts, few-shot examples, and varied output lengths. Log anonymized production prompts and replay them in your benchmark.

3. Report Distributions, Not Averages

TTFT distributions are typically right-skewed. A mean of 400ms can hide a P99 of 3 seconds. Always report percentiles (P50, P90, P99) alongside the mean, and visualize histograms to spot multi-modal behavior caused by batching or caching effects.

4. Control for Client-Side Overhead

SDK parsing, JSON serialization, and TLS handshake all add latency. For the most accurate server-side TTFT, measure from a client on the same machine or network segment, and subtract a baseline measured with an empty prompt if needed.

5. Isolate Variables

When comparing configurations (e.g., FP16 vs. INT8), change one variable at a time. Run benchmarks on dedicated hardware with no other GPU workloads. Pin CPU cores and disable frequency scaling to reduce variance:

# Lock GPU clocks for consistent measurements
# nvidia-smi -lgc 1410,1410
# nvidia-smi --persistence-mode=1

6. Benchmark Continuously in CI

TTFT regressions sneak in with model updates, framework upgrades, and traffic pattern shifts. Integrate your benchmark harness into CI and alert on P99 TTFT increases beyond a threshold. Store results in a time-series database to track trends over weeks and months.

7. Measure Under Realistic Load

Idle benchmarks are misleading. Replay production traffic patterns — including bursty arrivals and mixed prompt lengths — to understand how TTFT behaves when the system is actually stressed. Tools like vegeta, k6, or custom async load generators work well here.

Optimizing TTFT

Once you can measure TTFT reliably, common optimization strategies include:

Conclusion

Time to First Token is the single most impactful latency metric for streaming LLM applications because it directly governs perceived responsiveness. By building a disciplined benchmarking harness that measures TTFT across realistic prompts, concurrency levels, and percentile thresholds, you gain the visibility needed to diagnose regressions and guide optimization. Pair accurate measurement with targeted optimizations — prefix caching, prompt engineering, quantization, and capacity tuning — and you can deliver streaming experiences that feel instantaneous to users. Treat TTFT as a first-class citizen in your observability stack and CI pipeline, and your LLM-powered products will remain fast and trustworthy as they scale.

— Ad —

Google AdSense will appear here after approval

← Back to all articles