How to Benchmark Local LLM Performance with llama-bench

When you run large language models locally, raw throughput matters as much as model quality. A model that produces brilliant answers at 2 tokens per second may be unusable for interactive applications, while a slightly less capable model running at 60 tokens per second could deliver a far better user experience. llama-bench is the official benchmarking tool bundled with the llama.cpp project, designed to give you reproducible, comparable performance numbers across models, prompt sizes, and hardware configurations.

What Is llama-bench?

llama-bench is a command-line utility shipped with llama.cpp that measures two critical metrics for local LLM inference:

tg128 — Token generation speed (tokens/second) for 128 output tokens, representing the model's decoding throughput.
pp512 — Prompt processing speed (tokens/second) for 512 input tokens, representing how fast the model ingests context.

Unlike ad-hoc timing scripts, llama-bench runs multiple iterations, reports averages, and outputs results in a structured format that can be exported as CSV or Markdown. This makes it suitable for systematic hardware comparisons, regression testing, and sharing reproducible results with the community.

Why Benchmarking Matters

Local LLM performance is influenced by a tangled web of factors: quantization level, context length, batch size, thread count, GPU offload layers, memory bandwidth, and even CPU thermal throttling. Without a standardized benchmark, you are left guessing which configuration actually works best on your machine.

Concrete reasons to benchmark include:

Choosing the right quantization trade-off between quality and speed.
Determining how many layers to offload to your GPU before memory or bandwidth becomes a bottleneck.
Comparing CPU-only inference against GPU acceleration to justify hardware upgrades.
Detecting performance regressions when updating llama.cpp or switching model formats.
Estimating whether a model can sustain real-time chat latency for your application.

Installing llama-bench

Because llama-bench is part of llama.cpp, you build it from source. The following commands clone the repository and compile with CUDA support enabled. Adjust the backend flags for your hardware (Metal, ROCm, Vulkan, or CPU-only).

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# The binary lives here
ls build/bin/llama-bench

On macOS with Apple Silicon, the build is simpler because Metal is enabled by default:

cmake -B build
cmake --build build --config Release -j

Verify the installation by printing the help text:

./build/bin/llama-bench --help

Running Your First Benchmark

The simplest invocation points llama-bench at a single GGUF model file and uses sensible defaults for thread count and output length:

./build/bin/llama-bench -m models/llama-3-8b-q4_k_m.gguf

This produces output similar to:

| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------ | -------------------: |
| llama 3B Q4_K_M - 4.69 GiB     |     4.69 GiB |     8.03 B | CUDA       |       8 |         pp512 |       2341.82 ± 18.41 |
| llama 3B Q4_K_M - 4.69 GiB     |     4.69 GiB |     8.03 B | CUDA       |       8 |         tg128 |         52.14 ± 0.31  |

The pp512 row tells you how fast the model processes input prompts, while tg128 tells you how fast it generates new tokens. For interactive chat, tg128 is the number that determines perceived latency.

Comparing Multiple Models

One of the most powerful features of llama-bench is the ability to sweep across multiple models and configurations in a single run. Pass several -m flags, and the tool iterates over each combination:

./build/bin/llama-bench \
  -m models/llama-3-8b-q4_k_m.gguf \
  -m models/llama-3-8b-q8_0.gguf \
  -m models/mistral-7b-q4_k_m.gguf \
  -p 512 -n 128

You can also sweep over parameters like thread count, batch size, and GPU offload layers using repeated flags:

./build/bin/llama-bench \
  -m models/llama-3-8b-q4_k_m.gguf \
  -t 4,8,16 \
  -ngl 0,10,20,33 \
  -p 512 -n 128

This single command runs twelve combinations, letting you pinpoint the exact offload depth where performance plateaus — a critical insight for machines with limited VRAM.

Exporting Results

For documentation and analysis, export results to CSV or Markdown using the -oe flag:

./build/bin/llama-bench \
  -m models/llama-3-8b-q4_k_m.gguf \
  -p 512 -n 128 \
  -oe csv > results.csv

# Or Markdown for pasting into GitHub issues
./build/bin/llama-bench \
  -m models/llama-3-8b-q4_k_m.gguf \
  -p 512 -n 128 \
  -oe md > results.md

The CSV output includes columns for every parameter, making it easy to load into pandas or a spreadsheet for visualization:

import pandas as pd

df = pd.read_csv("results.csv")
print(df[["model", "n_gpu_layers", "t/s"]].sort_values("t/s", ascending=False))

Controlling Test Parameters

Fine-grained control over the benchmark workload is essential for matching real-world usage. The most important flags are:

-p <n> — Prompt length in tokens for the prompt-processing test.
-n <n> — Number of tokens to generate for the generation test.
-t <n> — Number of CPU threads.
-ngl <n> — Number of layers to offload to the GPU.
-b <n> — Batch size for prompt processing.
-ub <n> — Physical batch size (useful for memory-constrained GPUs).
-r <n> — Number of repetitions; higher values reduce variance.
--mlock — Lock model in RAM to prevent swapping.
--no-mmap — Disable memory mapping; loads the full model into RAM.

For a chat-style workload, use a longer prompt and shorter generation:

./build/bin/llama-bench \
  -m models/llama-3-8b-q4_k_m.gguf \
  -p 2048 -n 64 -r 5 -ngl 33

For a long-form generation workload, invert the ratio:

./build/bin/llama-bench \
  -m models/llama-3-8b-q4_k_m.gguf \
  -p 128 -n 512 -r 5 -ngl 33

Best Practices

To get trustworthy, reproducible numbers, follow these guidelines:

Warm up before measuring. The first run often triggers model loading, kernel compilation, or thermal ramp-up. Use -r 5 or higher so the reported average excludes cold-start effects.
Pin thread count to physical cores. Using more threads than physical cores usually degrades performance due to hyperthreading overhead. Experiment with -t values equal to and just below your physical core count.
Find the offload sweet spot. Sweeping -ngl reveals where VRAM bandwidth saturates. Beyond a certain point, adding more layers to the GPU yields diminishing returns or even slowdowns.
Disable background workloads. Close browsers, compilers, and other CPU-intensive processes. Even moderate background load can swing results by 20% or more.
Keep the model file consistent. When comparing hardware or configurations, use the exact same GGUF file. Different quantization levels change memory access patterns and cache behavior.
Report environment details. When sharing results, include CPU model, GPU, RAM type, OS, llama.cpp commit hash, and compiler flags. Without context, raw numbers are meaningless.
Use --mlock on servers. On shared or memory-pressured systems, locking the model prevents the OS from paging it out between runs, which would otherwise produce erratic timings.

Interpreting the Numbers

Understanding what the numbers mean for your application is just as important as collecting them. As a rough guide for interactive chat:

Below 10 t/s generation: noticeable lag, suitable only for background tasks.
10–25 t/s: usable but visibly slower than reading speed.
25–50 t/s: comfortable for most chat interfaces.
Above 50 t/s: feels instantaneous for typical responses.

Prompt processing speed matters most when your application sends large context windows, such as document summarization or retrieval-augmented generation. A model that generates quickly but processes prompts at 200 t/s will feel sluggish when you paste a 4,000-token document, because the user waits 20 seconds before the first token appears.

Conclusion

llama-bench transforms local LLM performance tuning from guesswork into a measurable, repeatable process. By systematically sweeping across models, quantizations, thread counts, and GPU offload depths, you can identify the configuration that delivers the best balance of speed and quality for your specific hardware and workload. The key is consistency: keep your test parameters fixed, run enough repetitions to smooth out variance, and always record your environment alongside the results. With disciplined benchmarking, you can confidently choose models, justify hardware decisions, and catch performance regressions before they reach your users.