Benchmarking Inference Speed: llama.cpp vs Ollama

Introduction to Benchmarking LLM Inference Speed

When deploying Large Language Models (LLMs) locally, inference speed is often the deciding factor between a smooth user experience and a frustrating wait. Two of the most popular tools for running LLMs locally are llama.cpp and Ollama. Both are built on top of the same underlying C/C++ inference engine, but they expose different interfaces, abstractions, and overhead. This tutorial walks you through benchmarking both tools so you can make an informed decision for your specific use case.

What Is Inference Benchmarking?

Inference benchmarking is the process of measuring how fast a model generates tokens when given a prompt. The two key metrics you will encounter are:

• Tokens per second (tok/s): How many tokens the model generates per second during decoding.
• Time to first token (TTFT): The latency between sending the prompt and receiving the first generated token, which is dominated by prompt processing (prefill).
• Prompt processing speed: How quickly the model ingests the input prompt, measured in tokens per second.

These metrics matter because they directly affect perceived performance. A chatbot that takes 10 seconds to produce its first word feels broken, even if it then streams text quickly.

Why llama.cpp vs Ollama?

llama.cpp is the low-level C/C++ library that implements efficient inference for GGUF models. It exposes a CLI binary (llama-cli), a server (llama-server), and a benchmarking tool (llama-bench). It gives you fine-grained control over thread counts, batch sizes, GPU layers, and memory layout.

Ollama is a higher-level wrapper that bundles a model registry, a REST API, and a CLI. Under the hood, it uses llama.cpp for inference, but it adds its own process management, model loading, and API layer. This convenience can introduce overhead, which is exactly what we want to measure.

Prerequisites and Environment Setup

Before benchmarking, you need a consistent environment. Variations in hardware, OS, background processes, and even CPU thermals can skew results. For reproducible benchmarks, follow these steps:

System Requirements

• A machine with a modern CPU (AVX2 support recommended) or a GPU (CUDA, Metal, or ROCm).
• At least 16 GB of RAM for 7B parameter models in Q4 quantization.
• Linux or macOS recommended for the most consistent results.
• Both llama.cpp and Ollama installed from source or official releases.

Installing llama.cpp

Build llama.cpp from source to ensure you have the latest optimizations and the benchmarking tools:

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Build with CPU support (default)
cmake -B build
cmake --build build --config Release

# Or build with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Verify the benchmark tool exists
./build/bin/llama-bench --help

Installing Ollama

Ollama provides a simple install script for Linux and macOS:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Pull the same model you will use for llama.cpp benchmarks
ollama pull llama3.2:3b

Downloading a GGUF Model for llama.cpp

To make a fair comparison, use the exact same model and quantization in both tools. Download a GGUF file from Hugging Face:

# Download a Q4_K_M quantized model
wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

# Move it to a known location
mkdir -p ~/models
mv Llama-3.2-3B-Instruct-Q4_K_M.gguf ~/models/

For Ollama, the ollama pull llama3.2:3b command downloads a Q4_K_M quantized version by default, which matches what we use for llama.cpp. Always verify the quantization level matches by inspecting the model details on both platforms.

Benchmarking llama.cpp

llama.cpp ships with a dedicated benchmarking binary called llama-bench. This tool runs multiple iterations and reports prompt processing speed and generation speed in a clean tabular format.

Using llama-bench

The llama-bench tool is the most reliable way to benchmark llama.cpp because it handles warmup runs, multiple repetitions, and statistical reporting automatically:

# Basic CPU benchmark
./build/bin/llama-bench \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p 512 \
  -n 128 \
  -r 5

# Benchmark with GPU offloading (CUDA)
./build/bin/llama-bench \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p 512 \
  -n 128 \
  -ngl 99 \
  -r 5

The flags mean the following:

• -m: Path to the model file.
• -p 512: Process a 512-token prompt (measures prompt processing speed).
• -n 128: Generate 128 tokens (measures generation speed).
• -ngl 99: Offload all layers to GPU.
• -r 5: Repeat the benchmark 5 times and report averages.

The output will look something like this:

| model                          |       size |     params | backend    | ngl | test       | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------- | --: |
| llama 3B Q4_K_M                |  1.99 GiB  |     3.21 B | CUDA       |  99 | pp 512     | 4521.23 |
| llama 3B Q4_K_M                |  1.99 GiB  |     3.21 B | CUDA       |  99 | tg 128     |  142.87 |

Here, pp 512 is the prompt processing speed (4521 tokens/s) and tg 128 is the generation speed (142.87 tokens/s).

Benchmarking with llama-cli

For a more realistic end-to-end benchmark that includes actual text generation, use llama-cli:

# Time a single generation
time ./build/bin/llama-cli \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p "Explain quantum computing in three sentences." \
  -n 200 \
  -ngl 99 \
  -t 4 \
  --no-display-prompt \
  2>&1 | tail -5

The llama-cli output includes a summary line at the end:

llama_print_timings:        load time =     342.11 ms
llama_print_timings:      prompt eval time =      12.45 ms /   10 tokens (    1.25 ms per token,   803.21 tokens per second)
llama_print_timings:        eval time =    1402.33 ms /   200 tokens (    7.01 ms per token,   142.62 tokens per second)
llama_print_timings:       total time =    1414.78 ms /   210 tokens

This gives you the prompt eval speed (prefill) and the generation speed separately, which is useful for understanding where time is spent.

Benchmarking Ollama

Ollama does not ship with a dedicated benchmarking tool, so we need to build one. The most reliable approach is to use the Ollama REST API with a Python script that measures timing precisely.

Using the Ollama CLI for a Quick Test

For a quick sanity check, you can time the CLI directly:

# Time a generation with Ollama CLI
time ollama run llama3.2:3b "Explain quantum computing in three sentences."

However, this includes CLI startup overhead and does not give you tokens-per-second metrics. For accurate benchmarking, use the API.

Writing a Python Benchmark Script for Ollama

The following Python script uses the Ollama API to measure prompt processing time, generation speed, and time to first token:

import requests
import time
import json

OLLAMA_URL = "http://localhost:8343/api/generate"
MODEL = "llama3.2:3b"
PROMPT = "Explain quantum computing in three sentences."
NUM_PREDICT = 200
WARMUP_RUNS = 2
BENCHMARK_RUNS = 5

def benchmark_ollama():
    results = {"ttft": [], "gen_speed": [], "total_time": []}

    # Warmup runs to load the model into memory
    for _ in range(WARMUP_RUNS):
        requests.post(OLLAMA_URL, json={
            "model": MODEL,
            "prompt": "Hello",
            "stream": False,
            "options": {"num_predict": 5}
        })

    # Benchmark runs
    for i in range(BENCHMARK_RUNS):
        start_time = time.perf_counter()
        first_token_time = None
        token_count = 0

        response = requests.post(OLLAMA_URL, json={
            "model": MODEL,
            "prompt": PROMPT,
            "stream": True,
            "options": {"num_predict": NUM_PREDICT}
        }, stream=True)

        for line in response.iter_lines():
            if line:
                data = json.loads(line)
                if first_token_time is None and data.get("response"):
                    first_token_time = time.perf_counter()
                if data.get("response"):
                    token_count += 1
                if data.get("done"):
                    break

        end_time = time.perf_counter()

        ttft = first_token_time - start_time
        total_time = end_time - start_time
        gen_time = end_time - first_token_time
        gen_speed = token_count / gen_time if gen_time > 0 else 0

        results["ttft"].append(ttft)
        results["gen_speed"].append(gen_speed)
        results["total_time"].append(total_time)

        print(f"Run {i+1}: TTFT={ttft:.3f}s, "
              f"Gen={gen_speed:.2f} tok/s, "
              f"Total={total_time:.3f}s")

    # Report averages
    avg_ttft = sum(results["ttft"]) / len(results["ttft"])
    avg_gen = sum(results["gen_speed"]) / len(results["gen_speed"])
    avg_total = sum(results["total_time"]) / len(results["total_time"])

    print(f"\n--- Average over {BENCHMARK_RUNS} runs ---")
    print(f"Time to first token: {avg_ttft:.3f}s")
    print(f"Generation speed:    {avg_gen:.2f} tok/s")
    print(f"Total time:          {avg_total:.3f}s")

if __name__ == "__main__":
    benchmark_ollama()

Run the script after ensuring Ollama is running:

# Start Ollama server (if not already running)
ollama serve &

# Run the benchmark
python benchmark_ollama.py

Sample output:

Run 1: TTFT=0.142s, Gen=138.21 tok/s, Total=1.591s
Run 2: TTFT=0.138s, Gen=139.85 tok/s, Total=1.572s
Run 3: TTFT=0.145s, Gen=137.92 tok/s, Total=1.601s
Run 4: TTFT=0.140s, Gen=139.10 tok/s, Total=1.583s
Run 5: TTFT=0.143s, Gen=138.55 tok/s, Total=1.590s

--- Average over 5 runs ---
Time to first token: 0.142s
Generation speed:    138.73 tok/s
Total time:          1.587s

Using Ollama's Built-in Stats

Ollama also returns timing statistics in its non-streaming response. You can access these directly:

import requests
import json

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.2:3b",
    "prompt": "Explain quantum computing in three sentences.",
    "stream": False,
    "options": {"num_predict": 200}
})

data = response.json()
print(f"Prompt eval: {data['prompt_eval_count']} tokens in "
      f"{data['prompt_eval_duration']/1e9:.3f}s "
      f"({data['prompt_eval_count']/(data['prompt_eval_duration']/1e9):.2f} tok/s)")
print(f"Generation:  {data['eval_count']} tokens in "
      f"{data['eval_duration']/1e9:.3f}s "
      f"({data['eval_count']/(data['eval_duration']/1e9):.2f} tok/s)")
print(f"Total load time: {data['load_duration']/1e9:.3f}s")

This is the simplest way to get Ollama's own reported metrics, which you can then compare directly with llama-bench output.

Comparing Results Fairly

To make a valid comparison between llama.cpp and Ollama, you must control several variables. A common mistake is to compare a CPU-only llama.cpp run against a GPU-accelerated Ollama run, or vice versa.

Variables to Control

• Model and quantization: Use the same GGUF file or verify that Ollama's pulled model uses the same quantization.
• GPU offload: Set -ngl in llama.cpp to match Ollama's default behavior (Ollama offloads all layers to GPU when possible).
• Thread count: Match CPU thread counts. Ollama auto-detects, so check its logs for the thread count it uses.
• Prompt length: Use identical prompts with the same token count.
• Number of generated tokens: Keep num_predict consistent across both tools.
• Batch size: Ollama uses default batch sizes; match them in llama.cpp with -b and -ub flags if needed.

Creating a Side-by-Side Comparison Script

The following bash script runs both benchmarks and saves results for comparison:

#!/bin/bash
# compare_benchmarks.sh
# Run this after both llama.cpp and Ollama are installed and the model is pulled

MODEL_PATH="$HOME/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf"
OLLAMA_MODEL="llama3.2:3b"
PROMPT_TOKENS=512
GEN_TOKENS=128
RESULTS_FILE="benchmark_results.txt"

echo "=== Benchmark Comparison: llama.cpp vs Ollama ===" > "$RESULTS_FILE"
echo "Date: $(date)" >> "$RESULTS_FILE"
echo "" >> "$RESULTS_FILE"

# --- llama.cpp benchmark ---
echo "--- llama.cpp ---" >> "$RESULTS_FILE"
./build/bin/llama-bench \
  -m "$MODEL_PATH" \
  -p "$PROMPT_TOKENS" \
  -n "$GEN_TOKENS" \
  -ngl 99 \
  -r 5 >> "$RESULTS_FILE" 2>&1

echo "" >> "$RESULTS_FILE"

# --- Ollama benchmark ---
echo "--- Ollama ---" >> "$RESULTS_FILE"
python3 -c "
import requests, json

response = requests.post('http://localhost:11434/api/generate', json={
    'model': '$OLLAMA_MODEL',
    'prompt': 'The quick brown fox jumps over the lazy dog. ' * 40,
    'stream': False,
    'options': {'num_predict': $GEN_TOKENS}
})

data = response.json()
pp_speed = data['prompt_eval_count'] / (data['prompt_eval_duration'] / 1e9)
tg_speed = data['eval_count'] / (data['eval_duration'] / 1e9)

print(f'| ollama | pp $PROMPT_TOKENS | {pp_speed:.2f} t/s |')
print(f'| ollama | tg $GEN_TOKENS  | {tg_speed:.2f} t/s |')
" >> "$RESULTS_FILE" 2>&1

echo "" >> "$RESULTS_FILE"
echo "Results saved to $RESULTS_FILE"
cat "$RESULTS_FILE"

Interpreting the Results

Typically, you will observe the following patterns:

• Generation speed: llama.cpp and Ollama are usually within 2-5% of each other when configured identically, because Ollama uses llama.cpp under the hood.
• Prompt processing: llama.cpp may be slightly faster due to less overhead in the request pipeline.
• Time to first token: Ollama adds HTTP API overhead, typically 10-50 ms, which is negligible for long generations but noticeable for short ones.
• Cold start: Ollama may unload models from memory after an idle period, causing slower first requests. Use OLLAMA_KEEP_ALIVE=-1 to prevent this.

Best Practices for Reliable Benchmarks

Always Warm Up

The first inference after loading a model is always slower because memory is being allocated, caches are cold, and the CPU/GPU is ramping up. Always run 2-3 warmup iterations before recording measurements:

# Warmup before the real benchmark
./build/bin/llama-bench -m ~/models/model.gguf -p 64 -n 16 -r 1

# Real benchmark
./build/bin/llama-bench -m ~/models/model.gguf -p 512 -n 128 -r 5

Pin CPU Frequency

CPU frequency scaling can cause variance between runs. On Linux, you can pin the CPU governor to performance mode:

# Set performance governor (requires root)
sudo cpupower frequency-set -g performance

# Or manually set a fixed frequency
sudo cpupower frequency-set -f 3.5GHz

# Revert after benchmarking
sudo cpupower frequency-set -g ondemand

Isolate the System

Close background applications, disable unnecessary services, and avoid running benchmarks while updates or backups are active. Even a browser tab playing video can steal CPU cycles and skew results.

Use Consistent Prompt Token Counts

Always verify the actual token count of your prompt, not just an estimate. Use a tokenizer to confirm:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
prompt = "Explain quantum computing in three sentences."
token_count = len(tokenizer.encode(prompt))
print(f"Prompt token count: {token_count}")

Report All Relevant Context

When sharing benchmark results, always include the following context so others can reproduce your findings:

• Hardware: CPU model, GPU model, RAM amount, RAM bandwidth.
• Software: OS, llama.cpp commit hash, Ollama version, CUDA/driver version.
• Model: name, parameter count, quantization level, file size.
• Configuration: thread count, GPU layers offloaded, batch size, context length.
• Environment: CPU governor, background load, temperature throttling status.

Benchmark Different Prompt Lengths

Real-world workloads vary. A RAG application might send 2000-token prompts, while a chatbot sends 50-token prompts. Benchmark across a range of prompt sizes:

#!/bin/bash
# Benchmark across different prompt sizes
for PP in 64 128 256 512 1024 2048; do
  echo "=== Prompt size: $PP ==="
  ./build/bin/llama-bench \
    -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
    -p "$PP" \
    -n 128 \
    -ngl 99 \
    -r 3
  echo ""
done

Monitor Thermals

Thermal throttling can cause later runs to be slower than earlier ones. Monitor CPU and GPU temperatures during benchmarking:

# Monitor CPU temperature (Linux)
watch -n 1 'sensors | grep -i core'

# Monitor GPU temperature (NVIDIA)
watch -n 1 'nvidia-smi --query-gpu=temperature.gpu,clocks.sm,clocks.mem --format=csv'

If you see temperatures exceeding 85°C on CPU or 80°C on GPU, your results may be affected by throttling. Improve cooling or reduce clock speeds for consistent measurements.

Advanced: Benchmarking the Ollama API Server vs llama-server

Both tools provide HTTP server modes. For a fair API-to-API comparison, run llama-server and benchmark it against Ollama's server using the same client script.

Starting llama-server

# Start llama.cpp server with GPU offload
./build/bin/llama-server \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -ngl 99 \
  --port 8080 \
  --host 0.0.0.0

Benchmarking Both Servers

import requests
import time
import json

def benchmark_server(url, model_name, prompt, num_predict, runs=5):
    """Benchmark either llama-server or Ollama server."""
    speeds = []

    # Warmup
    for _ in range(2):
        requests.post(f"{url}/api/generate", json={
            "model": model_name,
            "prompt": "Hello",
            "stream": False,
            "options": {"num_predict": 5}
        })

    for i in range(runs):
        start = time.perf_counter()
        response = requests.post(f"{url}/api/generate", json={
            "model": model_name,
            "prompt": prompt,
            "stream": False,
            "options": {"num_predict": num_predict}
        })
        elapsed = time.perf_counter() - start
        data = response.json()

        if "eval_count" in data and "eval_duration" in data:
            speed = data["eval_count"] / (data["eval_duration"] / 1e9)
        else:
            speed = num_predict / elapsed

        speeds.append(speed)
        print(f"  Run {i+1}: {speed:.2f} tok/s ({elapsed:.3f}s)")

    avg = sum(speeds) / len(speeds)
    print(f"  Average: {avg:.2f} tok/s")
    return avg

PROMPT = "Explain quantum computing in three sentences." * 10
NUM_PREDICT = 200

print("=== llama-server ===")
llama_avg = benchmark_server(
    "http://localhost:8080", "default", PROMPT, NUM_PREDICT
)

print("\n=== Ollama ===")
ollama_avg = benchmark_server(
    "http://localhost:11434", "llama3.2:3b", PROMPT, NUM_PREDICT
)

print(f"\n=== Summary ===")
print(f"llama-server: {llama_avg:.2f} tok/s")
print(f"Ollama:       {ollama_avg:.2f} tok/s")
print(f"Difference:   {((llama_avg - ollama_avg) / ollama_avg * 100):+.1f}%")

This script gives you a direct apples-to-apples comparison of the HTTP API overhead introduced by each tool.

Conclusion

Benchmarking llama.cpp and Ollama reveals that both tools deliver nearly identical raw inference performance when configured with the same model, quantization, and hardware settings, which makes sense given that Ollama is built on top of llama.cpp. The differences that do exist come from the abstraction layers Ollama adds: HTTP API overhead, model lifecycle management, and automatic configuration detection. For maximum performance and control, llama.cpp with llama-bench is the right choice, especially in production environments where every millisecond counts. For developer convenience, rapid prototyping, and ease of deployment, Ollama's overhead of a few percent is almost always an acceptable tradeoff. The key to meaningful benchmarks is consistency: control your variables, warm up before measuring, monitor thermals, and always report the full context of your test environment so that results are reproducible and trustworthy.