GPU vs CPU for Self-Hosted Agent Inference: Real Numbers

GPU vs CPU for Self-Hosted Agent Inference: Real Numbers

What Is Self-Hosted Agent Inference?

Self-hosted agent inference means running large language models (LLMs) or AI agents on your own hardware rather than calling a cloud API like OpenAI or Anthropic. The inference step is where the model processes a prompt and generates tokens one by one—this is the core loop every agent executes when reasoning, tool-calling, or generating responses.

When you self-host, you face an immediate architectural decision: should inference run on CPU (using system RAM and x86/ARM cores) or GPU (using VRAM on CUDA-capable cards)? The answer depends on model size, batch size, acceptable latency, and budget. This tutorial gives you real benchmark numbers, working code, and a decision framework to make that choice confidently.

Why This Decision Matters

Choosing the wrong compute target can silently multiply your infrastructure costs by 3× to 10× while delivering unacceptable latency. A 7B-parameter model that runs happily at 50 tokens/second on a $400 consumer GPU might crawl at 3 tokens/second on a $3,000 Xeon server. Conversely, a CPU-only setup with enough RAM can run a 70B model that no consumer GPU can fit, for a fraction of cloud GPU pricing. Understanding the real numbers lets you:

• Right-size hardware — avoid overspending on GPUs when CPUs suffice for batch workloads
• Meet latency SLOs — know exactly what hardware delivers sub-2-second time-to-first-token
• Optimize agent loops — agents often call models dozens of times per task; a 500ms difference per call compounds dramatically
• Plan capacity — know how many concurrent users a single node can serve

The Real Numbers: GPU vs CPU Benchmarks

Below are measured numbers from a typical developer workstation (Ryzen 7950X, 64GB DDR5) and a single RTX 4090 (24GB VRAM). The model is Llama-3.1-8B-Instruct at FP16 (GPU) and Q4_K_M quantized (CPU via llama.cpp). All measurements use batch size 1, 256 input tokens, generating 128 output tokens.

Key takeaway: For interactive agent workloads where latency matters, GPU delivers an unmistakably better experience. For overnight batch processing, model experimentation with large models, or cost-sensitive deployments, CPU inference is entirely viable and often cheaper.

Understanding the Performance Gap

The GPU advantage comes from two architectural facts:

• Memory bandwidth: Token generation is memory-bandwidth-bound. An RTX 4090 has ~1,008 GB/s of memory bandwidth; dual-channel DDR5-6000 delivers ~96 GB/s. That 10× gap directly determines tokens-per-second on decode.
• Parallel compute for prefill: The prompt prefill phase processes all input tokens in parallel. A GPU with thousands of CUDA cores crushes this; a CPU must churn through it sequentially across a handful of cores.

Quantization (Q4_K_M) narrows the memory bandwidth gap by shrinking the model, but it cannot overcome a 10× bandwidth deficit. However, quantization also enables a single CPU node to run 70B-class models that would require multiple GPUs—an expensive proposition.

Practical Code: Benchmarking Your Own Hardware

Before committing to a deployment strategy, measure your own hardware. Below is a complete Python script that benchmarks TTFT and tokens-per-second for any OpenAI-compatible endpoint (vLLM, llama.cpp server, text-generation-inference).

#!/usr/bin/env python3
"""
benchmark_inference.py
Measures TTFT and decode speed for a local inference server.
Works with vLLM, llama.cpp server, TGI, or any OpenAI-compatible API.
"""

import time
import requests
from typing import Tuple

# ---- CONFIGURATION ----
API_URL = "http://localhost:8000/v1/completions"
MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct"
PROMPT = "Explain the architecture of a transformer model in detail, covering attention, feed-forward layers, and positional encoding. Be thorough."
MAX_TOKENS = 256
WARMUP_RUNS = 2
MEASURED_RUNS = 5
# -----------------------

def measure_inference(api_url: str, prompt: str, max_tokens: int) -> Tuple[float, float, int, int]:
    """
    Returns: (ttft_ms, total_ms, prompt_tokens, completion_tokens)
    """
    payload = {
        "model": MODEL_NAME,
        "prompt": prompt,
        "max_tokens": max_tokens,
        "temperature": 0.0,
        "stream": True
    }

    start_time = time.perf_counter()
    first_token_time = None
    last_token_time = None
    completion_tokens = 0
    prompt_tokens = 0

    with requests.post(api_url, json=payload, stream=True, timeout=120) as resp:
        resp.raise_for_status()
        for line in resp.iter_lines(decode_unicode=True):
            if not line or not line.startswith("data: "):
                continue
            data_str = line[6:]
            if data_str == "[DONE]":
                break
            import json
            try:
                chunk = json.loads(data_str)
            except json.JSONDecodeError:
                continue
            choices = chunk.get("choices", [])
            if not choices:
                continue
            usage = chunk.get("usage", {})
            if usage:
                prompt_tokens = usage.get("prompt_tokens", prompt_tokens)
            choice = choices[0]
            text = choice.get("text", "")
            if text and first_token_time is None:
                first_token_time = time.perf_counter()
                ttft = (first_token_time - start_time) * 1000.0
            if text:
                completion_tokens += 1
                last_token_time = time.perf_counter()

    total_time_ms = (last_token_time - start_time) * 1000.0 if last_token_time else 0
    ttft_ms = ttft if first_token_time else 0
    return ttft_ms, total_time_ms, prompt_tokens, completion_tokens

def main():
    print("Warming up...")
    for i in range(WARMUP_RUNS):
        print(f"  Warmup run {i+1}/{WARMUP_RUNS}")
        measure_inference(API_URL, PROMPT, 64)

    print("\nRunning measured benchmarks...")
    results = []
    for i in range(MEASURED_RUNS):
        print(f"  Run {i+1}/{MEASURED_RUNS}")
        ttft, total, prompt_tok, comp_tok = measure_inference(API_URL, PROMPT, MAX_TOKENS)
        results.append((ttft, total, prompt_tok, comp_tok))
        print(f"    TTFT: {ttft:.1f}ms | Total: {total:.1f}ms | Tokens: {comp_tok}")

    # Aggregate
    ttfts = [r[0] for r in results]
    totals = [r[1] for r in results]
    comp_tokens = [r[3] for r in results]
    avg_ttft = sum(ttfts) / len(ttfts)
    avg_total = sum(totals) / len(totals)
    avg_comp_tokens = sum(comp_tokens) / len(comp_tokens)
    decode_time_ms = avg_total - avg_ttft
    tokens_per_second = (avg_comp_tokens / decode_time_ms * 1000) if decode_time_ms > 0 else 0

    print("\n===== RESULTS =====")
    print(f"  Avg TTFT:           {avg_ttft:.1f} ms")
    print(f"  Avg total latency:  {avg_total:.1f} ms")
    print(f"  Avg completion tok: {avg_comp_tokens:.0f}")
    print(f"  Tokens/sec (decode): {tokens_per_second:.1f}")
    print(f"  Runs:               {MEASURED_RUNS}")
    print("===================")

if __name__ == "__main__":
    main()

Run this against your local inference server. For CPU testing, launch llama.cpp server first:

# Terminal 1: Start llama.cpp server on CPU with 8 threads
./llama-server \
  -m ./models/Llama-3.1-8B-Q4_K_M.gguf \
  -t 8 \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8000

For GPU testing, launch vLLM:

# Terminal 1: Start vLLM with the same model (FP16)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype float16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.95 \
  --port 8000

Then run the benchmark script in Terminal 2:

python benchmark_inference.py

Memory Bandwidth Quick-Test Script

If you want a raw memory bandwidth number—which is the single best predictor of token generation speed—use this compact C program. Compile and run it directly on your target hardware.

/*
 * mem_bandwidth_test.c
 * Compile: gcc -O3 -march=native -pthread mem_bandwidth_test.c -o mem_bw
 * Run: ./mem_bw
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <pthread.h>

#define ARRAY_SIZE (512L * 1024L * 1024L)  // 512 MB
#define ITERATIONS 10

double get_time_seconds() {
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
    return ts.tv_sec + ts.tv_nsec / 1e9;
}

int main() {
    printf("Allocating %ld MB buffer...\n", ARRAY_SIZE / (1024*1024));
    char *src = (char*)aligned_alloc(4096, ARRAY_SIZE);
    char *dst = (char*)aligned_alloc(4096, ARRAY_SIZE);
    if (!src || !dst) { perror("aligned_alloc"); return 1; }

    // Fill source with random data to defeat any caching tricks
    for (long i = 0; i < ARRAY_SIZE; i++) src[i] = (char)(i & 0xFF);

    double best_bw = 0.0;
    for (int iter = 0; iter < ITERATIONS; iter++) {
        double start = get_time_seconds();
        // memcpy with non-temporal stores on supported platforms
        memcpy(dst, src, ARRAY_SIZE);
        double elapsed = get_time_seconds() - start;
        double bw_gb_per_sec = (ARRAY_SIZE / (1024.0*1024.0*1024.0)) / elapsed;
        if (bw_gb_per_sec > best_bw) best_bw = bw_gb_per_sec;
        printf("  Run %d: %.2f GB/s\n", iter+1, bw_gb_per_sec);
    }
    printf("\nBest achievable memory bandwidth: %.2f GB/s\n", best_bw);
    printf("Estimated max tokens/sec (Q4_K_M, ~5 GB model): %.1f tok/s\n",
           best_bw / 5.0);
    free(src); free(dst);
    return 0;
}

On a typical dual-channel DDR5-6000 system, expect ~45–50 GB/s. An RTX 4090 will show ~950–1,000 GB/s. Divide your measured bandwidth by the model's per-token memory footprint (roughly model_size_in_GB / number_of_parameters_in_billions, about 0.7–1.0 GB per billion parameters for Q4 quantized) to estimate your ceiling token rate.

Cost Analysis: GPU vs CPU Over Time

Real numbers on cost often tip the decision. Below is a Python calculator that estimates monthly cost for a given workload.

#!/usr/bin/env python3
"""
cost_calculator.py
Estimates monthly infrastructure cost for self-hosted inference.
"""

# ---- INPUT PARAMETERS ----
REQUESTS_PER_HOUR = 5000          # Agent calls per hour
TOKENS_PROMPT_PER_REQUEST = 350   # Average prompt length
TOKENS_COMPLETION_PER_REQUEST = 150 # Average response length
GPU_TOKENS_PER_SECOND = 110       # Measured from benchmark
CPU_TOKENS_PER_SECOND = 8         # Measured from benchmark
GPU_HW_COST = 2400                # USD, one-time
CPU_HW_COST = 1200                # USD, one-time
GPU_POWER_WATTS = 400             # Under load
CPU_POWER_WATTS = 170             # Under load
POWER_COST_PER_KWH = 0.12         # USD
GPU_AMORTIZATION_MONTHS = 36      # Depreciation period
CPU_AMORTIZATION_MONTHS = 36
GPU_MAINTENANCE_MONTHLY = 15      # Cooling, spare parts
CPU_MAINTENANCE_MONTHLY = 5
# ---------------------------------

def compute_monthly_cost(hw_cost, amort_months, power_w, power_cost_kwh,
                         maint_monthly, tok_per_sec, req_per_hour,
                         tok_prompt, tok_completion):
    # Time per request in seconds
    total_tokens = tok_prompt + tok_completion
    seconds_per_request = total_tokens / tok_per_sec
    # Hours of compute needed per month
    compute_hours_per_month = (req_per_hour * seconds_per_request / 3600.0) * 730
    # Power cost
    kwh_per_month = compute_hours_per_month * power_w / 1000.0
    power_cost = kwh_per_month * power_cost_kwh
    # Hardware amortization
    hw_monthly = hw_cost / amort_months
    # Total
    total = hw_monthly + power_cost + maint_monthly
    # Throughput check: can one instance keep up?
    max_req_per_hour = 3600 / seconds_per_request
    instances_needed = req_per_hour / max_req_per_hour if max_req_per_hour > 0 else 999
    return total, instances_needed, compute_hours_per_month, power_cost

gpu_total, gpu_instances, gpu_hours, gpu_power_cost = compute_monthly_cost(
    GPU_HW_COST, GPU_AMORTIZATION_MONTHS, GPU_POWER_WATTS, POWER_COST_PER_KWH,
    GPU_MAINTENANCE_MONTHLY, GPU_TOKENS_PER_SECOND,
    REQUESTS_PER_HOUR, TOKENS_PROMPT_PER_REQUEST, TOKENS_COMPLETION_PER_REQUEST
)

cpu_total, cpu_instances, cpu_hours, cpu_power_cost = compute_monthly_cost(
    CPU_HW_COST, CPU_AMORTIZATION_MONTHS, CPU_POWER_WATTS, POWER_COST_PER_KWH,
    CPU_MAINTENANCE_MONTHLY, CPU_TOKENS_PER_SECOND,
    REQUESTS_PER_HOUR, TOKENS_PROMPT_PER_REQUEST, TOKENS_COMPLETION_PER_REQUEST
)

print("===== MONTHLY COST ESTIMATE =====")
print(f"Workload: {REQUESTS_PER_HOUR} requests/hour")
print(f"Tokens per request: {TOKENS_PROMPT_PER_REQUEST + TOKENS_COMPLETION_PER_REQUEST}")
print()
print(f"GPU option:")
print(f"  Instances needed:        {gpu_instances:.1f}")
print(f"  Compute hours/month:     {gpu_hours:.1f}")
print(f"  Power cost/month:        ${gpu_power_cost:.2f}")
print(f"  HW amortization/month:   ${GPU_HW_COST/GPU_AMORTIZATION_MONTHS:.2f}")
print(f"  Total monthly cost:      ${gpu_total:.2f}")
print()
print(f"CPU option:")
print(f"  Instances needed:        {cpu_instances:.1f}")
print(f"  Compute hours/month:     {cpu_hours:.1f}")
print(f"  Power cost/month:        ${cpu_power_cost:.2f}")
print(f"  HW amortization/month:   ${CPU_HW_COST/CPU_AMORTIZATION_MONTHS:.2f}")
print(f"  Total monthly cost:      ${cpu_total:.2f}")
print()
print("===== RECOMMENDATION =====")
if gpu_total < cpu_total and gpu_instances <= 2:
    print("GPU is cheaper AND meets throughput. Go GPU.")
elif cpu_instances <= 1 and cpu_total < gpu_total:
    print("CPU is cheaper and a single node handles the load. Go CPU.")
elif gpu_instances > 4:
    print("GPU throughput is excellent but you need many instances. Consider CPU scaling.")
else:
    print("Evaluate based on latency requirements first, then cost.")

Run this with your own measured numbers. You'll often find that for bursty, latency-sensitive agent workloads, a single GPU comfortably handles thousands of requests per hour, while CPU would need multiple nodes to keep up—and still fail latency targets.

Decision Framework: When to Use CPU, When GPU

Here is a practical decision matrix based on real-world agent deployment patterns:

Best Practices for CPU Inference

• Use quantized models: Q4_K_M offers the best balance of quality and speed. Q8_0 gives near-FP16 quality at half the memory. Never run FP32 on CPU—it wastes bandwidth.
• Bind threads to NUMA nodes: On multi-socket servers, pin llama.cpp threads to a single NUMA node to avoid cross-socket memory latency penalties that can halve throughput.
• Use mmap and numa affinity: Launch llama.cpp with --numa and ensure the model file is on local SSD, not network storage.
• Prefer Intel AMX or Apple AMX: Sapphire Rapids Xeons with AMX intrinsics, or Apple M-series chips with AMX coprocessors, deliver 2×–3× the token rate of vanilla AVX2 cores.
• Batch when possible: llama.cpp supports batch sizes >1. Even batch=2 improves aggregate throughput by 30–60% on many-core CPUs.

Best Practices for GPU Inference

• Use a production-grade engine: vLLM, TensorRT-LLM, or text-generation-inference (TGI) give continuous batching, paged attention, and much higher throughput than naive HuggingFace pipelines.
• Tune GPU memory utilization: Set --gpu-memory-utilization 0.95 in vLLM to maximize batch capacity without hitting OOM.
• Enable FP8 or FP16 KV cache: On Ampere/Hopper GPUs, FP8 KV cache cuts memory by 50% with negligible quality loss, doubling your batch capacity.
• Profile with NVIDIA Nsight: Identify whether you're compute-bound or memory-bound. Most LLM inference is memory-bound; if you're compute-bound, you may have a kernel issue.
• Monitor VRAM fragmentation: Long-running servers can fragment VRAM. Implement periodic restarts (every 24–48 hours) or use vLLM's prefix caching to mitigate.

The Hybrid Approach: CPU + GPU Together

Many real-world deployments combine both. A common pattern: use GPU for the primary agent model (fast, interactive) and CPU for auxiliary models like guardrails, intent classifiers, or embedding extraction. llama.cpp can offload some layers to GPU while keeping the rest on CPU, enabling models that exceed VRAM capacity:

# Offload 20 of 32 layers to GPU, keep rest on CPU RAM
./llama-server \
  -m ./models/Llama-3.1-70B-Q4_K_M.gguf \
  -t 16 \
  --n-gpu-layers 20 \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8000

This gives you the best of both worlds: GPU-accelerated attention on the most compute-intensive layers, while CPU handles the rest. On a 4090 + 64GB system, a 70B Q4 model can run at 4–6 tok/s—usable for batch work.

Apple Silicon: A Special Case

Apple M-series chips (M2 Ultra, M4 Max) occupy a unique middle ground. With unified memory sporting bandwidths of 100–400 GB/s, they outperform x86 CPUs by 3×–6× on token generation while consuming 40–90W. An M2 Ultra with 192GB RAM can run a 70B Q4 model at 8–12 tok/s—comparable to an entry-level GPU but with vastly more RAM. For developers who want a single quiet machine that handles everything from 1B to 70B models, Apple Silicon is the pragmatic choice.

# On an M2 Ultra Mac Studio, llama.cpp with Metal backend
./llama-server \
  -m ./models/Llama-3.1-70B-Q4_K_M.gguf \
  -t 16 \
  -ngl 32 \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8000
# Typical result: 9.3 tok/s, TTFT ~600ms, power draw 65W

Profiling Agent Workloads End-to-End

Agents don't just call the model once—they loop. A typical ReAct agent might call the LLM 3–8 times per user request. That means the latency gap multiplies. Here's a quick profiling snippet you can drop into your agent loop:

import time
import contextlib

@contextlib.contextmanager
def timed_step(name: str, log: list):
    start = time.perf_counter()
    yield
    elapsed_ms = (time.perf_counter() - start) * 1000.0
    log.append((name, elapsed_ms))

# Inside your agent loop:
agent_trace = []
for turn in range(max_turns):
    with timed_step("llm_call", agent_trace):
        response = llm.generate(prompt)  # Your inference call
    with timed_step("tool_execution", agent_trace):
        result = execute_tool(response.tool_calls)
    with timed_step("parse_output", agent_trace):
        parsed = parse_result(result)

# After agent finishes, print summary
total_llm_ms = sum(ms for name, ms in agent_trace if name == "llm_call")
total_tool_ms = sum(ms for name, ms in agent_trace if name == "tool_execution")
print(f"Agent finished {len(agent_trace)//3} turns")
print(f"  LLM time total: {total_llm_ms/1000:.2f}s ({total_llm_ms/len(agent_trace)*3:.0f}ms avg)")
print(f"  Tool time total: {total_tool_ms/1000:.2f}s")

Run this with CPU and then GPU. The difference in total_llm_ms will make the decision obvious. A 6-turn agent that spends 17 seconds in CPU inference versus 1.1 seconds on GPU is the difference between a snappy product and an unusable one.

Conclusion

The GPU-versus-CPU decision for self-hosted agent inference reduces to three numbers you can measure yourself: memory bandwidth (predicts decode speed), time-to-first-token (predicts perceived responsiveness), and dollars per token per month (predicts total cost). GPU dominates on the first two by wide margins—40× faster TTFT and 10×–14× higher token rates—making it the default choice for interactive agents. CPU excels on capacity and cost for batch workloads and large models. By running the benchmarks in this tutorial on your own hardware, you replace guesswork with data and make an informed choice that balances latency, throughput, and budget for your specific agent workload.