← Back to DevBytes
Developer Tutorial August 17, 2026 7 min read

Function Calling at Scale with llama.cpp: Complete Guide

Function Calling at Scale with llama.cpp: Complete Guide

Function calling — also known as tool use — has become one of the most important capabilities for production LLM applications. While cloud APIs like OpenAI popularized the feature, llama.cpp brings the same power to local, self-hosted models running on commodity hardware. This guide walks through everything you need to know to implement function calling at scale using llama.cpp, from basic concepts to production-grade batching strategies.

What Is Function Calling in llama.cpp?

Function calling is the mechanism by which a language model can emit structured requests to invoke external tools — database queries, API calls, calculators, search engines — rather than only generating free-form text. In llama.cpp, this is supported natively through the llama-chat and server APIs, which expose a tools parameter compatible with the OpenAI function-calling schema.

Under the hood, llama.cpp injects a special system prompt describing available tools, parses the model's output for structured tool-call tokens, and returns them as a structured JSON object. The application then executes the function and feeds the result back into the conversation.

Why It Matters

Prerequisites

Build llama.cpp with the server target enabled. The server component exposes an OpenAI-compatible REST endpoint that supports the tools field.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_BUILD_SERVER=ON
cmake --build build --config Release -j

Download a model that has been trained or fine-tuned for tool use. Good options include Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, or Hermes-3-Llama-3.1-8B, all available in GGUF format.

Starting the Server

Launch the llama-server binary with a model that supports tool calling. The --jinja flag is essential — it enables the chat template that injects tool definitions correctly.

./build/bin/llama-server \
  -m models/Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --port 8080 \
  --ctx-size 8192 \
  --jinja \
  --parallel 4

The --parallel 4 flag is the first step toward scaling: it allows the server to process four concurrent sequences within a single batch, sharing the KV cache memory across requests.

Defining Tools

Tools are defined using JSON Schema, identical to the OpenAI format. Each tool has a name, description, and a parameters schema.

{
  "model": "local-model",
  "messages": [
    {"role": "user", "content": "What is the weather in Tokyo and Paris?"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {"type": "string", "description": "City name"},
            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

Making a Function Call Request

Send the request to the OpenAI-compatible /v1/chat/completions endpoint:

import requests

resp = requests.post("http://localhost:8080/v1/chat/completions", json={
    "model": "local-model",
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
    "tools": [{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    }],
    "tool_choice": "auto"
})

data = resp.json()
msg = data["choices"][0]["message"]
if msg.get("tool_calls"):
    for call in msg["tool_calls"]:
        print(call["function"]["name"], call["function"]["arguments"])

The response will contain a tool_calls array with the function name and JSON-encoded arguments. Your application executes the function, then appends a tool role message with the result to continue the conversation.

Completing the Tool Loop

import json

messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]
messages.append(msg)  # assistant message with tool_calls

# Execute the function locally
result = {"city": "Tokyo", "temp_c": 22, "condition": "clear"}

messages.append({
    "role": "tool",
    "tool_call_id": msg["tool_calls"][0]["id"],
    "content": json.dumps(result)
})

final = requests.post("http://localhost:8080/v1/chat/completions", json={
    "model": "local-model",
    "messages": messages,
    "tools": []  # tools can be omitted on follow-up
}).json()

print(final["choices"][0]["message"]["content"])

Scaling to Concurrent Requests

Single-request function calling is straightforward. Scaling to hundreds or thousands of concurrent agent loops requires a different architecture. The key techniques are connection pooling, batched inference, and asynchronous orchestration.

Asynchronous Agent Pool

Use asyncio with aiohttp to maintain many in-flight conversations simultaneously. The llama.cpp server handles multiplexing internally when launched with --parallel N.

import asyncio
import aiohttp
import json

SERVER = "http://localhost:8080/v1/chat/completions"

WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
        }
    }
}

async def run_agent(session, query):
    messages = [{"role": "user", "content": query}]
    payload = {
        "model": "local-model",
        "messages": messages,
        "tools": [WEATHER_TOOL],
        "tool_choice": "auto"
    }
    async with session.post(SERVER, json=payload) as r:
        data = await r.json()
    msg = data["choices"][0]["message"]
    if not msg.get("tool_calls"):
        return msg["content"]

    messages.append(msg)
    for call in msg["tool_calls"]:
        args = json.loads(call["function"]["arguments"])
        tool_result = {"city": args["city"], "temp_c": 21, "condition": "clear"}
        messages.append({
            "role": "tool",
            "tool_call_id": call["id"],
            "content": json.dumps(tool_result)
        })

    payload2 = {"model": "local-model", "messages": messages}
    async with session.post(SERVER, json=payload2) as r:
        data = await r.json()
    return data["choices"][0]["message"]["content"]

async def main():
    queries = [f"What is the weather in city {i}?" for i in range(50)]
    connector = aiohttp.TCPConnector(limit=20)
    async with aiohttp.ClientSession(connector=connector) as session:
        results = await asyncio.gather(*[run_agent(session, q) for q in queries])
    for q, r in zip(queries, results):
        print(q, "->", r[:60])

asyncio.run(main())

Multi-Server Horizontal Scaling

A single llama.cpp instance is bounded by one GPU's memory and compute. For true scale, run multiple server instances behind a load balancer. A simple round-robin dispatcher works well because each request is independent.

import itertools
import aiohttp

SERVERS = itertools.cycle([
    "http://gpu-node-1:8080/v1/chat/completions",
    "http://gpu-node-2:8080/v1/chat/completions",
    "http://gpu-node-3:8080/v1/chat/completions",
])

async def dispatch(session, payload):
    url = next(SERVERS)
    async with session.post(url, json=payload) as r:
        return await r.json()

For production deployments, use a proper reverse proxy like nginx or HAProxy with health checks and least-connections routing. Each GPU node should run with --parallel set to a value that fits comfortably in VRAM — typically 2 to 8 concurrent sequences for an 8B model at Q4 quantization on a 24 GB card.

Grammar-Constrained Tool Calls

One advantage of llama.cpp over cloud APIs is the ability to constrain decoding with a GBNF grammar. This guarantees that tool-call arguments are valid JSON, eliminating parse failures at scale.

{
  "name": "tool_call_grammar",
  "grammar": """
root ::= "{" ws "\"city\"" ws ":" ws string ws "}"
string ::= "\"" ([^"\\] | "\\" .)* "\""
ws ::= [ \t\n]*
"""
}

Pass the grammar via the grammar field in the request body. This is especially valuable when running thousands of unattended agent loops where a single malformed JSON argument could crash a pipeline.

Best Practices

Monitoring Throughput

The llama-server exposes a /metrics endpoint compatible with Prometheus. Track tokens-per-second, queue depth, and active sequences to identify bottlenecks before they affect users.

# Prometheus scrape config
scrape_configs:
  - job_name: llama
    static_configs:
      - targets: ["gpu-node-1:8080", "gpu-node-2:8080"]
    metrics_path: /metrics

Key metrics to watch include llama_tokens_predicted_total for generation throughput and llama_tokens_predicted_seconds for per-request latency. If queue depth grows unbounded, add more GPU nodes or reduce --parallel per node to stabilize latency.

Conclusion

Function calling with llama.cpp gives developers a fully local, privacy-preserving, and cost-effective alternative to cloud APIs for building tool-using agents. By combining the OpenAI-compatible server interface with asynchronous orchestration, horizontal scaling across GPU nodes, and grammar-constrained decoding, you can run thousands of concurrent agent loops reliably on your own infrastructure. Start with a single server and the basic tool loop, then scale outward with connection pooling and load balancing as your workload grows. The result is an agentic pipeline that is both economically sustainable and fully under your control.

— Ad —

Google AdSense will appear here after approval

← Back to all articles