Introduction to Tool Use with llama.cpp
Tool use, also known as function calling, is one of the most powerful capabilities you can add to a local LLM deployment. With llama.cpp, you can run capable models entirely on your own hardware while still giving them the ability to call external functions, query databases, fetch live data, and orchestrate complex workflows. This guide walks through the patterns, mechanics, and best practices for implementing tool use with llama.cpp.
What Is Tool Use?
Tool use is a pattern where the language model generates structured output (typically JSON) that represents a request to call an external function. The host application executes that function and returns the result back to the model, which then continues reasoning. This loop — model proposes a call, application executes it, model consumes the result — is the foundation of agentic behavior.
Why It Matters for llama.cpp Users
Running models locally with llama.cpp gives you privacy, cost control, and offline capability. But a model in isolation is limited to its training data. Tool use bridges that gap, letting local models interact with your filesystem, APIs, databases, and custom business logic. This combination is especially valuable for privacy-sensitive domains like healthcare, finance, and internal enterprise tooling where sending data to cloud APIs is not an option.
How llama.cpp Handles Tool Use
llama.cpp supports tool use through its server component (llama-server) and through the underlying C/C++ API. The server exposes an OpenAI-compatible endpoint at /v1/chat/completions that accepts a tools parameter. When the model decides to call a tool, the response includes a tool_calls array instead of (or alongside) regular content.
Under the hood, llama.cpp uses chat templates embedded in the GGUF model metadata. Models fine-tuned for function calling — such as Qwen2.5, Hermes, Llama 3.1, and Mistral function-calling variants — include templates that format tool definitions and parse tool-call outputs correctly. The server handles the parsing, so your application code can work with a clean JSON interface.
Starting the Server with Tool Support
First, build or obtain llama.cpp and start the server with a function-calling-capable model:
# Clone and build llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release
# Start the server with a function-calling model
./build/bin/llama-server \
-m models/qwen2.5-7b-instruct-q5_k_m.gguf \
--host 0.0.0.0 \
--port 8080 \
--ctx-size 8192 \
--n-gpu-layers 35
The key requirement is that your model supports function calling through its chat template. You can verify this by checking the model's documentation or inspecting the GGUF metadata with llama-gguf.
Basic Tool Use Pattern
Let's start with a minimal example. We'll define a single tool — a weather lookup — and let the model call it. Here's a Python client using the standard requests library:
import requests
import json
LLAMA_URL = "http://localhost:8080/v1/chat/completions"
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a given city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city name, e.g. 'San Francisco'"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["city"]
}
}
}
]
def get_weather(city: str, unit: str = "celsius") -> dict:
"""Simulated weather function."""
return {
"city": city,
"temperature": 22 if unit == "celsius" else 72,
"unit": unit,
"condition": "partly cloudy"
}
def chat_with_tools(messages, tools):
response = requests.post(LLAMA_URL, json={
"model": "qwen2.5",
"messages": messages,
"tools": tools,
"tool_choice": "auto",
"temperature": 0.7
})
return response.json()
# Initial user request
messages = [
{"role": "user", "content": "What's the weather in Tokyo in celsius?"}
]
result = chat_with_tools(messages, tools)
message = result["choices"][0]["message"]
print("Model response:", json.dumps(message, indent=2))
The model's response will look something like this:
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{\"city\": \"Tokyo\", \"unit\": \"celsius\"}"
}
}
]
}
Completing the Tool Call Loop
The model has proposed a tool call, but nothing has happened yet. Your application must execute the function and feed the result back. Here's the complete loop:
# Execute the tool call
if message.get("tool_calls"):
# Add the assistant's tool call to conversation
messages.append(message)
for tool_call in message["tool_calls"]:
func_name = tool_call["function"]["name"]
func_args = json.loads(tool_call["function"]["arguments"])
# Dispatch to the actual function
if func_name == "get_weather":
tool_result = get_weather(**func_args)
else:
tool_result = {"error": f"Unknown function: {func_name}"}
# Append the tool result
messages.append({
"role": "tool",
"tool_call_id": tool_call["id"],
"content": json.dumps(tool_result)
})
# Send the result back to the model for a final response
final_result = chat_with_tools(messages, tools)
final_message = final_result["choices"][0]["message"]
print("Final answer:", final_message["content"])
else:
print("Direct answer:", message["content"])
The model now receives the weather data and generates a natural language response like: "The current weather in Tokyo is 22°C with partly cloudy skies."
Multi-Tool and Parallel Call Patterns
Real applications rarely have just one tool. Let's expand to multiple tools and handle the case where the model calls several in a single turn.
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string"}
},
"required": ["city"]
}
}
},
{
"type": "function",
"function": {
"name": "search_documents",
"description": "Search internal documents by query",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"limit": {"type": "integer", "default": 5}
},
"required": ["query"]
}
}
},
{
"type": "function",
"function": {
"name": "send_email",
"description": "Send an email to a recipient",
"parameters": {
"type": "object",
"properties": {
"to": {"type": "string"},
"subject": {"type": "string"},
"body": {"type": "string"}
},
"required": ["to", "subject", "body"]
}
}
}
]
# Tool registry for dispatch
TOOL_REGISTRY = {
"get_weather": get_weather,
"search_documents": lambda query, limit=5: {"results": [
{"title": f"Doc {i}", "snippet": f"Content about {query}..."}
for i in range(min(limit, 3))
]},
"send_email": lambda to, subject, body: {"status": "sent", "to": to}
}
Handling Parallel Tool Calls
Some models can emit multiple tool calls in a single response. Your loop should handle all of them before sending results back:
def run_agent(user_message, tools, max_iterations=5):
messages = [{"role": "user", "content": user_message}]
for iteration in range(max_iterations):
result = chat_with_tools(messages, tools)
message = result["choices"][0]["message"]
# If no tool calls, we have a final answer
if not message.get("tool_calls"):
return message["content"]
# Add assistant message with tool calls
messages.append(message)
# Process ALL tool calls in this turn
for tool_call in message["tool_calls"]:
func_name = tool_call["function"]["name"]
func_args = json.loads(tool_call["function"]["arguments"])
print(f" Calling {func_name}({func_args})")
if func_name in TOOL_REGISTRY:
try:
tool_result = TOOL_REGISTRY[func_name](**func_args)
except Exception as e:
tool_result = {"error": str(e)}
else:
tool_result = {"error": f"Unknown function: {func_name}"}
messages.append({
"role": "tool",
"tool_call_id": tool_call["id"],
"content": json.dumps(tool_result)
})
print(f" Iteration {iteration + 1} complete, continuing...")
return "Max iterations reached without final answer."
# Example: the model may call get_weather AND search_documents
answer = run_agent(
"Check the weather in Berlin and search our docs for 'vacation policy'",
tools
)
print("Agent answer:", answer)
Chained Tool Calls
More complex tasks require the model to chain tool calls — using the output of one call as input to the next. For example, a user might ask the model to "find a document about Q3 revenue and email a summary to the finance team." The model needs to search, read the results, compose a summary, and then send an email.
answer = run_agent(
"Search our documents for 'Q3 revenue report', then send an email "
"to finance@company.com with a summary of what you find. "
"Use 'Q3 Revenue Summary' as the subject.",
tools
)
print(answer)
The agent loop handles this naturally. On the first iteration, the model calls search_documents. On the second iteration, after receiving the search results, it calls send_email with a composed summary. On the third iteration, after receiving the email confirmation, it produces a final text response. The max_iterations parameter prevents infinite loops.
Forced Tool Selection
Sometimes you want to force the model to use a specific tool rather than letting it choose. The tool_choice parameter controls this:
# Force a specific tool
response = requests.post(LLAMA_URL, json={
"model": "qwen2.5",
"messages": [{"role": "user", "content": "Tell me about Paris"}],
"tools": tools,
"tool_choice": {"type": "function", "function": {"name": "get_weather"}},
"temperature": 0.0
})
# Force any tool (model picks which one)
response = requests.post(LLAMA_URL, json={
"model": "qwen2.5",
"messages": [{"role": "user", "content": "Tell me about Paris"}],
"tools": tools,
"tool_choice": "required",
"temperature": 0.0
})
# No tools at all (force a text response)
response = requests.post(LLAMA_URL, json={
"model": "qwen2.5",
"messages": [{"role": "user", "content": "Tell me about Paris"}],
"tools": tools,
"tool_choice": "none",
"temperature": 0.0
})
Using the C++ API Directly
If you're embedding llama.cpp directly in a C++ application rather than using the server, you can implement tool use by managing the chat template manually. Here's a simplified pattern:
#include "llama.h"
#include <string>
#include <vector>
#include <json.hpp>
using json = nlohmann::json;
class ToolAgent {
llama_model* model;
llama_context* ctx;
std::vector<llama_chat_message> chat_history;
public:
ToolAgent(const std::string& model_path) {
llama_backend_init();
llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = 35;
model = llama_model_load_from_file(model_path.c_str(), model_params);
llama_context_params ctx_params = llama_context_default_params();
ctx_params.n_ctx = 8192;
ctx = llama_init_from_model(model, ctx_params);
}
std::string format_with_tools(const json& tools) {
// Build the messages array including a system message
// with tool definitions, then apply the chat template
std::vector<llama_chat_message> msgs;
// Add tool definitions as a system message
std::string tool_system = "You have access to these tools:\n"
+ tools.dump(2) +
"\nCall tools by generating the appropriate format.";
msgs.push_back({"system", tool_system.c_str()});
for (auto& m : chat_history) {
msgs.push_back(m);
}
// Apply chat template
std::vector<char> buffer(8192);
int len = llama_chat_apply_template(
model,
nullptr, // use built-in template
msgs.data(),
msgs.size(),
true, // add_assistant
buffer.data(),
buffer.size()
);
return std::string(buffer.data(), len);
}
~ToolAgent() {
llama_free(ctx);
llama_model_free(model);
llama_backend_free();
}
};
Note that the C++ approach requires you to handle tool-call parsing yourself unless your model's template includes special tokens for tool calls. For most applications, using the HTTP server is significantly simpler and more maintainable.
Best Practices
Write Precise Tool Descriptions
The tool description is the model's only guide for when and how to use a function. Be specific about behavior, edge cases, and expected formats:
{
"type": "function",
"function": {
"name": "query_database",
"description": (
"Execute a read-only SQL query against the analytics database. "
"Only SELECT statements are allowed. Returns up to 100 rows. "
"Use this for retrieving metrics, counts, and aggregations. "
"Do NOT use for INSERT, UPDATE, or DELETE operations."
),
"parameters": {
"type": "object",
"properties": {
"sql": {
"type": "string",
"description": "A valid SQL SELECT query"
}
},
"required": ["sql"]
}
}
}
Validate and Sanitize Arguments
Never trust model-generated arguments blindly. Validate types, ranges, and content before executing:
def safe_dispatch(func_name, func_args, registry):
if func_name not in registry:
return {"error": f"Unknown function: {func_name}"}
# Validate argument types against schema
schema = TOOL_SCHEMAS.get(func_name, {})
for required in schema.get("required", []):
if required not in func_args:
return {"error": f"Missing required parameter: {required}"}
# Sanitize string inputs
for key, val in func_args.items():
if isinstance(val, str) and len(val) > 10000:
return {"error": f"Parameter {key} exceeds maximum length"}
try:
return registry[func_name](**func_args)
except TypeError as e:
return {"error": f"Invalid arguments: {str(e)}"}
except Exception as e:
return {"error": f"Execution failed: {str(e)}"}
Use Low Temperature for Tool Calls
Tool calling is a structured task. High temperature introduces variability that can break JSON parsing or cause the model to hallucinate parameters. Use temperature: 0.0 to 0.3 for the tool-calling turns. You can increase temperature for the final natural-language response if you want more creative output.
Implement Timeouts and Limits
Always cap the number of tool-call iterations and add timeouts to prevent runaway agents:
import signal
class TimeoutError(Exception):
pass
def with_timeout(seconds, func, *args, **kwargs):
def handler(signum, frame):
raise TimeoutError("Function timed out")
signal.signal(signal.SIGALRM, handler)
signal.alarm(seconds)
try:
result = func(*args, **kwargs)
finally:
signal.alarm(0)
return result
# In your agent loop:
tool_result = with_timeout(10, safe_dispatch, func_name, func_args, TOOL_REGISTRY)
Log Everything for Debugging
Tool-use chains can be hard to debug. Log every step — the model's raw output, parsed arguments, function results, and the full message history:
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')
logger = logging.getLogger("tool_agent")
def run_agent_with_logging(user_message, tools, max_iterations=5):
messages = [{"role": "user", "content": user_message}]
logger.info(f"Starting agent with: {user_message}")
for iteration in range(max_iterations):
logger.info(f"--- Iteration {iteration + 1} ---")
result = chat_with_tools(messages, tools)
message = result["choices"][0]["message"]
logger.info(f"Model output: {json.dumps(message, indent=2)}")
if not message.get("tool_calls"):
logger.info("No tool calls, returning final answer")
return message["content"]
messages.append(message)
for tool_call in message["tool_calls"]:
func_name = tool_call["function"]["name"]
func_args = json.loads(tool_call["function"]["arguments"])
logger.info(f"Executing {func_name} with {func_args}")
tool_result = safe_dispatch(func_name, func_args, TOOL_REGISTRY)
logger.info(f"Result: {tool_result}")
messages.append({
"role": "tool",
"tool_call_id": tool_call["id"],
"content": json.dumps(tool_result)
})
logger.warning("Max iterations reached")
return "Max iterations reached."
Choose the Right Model
Not all GGUF models support tool use equally well. Models known for strong function-calling performance with llama.cpp include:
- Qwen2.5-Instruct (7B, 14B, 32B, 72B) — excellent native function calling support
- Llama 3.1 / 3.3 Instruct (8B, 70B) — built-in tool calling format
- Hermes 3 — fine-tuned specifically for function calling with a custom template
- Mistral / Mixtral Instruct — supports a function-calling format
- Command R / Command R+ — designed for tool use and RAG
Smaller models (under 7B parameters) often struggle with complex multi-tool scenarios. For production tool-use applications, 14B or larger is recommended when hardware permits.
Handle Errors Gracefully
When a tool fails, return a clear error message as the tool result. The model can often recover by trying a different approach or informing the user:
{
"role": "tool",
"tool_call_id": "call_abc123",
"content": "{\"error\": \"City 'xyz' not found. Please provide a valid city name.\"}"
}
The model will typically respond with something like: "I couldn't find a city called 'xyz'. Could you double-check the spelling or provide a different city name?"
Advanced Pattern: Stateful Tool Sessions
For applications like code interpreters or database sessions, tools may need to maintain state across calls. Design your tools to accept and return session identifiers:
tools = [
{
"type": "function",
"function": {
"name": "open_db_session",
"description": "Open a database session, returns a session_id",
"parameters": {
"type": "object",
"properties": {
"database": {"type": "string"}
},
"required": ["database"]
}
}
},
{
"type": "function",
"function": {
"name": "query_with_session",
"description": "Run a SQL query using an existing session",
"parameters": {
"type": "object",
"properties": {
"session_id": {"type": "string"},
"sql": {"type": "string"}
},
"required": ["session_id", "sql"]
}
}
},
{
"type": "function",
"function": {
"name": "close_db_session",
"description": "Close and clean up a database session",
"parameters": {
"type": "object",
"properties": {
"session_id": {"type": "string"}
},
"required": ["session_id"]
}
}
}
]
# Session store
sessions = {}
def open_db_session(database):
session_id = f"sess_{len(sessions)}"
sessions[session_id] = {"db": database, "connected": True}
return {"session_id": session_id, "status": "connected"}
def query_with_session(session_id, sql):
if session_id not in sessions:
return {"error": "Invalid or expired session"}
# Execute query...
return {"rows": [{"count": 42}], "rowcount": 1}
def close_db_session(session_id):
if session_id in sessions:
del sessions[session_id]
return {"status": "closed"}
return {"error": "Session not found"}
Conclusion
Tool use transforms llama.cpp from a text generator into an interactive agent capable of real-world actions. By leveraging the OpenAI-compatible server endpoint, you get a clean JSON interface for defining tools, dispatching calls, and feeding results back — all while keeping your data on your own hardware. The key to success lies in writing precise tool descriptions, validating arguments rigorously, choosing a model with strong function-calling support, and implementing robust error handling and iteration limits. Start simple with a single tool, verify the loop works end to end, then gradually add complexity with multi-tool scenarios, chained calls, and stateful sessions. With these patterns in place, you can build private, capable AI agents that extend local models with whatever functionality your application requires.