Introduction to Tool Use with vLLM
Tool use (also known as function calling) has become one of the most important capabilities for production LLM applications. Instead of relying solely on the model's internal knowledge, tool use allows models to invoke external functions, query databases, call APIs, and interact with the real world. vLLM, the high-throughput inference engine, provides robust support for tool use through its OpenAI-compatible API, making it an excellent choice for deploying tool-augmented LLMs at scale.
In this guide, we'll explore the complete landscape of tool use patterns with vLLM — from basic function calling to advanced multi-step agent workflows. Whether you're building a simple chatbot that checks the weather or a complex autonomous agent that orchestrates dozens of tools, understanding these patterns will help you architect reliable, production-ready systems.
What Is Tool Use and Why It Matters
Tool use is the mechanism by which a language model can signal that it wants to call an external function rather than (or in addition to) generating a text response. The model receives a description of available tools, decides when a tool is needed, produces structured arguments, and then incorporates the tool's output into its reasoning. This transforms a static text generator into an interactive system that can access real-time data, perform computations, and take actions.
Why Tool Use Matters in Production
- Accuracy: Models can retrieve factual, up-to-date information instead of hallucinating from stale training data.
- Capability extension: Tools allow models to perform tasks they cannot do alone, such as running code, querying SQL databases, or making HTTP requests.
- Cost efficiency: Smaller models augmented with tools can outperform much larger models on specialized tasks.
- Safety and control: Explicit tool definitions give developers fine-grained control over what actions a model can take.
- Composability: Tools can be shared, reused, and combined across different agents and applications.
vLLM supports tool use through its OpenAI-compatible /v1/chat/completions endpoint. Models like Qwen2.5, Hermes (Llama-based), Mistral, and others have been trained with tool-use capabilities and work seamlessly with vLLM's tool calling API.
Setting Up vLLM for Tool Use
Before diving into patterns, let's set up a vLLM server with a tool-capable model. We'll use Qwen2.5-7B-Instruct as our example, but the same approach works with other tool-trained models.
Starting the vLLM Server
# Install vLLM
pip install vllm
# Start the server with a tool-capable model
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8000 \
--enable-auto-tool-choice \
--tool-call-parser hermes
The --enable-auto-tool-choice flag is critical — it enables automatic tool call parsing. The --tool-call-parser flag specifies which parser to use. Different model families use different formats for tool calls, so you must match the parser to your model:
hermes— for Hermes, Qwen2.5, and many Llama-based tool modelsmistral— for Mistral and Mixtral modelsllama3_json— for Llama 3.1+ models with JSON tool callinginternlm— for InternLM modelsjamba— for AI21 Jamba models
Installing the Client Library
pip install openai
We'll use the official OpenAI Python client, pointing it at our local vLLM server. Since vLLM is API-compatible, any OpenAI client library works without modification.
Pattern 1: Basic Single Tool Call
The simplest tool use pattern involves a single function call. The model decides to call one tool, receives the result, and produces a final answer. This is the foundation of all tool use.
Defining a Tool
import json
import requests
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
# Define the tool schema
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a given city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The name of the city, e.g. San Francisco"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["city"]
}
}
}
]
# Implement the actual function
def get_weather(city: str, unit: str = "celsius") -> str:
# In production, call a real weather API here
mock_data = {
"San Francisco": {"celsius": 18, "fahrenheit": 64, "condition": "Foggy"},
"Tokyo": {"celsius": 25, "fahrenheit": 77, "condition": "Clear"},
"London": {"celsius": 12, "fahrenheit": 54, "condition": "Rainy"},
}
city_data = mock_data.get(city, {"celsius": 20, "fahrenheit": 68, "condition": "Unknown"})
return json.dumps({"city": city, "temperature": city_data[unit], "unit": unit, "condition": city_data["condition"]})
Making the Tool Call Request
messages = [
{"role": "user", "content": "What's the weather like in Tokyo right now?"}
]
# Step 1: Send the request with tools
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto"
)
# Step 2: Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
print(f"Model wants to call: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
# Step 3: Execute the tool
args = json.loads(tool_call.function.arguments)
result = get_weather(**args)
print(f"Tool result: {result}")
# Step 4: Feed the result back to the model
messages.append(response.choices[0].message)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result
})
# Step 5: Get the final response
final_response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools
)
print(f"Final answer: {final_response.choices[0].message.content}")
else:
print(f"Direct answer: {response.choices[0].message.content}")
This five-step flow — send, detect, execute, feed back, finalize — is the core loop of every tool use system. More complex patterns are variations and extensions of this basic cycle.
Pattern 2: Parallel Tool Calls
Many modern tool-use models can call multiple tools in a single response. This is useful when a query requires information from several independent sources. For example, comparing weather across multiple cities or fetching data from multiple endpoints simultaneously.
messages = [
{"role": "user", "content": "Compare the weather in San Francisco, Tokyo, and London. Which is warmest?"}
]
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto"
)
assistant_message = response.choices[0].message
messages.append(assistant_message)
# Handle multiple tool calls
if assistant_message.tool_calls:
for tool_call in assistant_message.tool_calls:
func_name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
# Dispatch to the correct function
if func_name == "get_weather":
result = get_weather(**args)
else:
result = json.dumps({"error": f"Unknown function: {func_name}"})
# Append each tool result
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result
})
# Get the final synthesized response
final_response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools
)
print(final_response.choices[0].message.content)
The key insight here is that all tool results are appended as separate tool role messages, each referencing its corresponding tool_call_id. The model then synthesizes all results into a single coherent answer.
Pattern 3: Multi-Step Tool Chaining
Sometimes a single round of tool calls isn't enough. The model may need to call a tool, analyze the result, then call another tool based on what it learned. This requires a loop that continues until the model produces a final answer without any tool calls.
def run_agent(user_query: str, tools: list, tool_map: dict, max_iterations: int = 10):
"""Run a multi-step agent loop with tool use."""
messages = [
{"role": "system", "content": "You are a helpful assistant. Use tools when needed to answer accurately."},
{"role": "user", "content": user_query}
]
for iteration in range(max_iterations):
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto"
)
assistant_msg = response.choices[0].message
messages.append(assistant_msg)
# If no tool calls, we're done
if not assistant_msg.tool_calls:
print(f"[Iteration {iteration}] Final answer generated")
return assistant_msg.content
# Execute each tool call
for tool_call in assistant_msg.tool_calls:
func_name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
print(f"[Iteration {iteration}] Calling {func_name}({args})")
if func_name in tool_map:
try:
result = tool_map[func_name](**args)
except Exception as e:
result = json.dumps({"error": str(e)})
else:
result = json.dumps({"error": f"Unknown function: {func_name}"})
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
return "Max iterations reached without a final answer."
# Define multiple tools
tools = [
{
"type": "function",
"function": {
"name": "search_products",
"description": "Search for products by keyword",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search keyword"},
"limit": {"type": "integer", "description": "Max results to return"}
},
"required": ["query"]
}
}
},
{
"type": "function",
"function": {
"name": "get_product_details",
"description": "Get detailed info about a specific product by ID",
"parameters": {
"type": "object",
"properties": {
"product_id": {"type": "string", "description": "The product ID"}
},
"required": ["product_id"]
}
}
},
{
"type": "function",
"function": {
"name": "check_inventory",
"description": "Check if a product is in stock",
"parameters": {
"type": "object",
"properties": {
"product_id": {"type": "string", "description": "The product ID"}
},
"required": ["product_id"]
}
}
}
]
# Mock implementations
def search_products(query, limit=5):
return json.dumps([
{"id": "p001", "name": f"{query} Pro", "price": 299},
{"id": "p002", "name": f"{query} Lite", "price": 149}
][:limit])
def get_product_details(product_id):
db = {"p001": {"id": "p001", "name": "Widget Pro", "price": 299, "specs": "Premium model"},
"p002": {"id": "p002", "name": "Widget Lite", "price": 149, "specs": "Budget model"}}
return json.dumps(db.get(product_id, {"error": "Not found"}))
def check_inventory(product_id):
return json.dumps({"product_id": product_id, "in_stock": product_id == "p001", "quantity": 42 if product_id == "p001" else 0})
tool_map = {
"search_products": search_products,
"get_product_details": get_product_details,
"check_inventory": check_inventory
}
# Run the agent
answer = run_agent(
"Find me a widget, show me the details of the cheapest one, and tell me if it's in stock.",
tools, tool_map
)
print(answer)
In this pattern, the model first searches for products, then uses the returned product IDs to fetch details, and finally checks inventory — each step depending on the previous. The loop continues until the model decides no more tool calls are needed.
Pattern 4: Tool Choice Control
vLLM supports several tool_choice options that give you control over when and how tools are used. Understanding these options is essential for building predictable agent behavior.
# Option 1: "auto" — model decides whether to use tools
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Hello, how are you?"}],
tools=tools,
tool_choice="auto" # Model will likely NOT call a tool for this
)
# Option 2: "required" — force the model to call at least one tool
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=tools,
tool_choice="required" # Model MUST call a tool
)
# Option 3: "none" — explicitly prevent tool use
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "Explain how rainbows work."}],
tools=tools,
tool_choice="none" # Model will give a text-only answer
)
# Option 4: Specific tool — force a particular tool
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "I need weather data for Berlin."}],
tools=tools,
tool_choice={"type": "function", "function": {"name": "get_weather"}}
)
Using tool_choice="required" or specifying a particular function is especially useful in pipeline architectures where you want deterministic behavior rather than model discretion.
Pattern 5: Structured Output with Tools
Tool use can be leveraged as a structured output mechanism. Even when you don't need to call an external function, you can define a "tool" that simply enforces a JSON schema on the model's output. This is often more reliable than prompt-based JSON extraction.
extraction_tool = [
{
"type": "function",
"function": {
"name": "extract_entity",
"description": "Extract structured entity information from text",
"parameters": {
"type": "object",
"properties": {
"name": {"type": "string", "description": "Person's full name"},
"age": {"type": "integer", "description": "Person's age"},
"email": {"type": "string", "description": "Email address"},
"skills": {
"type": "array",
"items": {"type": "string"},
"description": "List of skills"
}
},
"required": ["name"]
}
}
}
]
text = "Jane Smith is a 32-year-old software engineer. You can reach her at jane.smith@example.com. She knows Python, Rust, and Kubernetes."
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": f"Extract entity information from this text:\n\n{text}"}],
tools=extraction_tool,
tool_choice={"type": "function", "function": {"name": "extract_entity"}}
)
if response.choices[0].message.tool_calls:
extracted = json.loads(response.choices[0].message.tool_calls[0].function.arguments)
print(json.dumps(extracted, indent=2))
# Output:
# {
# "name": "Jane Smith",
# "age": 32,
# "email": "jane.smith@example.com",
# "skills": ["Python", "Rust", "Kubernetes"]
# }
This pattern is powerful for data extraction pipelines, form filling, and any scenario where you need reliable structured output from unstructured text.
Pattern 6: Error Handling and Recovery
In production, tools fail. APIs return errors, arguments are malformed, and network timeouts occur. A robust tool use system must handle these gracefully and allow the model to recover.
def run_agent_with_recovery(user_query, tools, tool_map, max_iterations=10):
messages = [
{"role": "system", "content": """You are a helpful assistant with access to tools.
If a tool returns an error, try to fix your approach and retry, or explain the error to the user."""},
{"role": "user", "content": user_query}
]
for i in range(max_iterations):
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto"
)
msg = response.choices[0].message
messages.append(msg)
if not msg.tool_calls:
return msg.content
for tool_call in msg.tool_calls:
func_name = tool_call.function.name
# Parse arguments with error handling
try:
args = json.loads(tool_call.function.arguments)
except json.JSONDecodeError as e:
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({"error": f"Invalid JSON arguments: {e}"})
})
continue
# Validate required parameters
if func_name not in tool_map:
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({"error": f"Function '{func_name}' is not available. Available: {list(tool_map.keys())}"})
})
continue
# Execute with timeout and error capture
try:
result = tool_map[func_name](**args)
# Validate result is not empty
if not result:
result = json.dumps({"warning": "Tool returned empty result"})
except TypeError as e:
result = json.dumps({"error": f"Argument error: {e}. Check parameter names and types."})
except Exception as e:
result = json.dumps({"error": f"Execution failed: {str(e)}"})
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
return "Unable to complete the request after multiple attempts."
The key principle is that errors are fed back to the model as tool results. A well-trained tool-use model can read the error message, adjust its arguments, and retry — much like a human developer debugging.
Pattern 7: Streaming with Tool Calls
For responsive user interfaces, streaming is essential. vLLM supports streaming responses even with tool calls, though the handling is slightly different since tool call arguments arrive incrementally.
def stream_with_tools(user_query, tools, tool_map):
messages = [{"role": "user", "content": user_query}]
# First, get the tool call decision (non-streaming for reliability)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto"
)
msg = response.choices[0].message
messages.append(msg)
if msg.tool_calls:
for tool_call in msg.tool_calls:
args = json.loads(tool_call.function.arguments)
result = tool_map[tool_call.function.name](**args)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
# Stream the final response after tool results
stream = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="", flush=True)
print() # newline
else:
# No tools needed, stream directly
stream = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
tools=tools,
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
stream_with_tools("What's the weather in London?", tools, {"get_weather": get_weather})
Pattern 8: Building a Tool Registry
As your application grows, managing tools individually becomes unwieldy. A tool registry pattern centralizes tool definitions and implementations, making it easy to add, remove, and document tools.
from dataclasses import dataclass
from typing import Callable, Any
import inspect
@dataclass
class RegisteredTool:
name: str
description: str
func: Callable
parameters: dict
class ToolRegistry:
def __init__(self):
self._tools: dict[str, RegisteredTool] = {}
def register(self, func: Callable, description: str = None):
"""Register a function as a tool, auto-generating schema from type hints."""
sig = inspect.signature(func)
params = {"type": "object", "properties": {}, "required": []}
for param_name, param in sig.parameters.items():
param_type = param.annotation
json_type = self._python_type_to_json(param_type)
params["properties"][param_name] = {
"type": json_type,
"description": f"Parameter {param_name}"
}
if param.default == inspect.Parameter.empty:
params["required"].append(param_name)
desc = description or func.__doc__ or f"Tool: {func.__name__}"
self._tools[func.__name__] = RegisteredTool(
name=func.__name__,
description=desc,
func=func,
parameters=params
)
return func
def get_openai_tools(self) -> list:
"""Return tools in OpenAI API format."""
return [
{
"type": "function",
"function": {
"name": t.name,
"description": t.description,
"parameters": t.parameters
}
}
for t in self._tools.values()
]
def get_tool_map(self) -> dict:
"""Return a name -> function mapping for execution."""
return {name: t.func for name, t in self._tools.items()}
def _python_type_to_json(self, py_type) -> str:
mapping = {str: "string", int: "integer", float: "number", bool: "boolean"}
return mapping.get(py_type, "string")
# Usage
registry = ToolRegistry()
@registry.register
def calculate_mortgage(principal: float, annual_rate: float, years: int) -> str:
"""Calculate monthly mortgage payment."""
monthly_rate = annual_rate / 12 / 100
num_payments = years * 12
if monthly_rate == 0:
payment = principal / num_payments
else:
payment = principal * (monthly_rate * (1 + monthly_rate) ** num_payments) / ((1 + monthly_rate) ** num_payments - 1)
return json.dumps({"monthly_payment": round(payment, 2), "total_paid": round(payment * num_payments, 2)})
@registry.register
def convert_currency(amount: float, from_currency: str, to_currency: str) -> str:
"""Convert an amount from one currency to another."""
rates = {"USD": 1.0, "EUR": 0.92, "GBP": 0.79, "JPY": 150.0}
usd_amount = amount / rates.get(from_currency, 1.0)
converted = usd_amount * rates.get(to_currency, 1.0)
return json.dumps({"amount": round(converted, 2), "currency": to_currency})
# Now use the registry with the agent loop
tools = registry.get_openai_tools()
tool_map = registry.get_tool_map()
answer = run_agent(
"I have a $300,000 mortgage at 6.5% interest for 30 years. What's my monthly payment? Also convert it to EUR.",
tools, tool_map
)
print(answer)
This registry pattern auto-generates tool schemas from Python type hints, reducing boilerplate and ensuring your schema stays in sync with your implementation.
Best Practices for Tool Use with vLLM
Tool Design
- Write clear, specific descriptions: The model relies entirely on your tool descriptions to decide when and how to use them. Include what the tool does, when to use it, and when NOT to use it.
- Keep tools focused: Each tool should do one thing well. Avoid "god functions" with many optional parameters that confuse the model.
- Use descriptive parameter names:
city_nameis clearer thanq. The model has to guess less. - Provide parameter descriptions: Especially for enums and formats. Tell the model if a date should be "YYYY-MM-DD" format.
- Limit the number of tools: Most models handle 5-15 tools well. Beyond that, consider a two-stage approach where the model first selects a category.
Model Selection
- Use tool-trained models: Models specifically trained for function calling (Qwen2.5, Hermes, Mistral) perform dramatically better than base models.
- Match the parser to the model: Using the wrong
--tool-call-parserwill produce garbled output. Always verify the parser matches your model family. - Test with your specific tools: Different models have different strengths. Benchmark a few models on your actual tool set before committing.
Production Considerations
- Always set max_iterations: An infinite loop of tool calls can burn through compute. Cap iterations and handle the timeout gracefully.
- Validate tool arguments server-side: Never trust model-generated arguments blindly. Validate types, ranges, and permissions before executing.
- Log all tool calls: For debugging and auditing, log the function name, arguments, results, and timing for every tool invocation.
- Handle rate limits: If your tools call external APIs, implement backoff and circuit breakers. Feed rate-limit errors back to the model so it can adjust.
- Use streaming for long responses: After tool execution, stream the final response to keep users engaged.
- Cache tool results: If the same tool is called with the same arguments repeatedly, cache the results to reduce latency and API costs.
Prompt Engineering for Tools
- System prompts matter: A good system prompt that explains when to use tools and how to present results significantly improves quality.
- Encourage step-by-step reasoning: Phrases like "Think step by step about which tools to use" can improve tool selection accuracy.
- Discourage unnecessary calls: If the model over-uses tools, add instructions like "Only use tools when you cannot answer from your own knowledge."
Conclusion
Tool use transforms vLLM from a text generation engine into a versatile application platform. By mastering the patterns in this guide — from basic single calls to multi-step agent loops, from error recovery to structured output extraction — you can build sophisticated AI systems that leverage external data and actions reliably. The key to success lies in thoughtful tool design, robust error handling, and choosing the right model and parser combination for your use case. Start simple with the basic pattern, add complexity as needed, and always test thoroughly with real-world queries before deploying to production. With vLLM's high-throughput inference and OpenAI-compatible tool use API, you have everything you need to build production-grade tool-augmented LLM applications at scale.