Function Calling at Scale with vLLM: Complete Guide

Introduction to Function Calling with vLLM

Function calling has become one of the most powerful capabilities of modern large language models (LLMs). It allows models to interact with external tools, APIs, and systems by generating structured outputs that can be parsed and executed programmatically. When you combine this capability with vLLM — a high-throughput, memory-efficient inference engine — you unlock the ability to run function calling at scale, serving thousands of concurrent requests with minimal latency.

In this guide, we will explore what function calling is, why running it at scale matters, how to implement it using vLLM, and the best practices you should follow to build production-grade systems.

What Is Function Calling?

Function calling is a mechanism where an LLM is given a set of tool definitions (functions with names, descriptions, and parameter schemas) and, when prompted, the model decides whether to call one of those functions. Instead of generating free-form text, the model emits a structured JSON object that matches the function's parameter schema. The host application can then execute the function and return the result back to the model for further reasoning.

A typical function calling workflow looks like this:

• The user sends a message to the application.
• The application passes the message along with available function definitions to the LLM.
• The LLM decides to call one or more functions and returns structured arguments.
• The application executes the function(s) and collects results.
• The results are sent back to the LLM, which produces a final natural-language response.

This loop enables LLMs to act as intelligent agents that can fetch live data, perform calculations, query databases, and orchestrate complex workflows.

Why Function Calling at Scale Matters

Most tutorials cover function calling with a single request to an API like OpenAI. However, real-world applications — such as customer support agents, coding assistants, and autonomous workflows — require handling hundreds or thousands of concurrent users. This is where scale becomes a challenge.

Running function calling at scale introduces several concerns:

• Throughput: You need to process many requests per second without queuing delays.
• Latency: Users expect sub-second time-to-first-token, even under load.
• Cost: Self-hosting with vLLM can dramatically reduce per-token costs compared to managed APIs.
• Consistency: Structured outputs must be reliable; malformed JSON breaks downstream systems.
• Concurrency: The inference engine must handle parallel tool calls and multi-turn conversations efficiently.

vLLM addresses these concerns through PagedAttention, continuous batching, and optimized KV cache management, making it an ideal choice for high-volume function calling workloads.

How vLLM Supports Function Calling

vLLM supports function calling through several mechanisms. The most common approaches are:

• Guided decoding: vLLM integrates with libraries like outlines and lm-format-enforcer to constrain output to valid JSON matching a schema.
• Chat templates: Many popular models (such as Hermes, Qwen, and Llama 3.1) ship with chat templates that natively support tool calling syntax.
• Structured output APIs: vLLM's OpenAI-compatible server exposes a /v1/chat/completions endpoint with tools and tool_choice parameters.

This means you can often swap your OpenAI API client to point at a vLLM server with minimal code changes.

Setting Up vLLM for Function Calling

Installation

Start by installing vLLM. It is recommended to use a Python virtual environment and a machine with a CUDA-capable GPU.

pip install vllm
pip install outlines

Launching the OpenAI-Compatible Server

The easiest way to serve function calling is through vLLM's OpenAI-compatible API server. Launch it with a model that supports tool calling, such as Qwen/Qwen2.5-7B-Instruct or meta-llama/Llama-3.1-8B-Instruct.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The --enable-auto-tool-choice flag enables automatic tool selection, and --tool-call-parser specifies how the model's output should be parsed into structured tool calls. Different models use different parsers; common options include hermes, mistral, llama3_json, and internlm.

Defining Functions and Calling Them

Basic Example with the OpenAI Client

Because vLLM exposes an OpenAI-compatible API, you can use the standard openai Python library. Here is a complete example that defines a weather function, lets the model call it, and returns the result.

from openai import OpenAI
import json

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy-key"
)

# Define the tools available to the model
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a given city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The name of the city, e.g. San Francisco"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["city"]
            }
        }
    }
]

# Mock implementation of the function
def get_weather(city: str, unit: str = "celsius") -> dict:
    return {
        "city": city,
        "temperature": 22 if unit == "celsius" else 72,
        "condition": "sunny",
        "unit": unit
    }

# First turn: ask the model
messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

message = response.choices[0].message
messages.append(message)

# Check if the model wants to call a function
if message.tool_calls:
    for tool_call in message.tool_calls:
        name = tool_call.function.name
        args = json.loads(tool_call.function.arguments)
        print(f"Model requested: {name}({args})")

        # Execute the function
        if name == "get_weather":
            result = get_weather(**args)
        else:
            result = {"error": "Unknown function"}

        # Append the tool result back to the conversation
        messages.append({
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": json.dumps(result)
        })

    # Second turn: let the model produce a final answer
    final_response = client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct",
        messages=messages,
        tools=tools
    )
    print(final_response.choices[0].message.content)

This example demonstrates the full agentic loop: the model decides to call get_weather, the application executes it, and the model uses the result to answer the user.

Scaling Up with Concurrent Requests

Using Async Batching

To handle many users simultaneously, you should use asynchronous requests. vLLM's continuous batching engine is designed to process concurrent requests efficiently — new requests are merged into active batches without waiting for previous ones to finish.

import asyncio
import json
from openai import AsyncOpenAI

async_client = AsyncOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy-key"
)

async def handle_query(user_query: str, tools: list) -> str:
    messages = [{"role": "user", "content": user_query}]
    response = await async_client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct",
        messages=messages,
        tools=tools,
        tool_choice="auto"
    )
    msg = response.choices[0].message

    if msg.tool_calls:
        messages.append(msg)
        for tc in msg.tool_calls:
            args = json.loads(tc.function.arguments)
            # Replace with real function dispatch
            result = {"echo": args}
            messages.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "content": json.dumps(result)
            })
        followup = await async_client.chat.completions.create(
            model="Qwen/Qwen2.5-7B-Instruct",
            messages=messages,
            tools=tools
        )
        return followup.choices[0].message.content
    return msg.content

async def main():
    queries = [
        "What is the weather in Paris?",
        "What is the weather in New York in fahrenheit?",
        "What is the weather in Sydney?",
    ]
    tasks = [handle_query(q, tools) for q in queries]
    results = await asyncio.gather(*tasks)
    for q, r in zip(queries, results):
        print(f"Q: {q}\nA: {r}\n")

asyncio.run(main())

Because vLLM uses continuous batching, all three requests are processed together in a single batch, dramatically improving throughput compared to sequential processing.

Benchmarking Throughput

To understand your system's capacity, you should benchmark it. A simple approach is to use asyncio to fire N concurrent requests and measure total time.

import time

async def benchmark(n: int = 100):
    queries = ["What is the weather in London?"] * n
    start = time.time()
    await asyncio.gather(*[handle_query(q, tools) for q in queries])
    elapsed = time.time() - start
    print(f"Completed {n} requests in {elapsed:.2f}s "
          f"({n / elapsed:.1f} req/s)")

asyncio.run(benchmark(100))

Tune the concurrency level based on your GPU memory and model size. vLLM will automatically manage batching, but extremely high concurrency can cause queueing if the KV cache is exhausted.

Ensuring Structured Output Reliability

One of the biggest challenges at scale is ensuring the model produces valid, parseable JSON. vLLM offers guided decoding to enforce structure at the token level, preventing malformed outputs before they are generated.

Using Guided JSON with the Python API

If you use vLLM directly via its Python API rather than the OpenAI-compatible server, you can pass a JSON schema to constrain generation.

from vllm import LLM, SamplingParams
from pydantic import BaseModel

class WeatherArgs(BaseModel):
    city: str
    unit: str = "celsius"

llm = LLM(model="Qwen/Qwen2.5-7B-Instruct")

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=256,
    guided_decoding={"json": WeatherArgs.model_json_schema()}
)

prompt = "Extract the city and unit from: 'What is the weather in Berlin in fahrenheit?'"
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)

This guarantees the output will conform to the WeatherArgs schema, eliminating JSON parsing errors entirely.

Using Guided JSON via the Server API

You can also pass guided_json through the OpenAI-compatible endpoint using the extra_body parameter.

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "What is the weather in Berlin in fahrenheit?"}],
    extra_body={
        "guided_json": {
            "type": "object",
            "properties": {
                "city": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["city"]
        }
    }
)

Best Practices for Production

Choose the Right Model and Parser

Not all models support function calling equally well. Models specifically fine-tuned for tool use — such as Qwen2.5-Instruct, Hermes 3, and Llama 3.1 — produce more reliable structured outputs. Always match the --tool-call-parser to the model's training format.

Keep Tool Descriptions Clear and Concise

The model relies on tool descriptions to decide when and how to call functions. Write descriptions that clearly state the purpose, expected inputs, and any constraints. Ambiguous descriptions lead to incorrect or unnecessary calls.

Limit the Number of Tools

Passing too many tools increases context length and can confuse the model. Group related tools and use a retrieval step to select the most relevant subset per query. A practical limit is 10–20 tools per request.

Handle Errors Gracefully

Functions can fail — APIs time out, databases return errors, and arguments may be invalid. Always wrap function execution in error handling and return structured error messages back to the model so it can recover or ask the user for clarification.

def safe_execute(name: str, args: dict) -> dict:
    try:
        if name == "get_weather":
            return get_weather(**args)
        return {"error": f"Unknown function: {name}"}
    except TypeError as e:
        return {"error": f"Invalid arguments: {e}"}
    except Exception as e:
        return {"error": f"Execution failed: {e}"}

Set Appropriate Sampling Parameters

For function calling, use low or zero temperature to maximize determinism and reduce the chance of malformed outputs. Set a reasonable max_tokens limit to prevent runaway generation.

Monitor and Tune Concurrency

Use vLLM's built-in metrics endpoint (available at /metrics in Prometheus format) to track request latency, queue depth, and GPU utilization. Adjust your concurrency limits and consider horizontal scaling with multiple vLLM instances behind a load balancer when a single GPU is saturated.

Cache Function Results

Many function calls are repetitive (for example, weather lookups for the same city). Implement a caching layer to avoid redundant external API calls and reduce latency. This is especially important at scale where rate limits on external services can become a bottleneck.

Conclusion

Function calling transforms LLMs from passive text generators into active agents that can interact with the world. By pairing function calling with vLLM's high-throughput inference engine, you can serve agentic workloads at scale with low latency, predictable costs, and reliable structured outputs. The key to success lies in choosing the right model and parser, writing clear tool definitions, enforcing output structure with guided decoding, handling errors robustly, and continuously monitoring performance. With these practices in place, vLLM provides a powerful foundation for building production-grade AI agents that can handle thousands of concurrent users without breaking a sweat.