Function Calling at Scale with LlamaIndex: Complete Guide
Function calling has emerged as one of the most powerful capabilities of modern large language models (LLMs). When combined with LlamaIndex, a leading framework for building LLM applications, function calling becomes a scalable mechanism for connecting models to external tools, APIs, and data sources. This guide walks you through everything you need to know to implement function calling at scale using LlamaIndex, from foundational concepts to production-grade best practices.
What Is Function Calling?
Function calling is the ability of an LLM to recognize when a user's request requires an external action and to generate a structured response that specifies which function to invoke, along with the appropriate arguments. Instead of merely generating text, the model emits a structured payload — typically JSON — that your application can parse and execute against real-world tools.
For example, if a user asks, "What's the weather in Tokyo?", a function-calling-capable model can return a structured call like get_weather(location="Tokyo") rather than hallucinating a forecast. Your application then executes the function, retrieves real data, and feeds the result back to the model for a natural-language response.
Why Function Calling Matters at Scale
While a single function call is straightforward, running function calling at scale introduces a different set of challenges. Production systems often need to handle thousands of concurrent requests, manage dozens or hundreds of available tools, route calls intelligently, and maintain observability across complex agent workflows. LlamaIndex addresses these concerns with a unified abstraction layer that supports multiple model providers, structured output parsing, and composable agent architectures.
- Tool management: Register and dispatch many functions without bloating prompts.
- Provider flexibility: Switch between OpenAI, Anthropic, Mistral, and open-source models with minimal code changes.
- Structured outputs: Enforce schemas so downstream systems receive predictable data.
- Concurrency: Leverage async execution to handle high-throughput workloads.
- Observability: Integrate with LlamaIndex's callback handlers and tracing tools.
Setting Up Your Environment
Before writing any code, install the necessary packages. LlamaIndex is modular, so you only need to install the components you plan to use.
pip install llama-index llama-index-core llama-index-llms-openai
pip install llama-index-llms-anthropic llama-index-llms-mistralai
Set your API keys as environment variables so the LLM clients can authenticate automatically.
import os
os.environ["OPENAI_API_KEY"] = "sk-your-key-here"
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-your-key-here"
Defining Tools with FunctionTool
LlamaIndex provides the FunctionTool class, which wraps a standard Python function into a tool the LLM can call. The function's docstring and type hints are automatically converted into a schema the model uses to decide when and how to invoke the tool.
from llama_index.core.tools import FunctionTool
def get_weather(location: str, unit: str = "celsius") -> str:
"""Get the current weather for a given location.
Args:
location: The city name, e.g. "Tokyo" or "New York".
unit: Temperature unit, either "celsius" or "fahrenheit".
Returns:
A string describing the current weather.
"""
# In production, call a real weather API here
return f"The weather in {location} is 22 degrees {unit}, partly cloudy."
def search_documents(query: str, top_k: int = 5) -> str:
"""Search internal documents for a given query.
Args:
query: The search query string.
top_k: Number of top results to return.
Returns:
A string containing the search results.
"""
# In production, query a vector store here
return f"Top {top_k} results for '{query}': [doc1, doc2, doc3...]"
weather_tool = FunctionTool.from_defaults(fn=get_weather)
search_tool = FunctionTool.from_defaults(fn=search_documents)
Notice how the type annotations and docstrings are doing the heavy lifting. LlamaIndex inspects these to generate the JSON schema that gets sent to the model. Clear, descriptive docstrings directly improve the model's ability to select the right tool.
Building a Function-Calling Agent
With tools defined, you can now create an agent that uses function calling to decide which tool to invoke. LlamaIndex's FunctionAgent is designed specifically for models that support native function calling.
from llama_index.core.agent import FunctionAgent
from llama_index.llms.openai import OpenAI
llm = OpenAI(model="gpt-4o")
agent = FunctionAgent(
tools=[weather_tool, search_tool],
llm=llm,
system_prompt=(
"You are a helpful assistant. Use the available tools to answer "
"user questions accurately. Always prefer real data over guessing."
),
)
response = agent.run("What's the weather in Berlin right now?")
print(response)
The agent handles the full loop: it sends the user query and tool schemas to the model, receives the function call, executes the corresponding Python function, and feeds the result back to the model for a final natural-language answer.
Scaling Up with Multiple Tools
At scale, you may have dozens or hundreds of tools. Sending every tool schema in every request wastes tokens and can confuse the model. LlamaIndex offers several strategies to manage large tool sets effectively.
Tool Retrieval with ObjectIndex
One powerful approach is to index your tools themselves and retrieve only the relevant ones per query. LlamaIndex's ObjectIndex lets you treat tools as retrievable objects.
from llama_index.core import VectorStoreIndex
from llama_index.core.objects import ObjectIndex, SimpleToolNodeMapping
# Assume you have a large list of tools
all_tools = [weather_tool, search_tool] # ... plus many more
tool_mapping = SimpleToolNodeMapping.from_objects(all_tools)
obj_index = ObjectIndex.from_objects(
all_tools,
tool_mapping,
VectorStoreIndex,
)
# Retrieve only relevant tools for a given query
retrieved_tools = obj_index.as_retriever().retrieve("weather forecast")
print([t.metadata.name for t in retrieved_tools])
You can then pass only the retrieved tools to the agent for that specific request, dramatically reducing token usage and improving tool selection accuracy.
Async Execution for High Throughput
For production workloads, synchronous execution becomes a bottleneck. LlamaIndex supports async throughout the stack, allowing you to handle many concurrent requests efficiently.
import asyncio
from llama_index.core.agent import FunctionAgent
from llama_index.llms.openai import AsyncOpenAI
async def get_stock_price(symbol: str) -> str:
"""Get the current stock price for a given ticker symbol.
Args:
symbol: The stock ticker symbol, e.g. "AAPL".
"""
# Simulate an async API call
await asyncio.sleep(0.1)
return f"{symbol} is currently trading at $185.42"
stock_tool = FunctionTool.from_defaults(fn=get_stock_price, async_mode=True)
async_llm = OpenAI(model="gpt-4o", async_mode=True)
async_agent = FunctionAgent(
tools=[stock_tool, weather_tool, search_tool],
llm=async_llm,
system_prompt="You are a helpful financial and general assistant.",
)
async def handle_batch(queries: list[str]) -> list[str]:
tasks = [async_agent.achat(q) for q in queries]
return await asyncio.gather(*tasks)
queries = [
"What's the price of AAPL?",
"What's the weather in Paris?",
"Search for documents about Q3 earnings",
]
results = asyncio.run(handle_batch(queries))
for q, r in zip(queries, results):
print(f"Q: {q}\nA: {r}\n")
By using achat and asyncio.gather, you can process many user requests concurrently, which is essential when serving real-time applications.
Structured Output Enforcement
Function calling is also a reliable way to enforce structured output. Instead of hoping the model returns valid JSON, you define a Pydantic schema and let the model fill it in via a tool call.
from pydantic import BaseModel, Field
from llama_index.core.tools import FunctionTool
class CustomerSummary(BaseModel):
name: str = Field(description="Customer's full name")
sentiment: str = Field(description="positive, neutral, or negative")
key_concerns: list[str] = Field(description="Main issues mentioned")
recommended_action: str = Field(description="Suggested next step")
def extract_customer_summary(
name: str,
sentiment: str,
key_concerns: list[str],
recommended_action: str,
) -> CustomerSummary:
"""Extract a structured customer summary from a support conversation."""
return CustomerSummary(
name=name,
sentiment=sentiment,
key_concerns=key_concerns,
recommended_action=recommended_action,
)
summary_tool = FunctionTool.from_defaults(fn=extract_customer_summary)
extraction_agent = FunctionAgent(
tools=[summary_tool],
llm=llm,
system_prompt=(
"Analyze the customer conversation and call the "
"extract_customer_summary tool with the extracted information."
),
)
conversation = """
Customer Jane Doe reached out frustrated about a delayed shipment.
She mentioned the package was supposed to arrive 5 days ago and she
needs it for an event tomorrow. She also asked about a refund policy.
"""
result = extraction_agent.run(conversation)
print(result)
This pattern guarantees that downstream systems receive a validated, typed object rather than free-form text.
Working with Multiple Model Providers
One of LlamaIndex's strengths is provider abstraction. The same tools and agent logic work across different LLM providers with minimal changes.
from llama_index.llms.anthropic import Anthropic
from llama_index.llms.mistralai import MistralAI
# OpenAI
openai_agent = FunctionAgent(
tools=[weather_tool, search_tool],
llm=OpenAI(model="gpt-4o"),
system_prompt="You are a helpful assistant.",
)
# Anthropic
anthropic_agent = FunctionAgent(
tools=[weather_tool, search_tool],
llm=Anthropic(model="claude-3-5-sonnet-20241022"),
system_prompt="You are a helpful assistant.",
)
# Mistral
mistral_agent = FunctionAgent(
tools=[weather_tool, search_tool],
llm=MistralAI(model="mistral-large-latest"),
system_prompt="You are a helpful assistant.",
)
# All three agents use the same tools and logic
for name, agent in [("OpenAI", openai_agent),
("Anthropic", anthropic_agent),
("Mistral", mistral_agent)]:
response = agent.run("What's the weather in Sydney?")
print(f"{name}: {response}")
This abstraction lets you benchmark providers, implement fallback strategies, or route requests to different models based on cost and latency requirements.
Adding Observability and Tracing
At scale, understanding what your agents are doing is critical. LlamaIndex integrates with callback handlers and observability platforms to trace every tool call.
from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler
debug_handler = LlamaDebugHandler(print_trace_on_end=True)
callback_manager = CallbackManager([debug_handler])
llm_with_callbacks = OpenAI(model="gpt-4o", callback_manager=callback_manager)
observable_agent = FunctionAgent(
tools=[weather_tool, search_tool, stock_tool],
llm=llm_with_callbacks,
system_prompt="You are a helpful assistant.",
)
response = observable_agent.run("Compare the weather in London and the price of MSFT.")
# Inspect all events after execution
for event in debug_handler.get_events():
print(event)
For production, you can replace LlamaDebugHandler with integrations for Langfuse, Arize Phoenix, or OpenTelemetry-compatible backends to get distributed tracing across your entire agent fleet.
Best Practices for Function Calling at Scale
- Write precise docstrings: The model relies on your function descriptions to decide when to call a tool. Be specific about inputs, outputs, and use cases.
- Use narrow, single-purpose tools: A tool that does one thing well is easier for the model to select correctly than a multi-purpose function with many optional parameters.
- Validate inputs and outputs: Use Pydantic models or explicit validation inside your tool functions to catch malformed calls before they reach external systems.
- Implement retries and timeouts: External APIs fail. Wrap tool logic with retry mechanisms and timeouts to prevent agents from hanging indefinitely.
- Limit tool count per request: Use tool retrieval to send only relevant tools, keeping token usage and selection accuracy under control.
- Log every tool call: Maintain an audit trail of which tools were called, with what arguments, and what they returned. This is essential for debugging and compliance.
- Test with edge cases: Verify behavior when the model calls the wrong tool, passes invalid arguments, or calls multiple tools in sequence.
- Cache deterministic results: If a tool returns the same output for the same input, cache it to reduce latency and API costs.
- Set max iterations: Configure a maximum number of agent steps to prevent infinite loops in pathological cases.
Putting It All Together: A Production Pattern
Here is a condensed example that combines async execution, tool retrieval, structured output, and observability into a single production-ready pattern.
import asyncio
import os
from llama_index.core import VectorStoreIndex
from llama_index.core.agent import FunctionAgent
from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler
from llama_index.core.objects import ObjectIndex, SimpleToolNodeMapping
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI
# --- Define tools ---
async def get_weather(location: str) -> str:
"""Get current weather for a location."""
await asyncio.sleep(0.05)
return f"Weather in {location}: 20C, sunny."
async def get_stock(symbol: str) -> str:
"""Get current stock price for a ticker symbol."""
await asyncio.sleep(0.05)
return f"{symbol}: $185.42"
async def search_docs(query: str, top_k: int = 3) -> str:
"""Search internal documents."""
await asyncio.sleep(0.05)
return f"Results for '{query}': doc1, doc2, doc3"
async def send_email(to: str, subject: str, body: str) -> str:
"""Send an email to a recipient."""
await asyncio.sleep(0.05)
return f"Email sent to {to} with subject '{subject}'."
tools = [
FunctionTool.from_defaults(fn=get_weather, async_mode=True),
FunctionTool.from_defaults(fn=get_stock, async_mode=True),
FunctionTool.from_defaults(fn=search_docs, async_mode=True),
FunctionTool.from_defaults(fn=send_email, async_mode=True),
]
# --- Build tool index for retrieval ---
tool_mapping = SimpleToolNodeMapping.from_objects(tools)
obj_index = ObjectIndex.from_objects(
tools, tool_mapping, VectorStoreIndex
)
tool_retriever = obj_index.as_retriever(similarity_top_k=3)
# --- Set up observability ---
debug_handler = LlamaDebugHandler(print_trace_on_end=False)
callback_manager = CallbackManager([debug_handler])
llm = OpenAI(model="gpt-4o", callback_manager=callback_manager)
# --- Async handler with dynamic tool selection ---
async def handle_request(query: str) -> str:
relevant_tools = tool_retriever.retrieve(query)
agent = FunctionAgent(
tools=relevant_tools,
llm=llm,
system_prompt=(
"You are a helpful assistant. Use tools when needed. "
"Be concise and accurate."
),
max_iterations=5,
)
return await agent.achat(query)
async def main():
queries = [
"What's the weather in Tokyo?",
"Get the price of GOOGL and email summary to boss@company.com",
"Search docs about onboarding process",
]
results = await asyncio.gather(*[handle_request(q) for q in queries])
for q, r in zip(queries, results):
print(f"Q: {q}\nA: {r}\n{'-'*50}")
asyncio.run(main())
Conclusion
Function calling at scale with LlamaIndex gives you a robust, provider-agnostic framework for connecting LLMs to real-world tools and data. By combining well-documented FunctionTool definitions, dynamic tool retrieval, async execution, structured output enforcement, and comprehensive observability, you can build agent systems that handle high-throughput production workloads reliably. The key to success lies in treating tools as first-class, well-tested components: write precise schemas, validate inputs, limit tool scope, and trace every call. With these patterns in place, LlamaIndex enables you to move from simple chatbot demos to scalable, tool-augmented AI systems that deliver real business value.