Comparing Agent Frameworks: LangChain vs LlamaIndex vs Haystack

Comparing Agent Frameworks: LangChain vs LlamaIndex vs Haystack

Building production-grade applications on top of large language models (LLMs) requires more than just calling an API. You need orchestration: connecting models to data sources, managing memory, chaining operations, routing requests, and exposing tools that the model can call. Agent frameworks exist to solve this orchestration problem. In this tutorial, we compare three of the most popular open-source frameworks — LangChain, LlamaIndex, and Haystack — and show you how to build a working agent in each.

What Is an Agent Framework?

An agent framework is a software library that provides reusable abstractions for building LLM-powered applications. At minimum, it offers primitives for prompts, model wrappers, memory, retrievers, and tool calling. More advanced frameworks add agent loops — where the model decides which tool to invoke, observes the result, and iterates until it reaches an answer.

The three frameworks we examine here take different philosophical approaches:

• LangChain — A general-purpose composition framework with a huge ecosystem of integrations and a flexible (sometimes complex) abstraction layer.
• LlamaIndex — Originally focused on data ingestion and retrieval (RAG), now expanded into a full agent framework with strong document-centric workflows.
• Haystack — Built by deepset, it emphasizes pipeline-based architecture, type safety, and production reliability.

Why It Matters

Choosing the right framework affects your development velocity, debugging experience, and ability to scale. A framework that fits your use case will let you ship features quickly; the wrong one will fight you at every turn. If your application is primarily a retrieval-augmented chatbot, LlamaIndex's document abstractions may save you weeks. If you need complex multi-tool orchestration with many integrations, LangChain's breadth is hard to beat. If you value explicit, testable pipelines and clean separation of concerns, Haystack's design will feel natural.

Other factors that matter in practice: community size, documentation quality, frequency of breaking changes, and how easily you can swap underlying model providers. All three frameworks support OpenAI, Anthropic, and open-source models via local runtimes, but the ergonomics differ.

LangChain: The General-Purpose Composer

What It Is

LangChain is the oldest and most widely adopted of the three. It provides abstractions like ChatModel, PromptTemplate, Memory, Retriever, Tool, and AgentExecutor. The newer LangGraph library extends LangChain with stateful, cyclic graphs for more sophisticated agent workflows.

Why It Matters

LangChain's biggest strength is its integration catalog — hundreds of document loaders, vector stores, tools, and model providers. If you need to connect to an obscure SaaS API or a niche vector database, LangChain probably already has a connector. The trade-off is that the abstraction layer can feel leaky and the rapid release cadence has historically introduced breaking changes.

How to Use It

Install the core package and the OpenAI integration:

pip install langchain langchain-openai langchain-community

Here is a minimal agent that can search the web and do math:

from langchain_openai import ChatOpenAI
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain.tools import tool
from langchain_core.prompts import ChatPromptTemplate

# Define tools
@tool
def add(a: float, b: float) -> float:
    """Add two numbers together."""
    return a + b

@tool
def multiply(a: float, b: float) -> float:
    """Multiply two numbers."""
    return a * b

tools = [add, multiply]

# Set up the model and prompt
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful math assistant. Use tools when needed."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

# Create and run the agent
agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

result = executor.invoke({"input": "What is 23 times 17, then add 100?"})
print(result["output"])

The @tool decorator inspects the function's type hints and docstring to generate the tool schema automatically. The AgentExecutor runs the loop: it calls the model, parses tool calls, executes them, feeds results back, and repeats until the model produces a final answer.

Best Practices for LangChain

• Pin your dependency versions. LangChain moves fast; use a lockfile to avoid surprise breakages.
• Prefer LangGraph for anything beyond simple chains. It gives you explicit state management and avoids the hidden complexity of AgentExecutor.
• Use structured output parsing (with_structured_output) instead of regex-based parsing when you need typed responses.
• Keep tool functions small and pure. Side effects make agent loops harder to debug.

LlamaIndex: The Data-Centric Agent Platform

What It Is

LlamaIndex started as a framework for connecting LLMs to private data — indexing documents, building retrieval pipelines, and generating grounded answers. It has since grown into a full agent framework with FunctionAgent, ReActAgent, and a workflows engine for event-driven orchestration.

Why It Matters

If your application revolves around documents — knowledge bases, research assistants, enterprise search — LlamaIndex's data primitives are best-in-class. Its Document, Node, and Index abstractions handle chunking, embedding, and retrieval with sensible defaults. The agent layer sits naturally on top of this, so building a "chat with your documents" agent requires very little glue code.

How to Use It

Install the package:

pip install llama-index llama-index-llms-openai

Here is a ReAct agent with custom tools:

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI
from llama_index.core.tools import FunctionTool

# Define tool functions
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    # In production, call a real weather API
    return f"The weather in {city} is sunny and 22°C."

def calculate_tip(bill: float, percentage: float) -> str:
    """Calculate a tip given a bill amount and tip percentage."""
    tip = bill * (percentage / 100)
    return f"A {percentage}% tip on ${bill:.2f} is ${tip:.2f}."

# Wrap as tools
weather_tool = FunctionTool.from_defaults(fn=get_weather)
tip_tool = FunctionTool.from_defaults(fn=calculate_tip)

# Create the agent
llm = OpenAI(model="gpt-4o-mini", temperature=0)
agent = ReActAgent.from_tools(
    [weather_tool, tip_tool],
    llm=llm,
    verbose=True,
    system_prompt="You are a helpful assistant. Use tools to answer accurately.",
)

response = agent.chat("What's the weather in Tokyo, and if I had a $45 dinner bill there, what's a 15% tip?")
print(response)

To add document retrieval, you build an index and expose a query engine as a tool:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

# Wrap the query engine as a tool the agent can call
from llama_index.core.tools import QueryEngineTool

doc_tool = QueryEngineTool.from_defaults(
    query_engine=query_engine,
    name="document_search",
    description="Search internal documents for information.",
)

doc_agent = ReActAgent.from_tools([doc_tool], llm=llm, verbose=True)
answer = doc_agent.chat("What does our refund policy say about digital products?")
print(answer)

Best Practices for LlamaIndex

• Use FunctionAgent (newer) instead of ReActAgent when your LLM supports native tool calling — it is faster and more reliable.
• Experiment with chunk sizes and overlap. LlamaIndex's defaults are reasonable but rarely optimal for your specific corpus.
• Use the workflows engine for multi-step pipelines that need branching or human-in-the-loop checkpoints.
• Store and reload indexes using StorageContext to avoid re-embedding on every startup.

Haystack: The Pipeline-First Framework

What It Is

Haystack, maintained by deepset, models LLM applications as explicit pipelines composed of typed components. Each component declares its input and output types, and the framework validates connections at build time. Haystack 2.x introduced a clean component architecture where everything — retrievers, generators, converters, joiners — is a node in a directed graph.

Why It Matters

Haystack's design philosophy favors explicitness and testability. You can see exactly what data flows between components, and the type system catches wiring errors before runtime. This makes it especially attractive for teams building production systems where reliability and auditability matter. The trade-off is more boilerplate for simple use cases.

How to Use It

Install Haystack and the OpenAI integration:

pip install haystack-ai haystack-ai-openai

Here is a basic RAG pipeline:

from haystack import Pipeline, Document
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder
from haystack_integrations.components.generators.openai import OpenAIGenerator

# Prepare a document store with sample data
store = InMemoryDocumentStore()
docs = [
    Document(content="Our refund policy allows returns within 30 days of purchase."),
    Document(content="Digital products are non-refundable once downloaded."),
    Document(content="Shipping is free for orders over $50."),
]
store.write_documents(docs)

# Build the pipeline
prompt_template = """
Given these documents, answer the question.
Documents:
{% for doc in documents %}
    {{ doc.content }}
{% endfor %}
Question: {{ question }}
Answer:
"""

pipeline = Pipeline()
pipeline.add_component("retriever", InMemoryBM25Retriever(document_store=store))
pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
pipeline.add_component("generator", OpenAIGenerator(model="gpt-4o-mini"))

pipeline.connect("retriever.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder.prompt", "generator.prompt")

# Run it
result = pipeline.run({
    "retriever": {"query": "What is the refund policy for digital products?"},
    "prompt_builder": {"question": "What is the refund policy for digital products?"},
})
print(result["generator"]["replies"][0])

For agent-style tool use, Haystack provides a ToolInvoker component:

from haystack.components.agents import Agent
from haystack.components.tools import ToolInvoker
from haystack.dataclasses import Tool
from haystack.tools import ComponentTool
import random

def get_stock_price(symbol: str) -> str:
    """Get the current stock price for a given symbol."""
    price = round(random.uniform(50, 500), 2)
    return f"The current price of {symbol} is ${price}"

stock_tool = Tool(
    name="get_stock_price",
    description="Get the current stock price for a ticker symbol.",
    parameters={"symbol": {"type": "string", "description": "Stock ticker symbol"}},
    function=get_stock_price,
)

tool_invoker = ToolInvoker(tools=[stock_tool])
generator = OpenAIGenerator(model="gpt-4o-mini")

agent_pipeline = Pipeline()
agent_pipeline.add_component("generator", generator)
agent_pipeline.add_component("tool_invoker", tool_invoker)

agent_pipeline.connect("generator.replies", "tool_invoker.replies")

# First call: ask the model
first = agent_pipeline.run({
    "generator": {"prompt": "What is the stock price of AAPL? Use the tool."}
})

# The tool_invoker executes any tool calls in the model's reply
print(first["tool_invoker"]["tool_results"])

Best Practices for Haystack

• Lean into the pipeline metaphor. Break your application into small, single-responsibility components — this makes testing and debugging far easier.
• Use Pipeline.dumps() and Pipeline.loads() to serialize your pipeline to YAML. This enables version control and deployment reproducibility.
• Take advantage of input/output type validation. It catches integration bugs early.
• For complex agent loops, combine ToolInvoker with a loop construct or use Haystack's experimental agent components that handle multi-turn tool calling.

Side-by-Side Comparison

Abstraction Style

LangChain uses chains and agents with implicit execution loops. LlamaIndex centers on data structures (documents, nodes, indices) with agents layered on top. Haystack uses explicit, typed pipelines where you wire components together manually.

Ecosystem and Integrations

LangChain has the largest integration catalog by a wide margin. LlamaIndex has strong coverage for data sources and vector stores but fewer general-purpose tool integrations. Haystack has a smaller but well-curated set of integrations, with a focus on quality over quantity.

Learning Curve

Haystack is the most explicit and arguably the easiest to reason about for developers who prefer seeing the full data flow. LlamaIndex is approachable for RAG-centric tasks but can get complex when you dive into advanced workflows. LangChain has the steepest learning curve due to its many overlapping abstractions and multiple ways to accomplish the same task.

Production Readiness

All three are used in production. Haystack's typed pipelines and YAML serialization give it an edge for teams that need auditability and reproducibility. LangChain paired with LangGraph offers the most sophisticated stateful agent capabilities. LlamaIndex excels in production RAG systems with its mature indexing and retrieval tooling.

Choosing the Right Framework

• Choose LangChain if you need maximum integration flexibility, complex multi-agent orchestration (via LangGraph), or you are already invested in its ecosystem.
• Choose LlamaIndex if your application is document-heavy, RAG-centric, or involves complex data ingestion and retrieval patterns.
• Choose Haystack if you value explicit pipelines, type safety, reproducibility, and clean production architecture over rapid prototyping convenience.

Conclusion

There is no single "best" agent framework — the right choice depends on your application's shape, your team's preferences, and your production requirements. LangChain offers unmatched breadth and is ideal when you need to glue together many disparate services. LlamaIndex shines when documents and retrieval are at the heart of your product. Haystack provides the most disciplined, production-oriented architecture for teams that prioritize maintainability and explicit control. The good news is that all three are open source, actively maintained, and interoperable at the API level — so you can prototype in one and migrate to another as your needs evolve. Start with the framework that matches your immediate use case, build a small end-to-end prototype, and let real requirements guide your long-term choice.