Introduction to Multi-Agent Research Pipelines
Modern research workflows rarely fit neatly into a single prompt-response cycle. Whether you're conducting market analysis, summarizing scientific literature, or building competitive intelligence reports, the work demands planning, searching, reading, synthesizing, and reviewing — each step requiring different skills. CrewAI is a Python framework that lets you orchestrate multiple specialized AI agents, each with a defined role, goal, and set of tools, working together to complete complex research tasks.
In this guide, you'll build a complete multi-agent research pipeline from scratch. You'll learn how to define agents, assign tasks, equip them with tools, and orchestrate their collaboration to produce a polished research report. By the end, you'll have a reusable pattern you can adapt to any domain.
What Is CrewAI?
CrewAI is an open-source orchestration framework for building autonomous AI agent systems. Unlike single-agent chatbots, CrewAI lets you compose teams of agents that communicate, delegate, and collaborate. Each agent has a role, a goal, a backstory, and access to tools. Tasks define what needs to be done, and a Crew ties agents and tasks together with a defined process.
The framework supports two main process types: sequential, where tasks execute one after another, and hierarchical, where a manager agent delegates and coordinates. For research pipelines, sequential processes are usually the right starting point because research naturally flows through stages.
Why Multi-Agent Pipelines Matter
- Specialization: Each agent focuses on one job, producing higher-quality output than a single generalist prompt.
- Separation of concerns: Searching, analyzing, and writing are distinct skills — splitting them improves reliability.
- Verifiability: A reviewer agent can critique and refine output before delivery, reducing hallucinations.
- Scalability: You can add new agents or tools without rewriting the entire pipeline.
- Observability: CrewAI logs each agent's reasoning, making it easier to debug failures.
Prerequisites and Setup
You'll need Python 3.10 or later, an OpenAI API key (or another supported LLM provider), and basic familiarity with Python. Start by creating a virtual environment and installing CrewAI.
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install crewai crewai-tools
Set your API key as an environment variable. CrewAI uses this to power the underlying language model for each agent.
export OPENAI_API_KEY="sk-your-key-here"
Create a project directory and add a file named research_pipeline.py. This single file will hold your entire pipeline for clarity, though in production you'd split it into modules.
Designing the Research Pipeline
Before writing code, sketch the pipeline. A solid research workflow has four stages:
- Planning: A research strategist breaks the topic into sub-questions.
- Searching: A research analyst gathers information using web search.
- Writing: A report writer synthesizes findings into a structured report.
- Reviewing: A quality reviewer checks for accuracy, gaps, and clarity.
Each stage maps to one agent and one task. The output of each task feeds the next, creating a chain of refined artifacts.
Defining the Agents
Agents are the workers in your crew. Define them with clear roles and backstories — these aren't just flavor text. The backstory shapes how the LLM reasons about its role, so invest time in writing them well.
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool
# Tool for web search
search_tool = SerperDevTool()
research_strategist = Agent(
role="Research Strategist",
goal="Break down a research topic into specific, answerable sub-questions "
"that guide the research process.",
backstory="You are a senior research strategist with 15 years of experience "
"designing research plans for think tanks. You excel at decomposing "
"broad topics into focused, actionable questions.",
verbose=True,
allow_delegation=False,
)
research_analyst = Agent(
role="Research Analyst",
goal="Gather accurate, up-to-date information from the web to answer each "
"sub-question thoroughly.",
backstory="You are a meticulous research analyst who cross-references sources "
"and prioritizes primary data. You never fabricate information and "
"always cite where findings come from.",
verbose=True,
tools=[search_tool],
allow_delegation=False,
)
report_writer = Agent(
role="Report Writer",
goal="Synthesize research findings into a clear, well-structured report "
"with an executive summary, key findings, and recommendations.",
backstory="You are an award-winning business writer who transforms dense "
"research into compelling, accessible reports for executives.",
verbose=True,
allow_delegation=False,
)
quality_reviewer = Agent(
role="Quality Reviewer",
goal="Review the report for accuracy, completeness, logical flow, and "
"clarity. Return the final polished version.",
backstory="You are a demanding editor with a reputation for catching "
"inconsistencies and weak arguments. You improve every document "
"you touch.",
verbose=True,
allow_delegation=False,
)
Notice that only the research analyst has the search_tool. Giving agents only the tools they need keeps them focused and prevents scope creep. The allow_delegation=False flag prevents agents from handing work to each other unpredictably — in a sequential pipeline, the process itself handles handoffs.
Defining the Tasks
Tasks describe concrete deliverables. Each task has a description, an expected output, and the agent responsible. The expected_output field is important — it tells the agent what format to produce, which dramatically improves consistency.
planning_task = Task(
description=(
"Analyze the following research topic and break it down into 4-6 "
"specific sub-questions that a research analyst can investigate.\n\n"
"Topic: {topic}\n\n"
"For each sub-question, explain briefly why it matters to the overall "
"research goal. Return the sub-questions as a numbered list."
),
expected_output=(
"A numbered list of 4-6 specific sub-questions with a one-sentence "
"justification for each."
),
agent=research_strategist,
)
research_task = Task(
description=(
"Using the sub-questions provided by the research strategist, conduct "
"web research to answer each one. For every finding, note the source "
"URL and a short quote or summary of the relevant information.\n\n"
"Be thorough. If a sub-question cannot be answered with available "
"sources, say so explicitly rather than guessing."
),
expected_output=(
"A structured research document with one section per sub-question. "
"Each section contains the answer, supporting evidence, and source URLs."
),
agent=research_analyst,
)
writing_task = Task(
description=(
"Using the research document from the analyst, write a polished "
"research report. Structure it as follows:\n"
"1. Executive Summary (3-4 sentences)\n"
"2. Key Findings (one subsection per sub-question)\n"
"3. Recommendations (actionable next steps)\n"
"4. Sources (list of URLs referenced)\n\n"
"Write in a professional, objective tone. Avoid speculation."
),
expected_output=(
"A markdown-formatted research report with the four sections described "
"above, approximately 800-1200 words."
),
agent=report_writer,
)
review_task = Task(
description=(
"Review the research report for accuracy, completeness, logical flow, "
"and clarity. Fix any issues you find. Ensure every claim is supported "
"by the research and that the executive summary accurately reflects "
"the findings.\n\n"
"Return the final, polished version of the report."
),
expected_output=(
"The final reviewed and polished research report in markdown format, "
"ready for delivery to a stakeholder."
),
agent=quality_reviewer,
)
The {topic} placeholder in the planning task is filled at runtime through the crew's inputs. This makes the pipeline reusable across topics without code changes.
Assembling and Running the Crew
Now combine the agents and tasks into a crew. The order of tasks in the list determines execution order in a sequential process.
research_crew = Crew(
agents=[research_strategist, research_analyst, report_writer, quality_reviewer],
tasks=[planning_task, research_task, writing_task, review_task],
process=Process.sequential,
verbose=True,
)
result = research_crew.kickoff(inputs={"topic": "The impact of AI on software development productivity in 2024"})
print("===== FINAL REPORT =====")
print(result.raw)
Run the script and watch the output. CrewAI prints each agent's reasoning and actions, so you can see the strategist decompose the topic, the analyst search the web, the writer compose the report, and the reviewer refine it. The final result.raw contains the polished report.
Adding Custom Tools
Beyond web search, you can build custom tools for specialized data sources. CrewAI tools are simple Python classes decorated with @tool. Here's an example tool that queries a hypothetical internal knowledge base.
from crewai.tools import tool
import requests
@tool("Internal Knowledge Base Search")
def search_knowledge_base(query: str) -> str:
"""Search the company's internal knowledge base for documents matching the query.
Use this when researching internal policies, past projects, or company data.
"""
response = requests.post(
"https://internal-api.example.com/search",
json={"query": query},
headers={"Authorization": "Bearer " + os.environ["KB_API_KEY"]},
timeout=30,
)
response.raise_for_status()
results = response.json().get("documents", [])
if not results:
return "No documents found."
return "\n\n".join(
f"Title: {doc['title']}\n{doc['content'][:500]}" for doc in results[:5]
)
Add the tool to an agent's tools list just like the built-in search tool. The docstring is critical — it's what the agent reads to decide when and how to use the tool. Write it as if you're instructing a new colleague.
Handling Memory and Context
By default, each agent in a sequential crew sees the output of the previous task. This is usually sufficient, but for longer pipelines you may want agents to have access to earlier outputs too. CrewAI supports short-term, long-term, and entity memory.
research_crew = Crew(
agents=[research_strategist, research_analyst, report_writer, quality_reviewer],
tasks=[planning_task, research_task, writing_task, review_task],
process=Process.sequential,
memory=True,
verbose=True,
)
Enabling memory lets agents recall context from earlier in the run and, with long-term memory configured, from previous runs. This is useful when building a research assistant that learns over time.
Best Practices
Write Precise Agent Backstories
The backstory is the single biggest lever on agent behavior. A vague backstory produces vague output. Instead of "you are a helpful assistant," write "you are a financial analyst at a hedge fund who specializes in semiconductor supply chains and distrusts unverified claims." Specificity breeds focus.
Constrain Output Formats
Always set expected_output with a concrete format description. If you need structured data, ask for JSON explicitly and parse it downstream. This prevents agents from producing free-form text that breaks your pipeline.
Limit Tool Access
Give each agent only the tools it needs. An agent with a web search tool but no reason to use it may still call it, wasting tokens and time. Tool access should match the agent's role exactly.
Use a Reviewer Agent
A dedicated reviewer agent catches errors that slip through earlier stages. This is the multi-agent equivalent of self-reflection prompting, but more powerful because the reviewer has a distinct persona and no attachment to the draft it's critiquing.
Start Sequential, Move to Hierarchical When Needed
Sequential processes are easier to debug and reason about. Only move to hierarchical processes when you genuinely need dynamic delegation — for example, when the number or type of sub-tasks can't be known in advance.
Log and Inspect
Keep verbose=True during development. CrewAI's logs show each agent's thought process, tool calls, and outputs. When something goes wrong, the logs usually reveal whether the problem is in the prompt, the tool, or the task design.
Manage Costs
Multi-agent pipelines make many LLM calls. Monitor token usage, especially when agents have tools that trigger additional calls. Consider using a cheaper model for planning and a stronger model for synthesis and review. CrewAI supports per-agent LLM assignment.
from crewai import LLM
analyst_llm = LLM(model="gpt-4o-mini", temperature=0.3)
writer_llm = LLM(model="gpt-4o", temperature=0.7)
research_analyst = Agent(
role="Research Analyst",
goal="Gather accurate information from the web.",
backstory="You are a meticulous research analyst.",
tools=[search_tool],
llm=analyst_llm,
)
report_writer = Agent(
role="Report Writer",
goal="Synthesize findings into a clear report.",
backstory="You are an award-winning business writer.",
llm=writer_llm,
)
Extending the Pipeline
Once your basic pipeline works, you can extend it in several directions. Add a fact-checker agent that verifies claims against fresh searches before the reviewer runs. Add a formatting agent that converts the markdown report into a PDF or slide deck. Add a feedback loop where the reviewer can send the report back to the writer for revision if quality thresholds aren't met.
You can also parallelize. If the strategist produces six sub-questions, you can spawn six analyst agents to research them simultaneously, then merge results. CrewAI's hierarchical process supports this pattern through the manager agent's delegation.
Conclusion
Building a multi-agent research pipeline with CrewAI transforms a fragile, single-prompt approach into a robust, modular system where each agent owns a specific part of the workflow. By separating planning, research, writing, and review into distinct agents with clear roles and tools, you get higher-quality output, better observability, and a codebase you can extend without rewriting. Start with the sequential four-agent pattern shown here, tune the backstories and task descriptions to your domain, and add custom tools and memory as your needs grow. The result is a research pipeline that scales from a one-off report to a repeatable, production-grade intelligence system.