Introduction to Building a Documentation Generator with CrewAI
Documentation is often the most neglected part of the software development lifecycle. Developers write code, ship features, and move on — leaving behind outdated READMEs, missing API references, and confused future maintainers. CrewAI, an open-source framework for orchestrating role-playing autonomous AI agents, offers a powerful way to automate this tedious process. In this guide, you'll build a complete documentation generator that reads your codebase, analyzes its structure, and produces clean, structured documentation.
What Is CrewAI?
CrewAI is a Python framework that lets you create teams of AI agents, each with a specific role, goal, and backstory. These agents collaborate to complete complex tasks by passing information between one another, much like a real team of specialists. Unlike simple prompt chains, CrewAI agents can delegate work, share context, and produce coordinated outputs.
Why Use CrewAI for Documentation?
Generating good documentation requires multiple skills: understanding code structure, writing clear prose, formatting output, and reviewing for accuracy. A single LLM call struggles to do all of this well. CrewAI solves this by assigning each skill to a dedicated agent:
- Code Analysis Agent — reads source files and extracts structure, functions, classes, and dependencies.
- Technical Writer Agent — transforms raw analysis into readable documentation.
- Reviewer Agent — checks the documentation for accuracy, completeness, and clarity.
This separation produces higher-quality output than a monolithic prompt and makes the pipeline easier to debug and extend.
Prerequisites and Setup
Before you start, make sure you have Python 3.10 or higher installed. You'll also need an OpenAI API key (or another supported LLM provider). Create a new project directory and set up a virtual environment:
mkdir doc-generator
cd doc-generator
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install crewai crewai-tools
Create a .env file in your project root to store your API key:
OPENAI_API_KEY=sk-your-key-here
Install python-dotenv to load environment variables automatically:
pip install python-dotenv
Project Structure
Organize your project with the following layout:
doc-generator/
├── .env
├── main.py
├── crews/
│ └── doc_crew.py
├── agents/
│ └── agents.py
├── tasks/
│ └── tasks.py
└── sample_project/
├── calculator.py
└── utils.py
The sample_project/ directory contains the code you want to document. You'll point your crew at this directory and let the agents do the rest.
Creating Sample Code to Document
Before building the crew, create a small sample project so you have something to document. Add the following to sample_project/calculator.py:
"""A simple calculator module for demonstration purposes."""
class Calculator:
"""Performs basic arithmetic operations."""
def __init__(self, precision: int = 2):
"""Initialize the calculator with a given decimal precision.
Args:
precision: Number of decimal places to round results to.
"""
self.precision = precision
def add(self, a: float, b: float) -> float:
"""Return the sum of two numbers."""
return round(a + b, self.precision)
def divide(self, a: float, b: float) -> float:
"""Return the quotient of a divided by b.
Raises:
ZeroDivisionError: If b is zero.
"""
if b == 0:
raise ZeroDivisionError("Cannot divide by zero.")
return round(a / b, self.precision)
And add this to sample_project/utils.py:
"""Utility helpers used across the project."""
from typing import List
def flatten(nested: List[List]) -> List:
"""Flatten a list of lists into a single list.
Args:
nested: A list containing sub-lists.
Returns:
A flat list with all elements.
"""
return [item for sublist in nested for item in sublist]
def chunk(items: List, size: int) -> List[List]:
"""Split a list into chunks of the given size."""
return [items[i:i + size] for i in range(0, len(items), size)]
Defining the Agents
Now create agents/agents.py. Each agent needs a role, a goal, a backstory, and an LLM. The backstory shapes the agent's tone and expertise, so write it carefully.
from crewai import Agent, LLM
llm = LLM(model="gpt-4o", temperature=0.3)
code_analyst = Agent(
role="Senior Code Analyst",
goal="Analyze source code files and extract their structure, "
"including classes, functions, parameters, and dependencies.",
backstory="You are a meticulous software engineer with 15 years of "
"experience reading and understanding codebases. You never "
"miss a function signature or a hidden dependency.",
llm=llm,
verbose=True,
)
technical_writer = Agent(
role="Technical Writer",
goal="Transform raw code analysis into clear, well-structured "
"documentation in Markdown format.",
backstory="You are an award-winning technical writer who specializes "
"in developer documentation. You write concisely, avoid "
"jargon, and always include practical examples.",
llm=llm,
verbose=True,
)
reviewer = Agent(
role="Documentation Reviewer",
goal="Review generated documentation for accuracy, completeness, "
"and clarity, then produce the final polished version.",
backstory="You are a senior developer advocate who reviews "
"documentation before it ships. You catch inaccuracies, "
"fill gaps, and ensure every function is documented.",
llm=llm,
verbose=True,
)
Why Three Agents Instead of One?
Splitting responsibilities across agents gives each one a focused context window. The code analyst only thinks about parsing code, the writer only thinks about prose, and the reviewer only thinks about quality. This mirrors how human teams work and consistently produces better results than asking a single model to do everything at once.
Defining the Tasks
Create tasks/tasks.py. Each task is assigned to an agent and has a description and an expected output. Tasks can reference outputs from previous tasks using CrewAI's context-passing mechanism.
from crewai import Task
from agents.agents import code_analyst, technical_writer, reviewer
analysis_task = Task(
description=(
"Analyze the following source code files and produce a structured "
"summary. For each file, list every class and function with its "
"signature, parameters, return type, and a one-line description "
"of what it does. Also note any imports and dependencies.\n\n"
"Source code:\n{source_code}"
),
expected_output=(
"A structured text summary of every file, class, and function "
"with signatures and brief descriptions."
),
agent=code_analyst,
)
writing_task = Task(
description=(
"Using the code analysis provided, write complete Markdown "
"documentation. Include:\n"
"- A module-level overview for each file\n"
"- A class section with method tables\n"
"- Function signatures with parameter descriptions\n"
"- At least one usage example per public function\n"
"- A table of contents at the top\n\n"
"Analysis:\n{analysis_result}"
),
expected_output="A complete Markdown documentation file.",
agent=technical_writer,
)
review_task = Task(
description=(
"Review the following documentation for accuracy and completeness. "
"Fix any errors, fill in missing details, and ensure the formatting "
"is consistent. Output only the final, polished Markdown.\n\n"
"Documentation:\n{documentation}"
),
expected_output="The final, reviewed and corrected Markdown documentation.",
agent=reviewer,
)
Assembling the Crew
Create crews/doc_crew.py to wire the agents and tasks together. The crew defines the process type — sequential means tasks run one after another, with each task's output available to the next.
from crewai import Crew, Process
from tasks.tasks import analysis_task, writing_task, review_task
from agents.agents import code_analyst, technical_writer, reviewer
def build_crew():
return Crew(
agents=[code_analyst, technical_writer, reviewer],
tasks=[analysis_task, writing_task, review_task],
process=Process.sequential,
verbose=True,
)
Reading Source Files and Running the Crew
Now create main.py. This script reads all Python files from the target directory, passes their contents into the first task, runs the crew, and saves the final documentation.
import os
from pathlib import Path
from dotenv import load_dotenv
from crews.doc_crew import build_crew
from tasks.tasks import analysis_task, writing_task, review_task
load_dotenv()
def read_source_files(directory: str) -> str:
"""Read all Python files in a directory and return their contents."""
source_parts = []
for path in sorted(Path(directory).rglob("*.py")):
relative = path.relative_to(directory)
content = path.read_text(encoding="utf-8")
source_parts.append(f"### File: {relative}\npython\n{content}\n")
return "\n\n".join(source_parts)
def main():
target_dir = "sample_project"
source_code = read_source_files(target_dir)
if not source_code.strip():
print("No Python files found in the target directory.")
return
# Inject source code into the first task description
analysis_task.description = analysis_task.description.format(
source_code=source_code
)
crew = build_crew()
result = crew.kickoff()
# Save the final output
output_path = Path("DOCUMENTATION.md")
output_path.write_text(str(result), encoding="utf-8")
print(f"\nDocumentation saved to {output_path}")
if __name__ == "__main__":
main()
Handling Context Passing Between Tasks
In the task definitions above, placeholders like {analysis_result} and {documentation} appear in the descriptions. CrewAI's sequential process automatically passes each task's output to the next task. However, to make the placeholders resolve correctly, you can use the context parameter to explicitly link tasks. Update your tasks to include context references:
writing_task = Task(
description=(
"Using the code analysis provided, write complete Markdown "
"documentation. Include module overviews, class sections with "
"method tables, function signatures, and usage examples.\n\n"
"Analysis:\n{analysis_task_output}"
),
expected_output="A complete Markdown documentation file.",
agent=technical_writer,
context=[analysis_task],
)
review_task = Task(
description=(
"Review the documentation for accuracy and completeness. "
"Fix errors and ensure consistent formatting. Output only "
"the final polished Markdown.\n\n"
"Documentation:\n{writing_task_output}"
),
expected_output="The final, reviewed Markdown documentation.",
agent=reviewer,
context=[writing_task],
)
When you pass context=[previous_task], CrewAI automatically injects the previous task's output into the current task, making the data flow explicit and reliable.
Running the Generator
With everything in place, run the generator from your project root:
python main.py
You'll see verbose output in your terminal as each agent takes its turn. When the process finishes, open DOCUMENTATION.md to see the result. You should find a structured Markdown file with a table of contents, module overviews, function signatures, parameter tables, and usage examples.
Best Practices
1. Keep Agents Focused
Each agent should have one clear responsibility. Resist the temptation to create a "do everything" agent. Focused agents produce more accurate and consistent output because their context window isn't diluted with unrelated instructions.
2. Use Low Temperature for Analysis
Set a low temperature (0.2–0.4) for agents that extract facts from code. Higher temperatures introduce creativity, which is desirable for the writer but harmful for the analyst. You can even use different LLM configurations per agent:
analyst_llm = LLM(model="gpt-4o", temperature=0.1)
writer_llm = LLM(model="gpt-4o", temperature=0.7)
3. Validate Output Programmatically
Don't blindly trust the generated documentation. Add a validation step that checks whether every public function in the source code appears in the output. A simple script can parse the Markdown and compare it against the AST of your Python files:
import ast
from pathlib import Path
def get_public_functions(file_path: str):
tree = ast.parse(Path(file_path).read_text())
return [
node.name for node in ast.walk(tree)
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
and not node.name.startswith("_")
]
def check_coverage(doc_path: str, source_dir: str):
doc = Path(doc_path).read_text()
missing = []
for py_file in Path(source_dir).rglob("*.py"):
for func in get_public_functions(str(py_file)):
if func not in doc:
missing.append(f"{py_file.name}::{func}")
if missing:
print("Missing documentation for:")
for m in missing:
print(f" - {m}")
else:
print("All public functions are documented.")
4. Cache Intermediate Results
Running the full crew on every change is expensive. Consider caching the analysis task output so you only re-run the writer and reviewer when the source code hasn't changed. You can serialize task outputs to JSON and reload them on subsequent runs.
5. Handle Large Codebases with Chunking
If your project has dozens of files, passing everything into a single task will exceed context limits. Instead, process files in batches. Run the analysis task per file or per module, collect all analyses, then run the writer once with the combined results:
from tasks.tasks import analysis_task
from crews.doc_crew import build_crew
from crewai import Crew, Process
analyses = []
for py_file in Path("sample_project").rglob("*.py"):
content = py_file.read_text()
single_task = analysis_task.copy()
single_task.description = analysis_task.description.format(
source_code=f"### File: {py_file.name}\npython\n{content}\n"
)
crew = Crew(
agents=[code_analyst],
tasks=[single_task],
process=Process.sequential,
)
result = crew.kickoff()
analyses.append(str(result))
combined_analysis = "\n\n---\n\n".join(analyses)
# Now pass combined_analysis to the writer and reviewer crew
6. Choose the Right Model
GPT-4o works well for most projects, but for large codebases you may want a model with a larger context window, such as Claude 3.5 Sonnet or GPT-4 Turbo. CrewAI supports multiple providers — just change the LLM configuration:
from crewai import LLM
llm = LLM(
model="anthropic/claude-3-5-sonnet",
api_key=os.getenv("ANTHROPIC_API_KEY"),
temperature=0.3,
)
Extending the Generator
Once you have the basic pipeline working, you can extend it in several directions:
- Add a diagram agent that generates Mermaid diagrams of class relationships and module dependencies.
- Add a changelog agent that compares two versions of the codebase and documents what changed.
- Output multiple formats — add a final task that converts Markdown to HTML, reStructuredText, or a static site generator format like MkDocs or Docusaurus.
- Integrate with CI/CD — run the generator on every pull request and commit the updated documentation automatically.
Conclusion
Building a documentation generator with CrewAI demonstrates the real power of multi-agent orchestration. By breaking documentation into analysis, writing, and review stages — each handled by a specialized agent — you get output that is more accurate and more readable than what a single prompt can produce. The approach scales from small scripts to large codebases with simple adjustments like chunking and caching, and the modular design means you can swap models, add agents, or change output formats without rewriting the whole pipeline. Start with the sample project in this guide, then point the generator at your own codebase and watch your documentation write itself. With a few refinements and a CI integration, you can ensure your docs stay in sync with your code forever.