Introduction to Building a Documentation Generator with llama.cpp
Documentation is often the most neglected part of software development. Developers write code, ship features, and move on, leaving behind sparse README files and outdated comments. What if you could automatically generate high-quality documentation directly from your source code using a local, privacy-preserving language model? That is exactly what llama.cpp makes possible.
In this tutorial, you will learn how to build a complete documentation generator that reads source files, analyzes their structure, and produces clean, readable documentation using a locally hosted LLM. By the end, you will have a working tool you can run on your own machine without relying on any cloud API.
What is llama.cpp?
llama.cpp is an open-source C/C++ inference engine for running large language models locally. Originally created to run Meta's LLaMA models on consumer hardware, it has grown into one of the most popular projects for local LLM inference. It supports a wide range of model formats, including GGUF, and runs efficiently on CPUs as well as GPUs.
Unlike cloud-based APIs such as OpenAI or Anthropic, llama.cpp runs entirely on your machine. This means no data leaves your system, no API keys are required, and you have full control over the model and its behavior. For a documentation generator that will read your proprietary source code, this privacy guarantee is a significant advantage.
Key Features of llama.cpp
- Runs on CPU and GPU (CUDA, Metal, Vulkan, ROCm)
- Supports quantized models (GGUF format) for low memory usage
- Provides a lightweight HTTP server with an OpenAI-compatible API
- Works on Linux, macOS, and Windows
- No external dependencies required for the core build
Why Build a Documentation Generator?
Manual documentation is slow, error-prone, and tends to drift out of sync with the actual code. A documentation generator powered by an LLM can bridge this gap by understanding the intent behind your code and producing human-readable explanations. Here are the main reasons this approach matters:
- Consistency: Every function, class, and module gets documented in the same style.
- Speed: Generating docs for an entire project takes minutes instead of days.
- Context awareness: The LLM can infer purpose from variable names, control flow, and surrounding code.
- Privacy: With llama.cpp, your source code never leaves your machine.
- Cost: No per-token API charges. The only cost is your hardware and electricity.
Prerequisites and Setup
Before building the generator, you need to set up llama.cpp and download a model. This section walks through the complete setup process.
Building llama.cpp from Source
First, clone the repository and build the project. The following commands work on most Linux and macOS systems:
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make
If you have an NVIDIA GPU and want CUDA acceleration, build with the following flag:
make GGML_CUDA=1
On macOS with Apple Silicon, Metal support is enabled by default, so a plain make is sufficient.
Downloading a Model
You need a model in GGUF format. A good starting point is a quantized version of a smaller model like Llama 3.2 3B or Qwen 2.5 7B. You can find these on Hugging Face. Download one and place it in your project directory:
mkdir models
# Download a GGUF model from Hugging Face, for example:
# https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF
wget -O models/qwen2.5-7b-instruct-q4_k_m.gguf \
"https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf"
A 4-bit quantized 7B model requires roughly 4-5 GB of RAM, which is manageable on most modern developer machines.
Starting the Server
llama.cpp ships with an HTTP server that exposes an OpenAI-compatible API. Start it with the following command:
./llama-server \
--model models/qwen2.5-7b-instruct-q4_k_m.gguf \
--port 8080 \
--ctx-size 8192 \
--n-gpu-layers 35
The --ctx-size parameter controls the context window size. A larger context allows the model to process longer source files. The --n-gpu-layers parameter offloads layers to the GPU for faster inference. Adjust these values based on your hardware.
Architecture of the Documentation Generator
Our documentation generator will follow a straightforward pipeline:
- File Discovery: Scan the project directory for source files.
- Chunking: Split large files into manageable chunks that fit within the model's context window.
- Prompt Construction: Build a prompt that asks the model to generate documentation for each chunk.
- Inference: Send the prompt to the llama.cpp server and receive the generated documentation.
- Assembly: Combine the outputs into a final Markdown document.
We will implement this in Python because it offers excellent file handling, HTTP client libraries, and string manipulation capabilities. However, the same approach works in any language that can make HTTP requests.
Building the Generator
Project Structure
Create the following project structure:
docgen/
├── generate_docs.py
├── config.py
├── requirements.txt
└── output/
The requirements.txt file is minimal:
requests>=2.31.0
Configuration
Start with a configuration file that centralizes all settings:
# config.py
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
# llama.cpp server settings
LLAMA_SERVER_URL = "http://localhost:8080/v1/chat/completions"
MODEL_NAME = "qwen2.5-7b-instruct"
# File scanning settings
SOURCE_DIR = os.path.join(BASE_DIR, "..", "my_project")
OUTPUT_DIR = os.path.join(BASE_DIR, "output")
SUPPORTED_EXTENSIONS = [".py", ".js", ".ts", ".java", ".go", ".rs", ".cpp", ".c", ".h"]
# Chunking settings
MAX_CHUNK_LINES = 150
MAX_CHUNK_CHARS = 6000
# Generation settings
TEMPERATURE = 0.2
MAX_TOKENS = 2048
Adjust SOURCE_DIR to point to the project you want to document. The TEMPERATURE is kept low because documentation should be factual and deterministic, not creative.
File Discovery and Chunking
Next, implement the file scanner and chunking logic. The scanner walks the source directory and collects files with supported extensions. The chunker splits files into pieces that fit within the model's context window.
# generate_docs.py - Part 1: File Discovery and Chunking
import os
import config
def discover_files(source_dir, extensions):
"""Walk the source directory and return a list of source files."""
files = []
for root, dirs, filenames in os.walk(source_dir):
# Skip common directories that should not be documented
dirs[:] = [d for d in dirs if d not in {
".git", "node_modules", "__pycache__", ".venv",
"venv", "dist", "build", ".idea", ".vscode"
}]
for filename in filenames:
ext = os.path.splitext(filename)[1]
if ext in extensions:
files.append(os.path.join(root, filename))
return files
def chunk_file(filepath, max_lines, max_chars):
"""Split a file into chunks that respect line and character limits."""
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
content = f.read()
lines = content.split("\n")
chunks = []
current_chunk = []
current_lines = 0
current_chars = 0
for line in lines:
line_len = len(line) + 1 # +1 for the newline
if (current_lines + 1 > max_lines or
current_chars + line_len > max_chars):
if current_chunk:
chunks.append("\n".join(current_chunk))
current_chunk = [line]
current_lines = 1
current_chars = line_len
else:
current_chunk.append(line)
current_lines += 1
current_chars += line_len
if current_chunk:
chunks.append("\n".join(current_chunk))
return chunks
The chunker respects both a maximum line count and a maximum character count. This dual limit ensures that no single chunk is too large for the model's context window, regardless of whether the file has many short lines or a few very long ones.
Prompt Construction
The prompt is the most critical part of the system. A well-crafted prompt produces clear, structured documentation. A poor prompt produces vague or incorrect output. Here is the prompt template we will use:
# generate_docs.py - Part 2: Prompt Construction
SYSTEM_PROMPT = """You are a technical documentation expert. Your task is to \
generate clear, accurate documentation for source code. Follow these rules:
1. Use Markdown formatting.
2. For each function, class, or method, provide:
- A one-sentence summary of what it does.
- A description of each parameter and its type.
- A description of the return value and its type.
- Any notable side effects or exceptions.
3. Do not include the original source code in your output.
4. Do not add introductory or concluding remarks.
5. Be concise but complete. Avoid filler words.
6. If a chunk contains only imports or configuration, briefly describe its purpose.
"""
def build_user_prompt(filepath, chunk_index, total_chunks, code_chunk):
"""Build the user prompt for a single code chunk."""
return f"""Document the following code from the file `{filepath}`.
This is chunk {chunk_index} of {total_chunks}.
Code:
{code_chunk}
Generate documentation in Markdown format:"""
The system prompt establishes the model's role and enforces a consistent output format. The user prompt provides the specific code chunk and contextual information about which part of the file is being documented.
Calling the llama.cpp Server
Now implement the function that sends requests to the llama.cpp server. Since the server exposes an OpenAI-compatible API, we use the standard chat completions format:
# generate_docs.py - Part 3: LLM Inference
import requests
import time
import config
def generate_documentation(filepath, chunk_index, total_chunks, code_chunk):
"""Send a code chunk to the llama.cpp server and get documentation back."""
user_prompt = build_user_prompt(filepath, chunk_index, total_chunks, code_chunk)
payload = {
"model": config.MODEL_NAME,
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt}
],
"temperature": config.TEMPERATURE,
"max_tokens": config.MAX_TOKENS,
}
max_retries = 3
for attempt in range(max_retries):
try:
response = requests.post(
config.LLAMA_SERVER_URL,
json=payload,
timeout=120
)
response.raise_for_status()
data = response.json()
return data["choices"][0]["message"]["content"].strip()
except requests.exceptions.RequestException as e:
print(f" Request failed (attempt {attempt + 1}/{max_retries}): {e}")
if attempt < max_retries - 1:
time.sleep(5)
else:
return f"> **Error generating documentation for this chunk:** {e}"
The retry logic handles transient network errors and server timeouts gracefully. The timeout is set to 120 seconds because local inference on CPU can be slow for large prompts.
Assembling the Final Document
The final step is to tie everything together. The main function discovers files, chunks them, generates documentation for each chunk, and writes the results to Markdown files in the output directory.
# generate_docs.py - Part 4: Main Assembly
import os
import config
def document_file(filepath):
"""Generate documentation for a single file."""
print(f"Processing: {filepath}")
chunks = chunk_file(
filepath,
config.MAX_CHUNK_LINES,
config.MAX_CHUNK_CHARS
)
if not chunks:
print(f" Skipping empty file: {filepath}")
return None
total_chunks = len(chunks)
doc_parts = []
for i, chunk in enumerate(chunks, 1):
print(f" Generating docs for chunk {i}/{total_chunks}...")
doc = generate_documentation(filepath, i, total_chunks, chunk)
doc_parts.append(doc)
# Build the final document for this file
relative_path = os.path.relpath(filepath, config.SOURCE_DIR)
header = f"# Documentation: `{relative_path}`\n\n"
body = "\n\n---\n\n".join(doc_parts)
return header + body
def main():
"""Main entry point for the documentation generator."""
os.makedirs(config.OUTPUT_DIR, exist_ok=True)
files = discover_files(config.SOURCE_DIR, config.SUPPORTED_EXTENSIONS)
print(f"Found {len(files)} source files to document.\n")
if not files:
print("No source files found. Check your SOURCE_DIR setting in config.py.")
return
all_docs = []
for filepath in files:
doc = document_file(filepath)
if doc:
all_docs.append(doc)
# Also write per-file documentation
relative_path = os.path.relpath(filepath, config.SOURCE_DIR)
safe_name = relative_path.replace(os.sep, "_").replace(".", "_") + ".md"
output_path = os.path.join(config.OUTPUT_DIR, safe_name)
with open(output_path, "w", encoding="utf-8") as f:
f.write(doc)
print(f" Written: {output_path}\n")
# Write combined documentation
combined_path = os.path.join(config.OUTPUT_DIR, "full_documentation.md")
with open(combined_path, "w", encoding="utf-8") as f:
f.write("# Project Documentation\n\n")
f.write("> This documentation was automatically generated using llama.cpp.\n\n")
f.write("---\n\n")
f.write("\n\n---\n\n".join(all_docs))
print(f"\nCombined documentation written to: {combined_path}")
print(f"Total files documented: {len(all_docs)}")
if __name__ == "__main__":
main()
Running the Generator
Make sure the llama.cpp server is running, then execute the generator:
pip install -r requirements.txt
python generate_docs.py
You should see output similar to:
Found 12 source files to document.
Processing: ../my_project/main.py
Generating docs for chunk 1/2...
Generating docs for chunk 2/2...
Written: output/main_py.md
Processing: ../my_project/utils.py
Generating docs for chunk 1/1...
Written: output/utils_py.md
...
Combined documentation written to: output/full_documentation.md
Total files documented: 12
Enhancing the Generator
The basic generator works, but there are several enhancements that can significantly improve the quality and usefulness of the output.
Adding an Index Page
An index page helps users navigate the generated documentation. Add this function to create a table of contents:
def generate_index(files, output_dir):
"""Generate an index page listing all documented files."""
index_path = os.path.join(output_dir, "index.md")
with open(index_path, "w", encoding="utf-8") as f:
f.write("# Documentation Index\n\n")
f.write("This index lists all files that have been documented.\n\n")
for filepath in sorted(files):
relative_path = os.path.relpath(filepath, config.SOURCE_DIR)
safe_name = relative_path.replace(os.sep, "_").replace(".", "_") + ".md"
f.write(f"- [{relative_path}]({safe_name})\n")
print(f"Index written to: {index_path}")
Call generate_index(files, config.OUTPUT_DIR) at the end of the main() function.
Handling Different Languages
Different programming languages have different documentation conventions. You can extend the prompt system to be language-aware:
LANGUAGE_PROMPTS = {
".py": "Use Python docstring conventions as inspiration for the style.",
".js": "Use JSDoc conventions as inspiration for the style.",
".ts": "Use TSDoc conventions and include TypeScript type information.",
".go": "Use Go documentation conventions. Document exported identifiers.",
".rs": "Use rustdoc conventions. Document public items with examples where helpful.",
".java": "Use Javadoc conventions as inspiration for the style.",
}
def build_system_prompt(file_extension):
"""Build a language-specific system prompt."""
base = SYSTEM_PROMPT
lang_note = LANGUAGE_PROMPTS.get(file_extension, "")
if lang_note:
base += f"\n\nAdditional guidance for this language:\n{lang_note}"
return base
Then update the generate_documentation function to use build_system_prompt(ext) instead of the static SYSTEM_PROMPT.
Adding Parallel Processing
Processing files sequentially can be slow for large projects. Since the llama.cpp server can handle concurrent requests, you can use Python's concurrent.futures module to parallelize the work:
from concurrent.futures import ThreadPoolExecutor, as_completed
def main_parallel():
"""Parallel version of the main function."""
os.makedirs(config.OUTPUT_DIR, exist_ok=True)
files = discover_files(config.SOURCE_DIR, config.SUPPORTED_EXTENSIONS)
print(f"Found {len(files)} source files to document.\n")
max_workers = 4 # Adjust based on your hardware
with ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_file = {
executor.submit(document_file, f): f for f in files
}
for future in as_completed(future_to_file):
filepath = future_to_file[future]
try:
doc = future.result()
if doc:
relative_path = os.path.relpath(filepath, config.SOURCE_DIR)
safe_name = relative_path.replace(os.sep, "_").replace(".", "_") + ".md"
output_path = os.path.join(config.OUTPUT_DIR, safe_name)
with open(output_path, "w", encoding="utf-8") as f:
f.write(doc)
print(f"Completed: {filepath}")
except Exception as e:
print(f"Error processing {filepath}: {e}")
Be careful not to set max_workers too high. Each concurrent request consumes memory and compute resources on the server. A value of 2 to 4 is a good starting point for most machines.
Best Practices
Based on experience building and running documentation generators with llama.cpp, here are the key best practices to follow:
Choose the Right Model
Model selection has the largest impact on output quality. For documentation generation, you want a model that is good at understanding code and producing structured text. Models in the 7B to 14B parameter range generally offer a good balance of quality and speed. Smaller models (3B and below) may struggle with complex code, while larger models (70B+) require significant hardware.
Use Low Temperature
Documentation should be factual and consistent. Set the temperature between 0.1 and 0.3. Higher temperatures introduce variability, which can cause the model to hallucinate function behavior or produce inconsistent formatting across files.
Keep Chunks Focused
Smaller, focused chunks produce better documentation than large, mixed chunks. Try to split files at natural boundaries such as function or class definitions. If you have control over the chunking logic, you can parse the source code and split at definition boundaries rather than using simple line counts.
Always Review Output
LLMs can and do make mistakes. They may invent parameters, misdescribe return values, or misunderstand complex logic. Treat generated documentation as a first draft, not a final product. Always have a human review the output before publishing it.
Cache Results
Generating documentation for a large project can take a long time. Implement a simple caching mechanism that stores the hash of each file along with its generated documentation. On subsequent runs, skip files that have not changed:
import hashlib
import json
CACHE_FILE = os.path.join(config.OUTPUT_DIR, ".docgen_cache.json")
def load_cache():
if os.path.exists(CACHE_FILE):
with open(CACHE_FILE, "r") as f:
return json.load(f)
return {}
def save_cache(cache):
with open(CACHE_FILE, "w") as f:
json.dump(cache, f, indent=2)
def file_hash(filepath):
with open(filepath, "rb") as f:
return hashlib.sha256(f.read()).hexdigest()
In document_file, check the cache before generating:
cache = load_cache()
file_hash_val = file_hash(filepath)
if filepath in cache and cache[filepath]["hash"] == file_hash_val:
print(f" Using cached documentation for: {filepath}")
return cache[filepath]["doc"]
# ... generate documentation ...
cache[filepath] = {"hash": file_hash_val, "doc": doc}
save_cache(cache)
Monitor Resource Usage
Running an LLM locally is resource-intensive. Monitor your CPU, memory, and GPU usage during generation. If you run out of memory, try a smaller model or a more aggressive quantization level. If generation is too slow, consider offloading more layers to the GPU or using a smaller context window.
Troubleshooting Common Issues
Server Connection Refused
If you see connection errors, verify that the llama.cpp server is running and listening on the correct port. Check with curl http://localhost:8080/v1/models. If the server is not responding, check its console output for errors.
Out of Memory Errors
If the server crashes with out-of-memory errors, reduce the context size with --ctx-size, use a smaller model, or increase quantization. A Q4_K_M quantized 7B model is a good starting point for machines with 8 GB of RAM.
Poor Quality Output
If the generated documentation is vague or incorrect, try the following: use a larger or more capable model, reduce the temperature, provide more context in the prompt, or use smaller chunks. Sometimes simply rephrasing the system prompt can make a significant difference.
Conclusion
Building a documentation generator with llama.cpp is a practical and powerful way to leverage local LLMs for a real development task. You get the benefits of AI-powered documentation without sending your proprietary source code to a third-party service. The generator you built in this tutorial can scan a project, chunk source files intelligently, generate structured documentation for each file, and assemble everything into a navigable set of Markdown documents. By following the best practices around model selection, temperature tuning, chunking, caching, and human review, you can produce documentation that genuinely helps developers understand and work with your codebase. As local LLMs continue to improve in capability and efficiency, tools like this will become an increasingly valuable part of the developer toolkit, bridging the gap between code and documentation in a way that is private, fast, and cost-effective.