LlamaIndex Architecture: Design Patterns and Project Structure

Understanding LlamaIndex Architecture

LlamaIndex (formerly GPT Index) is a data framework designed to bridge the gap between Large Language Models (LLMs) and external data sources. Its architecture revolves around a set of carefully crafted design patterns that make it extensible, modular, and adaptable to a wide variety of use cases—from simple document Q&A to complex multi-agent retrieval systems. Understanding these patterns and the recommended project structure is essential for any developer building production-grade LLM applications.

What Is LlamaIndex Architecture?

At its core, LlamaIndex follows a pipeline-based ingestion and retrieval architecture broken into distinct stages:

• Loading — Reading data from disparate sources (PDFs, APIs, databases, websites)
• Parsing — Converting raw documents into structured representations
• Indexing — Building searchable data structures (vector stores, trees, graphs)
• Querying — Retrieving relevant context and synthesizing responses via LLMs
• Evaluation — Measuring retrieval and response quality

The framework is built on a composable component model where every stage can be swapped, extended, or customized independently. This is achieved through a combination of well-known software design patterns that we'll explore next.

Core Design Patterns in LlamaIndex

LlamaIndex employs several key design patterns that every developer should understand. These patterns aren't just academic—they directly influence how you structure your code, extend functionality, and debug issues.

1. Builder Pattern — Constructing Complex Indexes Step by Step

The Builder pattern is used extensively for constructing indexes and query engines. Instead of requiring a monolithic constructor with dozens of parameters, LlamaIndex lets you build complex objects incrementally.

from llama_index.core import VectorStoreIndex
from llama_index.core import ServiceContext
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

# Builder pattern in action: assemble components, then build
llm = OpenAI(model="gpt-4", temperature=0.1)
embed_model = OpenAIEmbedding(model="text-embedding-3-small")

service_context = ServiceContext.from_defaults(
    llm=llm,
    embed_model=embed_model,
    chunk_size=512,
    chunk_overlap=20
)

documents = [...]  # your loaded documents

# The index is "built" using the assembled context
index = VectorStoreIndex.from_documents(
    documents,
    service_context=service_context,
    show_progress=True
)

# Further build a query engine on top
query_engine = index.as_query_engine(
    similarity_top_k=5,
    response_mode="tree_summarize"
)

The Builder pattern allows you to swap components (LLM, embedding model, chunk size) without touching the core index logic. This is critical when experimenting with different providers or optimizing performance.

2. Strategy Pattern — Interchangeable Algorithms

The Strategy pattern is perhaps the most pervasive design choice in LlamaIndex. It allows you to swap chunking strategies, retrieval strategies, synthesis strategies, and node parsing strategies at runtime.

from llama_index.core.node_parser import (
    SentenceSplitter,
    SimpleFileNodeParser,
    HierarchicalNodeParser,
    SemanticSplitterNodeParser
)

# Strategy 1: Simple sentence splitting
sentence_splitter = SentenceSplitter(
    chunk_size=1024,
    chunk_overlap=200,
    paragraph_separator="\n\n"
)

# Strategy 2: Semantic chunking (requires embedding model)
semantic_splitter = SemanticSplitterNodeParser(
    embed_model=embed_model,
    breakpoint_percentile_threshold=95
)

# Strategy 3: Hierarchical chunking for long documents
hierarchical_splitter = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)

# The same index builder accepts any node parser strategy
index = VectorStoreIndex.from_documents(
    documents,
    transformations=[sentence_splitter, embed_model]  # pluggable strategy
)

Similarly, retrieval strategies can be swapped when creating a query engine:

from llama_index.core.retrievers import (
    VectorIndexRetriever,
    KeywordTableRetriever,
    BM25Retriever
)

# Strategy selection at query-engine construction time
retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=10,
    vector_store_query_mode="hybrid"  # combines vector + keyword
)

query_engine = index.as_query_engine(
    retriever=retriever,
    response_synthesizer_mode="compact"  # synthesis strategy
)

3. Adapter Pattern — Unifying Diverse Data Sources

LlamaIndex uses the Adapter pattern to normalize heterogeneous data sources into a uniform Document / Node representation. Every connector—whether for Notion, Google Drive, SQL databases, or custom APIs—adapts its native format to LlamaIndex's internal data model.

from llama_index.readers.notion import NotionPageReader
from llama_index.readers.google_drive import GoogleDriveReader
from llama_index.readers.database import DatabaseReader

# Each reader adapts its source to a common Document interface
notion_reader = NotionPageReader(integration_token="secret_abc...")
drive_reader = GoogleDriveReader(credentials_path="./credentials.json")
db_reader = DatabaseReader(
    uri="postgresql://user:pass@localhost:5432/mydb"
)

# All load() methods return List[Document] — unified interface
notion_docs = notion_reader.load_data(page_ids=["page1", "page2"])
drive_docs = drive_reader.load_data(folder_id="1abc123xyz")
db_docs = db_reader.load_data(
    query="SELECT * FROM knowledge_base WHERE active = true"
)

# Combine adapted documents seamlessly
all_documents = notion_docs + drive_docs + db_docs

4. Observer Pattern — Logging and Callback Hooks

LlamaIndex implements a lightweight Observer pattern through its callback system. You can attach handlers that observe every stage of the pipeline—token consumption, retrieval steps, LLM calls—without modifying core logic.

from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
from llama_index.core import Settings

# Set up observers
token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4").encode
)

Settings.callback_manager = CallbackManager([token_counter])

# Now run queries normally — the observer tracks everything silently
response = query_engine.query("What is the revenue forecast for Q4?")

# Access observed data after execution
print(f"Embedding tokens: {token_counter.total_embedding_token_count}")
print(f"LLM prompt tokens: {token_counter.prompt_llm_token_count}")
print(f"LLM completion tokens: {token_counter.completion_llm_token_count}")

# Reset observers between runs
token_counter.reset()

5. Pipeline / Chain-of-Responsibility Pattern — Ingestion Pipeline

The ingestion process follows a Chain-of-Responsibility pattern where each transformation stage processes nodes and passes them to the next stage. This is formalized through the IngestionPipeline class.

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.extractors import (
    TitleExtractor,
    KeywordExtractor,
    QuestionsAnsweredExtractor
)

# Define a chain of transformations
pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512),
        TitleExtractor(llm=llm, max_tokens=50),
        KeywordExtractor(llm=llm, num_keywords=5),
        QuestionsAnsweredExtractor(llm=llm, questions=3),
        OpenAIEmbedding(model="text-embedding-3-small")
    ]
)

# Run the chain — each stage processes nodes in sequence
nodes = pipeline.run(documents=documents, show_progress=True)

# Nodes now contain enriched metadata from each pipeline stage
for node in nodes[:3]:
    print(f"Title: {node.metadata.get('title')}")
    print(f"Keywords: {node.metadata.get('keywords')}")
    print(f"Questions: {node.metadata.get('questions_this_excerpt_can_answer')}")

6. Factory Pattern — Centralized Component Creation

LlamaIndex uses factory methods extensively to create components from configurations. The Settings module acts as a global factory, while individual factories exist for embeddings, LLMs, and vector stores.

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.anthropic import Anthropic

# Global factory configuration — applies across your entire application
Settings.llm = Anthropic(model="claude-3-opus-20240229")
Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-large-en-v1.5",
    max_length=512
)
Settings.chunk_size = 1024
Settings.num_output_tokens = 2048

# Now all indexes and query engines inherit these factory defaults
index = VectorStoreIndex.from_documents(documents)  # uses Settings.* automatically
query_engine = index.as_query_engine()  # consistent defaults everywhere

Recommended Project Structure

A well-organized LlamaIndex project separates concerns into distinct layers. Based on real-world production deployments, here is the recommended directory layout:

llamaindex-project/
├── config/
│   ├── settings.py           # Global Settings, API keys, model configs
│   ├── logging_config.py     # Callback manager and observer setup
│   └── constants.py          # Chunk sizes, top_k, thresholds
│
├── loaders/
│   ├── __init__.py
│   ├── pdf_loader.py         # PDF-specific loading logic
│   ├── web_loader.py         # Web scraping / sitemap loaders
│   └── database_loader.py    # SQL / NoSQL source connectors
│
├── pipelines/
│   ├── __init__.py
│   ├── ingestion_pipeline.py # Custom IngestionPipeline assembly
│   └── transformations.py    # Custom node parsers, extractors
│
├── indices/
│   ├── __init__.py
│   ├── vector_index.py       # VectorStoreIndex wrappers
│   ├── summary_index.py      # SummaryIndex configurations
│   └── tree_index.py         # TreeIndex for hierarchical retrieval
│
├── query_engines/
│   ├── __init__.py
│   ├── router_query.py       # RouterQueryEngine with tool selection
│   ├── sub_question.py       # Decomposes complex queries
│   └── custom_retriever.py   # Custom retriever implementations
│
├── agents/
│   ├── __init__.py
│   ├── tool_definitions.py   # Function tools for agent use
│   └── agent_runner.py       # Agent loop and orchestrator
│
├── evaluation/
│   ├── __init__.py
│   ├── retriever_eval.py     # Hit rate, MRR, recall metrics
│   └── response_eval.py      # Faithfulness, relevancy checks
│
├── storage/
│   ├── __init__.py
│   ├── vector_store.py       # Vector store initialization (Pinecone, Chroma, etc.)
│   └── doc_store.py          # Document / key-value store setup
│
├── utils/
│   ├── __init__.py
│   ├── cache_utils.py        # Ingestion and query caching helpers
│   └── monitoring_utils.py   # Token tracking, cost estimation
│
└── main.py                   # Entry point: build and run the pipeline

This structure ensures that changing your embedding model, swapping vector stores, or adding new data sources requires modifying only the relevant module—not the entire codebase.

Practical Example: Building a Modular Ingestion Pipeline

Let's implement the pipelines/ingestion_pipeline.py module following the structure above:

# pipelines/ingestion_pipeline.py
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.extractors import (
    TitleExtractor,
    KeywordExtractor,
    SummaryExtractor
)
from llama_index.core.node_parser import SemanticSplitterNodeParser
from config.settings import llm, embed_model

class ProductionIngestionPipeline:
    """Encapsulates the full ingestion chain with metadata extraction."""
    
    def __init__(self, chunk_size: int = 512, chunk_overlap: int = 50):
        self.pipeline = IngestionPipeline(
            transformations=[
                SemanticSplitterNodeParser(
                    embed_model=embed_model,
                    breakpoint_percentile_threshold=95,
                    buffer_size=1,
                    chunk_size=chunk_size,
                    chunk_overlap=chunk_overlap
                ),
                TitleExtractor(
                    llm=llm,
                    max_tokens=50,
                    num_workers=4  # parallel extraction
                ),
                KeywordExtractor(
                    llm=llm,
                    num_keywords=8,
                    num_workers=4
                ),
                SummaryExtractor(
                    llm=llm,
                    summaries=["self", "prev", "next"],
                    num_workers=4
                ),
                embed_model  # final embedding stage
            ]
        )
    
    def run(self, documents):
        """Execute the full pipeline and return enriched nodes."""
        nodes = self.pipeline.run(
            documents=documents,
            show_progress=True,
            num_workers=4
        )
        return nodes
    
    async def arun(self, documents):
        """Async version for high-throughput ingestion."""
        nodes = await self.pipeline.arun(
            documents=documents,
            show_progress=True,
            num_workers=8
        )
        return nodes

Now the main.py entry point becomes clean and declarative:

# main.py
from loaders.pdf_loader import load_pdf_directory
from loaders.web_loader import load_sitemap
from pipelines.ingestion_pipeline import ProductionIngestionPipeline
from indices.vector_index import build_vector_index
from query_engines.router_query import create_router_engine

def main():
    # 1. Load documents from multiple sources
    pdf_docs = load_pdf_directory("./data/pdfs/")
    web_docs = load_sitemap("https://docs.example.com/sitemap.xml")
    all_docs = pdf_docs + web_docs
    
    # 2. Run ingestion pipeline
    pipeline = ProductionIngestionPipeline(chunk_size=768)
    nodes = pipeline.run(all_docs)
    
    # 3. Build vector index
    index = build_vector_index(nodes)
    
    # 4. Create query engine with routing
    query_engine = create_router_engine(index)
    
    # 5. Query
    response = query_engine.query(
        "Compare pricing between Plan A and Plan B from the docs"
    )
    print(response)

if __name__ == "__main__":
    main()

Advanced Pattern: Multi-Index Routing with Tool Selection

For complex applications, LlamaIndex supports a Router pattern where different query types are directed to specialized indexes. This is implemented via the RouterQueryEngine:

# query_engines/router_query.py
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector

def create_router_engine(summary_index, vector_index, sql_index):
    """Route queries to the most appropriate index based on query type."""
    
    summary_tool = QueryEngineTool(
        query_engine=summary_index.as_query_engine(
            response_mode="tree_summarize"
        ),
        metadata=ToolMetadata(
            name="summary_tool",
            description="Use for high-level overview questions, summaries, "
                        "and questions about entire document collections"
        )
    )
    
    vector_tool = QueryEngineTool(
        query_engine=vector_index.as_query_engine(
            similarity_top_k=5,
            response_mode="compact"
        ),
        metadata=ToolMetadata(
            name="vector_tool",
            description="Use for specific detail questions, fact-finding, "
                        "and queries requiring precise information chunks"
        )
    )
    
    sql_tool = QueryEngineTool(
        query_engine=sql_index.as_query_engine(),
        metadata=ToolMetadata(
            name="sql_tool",
            description="Use for structured data queries involving numbers, "
                        "dates, aggregations, and comparisons"
        )
    )
    
    router = RouterQueryEngine(
        selector=LLMSingleSelector.from_defaults(),
        query_engine_tools=[summary_tool, vector_tool, sql_tool],
        verbose=True  # see routing decisions in logs
    )
    
    return router

Custom Retriever: Implementing the Template Pattern

LlamaIndex allows you to create custom retrievers by extending base classes, following the Template Method pattern. Here's a hybrid retriever that combines vector search with keyword filtering:

# query_engines/custom_retriever.py
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.schema import QueryBundle, NodeWithScore
from typing import List
import re

class KeywordFilteredRetriever(BaseRetriever):
    """
    Retrieves nodes via vector similarity, then post-filters
    using keyword matching on node metadata.
    """
    
    def __init__(
        self,
        vector_index,
        required_keywords: List[str],
        similarity_top_k: int = 20,
        final_top_k: int = 5
    ):
        super().__init__()
        self.vector_index = vector_index
        self.required_keywords = required_keywords
        self.similarity_top_k = similarity_top_k
        self.final_top_k = final_top_k
    
    def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
        # Step 1: Vector retrieval (fetch more than needed)
        vector_retriever = self.vector_index.as_retriever(
            similarity_top_k=self.similarity_top_k
        )
        candidates = vector_retriever.retrieve(query_bundle)
        
        # Step 2: Keyword filtering
        filtered = []
        for node_with_score in candidates:
            node_text = node_with_score.node.get_content().lower()
            metadata_str = str(node_with_score.node.metadata).lower()
            combined = node_text + " " + metadata_str
            
            # All required keywords must be present
            if all(
                re.search(rf"\b{kw.lower()}\b", combined)
                for kw in self.required_keywords
            ):
                filtered.append(node_with_score)
        
        # Step 3: Return top matches after filtering
        return filtered[:self.final_top_k]
    
    async def _aretrieve(self, query_bundle: QueryBundle):
        # Async version for production
        vector_retriever = self.vector_index.as_retriever(
            similarity_top_k=self.similarity_top_k
        )
        candidates = await vector_retriever.aretrieve(query_bundle)
        
        filtered = [
            n for n in candidates
            if all(
                re.search(rf"\b{kw.lower()}\b", 
                          n.node.get_content().lower() + 
                          str(n.node.metadata).lower())
                for kw in self.required_keywords
            )
        ]
        return filtered[:self.final_top_k]

Best Practices for LlamaIndex Projects

1. Centralize Configuration with Settings

Use the Settings module as your single source of truth. Avoid scattering model configurations across files—this makes A/B testing embeddings or LLMs painful. Create a single config/settings.py that everything imports.

# config/settings.py — Single configuration source
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
import tiktoken

def configure_settings():
    """Call once at application startup."""
    Settings.llm = OpenAI(
        model="gpt-4o",
        temperature=0.1,
        max_tokens=1024
    )
    Settings.embed_model = OpenAIEmbedding(
        model="text-embedding-3-large",
        dimensions=1024
    )
    Settings.chunk_size = 768
    Settings.chunk_overlap = 100
    Settings.num_output_tokens = 1024
    
    token_counter = TokenCountingHandler(
        tokenizer=tiktoken.encoding_for_model("gpt-4o").encode
    )
    Settings.callback_manager = CallbackManager([token_counter])
    
    return Settings

# Call once in main.py
settings = configure_settings()

2. Separate Ingestion from Querying

Ingestion is I/O and CPU heavy; querying is latency-sensitive. Run ingestion as a batch process (possibly in a separate service or CI/CD pipeline) and expose the built index for querying. This aligns with the Command Query Responsibility Segregation (CQRS) pattern.

# Run as separate processes or services
# python ingest.py — runs once, builds and persists index
# python serve.py  — loads persisted index, serves queries via API

# ingest.py
pipeline = ProductionIngestionPipeline()
nodes = pipeline.run(documents)
index = VectorStoreIndex(nodes)
index.storage_context.persist(persist_dir="./storage/index")

# serve.py (separate process)
from llama_index.core import load_index_from_storage
index = load_index_from_storage(persist_dir="./storage/index")
query_engine = index.as_query_engine()
# ... expose via FastAPI / Flask endpoint

3. Implement Graceful Degradation with Fallbacks

Production systems should handle failures gracefully. Use the Fallback pattern to degrade from primary to secondary LLM providers:

from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import BaseRetriever
from llama_index.llms.openai import OpenAI
from llama_index.llms.anthropic import Anthropic

class FallbackQueryEngine:
    def __init__(self, retriever: BaseRetriever):
        self.retriever = retriever
        self.primary_llm = OpenAI(model="gpt-4o", max_retries=2)
        self.fallback_llm = Anthropic(model="claude-3-haiku-20240307")
    
    def query(self, query_str: str):
        try:
            engine = RetrieverQueryEngine(
                retriever=self.retriever,
                llm=self.primary_llm
            )
            return engine.query(query_str)
        except Exception as primary_error:
            print(f"Primary LLM failed: {primary_error}")
            print("Falling back to secondary LLM...")
            engine = RetrieverQueryEngine(
                retriever=self.retriever,
                llm=self.fallback_llm
            )
            return engine.query(query_str)

4. Cache Aggressively at Multiple Levels

LlamaIndex supports caching at the ingestion level (avoid re-embedding unchanged documents) and the query level (avoid redundant LLM calls for identical queries). Use both:

from llama_index.core import IngestionCache, QueryCache
from llama_index.core.storage.docstore import SimpleDocumentStore

# Ingestion cache — skip re-embedding documents with unchanged hash
ingest_cache = IngestionCache(
    cache=SimpleDocumentStore.from_persist_path("./cache/docstore.json"),
    vector_store=vector_store
)

# Query cache — avoid duplicate LLM calls
query_cache = QueryCache(
    cache_store=SimpleDocumentStore.from_persist_path("./cache/query_cache.json")
)

# Enable both in settings
Settings.ingestion_cache = ingest_cache
Settings.query_cache = query_cache

5. Monitor and Evaluate Continuously

Build evaluation into your pipeline from day one. Use LlamaIndex's evaluation modules to track retrieval quality and response faithfulness:

# evaluation/retriever_eval.py
from llama_index.core.evaluation import (
    RetrieverEvaluator,
    FaithfulnessEvaluator,
    RelevancyEvaluator
)

def evaluate_retrieval_pipeline(retriever, test_queries, ground_truth):
    """Run systematic evaluation on a set of test queries."""
    
    retriever_evaluator = RetrieverEvaluator.from_metric_names(
        ["hit_rate", "mrr", "precision", "recall"],
        retriever=retriever
    )
    
    results = retriever_evaluator.evaluate(
        queries=test_queries,
        expected_ids=ground_truth
    )
    
    # Aggregate metrics
    metrics = retriever_evaluator.aggregate_metrics(results)
    print(f"Hit Rate: {metrics['hit_rate']:.3f}")
    print(f"MRR: {metrics['mrr']:.3f}")
    print(f"Precision: {metrics['precision']:.3f}")
    
    return metrics

def evaluate_responses(query_engine, test_queries, reference_answers):
    """Evaluate faithfulness and relevancy of generated responses."""
    faithfulness_eval = FaithfulnessEvaluator(llm=Settings.llm)
    relevancy_eval = RelevancyEvaluator(llm=Settings.llm)
    
    for query, reference in zip(test_queries, reference_answers):
        response = query_engine.query(query)
        faith_result = faithfulness_eval.evaluate(
            response=response,
            contexts=response.source_nodes
        )
        relevancy_result = relevancy_eval.evaluate(
            response=response,
            query=query
        )
        print(f"Query: {query}")
        print(f"Faithfulness: {faith_result.passing}")
        print(f"Relevancy: {relevancy_result.passing}")

6. Version Your Indexes

Treat indexes as versioned artifacts. When you change chunking strategy or embedding model, create a new index version rather than overwriting:

# storage/vector_store.py
import hashlib
import json

def generate_index_version(config: dict) -> str:
    """Create a deterministic version hash from configuration."""
    config_str = json.dumps(config, sort_keys=True)
    return hashlib.sha256(config_str.encode()).hexdigest()[:8]

config = {
    "chunk_size": 768,
    "chunk_overlap": 100,
    "embedding_model": "text-embedding-3-large",
    "embedding_dimensions": 1024
}

version = generate_index_version(config)
persist_dir = f"./storage/index_v{version}"

# Now persist with versioned path
storage_context.persist(persist_dir=persist_dir)

Conclusion

LlamaIndex's architecture is a masterclass in composable, pattern-driven design for LLM applications. By internalizing its core patterns—Builder for flexible construction, Strategy for swappable algorithms, Adapter for unifying data sources, Observer for transparent monitoring, Chain-of-Responsibility for pipeline processing, and Factory for centralized configuration—you gain the ability to build systems that are modular, testable, and production-ready. Pair these patterns with a disciplined project structure that separates loaders, pipelines, indices, query engines, agents, and evaluation into distinct layers, and you'll have a codebase that scales gracefully from prototype to enterprise deployment. The best practices of centralized configuration, ingestion-query separation, graceful degradation, aggressive caching, continuous evaluation, and index versioning form the operational backbone that turns a working prototype into a reliable, maintainable system. As the LlamaIndex ecosystem continues to evolve, these architectural foundations will serve you well across versions and integrations.