LlamaIndex from Scratch: Hands-On Tutorial:

Introduction to LlamaIndex

LlamaIndex (formerly known as GPT Index) is a data framework designed to bridge the gap between large language models (LLMs) and external data sources. It provides a structured way to ingest, index, and query your own data — whether it's documents, databases, APIs, or websites — and feed it into LLMs for context-aware responses. Think of it as the connective tissue that lets you build retrieval-augmented generation (RAG) applications without reinventing the wheel every time.

In this tutorial, you'll build a complete LlamaIndex pipeline from scratch. We'll start with basic concepts, move through ingestion and indexing, explore querying strategies, and finish with production-ready best practices. Every code example is self-contained and ready to run.

What LlamaIndex Is, at Its Core

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

At a high level, LlamaIndex consists of three primary stages:

Loading / Ingestion: Pulling data from various sources (PDFs, web pages, databases, APIs) into a unified document format.
Indexing: Transforming those documents into searchable data structures — vector stores, tree indices, keyword tables, and more.
Querying: Retrieving relevant chunks of data and synthesizing a coherent response using an LLM.

What makes LlamaIndex special is that it handles the entire lifecycle: it doesn't just retrieve chunks, it also orchestrates how those chunks are combined, ranked, summarized, and presented to the LLM. It supports multiple retrieval modes — from simple top-k vector search to recursive tree traversal and hybrid keyword+vector approaches.

Why LlamaIndex Matters

Without a framework like LlamaIndex, building a RAG pipeline means manually stitching together:

A document parser (PyPDF2, Unstructured, BeautifulSoup)
A chunking strategy with overlap logic
An embedding provider (OpenAI, HuggingFace, Cohere)
A vector database (Pinecone, Weaviate, Chroma, Qdrant)
A retriever with ranking logic
A prompt templating system for the LLM
Response synthesis with context window management

LlamaIndex unifies all of these under a single, consistent API. It reduces boilerplate, standardizes interfaces, and gives you battle-tested defaults that work across domains. Whether you're building a chatbot over company docs, an analyst assistant over financial reports, or a codebase-aware developer tool, LlamaIndex accelerates your path to production.

Setting Up Your Environment

Before we dive into code, let's install the necessary packages. Create a fresh Python environment and run:

pip install llama-index-core
pip install llama-index-llms-openai
pip install llama-index-embeddings-openai
pip install openai

You'll also need an OpenAI API key. Set it as an environment variable:

export OPENAI_API_KEY="sk-your-key-here"

Alternatively, you can provide it directly in code — but never hardcode secrets in production. For this tutorial, we'll use the environment variable approach.

Step 1: Loading Documents from Scratch

The fundamental unit in LlamaIndex is a Document — a container that holds text content plus metadata. Let's start by creating documents programmatically and loading them from a directory.

Creating Documents In-Memory

from llama_index.core import Document

# Create individual documents manually
doc1 = Document(
    text="LlamaIndex is a data framework for LLM applications. It provides tools for ingestion, indexing, and querying.",
    metadata={"source": "intro", "topic": "overview"}
)

doc2 = Document(
    text="Retrieval-Augmented Generation (RAG) combines retrieval from external knowledge bases with generative LLMs.",
    metadata={"source": "rag-explanation", "topic": "architecture"}
)

doc3 = Document(
    text="Embeddings are dense vector representations of text that capture semantic meaning.",
    metadata={"source": "embeddings", "topic": "fundamentals"}
)

documents = [doc1, doc2, doc3]
print(f"Created {len(documents)} documents")

Loading from a Directory of Files

For real-world use, you'll load data from files. LlamaIndex provides SimpleDirectoryReader which automatically detects file types and extracts text:

from llama_index.core import SimpleDirectoryReader

# Load all readable files from a directory
# Supported formats: .pdf, .txt, .md, .csv, .html, .docx, and more
documents = SimpleDirectoryReader(
    input_dir="./data",
    recursive=True,          # scan subdirectories
    required_exts=[".txt", ".md", ".pdf"]  # optional: filter by extension
).load_data()

print(f"Loaded {len(documents)} documents from ./data")

Place a few text files and markdown files in a ./data folder to test this. Each file becomes one or more Document objects, with file_name and file_path automatically added to metadata.

Step 2: Understanding Nodes and Chunking

Documents are often too large to fit into an LLM's context window or to embed effectively. LlamaIndex splits documents into Nodes — smaller, self-contained chunks that preserve semantic boundaries. A Node is essentially a Document fragment with its own metadata and relationship pointers.

Basic Chunking with SentenceSplitter

from llama_index.core.node_parser import SentenceSplitter

# Create a node parser with configurable chunk size
node_parser = SentenceSplitter(
    chunk_size=512,        # target chunk size in tokens/characters
    chunk_overlap=50,      # overlap between consecutive chunks for context continuity
    separator="\n",        # preferred split point (falls back to sentence boundaries)
    paragraph_separator="\n\n"
)

# Parse documents into nodes
nodes = node_parser.get_nodes_from_documents(documents)
print(f"Split into {len(nodes)} nodes")

# Inspect a node
first_node = nodes[0]
print(f"Node text (first 100 chars): {first_node.text[:100]}...")
print(f"Node metadata: {first_node.metadata}")

The chunk_overlap parameter is critical — it ensures that concepts spanning chunk boundaries are still retrievable. A typical value is 10-20% of chunk_size.

Advanced Parsing with Hierarchical Nodes

For structured documents like manuals or research papers, you might want to preserve the document hierarchy (chapters, sections, subsections). LlamaIndex offers HierarchicalNodeParser for this:

from llama_index.core.node_parser import HierarchicalNodeParser

# Parse with multiple chunk sizes for different hierarchy levels
hierarchical_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]  # coarse → medium → fine
)

hierarchical_nodes = hierarchical_parser.get_nodes_from_documents(documents)
print(f"Hierarchical nodes: {len(hierarchical_nodes)}")

This creates parent-child relationships between nodes, enabling recursive retrieval strategies where you first match at a coarse level, then drill down.

Step 3: Embedding and Indexing

With nodes ready, the next step is to embed them into vectors and build an index for fast retrieval.

Setting Up the Embedding Model

from llama_index.embeddings.openai import OpenAIEmbedding

# Initialize the embedding model
embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",  # cost-effective, 1536 dimensions
    # Alternative: "text-embedding-3-large" for higher quality (3072 dims)
)

# Test a single embedding
test_embedding = embed_model.get_text_embedding("Hello world")
print(f"Embedding dimension: {len(test_embedding)}")
print(f"First 5 values: {test_embedding[:5]}")

Building a Vector Store Index

The most common index type is the VectorStoreIndex, which stores embeddings and performs similarity search:

from llama_index.core import VectorStoreIndex
from llama_index.core import StorageContext
from llama_index.core.settings import Settings

# Configure global settings (used as defaults throughout)
Settings.embed_model = embed_model
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# Build the index from documents — this handles chunking + embedding automatically
index = VectorStoreIndex.from_documents(
    documents,
    embed_model=embed_model,
    show_progress=True  # displays a progress bar for embedding
)

print(f"Index built with {len(index.docstore.docs)} underlying documents")

Under the hood, VectorStoreIndex.from_documents parses documents into nodes, generates embeddings for each node, and stores them in an in-memory vector store by default. For production, you'd swap this with a persistent vector database.

Persisting the Index to Disk

Embedding can be expensive — persist your index so you don't have to recompute:

# Save the index
index.storage_context.persist(persist_dir="./my_index")

# Later, load it back
from llama_index.core import load_index_from_storage

restored_index = load_index_from_storage(
    StorageContext.from_defaults(persist_dir="./my_index")
)
print("Index restored from disk")

Using External Vector Stores

For scale, plug in Chroma (local, open-source) or Pinecone (managed, cloud):

# Chroma example
import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore

chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("my_docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    embed_model=embed_model
)

Step 4: Querying the Index

Now the fun part — asking questions and getting synthesized answers grounded in your data.

Basic Query Engine

from llama_index.llms.openai import OpenAI

# Set up the LLM for response synthesis
llm = OpenAI(model="gpt-4o-mini", temperature=0.1)

# Create a query engine from the index
query_engine = index.as_query_engine(
    llm=llm,
    similarity_top_k=3,        # retrieve top 3 most relevant chunks
    response_mode="compact"    # fit as much context as possible into the prompt
)

# Execute a query
response = query_engine.query("What is LlamaIndex and how does it relate to RAG?")
print(f"Response: {response}")
print(f"\nSource nodes used: {len(response.source_nodes)}")
for node in response.source_nodes:
    print(f"  - Score: {node.score:.4f} | Text preview: {node.text[:80]}...")

Response Modes Explained

LlamaIndex offers several response synthesis strategies, each suited to different use cases:

compact: Fits retrieved chunks plus the query into the LLM's context window by packing them efficiently. Good for short-to-medium queries.
refine: Iteratively feeds chunks one at a time, refining an answer incrementally. Best for when you have many chunks and need thorough coverage.
tree_summarize: Recursively summarizes chunks in a tree structure, then produces a final answer. Ideal for very large document sets.
simple_summarize: Truncates chunks to fit context, losing some detail. Fast but less accurate.

# Compare response modes
for mode in ["compact", "refine", "tree_summarize"]:
    engine = index.as_query_engine(llm=llm, response_mode=mode, similarity_top_k=5)
    result = engine.query("Explain the key components of a RAG system.")
    print(f"\n--- Mode: {mode} ---")
    print(f"Response length: {len(str(result))} chars")
    print(f"Sources: {len(result.source_nodes)}")

Step 5: Chat Engines and Conversational Memory

For chatbot-style interactions, LlamaIndex provides a ChatEngine that maintains conversation history:

from llama_index.core.memory import ChatMemoryBuffer

# Create a chat engine with memory
chat_engine = index.as_chat_engine(
    llm=llm,
    chat_mode="context",          # uses retrieved context for each message
    memory=ChatMemoryBuffer.from_defaults(token_limit=2000),  # keep last ~2000 tokens of conversation
    similarity_top_k=3
)

# Simulate a conversation
response1 = chat_engine.chat("What is an embedding?")
print(f"User: What is an embedding?\nAssistant: {response1}\n")

response2 = chat_engine.chat("How does that relate to the vector search we discussed?")
print(f"User: How does that relate to the vector search we discussed?\nAssistant: {response2}\n")

response3 = chat_engine.chat("Can you summarize our conversation so far?")
print(f"User: Can you summarize our conversation so far?\nAssistant: {response3}\n")

# Inspect conversation history
print(f"Chat history contains {len(chat_engine.memory.get())} messages")

The chat_mode parameter controls behavior:

context: Retrieves fresh context for each message based on the full conversation history.
condense_question: Reformulates the current question plus conversation history into a standalone query before retrieval.
simple: Just uses the latest message without history-aware retrieval.

Step 6: Customizing the Pipeline — Advanced Patterns

Custom Prompt Templates

You can override the default prompts to control how the LLM synthesizes answers:

from llama_index.core.prompts import PromptTemplate

# Define a custom QA prompt
custom_qa_template = PromptTemplate(
    template=(
        "You are a precise technical assistant. Use ONLY the provided context to answer.\n"
        "If the context doesn't contain the answer, say 'I don't have enough information.'\n\n"
        "Context:\n"
        "{context_str}\n\n"
        "Question: {query_str}\n\n"
        "Answer (be concise, cite specific sources where possible):"
    )
)

query_engine = index.as_query_engine(
    llm=llm,
    text_qa_template=custom_qa_template,
    similarity_top_k=4
)

response = query_engine.query("What vector databases does LlamaIndex support?")
print(response)

Hybrid Retrieval: Combining Vector + Keyword Search

Vector search alone can miss exact keyword matches. A hybrid approach combines both:

from llama_index.core.retrievers import VectorIndexRetriever, KeywordTableSimpleRetriever
from llama_index.core.retrievers import RouterRetriever
from llama_index.core.tools import RetrieverTool

# Build a keyword index in addition to the vector index
from llama_index.core import KeywordTableIndex

keyword_index = KeywordTableIndex.from_documents(documents)

# Create retrievers
vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=3)
keyword_retriever = KeywordTableSimpleRetriever(index=keyword_index, num_chunks=3)

# Combine them
from llama_index.core.retrievers import QueryFusionRetriever

fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, keyword_retriever],
    similarity_top_k=5,
    num_queries=1,
    mode="reciprocal_rerank"  # merges results with reciprocal rank fusion
)

# Use in a query engine
query_engine = index.as_query_engine(
    llm=llm,
    retriever=fusion_retriever
)

response = query_engine.query("Find information about RAG architecture.")
print(response)

Sub-Question Query Engine (Decomposition)

For complex queries, LlamaIndex can decompose them into sub-questions, answer each independently, and synthesize a final response:

from llama_index.core.tools import QueryEngineTool, ToolMetadata

query_engine_tool = QueryEngineTool(
    query_engine=query_engine,
    metadata=ToolMetadata(
        name="document_qa",
        description="Answers questions about the loaded documents"
    )
)

from llama_index.core.query_engine import SubQuestionQueryEngine

sub_question_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[query_engine_tool],
    llm=llm,
    verbose=True  # shows the decomposition steps
)

response = sub_question_engine.query(
    "Compare and contrast LlamaIndex and LangChain. Then explain which is better for building a RAG pipeline over PDF documents."
)
print(f"\nFinal answer:\n{response}")

Step 7: Observability and Debugging

Understanding what happens under the hood is essential for debugging and optimization. LlamaIndex integrates with observability tools:

# Enable basic logging
import logging
logging.basicConfig(level=logging.INFO)

# For detailed trace visualization, use the built-in callback system
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
import tiktoken

token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4o-mini").encode,
    verbose=True
)

callback_manager = CallbackManager([token_counter])

# Use it in your query engine
query_engine = index.as_query_engine(
    llm=llm,
    callback_manager=callback_manager,
    similarity_top_k=3
)

response = query_engine.query("What is LlamaIndex?")
print(f"\nTotal tokens used: {token_counter.total_llm_token_count}")

For production, consider integrating with Langfuse, Phoenix (Arize), or Weights & Biases for full tracing dashboards.

Best Practices for Production LlamaIndex

1. Chunk Size Tuning

The single most impactful parameter is chunk size. Too small, and you lose context; too large, and retrieval becomes noisy. Experiment systematically:

# Evaluate multiple chunk sizes
for chunk_size in [256, 512, 1024, 2048]:
    node_parser = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=int(chunk_size * 0.2))
    nodes = node_parser.get_nodes_from_documents(documents)
    index = VectorStoreIndex(nodes, embed_model=embed_model)
    query_engine = index.as_query_engine(llm=llm)
    response = query_engine.query("Your test query here")
    print(f"Chunk size {chunk_size}: Response length {len(str(response))}, Sources: {len(response.source_nodes)}")

2. Embedding Model Selection

Match your embedding model to your domain. OpenAI's text-embedding-3-large gives higher accuracy but costs more. For sensitive data, use local models:

# Local embedding with HuggingFace (no API calls)
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

local_embed = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5",  # 384 dimensions, fast
    max_length=512
)
Settings.embed_model = local_embed

3. Metadata Filtering

Attach rich metadata to documents and filter at query time for precision:

# Documents with metadata
docs = [
    Document(text="Q1 revenue report...", metadata={"year": 2024, "quarter": "Q1", "type": "financial"}),
    Document(text="Q2 revenue report...", metadata={"year": 2024, "quarter": "Q2", "type": "financial"}),
    Document(text="HR policy update...", metadata={"year": 2024, "type": "hr"}),
]

index = VectorStoreIndex.from_documents(docs)

# Query with metadata filter
from llama_index.core.vector_stores.types import MetadataFilter, MetadataFilters

retriever = index.as_retriever(
    similarity_top_k=3,
    filters=MetadataFilters(
        filters=[
            MetadataFilter(key="year", value=2024),
            MetadataFilter(key="type", value="financial")
        ]
    )
)

query_engine = index.as_query_engine(llm=llm, retriever=retriever)
response = query_engine.query("What were the revenue trends?")
print(response)

4. Caching and Performance

Cache embeddings and LLM responses to reduce latency and cost:

from llama_index.core import Settings
from llama_index.core.cache import SimpleCache

# Simple in-memory cache for LLM responses
Settings.llm_cache = SimpleCache()

# First query — goes to LLM
response1 = query_engine.query("What is a vector embedding?")
# Second identical query — served from cache
response2 = query_engine.query("What is a vector embedding?")
print("Second query served from cache (near-instant)")

5. Async for Throughput

For high-throughput applications, use async variants:

import asyncio

async def async_queries():
    query_engine = index.as_query_engine(llm=llm)

    # Fire multiple queries concurrently
    queries = [
        "What is LlamaIndex?",
        "Explain RAG architecture.",
        "How do embeddings work?"
    ]

    tasks = [query_engine.aquery(q) for q in queries]
    responses = await asyncio.gather(*tasks)

    for q, r in zip(queries, responses):
        print(f"Q: {q}\nA: {str(r)[:100]}...\n")

asyncio.run(async_queries())

6. Evaluation and Grounding

Always evaluate retrieval quality before deploying. LlamaIndex provides built-in evaluators:

from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator

# Create evaluators
faithfulness_eval = FaithfulnessEvaluator(llm=llm)
relevancy_eval = RelevancyEvaluator(llm=llm)

# Run evaluation on a set of test queries
test_pairs = [
    ("What is LlamaIndex?", "LlamaIndex is a data framework for LLM applications..."),
]

for query, expected in test_pairs:
    response = query_engine.query(query)
    faith_result = faithfulness_eval.evaluate(
        query=query,
        response=str(response),
        contexts=[n.text for n in response.source_nodes]
    )
    relevancy_result = relevancy_eval.evaluate(
        query=query,
        response=str(response),
        contexts=[n.text for n in response.source_nodes]
    )
    print(f"Query: {query}")
    print(f"Faithfulness: {faith_result.passing} | Relevancy: {relevancy_result.passing}")

7. Ingestion Pipeline for Complex Documents

For PDFs with tables, images, and complex layouts, use the ingestion pipeline with transformations:

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.extractors import (
    TitleExtractor,
    QuestionsAnsweredExtractor,
    SummaryExtractor
)

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=50),
        TitleExtractor(llm=llm),                    # extract titles for each node
        QuestionsAnsweredExtractor(questions=3, llm=llm),  # generate questions each node answers
        SummaryExtractor(summaries=["prev", "self"], llm=llm),  # generate contextual summaries
        embed_model
    ]
)

# Run the pipeline
processed_nodes = pipeline.run(documents=documents)
print(f"Processed {len(processed_nodes)} nodes with enriched metadata")

These extractors add metadata that dramatically improves retrieval quality — the QuestionsAnsweredExtractor in particular helps bridge the gap between user query language and document language.

Complete End-to-End Example

Let's put everything together into a single, runnable script that loads documents, builds an index, and serves queries:

import os
from llama_index.core import (
    Document, VectorStoreIndex, SimpleDirectoryReader,
    StorageContext, Settings
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core.memory import ChatMemoryBuffer

# 1. Global configuration
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.chunk_size = 512
Settings.chunk_overlap = 50

# 2. Load documents (replace with your own data path)
documents = SimpleDirectoryReader(input_dir="./my_docs").load_data()
print(f"Loaded {len(documents)} documents")

# 3. Build vector index
index = VectorStoreIndex.from_documents(documents, show_progress=True)

# 4. Persist for reuse
index.storage_context.persist(persist_dir="./saved_index")

# 5. Create query engine
query_engine = index.as_query_engine(
    similarity_top_k=4,
    response_mode="compact"
)

# 6. Run queries
questions = [
    "Summarize the key points from the documents.",
    "What are the main technical concepts discussed?",
    "Find any mentions of deadlines or timelines."
]

for q in questions:
    response = query_engine.query(q)
    print(f"\nQ: {q}")
    print(f"A: {response}")
    print(f"Sources: {len(response.source_nodes)} chunks")

# 7. Interactive chat mode
chat_engine = index.as_chat_engine(
    chat_mode="condense_question",
    memory=ChatMemoryBuffer.from_defaults(token_limit=3000)
)

print("\n--- Chat mode ready (type 'exit' to quit) ---")
while True:
    user_input = input("\nYou: ")
    if user_input.lower() == "exit":
        break
    response = chat_engine.chat(user_input)
    print(f"Assistant: {response}")

Conclusion

You've now built a complete LlamaIndex pipeline from scratch — from document loading and chunking through embedding, indexing, and querying, all the way to production best practices like metadata filtering, caching, async processing, and evaluation. The framework's strength lies in its composability: each component (parser, embedder, retriever, synthesizer) can be swapped, customized, or extended independently while the rest of the pipeline continues to work seamlessly.

As you move toward production, focus on chunk size tuning for your specific data, embedding model selection based on your domain and privacy requirements, and rigorous evaluation of both retrieval relevance and answer faithfulness. Remember that LlamaIndex is not just a retrieval tool — it's an orchestration layer that coordinates the entire flow from raw data to grounded, trustworthy LLM responses. With the patterns covered here, you're well-equipped to build RAG applications that are accurate, scalable, and production-ready.