Testing LlamaIndex Applications: Unit Tests to Integration

Testing LlamaIndex Applications: From Unit Tests to Integration

Building applications with LlamaIndex — whether they are retrieval-augmented generation (RAG) pipelines, agents, or custom query engines — introduces a unique set of testing challenges. Unlike traditional software, LLM-powered applications involve probabilistic outputs, external API calls, vector databases, and embedding models. Without a deliberate testing strategy, these systems become brittle, expensive to iterate on, and difficult to trust in production.

This tutorial walks through a complete testing approach for LlamaIndex applications, starting from isolated unit tests and progressing to full integration tests. You will learn how to mock LLM and embedding calls, test retrieval logic, validate prompt construction, and verify end-to-end pipeline behavior.

Why Testing LlamaIndex Applications Matters

LLM applications fail in ways that traditional applications do not. A small change in a prompt can silently degrade retrieval quality. A new embedding model can shift similarity scores enough to break ranking. A dependency on an external API can turn a fast unit test into a slow, flaky, and costly integration test. Testing gives you:

Confidence during iteration: Refactor prompts, swap models, or change chunking strategies without fear of silent regressions.
Cost control: Mocked tests run for free and in milliseconds, so you can run them on every save.
Deterministic behavior: Pin down non-deterministic LLM outputs with fixed responses and assertions on structure rather than exact wording.
Faster debugging: Isolate whether a bug lives in retrieval, prompt construction, synthesis, or the LLM itself.

Project Setup

For this tutorial, install LlamaIndex along with pytest and a mocking helper. We will use the OpenAI integration as the default LLM and embedding provider, but the same patterns apply to other providers.

pip install llama-index llama-index-llms-openai llama-index-embeddings-openai pytest pytest-asyncio

Create a simple LlamaIndex application that we will test throughout. Save it as app.py:

from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding


def build_index(documents):
    splitter = SentenceSplitter(chunk_size=256, chunk_overlap=32)
    nodes = splitter.get_nodes_from_documents(documents)
    return VectorStoreIndex(nodes)


def create_query_engine(index, similarity_top_k=3):
    return index.as_query_engine(similarity_top_k=similarity_top_k)


def answer_question(query_engine, question: str) -> str:
    response = query_engine.query(question)
    return str(response)


def run(documents, question: str) -> str:
    index = build_index(documents)
    engine = create_query_engine(index)
    return answer_question(engine, question)

This is a minimal but realistic RAG pipeline: documents are split, indexed, and queried. Now let us test each layer in isolation before testing the whole thing together.

Unit Testing: Testing Pure Logic

The easiest layer to test is pure logic — functions that do not call LLMs, embeddings, or external services. In our example, the document splitting logic is a good candidate.

# test_unit.py
from llama_index.core import Document
from app import build_index


def test_build_index_creates_nodes():
    documents = [
        Document(text="LlamaIndex is a framework for building LLM applications. It supports RAG pipelines."),
        Document(text="Testing helps ensure reliability. Unit tests are fast and deterministic."),
    ]
    index = build_index(documents)

    # The index should contain nodes derived from our documents
    ref_doc_info = index.ref_doc_info
    assert len(ref_doc_info) == 2


def test_build_index_chunks_long_document():
    long_text = "Sentence one. " * 100
    documents = [Document(text=long_text)]
    index = build_index(documents)

    nodes = list(index.docstore.docs.values())
    assert len(nodes) > 1

These tests run without any API keys and complete in milliseconds. They verify that our chunking and indexing logic behaves as expected. Always start here: extract as much pure logic as possible and test it directly.

Mocking the LLM and Embedding Models

To test components that depend on LLMs or embeddings without making real API calls, use LlamaIndex's built-in mock classes. The MockLLM and MockEmbedding classes let you inject deterministic responses.

# test_mock_llm.py
from llama_index.core import Document, Settings
from llama_index.core.llms.mock import MockLLM
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from app import build_index, create_query_engine, answer_question


def setup_mock_environment():
    Settings.llm = MockLLM(max_tokens=128)
    Settings.embed_model = MockEmbedding(embed_dim=1536)


def test_answer_question_returns_string():
    setup_mock_environment()
    documents = [Document(text="The capital of France is Paris.")]
    index = build_index(documents)
    engine = create_query_engine(index)

    result = answer_question(engine, "What is the capital of France?")
    assert isinstance(result, str)
    assert len(result) > 0

The MockLLM returns a predictable, generic response. This is useful for verifying that the pipeline runs end-to-end and returns the expected type, but it does not verify the content of the response. For that, we need a custom mock.

Custom Mock LLM for Deterministic Responses

When you need to assert on the actual content of an LLM response, subclass MockLLM or build a custom LLM that returns canned answers based on the input prompt.

# custom_mock.py
from llama_index.core.llms.mock import MockLLM
from llama_index.core.llms import ChatMessage, CompletionResponse


class ScriptedLLM(MockLLM):
    """An LLM that returns scripted responses for specific prompts."""

    def __init__(self, scripts: dict):
        super().__init__()
        self.scripts = scripts
        self.calls = []

    def complete(self, prompt: str, **kwargs) -> CompletionResponse:
        self.calls.append(prompt)
        for key, response in self.scripts.items():
            if key in prompt:
                return CompletionResponse(text=response)
        return CompletionResponse(text="default mock response")

Now you can test that your application handles specific LLM outputs correctly:

# test_scripted_llm.py
import pytest
from llama_index.core import Document, Settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from app import build_index, create_query_engine, answer_question
from custom_mock import ScriptedLLM


@pytest.fixture
def scripted_engine():
    scripts = {
        "capital of France": "The capital of France is Paris.",
    }
    Settings.llm = ScriptedLLM(scripts=scripts)
    Settings.embed_model = MockEmbedding(embed_dim=1536)

    documents = [Document(text="France is a country in Europe. Its capital is Paris.")]
    index = build_index(documents)
    return create_query_engine(index)


def test_scripted_response_content(scripted_engine):
    result = answer_question(scripted_engine, "What is the capital of France?")
    assert "Paris" in result


def test_llm_was_called(scripted_engine):
    answer_question(scripted_engine, "What is the capital of France?")
    assert len(Settings.llm.calls) >= 1

This pattern is powerful because it decouples your test from the LLM provider entirely. You can test prompt handling, output parsing, and downstream logic deterministically.

Testing Prompt Construction

A common source of bugs in LlamaIndex applications is the prompt template. You want to verify that the right context and instructions are being sent to the LLM. Capture the prompt using your scripted mock and assert on its contents.

# test_prompts.py
def test_prompt_contains_context(scripted_engine):
    answer_question(scripted_engine, "What is the capital of France?")
    prompt = Settings.llm.calls[0]

    assert "France" in prompt
    assert "capital" in prompt.lower()
    assert "What is the capital of France?" in prompt

By asserting on the prompt rather than the response, you test what your application controls — the inputs it constructs — rather than the LLM's probabilistic output.

Testing Retrieval Logic

Retrieval is the heart of any RAG pipeline. To test it in isolation, build an index from known documents and query the retriever directly, bypassing the LLM entirely.

# test_retrieval.py
from llama_index.core import Document, Settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from app import build_index


def test_retriever_returns_relevant_nodes():
    Settings.embed_model = MockEmbedding(embed_dim=1536)

    documents = [
        Document(text="Python is a popular programming language."),
        Document(text="The Eiffel Tower is located in Paris."),
        Document(text="Photosynthesis converts sunlight into energy."),
    ]
    index = build_index(documents)
    retriever = index.as_retriever(similarity_top_k=2)

    nodes = retriever.retrieve("Tell me about Paris")
    assert len(nodes) == 2
    assert any("Paris" in node.node.text for node in nodes)

Note that MockEmbedding produces deterministic but non-semantic embeddings, so this test verifies structure rather than true semantic relevance. For semantic retrieval testing, you have two options: use a real embedding model in a slower integration test, or inject a custom embedding model that maps known keywords to known vectors.

Testing with a Custom Embedding Model

For more realistic retrieval tests without hitting an API, implement a simple keyword-based embedding model:

# keyword_embed.py
import hashlib
from llama_index.core.embeddings.mock_embed_model import MockEmbedding


class KeywordEmbedding(MockEmbedding):
    """Embeds text based on keyword presence for deterministic retrieval tests."""

    KEYWORDS = ["paris", "python", "photosynthesis", "eiffel", "language"]

    def _get_text_embedding(self, text: str):
        text_lower = text.lower()
        return [1.0 if kw in text_lower else 0.0 for kw in self.KEYWORDS]

# test_keyword_retrieval.py
from llama_index.core import Document, Settings
from app import build_index
from keyword_embed import KeywordEmbedding


def test_keyword_based_retrieval():
    Settings.embed_model = KeywordEmbedding(embed_dim=5)

    documents = [
        Document(text="Python is a popular programming language."),
        Document(text="The Eiffel Tower is located in Paris."),
        Document(text="Photosynthesis converts sunlight into energy."),
    ]
    index = build_index(documents)
    retriever = index.as_retriever(similarity_top_k=1)

    nodes = retriever.retrieve("paris eiffel")
    assert "Paris" in nodes[0].node.text

This gives you deterministic, semantically meaningful retrieval tests that run instantly and cost nothing.

Integration Testing with Real Models

Unit and mock tests cover logic and structure, but eventually you need to verify that your application works with real LLMs and embeddings. Integration tests use actual API calls and should be run less frequently — for example, before merging a pull request or in a nightly CI job.

# test_integration.py
import os
import pytest
from llama_index.core import Document, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from app import run


pytestmark = pytest.mark.integration


@pytest.fixture(autouse=True)
def real_models():
    Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
    Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")


def test_real_rag_pipeline():
    documents = [
        Document(text="The LlamaIndex framework was created to simplify building LLM applications."),
        Document(text="It provides tools for data ingestion, indexing, and querying."),
    ]
    result = run(documents, "What does LlamaIndex do?")
    assert isinstance(result, str)
    assert len(result) > 20
    assert "LlamaIndex" in result or "framework" in result.lower()

Mark these tests with pytest.mark.integration so you can exclude them during fast development cycles:

# Run only fast unit tests
pytest -m "not integration"

# Run everything, including integration tests
pytest

Configure the marker in pytest.ini to avoid warnings:

[pytest]
markers =
    integration: marks tests that call real LLM APIs (slow and costly)
asyncio_mode = auto

Testing Asynchronous Components

LlamaIndex supports async query engines. Testing them requires pytest-asyncio. The same mocking patterns apply.

# test_async.py
import pytest
from llama_index.core import Document, Settings
from llama_index.core.llms.mock import MockLLM
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from app import build_index, create_query_engine


@pytest.mark.asyncio
async def test_async_query():
    Settings.llm = MockLLM(max_tokens=64)
    Settings.embed_model = MockEmbedding(embed_dim=1536)

    documents = [Document(text="Async queries allow concurrent processing.")]
    index = build_index(documents)
    engine = create_query_engine(index)

    response = await engine.aquery("What is async?")
    assert response.response is not None
    assert len(str(response)) > 0

Testing Error Handling

Production applications must handle failures gracefully — API timeouts, rate limits, and malformed responses. Test these paths by configuring your mock to raise exceptions.

# test_errors.py
import pytest
from llama_index.core import Document, Settings
from llama_index.core.llms import CompletionResponse
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from app import build_index, create_query_engine, answer_question
from custom_mock import ScriptedLLM


class FailingLLM(ScriptedLLM):
    def complete(self, prompt: str, **kwargs):
        raise RuntimeError("Simulated API failure")


def test_error_propagation():
    Settings.llm = FailingLLM(scripts={})
    Settings.embed_model = MockEmbedding(embed_dim=1536)

    documents = [Document(text="Some content.")]
    index = build_index(documents)
    engine = create_query_engine(index)

    with pytest.raises(RuntimeError, match="Simulated API failure"):
        answer_question(engine, "Anything")

Once you confirm errors propagate, add retry logic or fallback responses to your application and test that those paths work as intended.

Snapshot Testing for Prompts

As prompts grow complex, snapshot testing helps catch unintended changes. Use pytest-snapshot or simply compare against a stored file.

# test_snapshot.py
import json
from pathlib import Path
from llama_index.core import Document, Settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from app import build_index, create_query_engine, answer_question
from custom_mock import ScriptedLLM

SNAPSHOT_DIR = Path(__file__).parent / "snapshots"


def test_prompt_snapshot():
    Settings.llm = ScriptedLLM(scripts={"default": "ok"})
    Settings.embed_model = MockEmbedding(embed_dim=1536)

    documents = [Document(text="Snapshot testing captures prompt structure.")]
    index = build_index(documents)
    engine = create_query_engine(index)

    answer_question(engine, "What is snapshot testing?")
    prompt = Settings.llm.calls[0]

    snapshot_path = SNAPSHOT_DIR / "prompt_snapshot.json"
    if not snapshot_path.exists():
        SNAPSHOT_DIR.mkdir(parents=True, exist_ok=True)
        snapshot_path.write_text(json.dumps({"prompt": prompt}, indent=2))
        pytest.skip("Snapshot created. Re-run to verify.")

    saved = json.loads(snapshot_path.read_text())
    assert prompt == saved["prompt"], "Prompt has changed. Update snapshot if intentional."

This catches accidental prompt drift when you modify templates, context formatting, or metadata injection.

Best Practices

Test in layers: Start with pure logic, then mocked components, then real integration tests. Each layer catches different bugs at different costs.
Prefer asserting on inputs over outputs: You control the prompts you build; you do not control LLM responses. Assert on prompt structure and parsed output format rather than exact wording.
Keep integration tests small and few: Use short documents, cheap models, and low token limits. Run them in CI, not on every save.
Pin your models in integration tests: Specify exact model names and set temperature=0 to reduce variance.
Use fixtures for shared setup: Build indexes and engines in fixtures so tests stay focused and fast.
Separate test suites by speed: Use pytest markers to run unit tests continuously and integration tests on demand.
Test edge cases: Empty documents, very long inputs, queries with no relevant context, and malformed LLM outputs all deserve coverage.
Monitor costs: Integration tests that call real APIs cost money. Track usage and consider setting spending limits on your API keys.
Version your test data: Keep fixture documents in version control so test behavior is reproducible across environments.

Conclusion

Testing LlamaIndex applications effectively means embracing a layered strategy: unit tests for pure logic, mocked tests for prompt construction and pipeline behavior, and integration tests for real-world validation. By mocking LLMs and embeddings, you gain fast, deterministic, and free test suites that catch regressions early. By reserving real API calls for a small set of integration tests, you keep costs and runtime manageable while still verifying that your application works end to end. With these patterns in place, you can iterate confidently on prompts, models, and retrieval strategies — knowing that your test suite will catch problems before your users do.