How to Evaluate RAG Systems with Ragas

Introduction to Evaluating RAG Systems with Ragas

Retrieval-Augmented Generation (RAG) systems have become a cornerstone of modern LLM applications, allowing models to ground their responses in external knowledge. However, building a RAG pipeline is only half the battle — knowing whether it actually works is the other half. Ragas (Retrieval-Augmented Generation Assessment) is an open-source framework designed specifically to evaluate RAG systems in a structured, reproducible, and automated way.

In this tutorial, you'll learn what Ragas is, why evaluation matters, how to integrate it into your workflow, and the best practices that will help you ship reliable RAG applications.

What Is Ragas?

Ragas is a Python library that provides a suite of metrics to evaluate RAG pipelines. Instead of relying on manual inspection or gut feeling, Ragas uses LLM-based evaluation techniques to score your retrieval and generation components across multiple dimensions. It works with popular frameworks like LangChain and LlamaIndex, and it can evaluate both the retrieval step (did we fetch the right context?) and the generation step (did the model use that context correctly?).

Core Metrics in Ragas

Ragas evaluates RAG systems using several key metrics, each targeting a specific aspect of the pipeline:

Faithfulness — Measures whether the generated answer is factually consistent with the retrieved context. It detects hallucinations by checking if every claim in the answer can be inferred from the context.
Answer Relevancy — Assesses whether the answer actually addresses the user's question. A high score means the response is on-topic and informative.
Context Precision — Evaluates whether the retrieved context is relevant to the question. It penalizes retrieving irrelevant documents that don't help answer the query.
Context Recall — Measures whether the retrieved context contains all the information needed to answer the question, compared to a ground-truth answer.
Context Relevancy — A broader measure of how relevant the retrieved chunks are to the query.

These metrics combine to give you a holistic view of your RAG system's performance, from retrieval quality to generation fidelity.

Why RAG Evaluation Matters

Without evaluation, RAG systems are a black box. You might retrieve documents that look relevant but actually mislead the model, or your LLM might hallucinate confidently even when the correct context is available. Common failure modes include:

Retrieval failures — The vector store returns chunks that are semantically similar but not actually useful for answering the question.
Hallucination — The model generates claims that aren't supported by the retrieved context.
Context overload — Too many irrelevant chunks dilute the signal, causing the model to ignore important information.
Answer drift — The model answers a slightly different question than what was asked.

Ragas helps you catch these issues early, compare different configurations (chunk sizes, embedding models, prompt templates), and track regressions as your system evolves. This is especially important in production, where silent quality degradation can erode user trust.

Getting Started with Ragas

Installation

Install Ragas along with LangChain, which it uses for LLM and embedding integrations:

pip install ragas langchain langchain-openai

Set your OpenAI API key as an environment variable:

import os
os.environ["OPENAI_API_KEY"] = "your-api-key-here"

Preparing Evaluation Data

Ragas requires a dataset with four key columns: question, answer, contexts, and ground_truth. The ground_truth is a reference answer that helps compute context recall. Here's how to build a sample evaluation dataset:

from datasets import Dataset

data = {
    "question": [
        "What is the capital of France?",
        "Who wrote the novel '1984'?"
    ],
    "answer": [
        "The capital of France is Paris.",
        "George Orwell wrote the novel '1984'."
    ],
    "contexts": [
        ["France is a country in Western Europe. Its capital and largest city is Paris."],
        ["'1984' is a dystopian social science fiction novel by English writer George Orwell, published in 1949."]
    ],
    "ground_truth": [
        "Paris is the capital of France.",
        "George Orwell is the author of '1984'."
    ]
}

eval_dataset = Dataset.from_dict(data)

Running the Evaluation

Once you have your dataset, running the evaluation is straightforward. You define which metrics to compute and pass them along with the dataset:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)

results = evaluate(
    dataset=eval_dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall
    ]
)

print(results)

The output will be a dictionary-like object showing the aggregate score for each metric across your dataset. You can also convert it to a pandas DataFrame for a row-by-row breakdown:

import pandas as pd

df = results.to_pandas()
print(df.head())

Integrating Ragas with a Real RAG Pipeline

In practice, you won't manually construct the evaluation dataset. Instead, you'll run your RAG pipeline on a set of test questions and collect the outputs. Here's a complete example using LangChain:

from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

# 1. Prepare your documents
documents = [
    Document(page_content="The Eiffel Tower is a wrought-iron lattice tower in Paris, France. It was constructed from 1887 to 1889."),
    Document(page_content="The Great Wall of China is a series of fortifications across northern China, built over many centuries."),
    Document(page_content="Mount Everest is Earth's highest mountain above sea level, located in the Himalayas on the border of Nepal and China.")
]

# 2. Split and embed
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
chunks = splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)

# 3. Build the RAG chain
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

# 4. Define test questions and ground truth
test_cases = [
    {"question": "Where is the Eiffel Tower located?", "ground_truth": "The Eiffel Tower is located in Paris, France."},
    {"question": "What is the highest mountain on Earth?", "ground_truth": "Mount Everest is the highest mountain on Earth."},
    {"question": "Where is the Great Wall of China?", "ground_truth": "The Great Wall of China is located across northern China."}
]

# 5. Run the pipeline and collect outputs
eval_data = {"question": [], "answer": [], "contexts": [], "ground_truth": []}

for case in test_cases:
    question = case["question"]
    retrieved_docs = retriever.invoke(question)
    contexts = [doc.page_content for doc in retrieved_docs]
    
    # Build a simple prompt with context
    context_text = "\n\n".join(contexts)
    prompt = f"Answer the question based on the context below.\n\nContext:\n{context_text}\n\nQuestion: {question}"
    answer = llm.invoke(prompt).content
    
    eval_data["question"].append(question)
    eval_data["answer"].append(answer)
    eval_data["contexts"].append(contexts)
    eval_data["ground_truth"].append(case["ground_truth"])

# 6. Evaluate with Ragas
eval_dataset = Dataset.from_dict(eval_data)
results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)

print(results)
df = results.to_pandas()
print(df[["question", "faithfulness", "answer_relevancy", "context_precision", "context_recall"]])

This example demonstrates the full loop: from document ingestion through retrieval and generation, all the way to automated evaluation. You can swap out the embedding model, chunk size, retriever settings, or LLM and immediately see how the metrics change.

Generating Synthetic Test Data

One challenge with RAG evaluation is creating a good set of test questions and ground-truth answers. Ragas includes a TestsetGenerator that can create synthetic evaluation datasets from your documents:

from ragas.testset import TestsetGenerator
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator_llm = ChatOpenAI(model="gpt-4o-mini")
critic_llm = ChatOpenAI(model="gpt-4o")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator.from_langchain(
    generator_llm=generator_llm,
    critic_llm=critic_llm,
    embeddings=embeddings
)

# Define the distribution of question types
distribution = {
    simple: 0.5,
    reasoning: 0.3,
    multi_context: 0.2
}

testset = generator.generate_with_langchain_docs(
    documents=chunks,
    test_size=10,
    distributions=distribution
)

test_df = testset.to_pandas()
print(test_df[["question", "ground_truth"]].head())

This generates questions of varying complexity — from simple factoid lookups to multi-hop reasoning questions — giving you a more robust evaluation set without manual effort.

Best Practices for RAG Evaluation

1. Evaluate Iteratively, Not Just Once

Run evaluations every time you change a component — chunk size, embedding model, prompt template, or LLM. Track scores over time to catch regressions. Consider integrating Ragas into your CI/CD pipeline so that quality drops are flagged before deployment.

2. Use a Strong Evaluator LLM

Ragas uses an LLM to judge faithfulness, relevancy, and other metrics. The quality of evaluation depends on the judge model. Use a capable model like GPT-4o or Claude 3.5 Sonnet for the evaluator, even if your production RAG system uses a smaller, cheaper model.

from langchain_openai import ChatOpenAI

evaluator_llm = ChatOpenAI(model="gpt-4o", temperature=0)

results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=evaluator_llm
)

3. Curate a Representative Test Set

Your evaluation is only as good as your test data. Include a mix of question types: simple factoid questions, multi-hop reasoning questions, ambiguous queries, and edge cases. Aim for at least 50–100 test questions for meaningful aggregate scores.

4. Don't Optimize for a Single Metric

It's tempting to maximize faithfulness, but doing so might make answers overly conservative and less relevant. Look at all metrics together. For example, high context precision with low faithfulness suggests your retrieval is good but the LLM is hallucinating — a prompt engineering issue rather than a retrieval issue.

5. Log Individual Examples for Debugging

Aggregate scores hide individual failures. Use the pandas DataFrame output to inspect low-scoring examples and understand what went wrong:

df = results.to_pandas()

# Find examples with low faithfulness (potential hallucinations)
low_faith = df[df["faithfulness"] < 0.5]
for _, row in low_faith.iterrows():
    print(f"Q: {row['question']}")
    print(f"A: {row['answer']}")
    print(f"Context: {row['contexts']}")
    print(f"Faithfulness: {row['faithfulness']}")
    print("---")

6. Combine Automated and Human Evaluation

Ragas is powerful but not perfect. LLM-based judges can have biases and blind spots. Use Ragas for rapid iteration and regression testing, but supplement it with periodic human review, especially for high-stakes applications like healthcare or legal Q&A.

Conclusion

Evaluating RAG systems is essential for building trustworthy LLM applications, and Ragas provides a practical, automated way to measure retrieval quality and generation fidelity. By integrating metrics like faithfulness, answer relevancy, context precision, and context recall into your development workflow, you can catch hallucinations, diagnose retrieval failures, and confidently iterate on your pipeline. Start with a small test set, run evaluations on every change, and gradually expand your evaluation coverage as your application grows. With consistent evaluation practices in place, you'll be able to ship RAG systems that are not only functional but genuinely reliable.