Introduction to RAG Evaluation
Retrieval-Augmented Generation (RAG) pipelines combine a retrieval step with a generative LLM to produce grounded, context-aware answers. Evaluating these systems requires a two-pronged approach: measuring how well your retriever finds relevant documents (retrieval quality) and assessing whether the generated answer is correct, faithful, and relevant (answer accuracy). Without rigorous evaluation, you're flying blind — you cannot know if changes to your chunking strategy, embedding model, or LLM prompt actually improve the user experience.
RAG evaluation quantifies both components so you can debug failures systematically. A retriever might return irrelevant chunks even when the LLM is capable, or the LLM might hallucinate despite perfect retrieval. By isolating each stage with targeted metrics, you gain the diagnostic precision needed to iterate effectively.
Why RAG Evaluation Matters
RAG systems fail in subtle ways that escape casual inspection:
- Silent hallucination: The LLM generates a plausible-sounding but factually wrong answer that cites retrieved documents — yet the documents don't actually support the claim.
- Retrieval mismatch: The retriever returns documents that are topically adjacent but miss the specific information needed, causing the LLM to extrapolate incorrectly.
- Order sensitivity: The rank order of retrieved chunks dramatically affects generation quality, but surface-level testing may not reveal this.
- Regressions from model swaps: Changing your embedding model or vector database configuration can silently degrade retrieval without obvious errors.
Systematic evaluation lets you catch these issues before they reach production, compare candidate models objectively, and build a regression suite that runs in CI/CD. It transforms RAG development from guesswork into an engineering discipline.
Core Metrics for Retrieval Quality
Retrieval evaluation measures whether your system surfaces the right documents from your knowledge base. You need a labeled dataset where each query is paired with a set of relevant document IDs (ground truth). These metrics operate on the ranked list of documents your retriever returns.
Precision@K and Recall@K
These are the simplest and most interpretable retrieval metrics. Precision@K answers: "Of the top K documents I retrieved, how many were actually relevant?" Recall@K answers: "Of all relevant documents that exist, how many did I find in the top K?"
import numpy as np
from typing import List, Set
def precision_at_k(retrieved_ids: List[str], relevant_ids: Set[str], k: int) -> float:
"""Compute Precision@K: fraction of top-k results that are relevant."""
if k <= 0:
return 0.0
top_k = retrieved_ids[:k]
if not top_k:
return 0.0
relevant_in_top_k = sum(1 for doc_id in top_k if doc_id in relevant_ids)
return relevant_in_top_k / k
def recall_at_k(retrieved_ids: List[str], relevant_ids: Set[str], k: int) -> float:
"""Compute Recall@K: fraction of all relevant docs found in top-k."""
if not relevant_ids:
return 1.0 # If nothing is expected, recall is perfect by convention
top_k = retrieved_ids[:k]
relevant_found = sum(1 for doc_id in top_k if doc_id in relevant_ids)
return relevant_found / len(relevant_ids)
# Example usage
retrieved = ["doc_7", "doc_3", "doc_9", "doc_1", "doc_5"]
relevant = {"doc_3", "doc_5", "doc_8", "doc_11"}
print(f"Precision@3: {precision_at_k(retrieved, relevant, k=3):.3f}") # doc_3 is relevant → 1/3 = 0.333
print(f"Recall@3: {recall_at_k(retrieved, relevant, k=3):.3f}") # doc_3 found, out of 4 → 1/4 = 0.250
print(f"Precision@5: {precision_at_k(retrieved, relevant, k=5):.3f}") # doc_3, doc_5 → 2/5 = 0.400
print(f"Recall@5: {recall_at_k(retrieved, relevant, k=5):.3f}") # doc_3, doc_5 → 2/4 = 0.500
Choose K based on your RAG architecture. If you inject the top 3 chunks into the prompt, evaluate Precision@3 and Recall@3. If you use a re-ranker that selects from a larger pool, you might evaluate Recall@20 to measure the retriever's recall ceiling before re-ranking.
Mean Reciprocal Rank (MRR)
MRR measures where the first relevant document appears in the ranked list. It's ideal when the user only needs one good document (e.g., FAQ retrieval, fact lookup). The reciprocal rank for a query is 1 / rank_of_first_relevant_doc. MRR averages this across all queries.
def reciprocal_rank(retrieved_ids: List[str], relevant_ids: Set[str]) -> float:
"""Return the reciprocal of the rank of the first relevant document (1-indexed)."""
for i, doc_id in enumerate(retrieved_ids, start=1):
if doc_id in relevant_ids:
return 1.0 / i
return 0.0 # No relevant document found
def mean_reciprocal_rank(
retrieved_per_query: List[List[str]],
relevant_per_query: List[Set[str]]
) -> float:
"""Compute MRR across multiple queries."""
if not retrieved_per_query:
return 0.0
rr_values = [
reciprocal_rank(retrieved, relevant)
for retrieved, relevant in zip(retrieved_per_query, relevant_per_query)
]
return np.mean(rr_values)
# Example with 3 queries
retrieved_batch = [
["doc_a", "doc_b", "doc_c", "doc_d"], # Query 1: first relevant at rank 3
["doc_x", "doc_y", "doc_z"], # Query 2: first relevant at rank 1
["doc_p", "doc_q", "doc_r"], # Query 3: no relevant doc found
]
relevant_batch = [
{"doc_c", "doc_f"},
{"doc_x"},
{"doc_s", "doc_t"},
]
mrr = mean_reciprocal_rank(retrieved_batch, relevant_batch)
print(f"MRR: {mrr:.4f}")
# Expected: (1/3 + 1/1 + 0) / 3 = (0.333 + 1.0 + 0) / 3 = 0.4444
Normalized Discounted Cumulative Gain (NDCG)
NDCG extends relevance from binary (relevant/not) to graded relevance. You can assign scores like 3=perfect, 2=highly relevant, 1=somewhat relevant, 0=not relevant. NDCG penalizes relevant documents that appear lower in the ranking and normalizes against an ideal ordering. This is powerful for RAG systems where some chunks are more informative than others.
def dcg(relevance_scores: List[float], k: int = None) -> float:
"""Compute Discounted Cumulative Gain for graded relevance scores."""
if k is not None:
relevance_scores = relevance_scores[:k]
gains = []
for i, rel in enumerate(relevance_scores):
# Discount factor: log2(rank + 1), with rank starting at 1
discount = np.log2(i + 2) # i+2 because i is 0-indexed, rank = i+1
gains.append((2 ** rel - 1) / discount)
return sum(gains)
def ndcg_at_k(retrieved_scores: List[float], ideal_scores: List[float], k: int) -> float:
"""Compute NDCG@K given retrieved relevance scores and ideal relevance scores."""
# DCG of the retrieved ranking at k
retrieved_dcg = dcg(retrieved_scores, k)
# DCG of the ideal ranking (sorted descending) at k
ideal_scores_sorted = sorted(ideal_scores, reverse=True)
ideal_dcg = dcg(ideal_scores_sorted, k)
if ideal_dcg == 0:
return 1.0 # If ideal DCG is zero, NDCG is perfect by convention
return retrieved_dcg / ideal_dcg
# Example: graded relevance for 6 retrieved documents
# Query has 4 relevant docs in the collection with scores: 3, 2, 1, 0 (others are 0)
retrieved_relevance = [3, 0, 2, 0, 1, 0] # Retrieved ranking relevance
all_relevant_scores = [3, 2, 1, 0, 0, 0, 0, 0] # All docs in collection (padded)
ndcg_3 = ndcg_at_k(retrieved_relevance, all_relevant_scores, k=3)
ndcg_6 = ndcg_at_k(retrieved_relevance, all_relevant_scores, k=6)
print(f"NDCG@3: {ndcg_3:.4f}")
print(f"NDCG@6: {ndcg_6:.4f}")
To use NDCG in practice, you need graded relevance judgments. You can derive these from annotator ratings, or automatically via a strong LLM scoring the relevance of each chunk to the query on a scale of 0–3.
Building a Retrieval Evaluation Harness
Here's a complete function that evaluates retrieval across multiple metrics given a test dataset:
from dataclasses import dataclass
from typing import Dict, List, Set
import numpy as np
@dataclass
class RetrievalEvalResult:
precision_at_k: Dict[int, float]
recall_at_k: Dict[int, float]
mrr: float
ndcg_at_k: Dict[int, float]
def evaluate_retrieval(
queries: List[str],
retrieved_per_query: List[List[str]],
relevant_per_query: List[Set[str]],
graded_relevance_per_query: List[Dict[str, int]] = None,
k_values: List[int] = [1, 3, 5, 10]
) -> RetrievalEvalResult:
"""
Evaluate retrieval quality across multiple metrics.
Args:
queries: List of query strings (for reference)
retrieved_per_query: For each query, ordered list of retrieved document IDs
relevant_per_query: For each query, set of relevant document IDs (binary)
graded_relevance_per_query: Optional, dict mapping doc_id → relevance grade (0–3)
k_values: List of K values for Precision@K and Recall@K
"""
n = len(queries)
# Precision@K and Recall@K (averaged across queries)
precision_results = {}
recall_results = {}
for k in k_values:
prec_values = [
precision_at_k(retrieved_per_query[i], relevant_per_query[i], k)
for i in range(n)
]
rec_values = [
recall_at_k(retrieved_per_query[i], relevant_per_query[i], k)
for i in range(n)
]
precision_results[k] = np.mean(prec_values)
recall_results[k] = np.mean(rec_values)
# MRR
mrr = mean_reciprocal_rank(retrieved_per_query, relevant_per_query)
# NDCG@K (only if graded relevance is provided)
ndcg_results = {}
if graded_relevance_per_query is not None:
for k in k_values:
ndcg_values = []
for i in range(n):
retrieved_scores = [
graded_relevance_per_query[i].get(doc_id, 0)
for doc_id in retrieved_per_query[i]
]
# Ideal scores: all relevant docs for this query, sorted
ideal_scores = list(graded_relevance_per_query[i].values())
ndcg_values.append(ndcg_at_k(retrieved_scores, ideal_scores, k))
ndcg_results[k] = np.mean(ndcg_values)
return RetrievalEvalResult(
precision_at_k=precision_results,
recall_at_k=recall_results,
mrr=mrr,
ndcg_at_k=ndcg_results
)
# Example test dataset
test_queries = [
"What is the refund policy for international orders?",
"How do I reset my password?",
"What are the system requirements for the desktop app?",
]
retrieved = [
["doc_refund_intl", "doc_shipping", "doc_returns_us", "doc_privacy", "doc_terms"],
["doc_pwd_reset", "doc_account_faq", "doc_2fa_setup", "doc_login_troubleshoot", "doc_email_verify"],
["doc_sysreq_v2", "doc_sysreq_legacy", "doc_install_guide", "doc_troubleshooting", "doc_changelog"],
]
relevant = [
{"doc_refund_intl", "doc_returns_us"},
{"doc_pwd_reset", "doc_account_faq", "doc_login_troubleshoot"},
{"doc_sysreq_v2", "doc_sysreq_legacy"},
]
# Optional graded relevance (3=perfect, 2=high, 1=somewhat, 0=none)
graded = [
{"doc_refund_intl": 3, "doc_returns_us": 2, "doc_shipping": 1},
{"doc_pwd_reset": 3, "doc_account_faq": 2, "doc_login_troubleshoot": 2, "doc_2fa_setup": 1},
{"doc_sysreq_v2": 3, "doc_sysreq_legacy": 2, "doc_install_guide": 1},
]
result = evaluate_retrieval(test_queries, retrieved, relevant, graded, k_values=[1, 3, 5])
print(f"Precision@3: {result.precision_at_k[3]:.3f}")
print(f"Recall@5: {result.recall_at_k[5]:.3f}")
print(f"MRR: {result.mrr:.3f}")
print(f"NDCG@3: {result.ndcg_at_k[3]:.3f}")
print(f"NDCG@5: {result.ndcg_at_k[5]:.3f}")
Core Metrics for Answer Accuracy
Once retrieval is measured, you must evaluate the final generated answer. This is more challenging because natural language is fuzzy. Modern approaches combine LLM-based evaluation with reference-based metrics.
Faithfulness: Does the Answer Derive from Retrieved Context?
Faithfulness (also called "groundedness" or "hallucination rate") checks whether every claim in the generated answer is supported by the retrieved documents. An unfaithful answer may be fluent and plausible but factually wrong. The most reliable method uses an LLM to decompose the answer into atomic claims and verify each against the context.
from typing import List, Dict
import json
# This is a simplified LLM-based faithfulness check.
# In production, use structured output with a capable model (GPT-4, Claude, etc.)
def extract_claims_via_llm(answer: str, llm_call) -> List[str]:
"""
Use an LLM to decompose the answer into standalone atomic claims.
Returns a list of claim strings.
"""
prompt = f"""
Decompose the following answer into a list of standalone, atomic factual claims.
Each claim should be a single, verifiable statement.
Return the claims as a JSON array of strings.
Answer: {answer}
Claims (JSON array):
"""
response = llm_call(prompt)
try:
claims = json.loads(response)
return claims
except json.JSONDecodeError:
# Fallback: split on newlines and filter empties
return [c.strip() for c in response.split('\n') if c.strip()]
def verify_claim_against_context(claim: str, contexts: List[str], llm_call) -> Dict[str, any]:
"""
Verify a single claim against the retrieved contexts.
Returns a dict with 'verdict' ('supported', 'contradicted', 'insufficient') and 'reason'.
"""
context_text = "\n\n---\n\n".join(contexts)
prompt = f"""
You are a careful fact-checker. Given a claim and the retrieved context documents,
determine whether the claim is:
- "supported": The context explicitly confirms the claim.
- "contradicted": The context explicitly contradicts the claim.
- "insufficient": The context does not contain enough information to verify.
Context:
{context_text}
Claim: {claim}
Respond with a JSON object: {{"verdict": "...", "reason": "..."}}
"""
response = llm_call(prompt)
return json.loads(response)
def faithfulness_score(answer: str, contexts: List[str], llm_call) -> float:
"""
Compute faithfulness as: (number of supported claims) / (total claims).
Returns a float between 0 and 1.
"""
claims = extract_claims_via_llm(answer, llm_call)
if not claims:
return 1.0 # No claims to verify; technically faithful
verdicts = [verify_claim_against_context(c, contexts, llm_call) for c in claims]
supported_count = sum(1 for v in verdicts if v['verdict'] == 'supported')
# Print detailed breakdown (useful for debugging)
for claim, verdict in zip(claims, verdicts):
print(f"Claim: {claim}\n Verdict: {verdict['verdict']} — {verdict['reason']}\n")
return supported_count / len(claims)
# Pseudocode usage (requires an actual LLM call implementation)
# faithfulness = faithfulness_score(
# answer="The refund policy allows returns within 30 days for international orders.",
# contexts=["International orders can be returned within 30 days of delivery. ..."],
# llm_call=your_llm_function
# )
# print(f"Faithfulness: {faithfulness:.2f}")
In production pipelines, you'd replace the llm_call placeholder with an actual API call to GPT-4, Claude, or a fine-tuned evaluation model. Many teams run faithfulness checks as an offline batch evaluation against a golden dataset.
Answer Relevance: Does the Answer Address the Query?
An answer can be faithful but irrelevant — for example, if the retriever returned documents about a different topic and the LLM dutifully summarized them. Answer relevance checks whether the generated response actually answers what the user asked. A common approach generates hypothetical questions from the answer and checks their semantic similarity to the original query.
from sentence_transformers import SentenceTransformer, util
from typing import List
# Load a sentence-transformer model for semantic similarity
model = SentenceTransformer('all-MiniLM-L6-v2')
def answer_relevance_score(
query: str,
answer: str,
llm_call,
num_questions: int = 3,
similarity_model: SentenceTransformer = model
) -> float:
"""
Compute answer relevance by generating questions from the answer
and measuring cosine similarity with the original query.
Higher score = more relevant answer.
"""
# Step 1: Generate synthetic questions that the answer could address
prompt = f"""
Given the following answer, generate {num_questions} different questions
that this answer could be responding to. Make them diverse.
Return as a JSON array of strings.
Answer: {answer}
Questions (JSON array):
"""
response = llm_call(prompt)
try:
generated_questions = json.loads(response)
except json.JSONDecodeError:
generated_questions = [line.strip() for line in response.split('\n') if '?' in line]
if not generated_questions:
return 0.0
# Step 2: Compute cosine similarities between original query and each generated question
query_embedding = similarity_model.encode(query, convert_to_tensor=True)
question_embeddings = similarity_model.encode(
generated_questions, convert_to_tensor=True
)
similarities = util.cos_sim(query_embedding, question_embeddings)[0]
# Step 3: Average the similarity scores
avg_similarity = similarities.mean().item()
return avg_similarity
# Example (requires LLM call implementation)
# score = answer_relevance_score(
# query="What is the capital of France?",
# answer="Paris is the capital and largest city of France.",
# llm_call=your_llm_function
# )
# print(f"Answer Relevance: {score:.3f}")
Semantic Answer Similarity (SAS) / BERTScore
When you have reference (ground-truth) answers, you can directly compare the generated answer to the reference using embedding similarity or token-level matching. BERTScore uses contextual embeddings from transformer models to compute precision, recall, and F1 between the generated and reference text — far more robust than n-gram overlap metrics like ROUGE.
from sentence_transformers import SentenceTransformer, util
def semantic_answer_similarity(
generated_answer: str,
reference_answer: str,
model: SentenceTransformer = None
) -> float:
"""
Compute cosine similarity between embeddings of generated and reference answers.
Simple but effective for high-level semantic matching.
"""
if model is None:
model = SentenceTransformer('all-MiniLM-L6-v2')
gen_embedding = model.encode(generated_answer, convert_to_tensor=True)
ref_embedding = model.encode(reference_answer, convert_to_tensor=True)
similarity = util.cos_sim(gen_embedding, ref_embedding).item()
return similarity
# Example
generated = "The refund policy allows returns within 30 days for most items."
reference = "Customers can return most products within 30 days of purchase for a full refund."
sas = semantic_answer_similarity(generated, reference)
print(f"Semantic Answer Similarity: {sas:.3f}")
For a more rigorous token-level evaluation, use the bert_score library which computes F1 with importance weighting and handles long texts gracefully:
# Install: pip install bert-score
from bert_score import score
def bertscore_evaluation(generated_answers: List[str], reference_answers: List[str]):
"""
Compute BERTScore precision, recall, and F1 for a batch of answers.
Returns averaged metrics.
"""
P, R, F1 = score(
generated_answers,
reference_answers,
lang="en",
model_type="microsoft/deberta-xlarge-mnli", # Strong model for evaluation
verbose=False
)
return {
"precision": P.mean().item(),
"recall": R.mean().item(),
"f1": F1.mean().item()
}
# Example batch evaluation
generated_batch = [
"Paris is the capital of France.",
"To reset your password, click the 'Forgot Password' link on the login page.",
]
reference_batch = [
"The capital of France is Paris.",
"Users can reset their password by clicking 'Forgot Password' on the login screen.",
]
results = bertscore_evaluation(generated_batch, reference_batch)
print(f"BERTScore Precision: {results['precision']:.3f}")
print(f"BERTScore Recall: {results['recall']:.3f}")
print(f"BERTScore F1: {results['f1']:.3f}")
Using Specialized RAG Evaluation Frameworks
Several open-source libraries bundle these metrics into unified pipelines, saving you from implementing everything from scratch.
RAGAS (RAG Assessment)
RAGAS is a dedicated library for RAG evaluation that computes retrieval metrics (context precision, context recall) and answer metrics (faithfulness, answer relevancy, answer correctness) using LLM-based judges. It requires a test dataset with questions, generated answers, retrieved contexts, and optionally ground-truth answers.
# Install: pip install ragas
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
answer_correctness,
)
from datasets import Dataset
import pandas as pd
# Prepare your evaluation dataset
eval_data = {
"question": [
"What is the company's remote work policy?",
"How do I request time off?",
"What are the health insurance options?",
],
"answer": [
"Employees may work remotely up to 3 days per week with manager approval.",
"Submit a request through the HR portal at least 2 weeks in advance.",
"We offer PPO and HSA plans through Blue Cross.",
],
"contexts": [
["Remote work policy: up to 3 days per week, manager approval required. ..."],
["Time off requests: submit via HR portal, 2 weeks advance notice. ..."],
["Health insurance: Blue Cross PPO and HSA options available. ..."],
],
"ground_truth": [
"The remote work policy allows up to 3 days per week with manager approval.",
"Employees should submit time-off requests via the HR portal at least two weeks ahead.",
"Health insurance includes Blue Cross PPO and HSA plans.",
],
}
# Convert to HuggingFace Dataset
dataset = Dataset.from_dict(eval_data)
# Run evaluation
metrics = [faithfulness, answer_relevancy, context_precision, context_recall, answer_correctness]
results = evaluate(dataset, metrics=metrics)
# Convert to pandas for analysis
df_results = results.to_pandas()
print(df_results[['question', 'faithfulness', 'answer_relevancy', 'context_precision']])
# Aggregate scores
print(f"\nAverage Faithfulness: {df_results['faithfulness'].mean():.3f}")
print(f"Average Answer Relevancy: {df_results['answer_relevancy'].mean():.3f}")
print(f"Average Context Precision: {df_results['context_precision'].mean():.3f}")
print(f"Average Context Recall: {df_results['context_recall'].mean():.3f}")
RAGAS uses your configured LLM (GPT-4, Claude, or open-source models) as an evaluator judge. The faithfulness metric implements claim extraction and verification similar to the manual implementation shown earlier, but with more sophisticated prompt engineering and built-in robustness.
DeepEval
DeepEval provides a broader evaluation framework with RAG-specific metrics, CI/CD integration, and a dashboard. It offers hallucination detection, faithfulness scoring, and contextual relevancy metrics out of the box.
# Install: pip install deepeval
from deepeval import evaluate
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric, ContextualRelevancyMetric
from deepeval.test_case import LLMTestCase
# Define test cases
test_cases = [
LLMTestCase(
input="What is the company's remote work policy?",
actual_output="Employees may work remotely up to 3 days per week with manager approval.",
expected_output="The remote work policy allows up to 3 days per week with manager approval.",
retrieval_context=[
"Remote work policy: up to 3 days per week, manager approval required.",
"All remote work arrangements must be documented in writing.",
],
),
LLMTestCase(
input="How do I request time off?",
actual_output="Submit a request through the HR portal at least 2 weeks in advance.",
expected_output="Employees should submit time-off requests via the HR portal at least two weeks ahead.",
retrieval_context=[
"Time off requests: submit via HR portal, 2 weeks advance notice required.",
"Managers must approve time off requests within 3 business days.",
],
),
]
# Initialize metrics
faithfulness = FaithfulnessMetric(threshold=0.7)
answer_relevancy = AnswerRelevancyMetric(threshold=0.7)
contextual_relevancy = ContextualRelevancyMetric(threshold=0.7)
# Run evaluation
evaluation_results = evaluate(
test_cases=test_cases,
metrics=[faithfulness, answer_relevancy, contextual_relevancy],
)
# Print results
for result in evaluation_results:
print(f"Test: {result.input}")
print(f" Faithfulness: {result.metrics[0].score:.3f} — {result.metrics[0].reason}")
print(f" Answer Relevancy: {result.metrics[1].score:.3f} — {result.metrics[1].reason}")
print(f" Contextual Relevancy: {result.metrics[2].score:.3f} — {result.metrics[2].reason}")
print()
Building a Complete Evaluation Pipeline
Here's a production-grade evaluation harness that combines retrieval and answer metrics, stores results, and generates a report. This pattern works for both ad-hoc evaluation and automated CI/CD regression testing.
import json
import time
from dataclasses import dataclass, field
from typing import List, Dict, Optional
from datetime import datetime
import numpy as np
@dataclass
class RAGEvaluationResult:
"""Structured container for all RAG evaluation metrics."""
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
num_queries: int = 0
# Retrieval metrics
precision_at_3: float = 0.0
precision_at_5: float = 0.0
recall_at_3: float = 0.0
recall_at_5: float = 0.0
recall_at_10: float = 0.0
mrr: float = 0.0
ndcg_at_5: float = 0.0
# Answer metrics
faithfulness: float = 0.0
answer_relevancy: float = 0.0
semantic_similarity: float = 0.0
# Per-query details for debugging
per_query_details: List[Dict] = field(default_factory=list)
class RAGEvaluationPipeline:
"""
Complete RAG evaluation pipeline combining retrieval and answer metrics.
"""
def __init__(self, retriever, generator, llm_evaluator, embedding_model=None):
"""
Args:
retriever: Function(query) → List[str] (ranked document IDs)
generator: Function(query, contexts) → str (answer)
llm_evaluator: Function(prompt) → str (for LLM-based metrics)
embedding_model: Optional SentenceTransformer for similarity metrics
"""
self.retriever = retriever
self.generator = generator
self.llm = llm_evaluator
self.embed_model = embedding_model or SentenceTransformer('all-MiniLM-L6-v2')
def evaluate(self, test_dataset: List[Dict]) -> RAGEvaluationResult:
"""
Run full evaluation on a test dataset.
Each item in test_dataset should have:
- "query": str
- "relevant_docs": Set[str] or List[str] (ground-truth relevant doc IDs)
- "reference_answer": str (optional, for similarity metrics)
- "graded_relevance": Dict[str, int] (optional)
"""
result = RAGEvaluationResult()
result.num_queries = len(test_dataset)
retrieved_all = []
relevant_all = []
graded_all = []
faithfulness_scores = []
relevancy_scores = []
similarity_scores = []
for item in test_dataset:
query = item["query"]
relevant = set(item.get("relevant_docs", []))
# Step 1: Retrieve
retrieved_docs = self.retriever(query)
retrieved_all.append(retrieved_docs)
relevant_all.append(relevant)
if "graded_relevance" in item:
graded_all.append(item["graded_relevance"])
# Step 2: Generate answer
contexts = [self._fetch_document(doc_id) for doc_id in retrieved_docs[:5]]
contexts = [c for c in contexts if c is not None]
answer = self.generator(query, contexts)
# Step 3: Evaluate faithfulness
faith_score = faithfulness_score(answer, contexts, self.llm)
faithfulness_scores.append(faith_score)
# Step 4: Evaluate answer relevancy
rel_score = answer_relevance_score(query, answer, self.llm, similarity_model=self.embed_model)
relevancy_scores.append(rel_score)
# Step 5: Semantic similarity (if reference exists)
if "reference_answer" in item:
sim_score = semantic_answer_similarity(answer, item