Build a comprehensive RAG evaluation suite measuring retrieval quality, answer faithfulness, and end-to-end system performance.
You are an AI evaluation specialist. Build a comprehensive RAG evaluation framework. ## Configuration - **RAG System**: [SYSTEM e.g. LangChain / LlamaIndex / custom] - **Evaluation Dataset**: [DATASET e.g. custom QA pairs with ground truth] - **LLM Judge**: [JUDGE e.g. Claude 4 Opus / GPT-5] - **Metrics Focus**: [FOCUS e.g. retrieval quality / answer quality / both] ## Evaluation Components ### 1. Retrieval Metrics (`retrieval/`) - **Hit Rate@k**: Does the correct document appear in top-k results? - **MRR (Mean Reciprocal Rank)**: Average 1/rank of first relevant result - **NDCG@k**: Normalized Discounted Cumulative Gain - **Precision@k**: Fraction of relevant documents in top-k - **Recall@k**: Fraction of relevant documents retrieved - **MAP**: Mean Average Precision across queries - **Context Relevance**: LLM-judged relevance of each retrieved chunk - **Context Coverage**: Does retrieved context contain all needed information? ### 2. Generation Metrics (`generation/`) - **Faithfulness**: Is the answer supported by retrieved context? (no hallucination) - **Answer Relevancy**: Does the answer address the question? - **Answer Correctness**: Does the answer match ground truth? - **Answer Completeness**: Are all aspects of the question addressed? - **Citation Accuracy**: Do cited sources actually support the claims? - **Factual Consistency**: NLI-based entailment checking ### 3. End-to-End Metrics (`e2e/`) - **BLEU/ROUGE**: N-gram overlap with reference answers - **BERTScore**: Semantic similarity with reference answers - **Exact Match / F1**: Token-level accuracy - **LLM-as-Judge**: Pairwise comparison with reference answers - **Latency**: P50, P95, P99 response times - **Cost**: Token usage and API costs per query ### 4. Synthetic Dataset Generation (`synthetic/`) - Generate QA pairs from document corpus using LLM - Create multi-hop questions requiring multiple documents - Generate adversarial questions (unanswerable, out-of-scope) - Ensure question diversity across topics and difficulty levels - Human validation workflow for generated pairs ### 5. Reporting (`reporting/`) - Per-query breakdown with pass/fail criteria - Aggregate metrics dashboard - Failure analysis: categorize errors by type - Regression detection: compare across system versions - A/B test comparison framework - Export to JSON, CSV, and HTML report Provide the complete framework with example evaluation datasets and benchmarks.
Free to copy and use. Compatible with Claude 4 Opus, GPT-5, Gemini 2.5 Pro.
Prepare evaluation QA pairs with ground truth answers and relevant document references. Run retrieval metrics first to optimize retrieval, then generation metrics for answer quality.
Initial release
Sign in and download this prompt to leave a review.