rag-eval
由 jonathanjing 提供
Coding Agents & IDEs简介
Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).
描述
该技能的功能
Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.
Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.
何时使用
- Checking if a chatbot's answers are supported by retrieved documents
- Detecting hallucinations in RAG-generated responses before shipping
- Running batch quality checks on a Q&A dataset ahead of deployment
- Identifying irrelevant context chunks reducing retrieval precision
- Tracking RAG output quality over time with daily scored logs
示例工作流
User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']
- 1
Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json
- 2
Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json
- 3
Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score
- 4
Delete the temp file
- 5
Present a human-readable summary with PASS/REVIEW/FAIL verdict
Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS
前置要求
- OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
- ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)
安装
clawhub install jonathanjing/rag-eval运行 CLI 命令,或把这段提示词粘贴到 Claude Code、Cursor、Codex 或你在用的任意编程智能体里。