rag-eval
oleh jonathanjing
Coding Agents & IDEsRingkasan
Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).
Deskripsi
Apa yang Dilakukan Skill Ini
Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.
Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.
Kapan Menggunakannya
- Checking if a chatbot's answers are supported by retrieved documents
- Detecting hallucinations in RAG-generated responses before shipping
- Running batch quality checks on a Q&A dataset ahead of deployment
- Identifying irrelevant context chunks reducing retrieval precision
- Tracking RAG output quality over time with daily scored logs
Contoh Alur Kerja
User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']
- 1
Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json
- 2
Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json
- 3
Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score
- 4
Delete the temp file
- 5
Present a human-readable summary with PASS/REVIEW/FAIL verdict
Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS
Persyaratan
- OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
- ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)
Instalasi
clawhub install jonathanjing/rag-evalJalankan perintah CLI-nya, atau tempel prompt ke Claude Code, Cursor, Codex, atau coding agent apa pun yang Anda pakai.