rag-eval
von jonathanjing
Coding Agents & IDEsKurzbeschreibung
Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).
Beschreibung
Was dieser Skill macht
Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.
Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.
Wann du ihn einsetzt
- Checking if a chatbot's answers are supported by retrieved documents
- Detecting hallucinations in RAG-generated responses before shipping
- Running batch quality checks on a Q&A dataset ahead of deployment
- Identifying irrelevant context chunks reducing retrieval precision
- Tracking RAG output quality over time with daily scored logs
Beispiel-Workflow
User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']
- 1
Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json
- 2
Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json
- 3
Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score
- 4
Delete the temp file
- 5
Present a human-readable summary with PASS/REVIEW/FAIL verdict
Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS
Voraussetzungen
- OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
- ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)
Installation
clawhub install jonathanjing/rag-evalFühre den CLI-Befehl aus oder füge den Prompt in Claude Code, Cursor, Codex oder deinen Coding-Agenten ein.