rag-eval
por jonathanjing
Coding Agents & IDEsResumo
Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).
Descrição
O que esta skill faz
Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.
Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.
Quando usá-la
- Checking if a chatbot's answers are supported by retrieved documents
- Detecting hallucinations in RAG-generated responses before shipping
- Running batch quality checks on a Q&A dataset ahead of deployment
- Identifying irrelevant context chunks reducing retrieval precision
- Tracking RAG output quality over time with daily scored logs
Exemplo de fluxo de trabalho
User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']
- 1
Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json
- 2
Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json
- 3
Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score
- 4
Delete the temp file
- 5
Present a human-readable summary with PASS/REVIEW/FAIL verdict
Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS
Requisitos
- OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
- ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)
Instalação
clawhub install jonathanjing/rag-evalExecute o comando da CLI ou cole o prompt no Claude Code, Cursor, Codex ou no agente de código que você usa.