본문으로 건너뛰기
TakoAPI
스킬로 돌아가기

rag-eval

jonathanjing 제작

Coding Agents & IDEs
381clawskills.sh가 보고한 다운로드 수
2스타
1조회수
좋아요 0개로그인하면 이 스킬에 좋아요를 누를 수 있습니다
—아직 평가 없음
내 평가

요약

Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).

설명

이 스킬이 하는 일

Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.

Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.

사용 시점

  • Checking if a chatbot's answers are supported by retrieved documents
  • Detecting hallucinations in RAG-generated responses before shipping
  • Running batch quality checks on a Q&A dataset ahead of deployment
  • Identifying irrelevant context chunks reducing retrieval precision
  • Tracking RAG output quality over time with daily scored logs

예시 워크플로

입력

User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']

에이전트
  1. 1

    Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json

  2. 2

    Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json

  3. 3

    Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score

  4. 4

    Delete the temp file

  5. 5

    Present a human-readable summary with PASS/REVIEW/FAIL verdict

출력

Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS

요구 사항

  • OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
  • ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)

설치

clawhub install jonathanjing/rag-eval

CLI 명령을 실행하거나, 프롬프트를 Claude Code, Cursor, Codex 등 사용하는 코딩 에이전트에 붙여넣으세요.

링크