Pular para o conteúdo
TakoAPI
Voltar às skills

rag-eval

por jonathanjing

Coding Agents & IDEs
381Downloads informados pelo clawskills.sh
2Estrelas
1Visualizações
0 curtidaEntre para curtir esta skill
—Ainda sem avaliações
Sua avaliação

Resumo

Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).

Descrição

O que esta skill faz

Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.

Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.

Quando usá-la

  • Checking if a chatbot's answers are supported by retrieved documents
  • Detecting hallucinations in RAG-generated responses before shipping
  • Running batch quality checks on a Q&A dataset ahead of deployment
  • Identifying irrelevant context chunks reducing retrieval precision
  • Tracking RAG output quality over time with daily scored logs

Exemplo de fluxo de trabalho

ENTRADA

User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']

AGENTE
  1. 1

    Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json

  2. 2

    Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json

  3. 3

    Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score

  4. 4

    Delete the temp file

  5. 5

    Present a human-readable summary with PASS/REVIEW/FAIL verdict

SAÍDA

Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS

Requisitos

  • OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
  • ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)

Instalação

clawhub install jonathanjing/rag-eval

Execute o comando da CLI ou cole o prompt no Claude Code, Cursor, Codex ou no agente de código que você usa.

Links