跳到正文
TakoAPI
返回技能列表

rag-eval

由 jonathanjing 提供

Coding Agents & IDEs
381下载量由 clawskills.sh 提供
2星标
1浏览量
0 个赞登录后即可为这个技能点赞
—暂无评分
你的评分

简介

Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).

描述

该技能的功能

Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.

Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.

何时使用

  • Checking if a chatbot's answers are supported by retrieved documents
  • Detecting hallucinations in RAG-generated responses before shipping
  • Running batch quality checks on a Q&A dataset ahead of deployment
  • Identifying irrelevant context chunks reducing retrieval precision
  • Tracking RAG output quality over time with daily scored logs

示例工作流

输入

User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']

智能体
  1. 1

    Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json

  2. 2

    Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json

  3. 3

    Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score

  4. 4

    Delete the temp file

  5. 5

    Present a human-readable summary with PASS/REVIEW/FAIL verdict

输出

Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS

前置要求

  • OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
  • ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)

安装

clawhub install jonathanjing/rag-eval

运行 CLI 命令,或把这段提示词粘贴到 Claude Code、Cursor、Codex 或你在用的任意编程智能体里。

链接