تخطَّ إلى المحتوى
TakoAPI
العودة إلى المهارات

rag-eval

بواسطة jonathanjing

Coding Agents & IDEs
381التنزيلات وفق ما يذكره clawskills.sh
2النجوم
1المشاهدات
لا إعجاباتسجّل الدخول للإعجاب بهذه المهارة
—لم يُقيَّم بعد
تقييمك

موجز

Evaluate your RAG pipeline quality using Ragas metrics (faithfulness, answer relevancy, context precision).

الوصف

ما الذي تفعله هذه المهارة

Evaluates RAG pipeline output quality using Ragas metrics: faithfulness, answer relevancy, and context precision. Requires an existing RAG system with a vector DB and retrieval pipeline. An LLM serves as the judge, configurable via OpenAI, Anthropic, or a local model like Ollama.

Ragas applies LLM-graded metrics for faithfulness and relevancy that manual spot-checking or simple string matching cannot reliably replicate.

متى تستخدمها

  • Checking if a chatbot's answers are supported by retrieved documents
  • Detecting hallucinations in RAG-generated responses before shipping
  • Running batch quality checks on a Q&A dataset ahead of deployment
  • Identifying irrelevant context chunks reducing retrieval precision
  • Tracking RAG output quality over time with daily scored logs

مثال على سير العمل

الإدخال

User asks: Evaluate this RAG response — question: 'What is the return policy?', answer: 'Returns are accepted within 30 days.', contexts: ['Our policy allows returns within 30 days of purchase.']

الوكيل
  1. 1

    Write the question, answer, and contexts as JSON to /tmp/rag-eval-input.json

  2. 2

    Run python3 scripts/run_eval.py --input-file /tmp/rag-eval-input.json

  3. 3

    Parse output scores for faithfulness, answer_relevancy, context_precision, and overall_score

  4. 4

    Delete the temp file

  5. 5

    Present a human-readable summary with PASS/REVIEW/FAIL verdict

الإخراج

Faithfulness: 0.95 ✅, Answer Relevancy: 0.91 ✅, Context Precision: 0.88 ✅, Overall: 0.91 — PASS

المتطلبات

  • OPENAI_API_KEY environment variable (default judge, uses GPT-4o)
  • ANTHROPIC_API_KEY environment variable (alternative judge, uses Claude Haiku)

التثبيت

clawhub install jonathanjing/rag-eval

شغّل أمر واجهة سطر الأوامر، أو الصق النص في Claude Code أو Cursor أو Codex أو أي وكيل برمجة تستخدمه.

الروابط