RAGScore

Generate QA datasets & evaluate RAG systems in 2 commands

🔒 Privacy-First • ⚡ Lightning Fast • 🤖 Any LLM • 🏠 Local or Cloud • 🌍 Multilingual

English | 中文 | 日本語 | Deutsch

---

⚡ 2-Line RAG Evaluation

# Step 1: Generate QA pairs from your docs

ragscore generate docs/

Step 2: Evaluate your RAG system

ragscore evaluate http://localhost:8000/query

That's it. Get accuracy scores and incorrect QA pairs instantly.

============================================================

✅ EXCELLENT: 85/100 correct (85.0%) Average Score: 4.20/5.0 ============================================================

❌ 15 Incorrect Pairs:

1. Q: "What is RAG?" Score: 2/5 - Factually incorrect

1. Q: "How does retrieval work?" Score: 3/5 - Incomplete answer

---

🚀 Quick Start

Install

pip install ragscore              # Core (works with Ollama)

pip install "ragscore[openai]" # + OpenAI support pip install "ragscore[notebook]" # + Jupyter/Colab support pip install "ragscore[all]" # + All providers

Already installed? Keep up to date — new versions add features like failure diagnosis and retrieved context capture:

`bash

pip install --upgrade ragscore

`

Option 1: Python API (Notebook-Friendly)

Perfect for Jupyter, Colab, and rapid iteration. Get instant visualizations.

from ragscore import quick_test

1. Audit your RAG in one line

result = quick_test( endpoint="http://localhost:8000/query", # Your RAG API docs="docs/", # Your documents n=10, # Number of test questions )

1b. Tailored QA — target specific audiences

result = quick_test( endpoint="http://localhost:8000/query", docs="docs/", audience="developers", # Who asks the questions? purpose="api-integration", # What's the document for? )

2. See the report

result.plot()

3. Inspect failures

bad_rows = result.df[result.df['score'] < 3] display(bad_rows[['question', 'rag_answer', 'reason']])

Rich Object API:

• result.accuracy - Accuracy score
• result.df - Pandas DataFrame of all results
• result.plot() - 3-panel visualization (4-panel with detailed=True)
• result.corrections - List of items to fix

Option 2: CLI (Production)

Generate QA Pairs

# Set API key (or use local Ollama - no key needed!)

export OPENAI_API_KEY="sk-..."

Generate from any document

ragscore generate paper.pdf ragscore generate docs/*.pdf --concurrency 10

Tailored QA generation — target specific audiences

ragscore generate docs/ --audience developers --purpose faq ragscore generate docs/ --audience customers --purpose "pre-sales" ragscore generate docs/ --audience "compliance auditors" --purpose "security audit"

Evaluate Your RAG

# Point to your RAG endpoint

ragscore evaluate http://localhost:8000/query

Custom options

ragscore evaluate http://api/ask --model gpt-4o --output results.json

---

🔬 Detailed Multi-Metric Evaluation

Go beyond a single score. Add detailed=True to get 5 diagnostic dimensions per answer — in the same single LLM call.

result = quick_test(

endpoint=my_rag, docs="docs/", n=10, detailed=True, # ⭐ Enable multi-metric evaluation )

Inspect per-question metrics

display(result.df[[ "question", "score", "correctness", "completeness", "relevance", "conciseness", "faithfulness" ]])

Radar chart + 4-panel visualization

result.plot()

================================================== ✅ PASSED: 9/10 correct (90%) Average Score: 4.3/5.0 Threshold: 70% ────────────────────────────────────────────────── Correctness: 4.5/5.0 Completeness: 4.2/5.0 Relevance: 4.8/5.0 Conciseness: 4.1/5.0 Faithfulness: 4.6/5.0 ==================================================

CLI:

ragscore evaluate http://localhost:8000/query --detailed

🔍 Failure Diagnosis (--diagnose)

When answers fail, --diagnose tells you why — retriever miss, generator hallucination, incomplete answer, or wrong interpretation:

ragscore evaluate http://localhost:8000/query --diagnose

🔍 Failure Diagnosis:

Retriever Miss: 3 (42.9%) Generator Hallucination: 2 (28.6%) Incomplete Answer: 1 (14.3%) Wrong Interpretation: 1 (14.3%)

Uses the support_span already generated with each QA pair to give the judge grounding context. Combine with --detailed for full diagnostics:

ragscore evaluate http://localhost:8000/query --diagnose --detailed -o results.json

📓 Full demo notebook — build a mini RAG and test it with detailed metrics.

🎯 Audience & Purpose demo — generate tailored QA for developers, customers, auditors, and more.

🏠 Ollama local demo — 100% private RAG evaluation with no API keys.

---

🏠 100% Private with Local LLMs

# Use Ollama - no API keys, no cloud, 100% private

ollama pull llama3.1 ragscore generate confidential_docs/*.pdf ragscore evaluate http://localhost:8000/query

Perfect for: Healthcare 🏥 • Legal ⚖️ • Finance 🏦 • Research 🔬

Ollama Model Recommendations

RAGScore generates complex structured QA pairs (question + answer + rationale + support span) in JSON format. This requires models with strong instruction-following and JSON output capabilities.

Minimum recommended: 8B+ models. Smaller models (1.5B–3B) produce lower quality support spans and may timeout on longer chunks.

Ollama Performance Guide

# Recommended: 8B model with concurrency 2 for local machines

ollama pull llama3.1:8b ragscore generate docs/ --provider ollama --model llama3.1:8b

GPU server (A10/L20): larger model with higher concurrency

ollama pull qwen2.5:32b ragscore generate docs/ --provider ollama --model qwen2.5:32b --concurrency 5

Expected performance (28 chunks, 5 QA pairs per chunk):

RAGScore auto-reduces concurrency to 2 for local Ollama to avoid GPU/CPU contention.

---

🔌 Supported LLMs

---

📊 Output Formats

Generated QA Pairs (output/generated_qas.jsonl)

{

"id": "abc123", "question": "What is RAG?", "answer": "RAG (Retrieval-Augmented Generation) combines...", "rationale": "This is explicitly stated in the introduction...", "support_span": "RAG systems retrieve relevant documents...", "difficulty": "medium", "source_path": "docs/rag_intro.pdf" }

Evaluation Results (--output results.json)

{

"summary": { "total": 100, "correct": 85, "incorrect": 15, "accuracy": 0.85, "avg_score": 4.2 }, "incorrect_pairs": [ { "question": "What is RAG?", "golden_answer": "RAG combines retrieval with generation...", "rag_answer": "RAG is a database system.", "score": 2, "reason": "Factually incorrect - RAG is not a database" } ] }

---

🧪 Python API

from ragscore import run_pipeline, run_evaluation

Generate QA pairs

run_pipeline(paths=["docs/"], concurrency=10)

Generate tailored QA pairs for specific audiences

run_pipeline( paths=["docs/"], audience="support engineers", purpose="fine-tuning a support chatbot", )

Evaluate RAG

results = run_evaluation( endpoint="http://localhost:8000/query", model="gpt-4o", # LLM for judging ) print(f"Accuracy: {results.accuracy:.1%}")

---

🤖 AI Agent Integration

RAGScore is designed for AI agents and automation:

# Structured CLI with predictable output

ragscore generate docs/ --concurrency 5 ragscore evaluate http://api/query --output results.json

Exit codes: 0 = success, 1 = error

JSON output for programmatic parsing

CLI Reference:

---

⚙️ Configuration

Zero config required. Optional environment variables:

export RAGSCORE_CHUNK_SIZE=512          # Chunk size for documents

export RAGSCORE_QUESTIONS_PER_CHUNK=5 # QAs