Evalsmith MCP

evalsmith-mcp

Evaluate RAG retrieval and AI agents with deterministic metrics, from any MCP client or from CI.

evalsmith answers four questions that come up every time a retriever, prompt or agent changes:

1. Is retrieval good enough? recall@k, precision@k, nDCG@k, hit rate and MRR from qrels and a ranked run.
2. Which configuration is better, and is the gap real? 2-3 configurations ranked with seeded paired-bootstrap confidence intervals.
3. Did the agent call the right tools? Trace vs. expected calls, strict or lax order, argument matchers.
4. Did the prompt change break anything? Golden-set checks on baseline vs. candidate outputs.

Everything is deterministic and runs offline. An LLM judge exists for a second opinion, but it is off by default and never decides the verdict. Results come out as a terminal table, Markdown, JSON, JUnit XML or a self-contained HTML report, and the GitHub Action turns them into a merge gate.

Interfaces over the same core

flowchart LR

subgraph Clients A[MCP client desktop app, IDE, agent] B[Remote MCP client or service] C[Terminal / CI job] D[GitHub workflow] end subgraph evalsmith S1[MCP server stdio] S2[MCP server Streamable HTTP, stateless] S3[CLI evalsmith run] S4[GitHub Action uses: evalsmith-mcp@v0] CORE[(src/core metrics, bootstrap, matchers, checks, gates, reports)] end A --> S1 --> CORE B --> S2 --> CORE C --> S3 --> CORE D --> S4 --> CORE S3 -. exit code + JUnit + HTML .-> C S4 -. outputs + job summary .-> D

All four call the same functions in src/core. The MCP tools are thin adapters: parse input with Zod, call the core, return structuredContent validated against an output schema.

MCP surface

flowchart TB

subgraph Tools T1[evaluate_retrieval] T2[compare_configs] T3[check_tool_calls] T4[detect_prompt_regression] T5[generate_report] T6[run_suite] end subgraph Resources R1["evalsmith://examples/{name} 12 synthetic datasets + suite"] end subgraph Prompts P1[compare_rag_configs] P2[audit_agent_trace] P3[review_prompt_change] end T1 -- structuredContent --> T5 T2 -- structuredContent --> T5 T3 -- structuredContent --> T5 T4 -- structuredContent --> T5 classDef tool fill:#ffedd5,stroke:#c2410c,color:#431407 classDef res fill:#e0f2fe,stroke:#0369a1,color:#082f49 classDef prom fill:#ede9fe,stroke:#6d28d9,color:#2e1065 class T1,T2,T3,T4,T5,T6 tool class R1 res class P1,P2,P3 prom

Tools

Every tool declares an input and an output Zod schema, returns structuredContent plus a JSON text block, and is annotated readOnly and non-destructive. Four tools are also idempotent and closed-world. detect_prompt_regression and run_suite are declared open-world and non-idempotent: annotations are per tool, and with the judge enabled they call an external model whose answers can vary. With the judge off (the default) they behave like the others.

run_suite runs the same suites as evalsmith run, so a client can evaluate data that lives in files instead of pasting it into the conversation. It only reads inside the workspace root (EVALSMITH_ROOT, the directory the server started in by default), following symlinks before it checks, and runs the bundled example suite when no config is given.

Bad input (unknown metric, duplicate config names, qrels with no relevant documents) comes back as an InvalidParams tool error with a message the model can act on. A judge endpoint that is down or returns an HTTP error is an InternalError.

Resources

evalsmith://examples/{name} lists and serves the synthetic datasets in examples/datasets, so a client can try every tool before wiring in its own data:

Prompts

A conversation with an MCP client

sequenceDiagram

autonumber actor User participant Host as MCP client (LLM host) participant ES as evalsmith (stdio) participant Core as src/core

User->>Host: "Is the reranker worth it on our help-center set?" Host->>ES: initialize, tools/list, resources/list ES-->>Host: 6 tools + JSON Schemas, 13 resources Host->>ES: resources/read evalsmith://examples/rag-qrels (+ 3 runs) ES-->>Host: qrels and runs as JSON Host->>ES: tools/call compare_configs {metric: "ndcg@5", configs: [...]} ES->>Core: compareConfigs(qrels, configs, "ndcg@5", {seed: 42}) Core-->>ES: ranking, intervals, paired delta ES-->>Host: structuredContent {winner: "hybrid+rerank", significant: true, ...} Host->>ES: tools/call generate_report {format: "markdown", comparison} ES-->>Host: Markdown report Host-->>User: "Yes: +0.153 nDCG@5 over hybrid, 95% CI [0.069, 0.251]."

Metrics

For a query $q$ with relevant set $R_q$ (documents with grade $> 0$) an