Evals MCP Server

@cyanheads/evals-mcp-server Author verifiable eval records through a draft → review → revise → submit loop with server-enforced graders; compile to JSONL/CSV/Inspect/lm-eval via MCP. STDIO or Streamable HTTP. 9 Tools • 1 Resource

Overview

Build verifiable eval records with server-enforced graders. Draft a record with its own executable grader, revise it field by field, and submit through a gate that requires the gold answer to pass, a negative case to fail, and an independent check to agree. Export to JSONL, CSV, Inspect AI, or lm-evaluation-harness.

Tools

Resources

All record data is also reachable through the tool surface — evals_get_record for a single record, evals_list_records to browse. The resource is a convenience mirror for clients that support resources, not the access path.

Capability reference

evals_describe_schema tool

• Accepts task_type: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, or free_response.
• Returns the gold shape, grader kinds, required/optional fields, and authoring notes from static schemas; mcq requires choices, and free_response requires llm_rubric.

evals_create_draft tool

• Accepts a record with its grader and discrimination cases, plus optional verification and captures (EvalsIDs); mcq requires choices, and free_response requires llm_rubric.
• Saves the record as a draft once its grader runs, and returns the normalized record, a review protocol, a verification-subagent prompt, and what submit still needs. A draft whose gold fails its grader is still saved.
• Typed errors: grader_unexecutable, task_type_constraint, mcq_choice_mismatch.

evals_get_record tool

• Reads a draft or submitted record by id, which stays stable across submit.
• Returns the full grader, discrimination cases, and verification evidence; not_found points to evals_list_records.

evals_revise_draft tool

• Applies set, append, and unset by dotted path to a draft; task_type and server-owned fields cannot be changed, and arrays are replaced whole or appended, not edited by index.
• Re-validates the record and grader before writing, then returns the record plus changed entries (op, path, before, after). An unknown path rejects the whole patch.
• Typed errors: not_found, record_frozen for a submitted id, invalid_patch_path, grader_unexecutable, task_type_constraint, mcq_choice_mismatch.

evals_discard_draft tool

• Deletes a draft by draft_id and returns discarded: true.
• A submitted record reports record_frozen; a missing or already discarded id reports not_found.

evals_run_check tool

• Runs a grader against one or more candidates without a saved record; supply gold for exact_match, while target-embedding kinds such as numeric and mcq ignore it.
• Returns PASS/REJECT, detail, and the resolved reference per candidate. llm_rubric can't be run here.
• Numeric grading accepts only finite real values; unit-bearing, complex, nonfinite, and non-scalar candidates are rejected. units is documentation only.
• Typed errors: grader_unexecutable, mcq_choice_mismatch.

evals_submit_draft tool

• Finalizes by draft_id: gold must PASS, ≥1 negative must be REJECTED, and decorrelated verification must agree. Also checks captures against EVALS_CAPTURE_DIR, rejects duplicate content_hash, and can ask for confirmation via confirm or EVALS_REQUIRE_CONFIRMATION.
• On success, freezes the record with submitted_at and checksum and returns status: submitted; a refusal leaves the draft intact. free_response records rely on verification alone and carry server_verified: false.
• Typed errors: not_found, record_frozen, verification_incomplete, grader_failed_on_gold, verification_disagrees_with_gold, missing_negative_case, negative_case_passed, duplicate, decorrelation_violation, capture_unresolved, submit_declined.

evals_list_records tool

• Filters by status (draft/submitted), domain, task_type, or tag; limit is a positive integer and defaults to 50.
• Returns summaries (id, status, task_type, domain, tags, timestamps), newest-first, with shown, cap, totalCount, and truncated.

evals_export_records tool

• Exports submitted records, optionally filtered by domain, task_type, or tag, as jsonl (lossless), csv (lossy summary), inspect (Inspect AI), or lm-eval (lm-evaluation-harness).
• Writes under exports/ and returns the path, record count, byte size, and preview.

eval://record/{id} resource

• Reads id from evals_list_records or a draft/submit response.
• Returns the evals_get_record payload as application/json, or not_found when no record matches.

Features

Built on @cyanheads/mcp-ts-core: stdio and Streamable HTTP transports, pluggable auth (none / jwt / oauth), swappable storage (in-memory, filesystem, Supabase, Cloudflare KV/R2/D1), structured logging with optional OpenTelemetry tracing.

Eval authoring:

• A draft → review → surgical-revise → submit loop, with the server both storing records and running each record's own grader to reject what doesn't hold up
• Seven task types: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response
• A grader serialized with each record; deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side, while llm_rubric relies on recorded independent verification
• A submit gate: the gold must pass its own grader, ≥1 negative must be rejected, and a recorded decorrelated verification must agree with the gold
• Plain JSON files under EVALS_DATA_DIR, with drafts, submitted records, and exports kept separate

Agent-friendly output:

• evals_create_draft and evals_revise_draft return the parsed record, a per-field review protocol, and a ready-to-paste verification-subagent prompt
• Draft and revise responses report pass/reject results for each positive and negative case
• evals_list_records reports shown / cap / total count when the limit is hit
• The submit gate fails with a typed reason plus a recovery hint

Getting started

Add the following to your MCP client configuration file. Set EVALS_DATA_DIR to a writable folder — the server manages drafts/, submitted/, and exports/ under it.

{

"mcpServers": { "evals-mcp-server": { "type": "stdio", "command": "bunx", "args": ["@cyanheads/evals-mcp-server@latest"], "env": { "MCP_TRANSPORT_TYPE": "stdio", "MCP_LOG_LEVEL": "info", "EVALS_DATA_DIR": "/absolute/path/to/evals-data" } } } }

Or with npx (no Bun required):

{

"mcpServers": { "evals-mcp-server": { "type": "stdio", "command": "npx", "args": ["-y", "@cyanheads/evals-mcp-server@latest"], "env": { "MCP_TRANSPORT_TYPE": "stdio", "MCP_LOG_LEVEL": "info", "EVALS_DATA_DIR": "/absolute/path/to/evals-data" } } } }

Or with Docker:

{

"mcpServers": { "evals-mcp-server": { "type": "stdio", "command": "docker", "args": [ "run", "-i", "--rm", "-e", "MCP_TRANSPORT_TYPE=stdio", "-v", "evals-data:/usr/src/app/evals-data", "ghcr.io/cyanheads/evals-mcp-server:latest" ] } } }

For Streamable HTTP, set the transport and start the server:

MCP_TRANSPORT_TYPE=http MCP_HTTP_PORT=3010 EVALS_DATA_DIR=./evals-data bun run start:http

Server listens at http://localhost:3010/mcp

Prerequisites

• Bun v1.4.0 or higher (or Node.js v24+).
• A writable directory for EVALS_DATA_DIR. No external API key is required.

Installation

1. Clone the repository:

git clone https://github.com/cyanheads/evals-mcp-server.git

1. Navigate into the directory:

cd evals-mcp-server

1. Install dependencies:

bun install

1. Configure environment:

cp .env.example .env

edit .env and set EVALS_DATA_DIR

Configuration

All server configuration is validated at startup via Zod schemas in src/config/server-config.ts.

| EVALS_DATA_DIR | Root folder for record JSON; the store manages drafts/, submitted/, and exports/ under