evalpair

Name: evalpair
Author: Srikarmk

by Srikarmkversion 0.1.1i★ 0iRuns on your computer

Registry nameio.github.Srikarmk/evalpair

How to install

What it does

Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor.

Connects Runs on your computer
Account or key Needs EVALPAIR_API_KEY
Category Search & AI memory

Install

Pick the AI app you use, then copy the settings.

Passed 3/3 checks · Runs on your computer · ✓ Commands checked Nov 14, 2026

First: which AI do you use?

ClaudeWeb · app · phoneChatGPTWebClaude CodeTerminalCursorCode editorVS CodeCode editorOtherOther apps

Pick your app and the steps appear below.

Works with Claude, with some setup first

You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time) · Only the Claude desktop app (Windows or Mac), not the web or phone

Let AI install it

Copy this message into a Claude chat. The AI checks your computer, tells you which file it will change, and waits for your OK every time.

Please install the MCP server evalpair for Claude on this computer, in this order: 1) Check uv with uv --version. If it is missing, tell me to install it from https://docs.astral.sh/uv/getting-started/installation/; do not install it yourself. 2) Use only this config (already checked by PopNx MCP): { "mcpServers": { "evalpair": { "command": "uvx", "args": [ "evalpair", "mcp" ], "env": { "EVALPAIR_API_KEY": "your-value" } } } } 3) Tell me the full path of the file you will change, or the command you will run, and wait for my OK. 4) For EVALPAIR_API_KEY, leave "your-value" in place and tell me to paste the real value myself into the file or the app’s field. Never ask me to type a key in chat. 5) Do what we agreed, then check that the evalpair tools really show up.

This message has no key in it. If a key is needed, the AI will ask you to paste it yourself into the file or the app’s field.

1
Install uv (if you don’t have it)
Already installed? Check first · Open Terminal, type this command and press Enter
uv --version
- A version number such as uv … = installed, skip this step
- "command not found" (on Windows: "is not recognized") = not installed yet, or installed but Terminal needs to be closed and reopened
How to open Terminal
Mac: press ⌘ + Space, type Terminal, press Enter
Windows: open Start, type Windows PowerShell, then Open
Not installed: uv has no click-through installer. Copy the one-line command for your system (macOS or Windows) from uv’s official page into Terminal · no separate Python install needed; uv downloads it when required
Open the uv download page ↗
When it’s done, close Terminal, open it again and run the same command. You should see a version number: uv --version
Install once; it works for everything that needs uv
Done
Sign up and create a evalpair key
Open the maker’s site, sign up and create an API key for the next step. Check the maker’s site for pricing and free plans.
How to get a key is in the maker’s README below
Done
Open the Claude desktop app’s settings file
SettingsDeveloperEdit Config
Done

Paste these settings and save the file

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair",
        "mcp"
      ],
      "env": {
        "EVALPAIR_API_KEY": "your-value"
      }
    }
  }
}

Paste the key only into the app’s field, never into a chat or to anyone

Done

Quit Claude and open it again
Done
Try it
Use evalpair and tell me what you can do for me, then show me one thing
✓ If Claude says it is using evalpair or shows a evalpair tool in the answer, you are connected

1 optional setting(s) (works without them)

The command above works without these. Add them only if you want to change something, following the maker’s description.

EVALPAIR_HOME Folder for the call cache, saved runs and your own suites (default ~/.evalpair).

Descriptions are as written by the maker.

Menu names checked against Anthropic’s help page on Nov 11, 2026 · your screen may differ slightly by app version

ChatGPT cannot connect evalpair yet

evalpair runs on your computer, but ChatGPT can only connect servers that run on the web

What you can do instead

A · Use an app that can connect evalpair

Steps for Claude →You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time) · Only the Claude desktop app (Windows or Mac), not the web or phone
Steps for Claude Code →You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time)
Steps for Cursor →You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time)
Steps for VS Code →You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time)

B · Others in Search & AI memory that work with ChatGPT

Picked from the most used in the same category, only ones that connect by address with no key

Context7✓ ChatGPT
OpenSEO✓ ChatGPT
AIHOT✓ ChatGPT

Works with Claude Code, with some setup first

You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time)

Let AI install it

Copy this message into a Claude Code chat. The AI checks your computer, tells you which file it will change, and waits for your OK every time.

Please install the MCP server evalpair for Claude Code on this computer, in this order: 1) Check uv with uv --version. If it is missing, tell me to install it from https://docs.astral.sh/uv/getting-started/installation/; do not install it yourself. 2) Use only this config (already checked by PopNx MCP): claude mcp add --scope user evalpair --env EVALPAIR_API_KEY=your-value -- uvx evalpair mcp 3) Tell me the full path of the file you will change, or the command you will run, and wait for my OK. 4) For EVALPAIR_API_KEY, leave "your-value" in place and tell me to paste the real value myself into the file or the app’s field. Never ask me to type a key in chat. 5) Do what we agreed, then check that the evalpair tools really show up.

This message has no key in it. If a key is needed, the AI will ask you to paste it yourself into the file or the app’s field.

1
Install uv (if you don’t have it)
Already installed? Check first · Open Terminal, type this command and press Enter
uv --version
- A version number such as uv … = installed, skip this step
- "command not found" (on Windows: "is not recognized") = not installed yet, or installed but Terminal needs to be closed and reopened
How to open Terminal
Mac: press ⌘ + Space, type Terminal, press Enter
Windows: open Start, type Windows PowerShell, then Open
Not installed: uv has no click-through installer. Copy the one-line command for your system (macOS or Windows) from uv’s official page into Terminal · no separate Python install needed; uv downloads it when required
Open the uv download page ↗
When it’s done, close Terminal, open it again and run the same command. You should see a version number: uv --version
Install once; it works for everything that needs uv
Done
Sign up and create a evalpair key
Open the maker’s site, sign up and create an API key for the next step. Check the maker’s site for pricing and free plans.
How to get a key is in the maker’s README below
Done
Open Terminal
Mac: press Command+Space and type Terminal · Windows: open PowerShell
Done
4
Paste this command and press Enter
```
claude mcp add --scope user evalpair --env EVALPAIR_API_KEY=your-value -- uvx evalpair mcp
```
Replace “your-value” with your key before pressing Enter
Paste the key only into the app’s field, never into a chat or to anyone
Done
Type /mcp in Claude Code
You should see evalpair listed as connected
Done
Try it
Use evalpair and tell me what you can do for me, then show me one thing
✓ If Claude Code says it is using evalpair or shows a evalpair tool in the answer, you are connected

1 optional setting(s) (works without them)

The command above works without these. Add them only if you want to change something, following the maker’s description.

EVALPAIR_HOME Folder for the call cache, saved runs and your own suites (default ~/.evalpair).

Descriptions are as written by the maker.

Menu names checked against Anthropic’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Works with Cursor, with some setup first

You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time)

Let AI install it

Copy this message into a Cursor chat. The AI checks your computer, tells you which file it will change, and waits for your OK every time.

Please install the MCP server evalpair for Cursor on this computer, in this order: 1) Check uv with uv --version. If it is missing, tell me to install it from https://docs.astral.sh/uv/getting-started/installation/; do not install it yourself. 2) Use only this config (already checked by PopNx MCP): { "mcpServers": { "evalpair": { "command": "uvx", "args": [ "evalpair", "mcp" ], "env": { "EVALPAIR_API_KEY": "your-value" } } } } 3) Tell me the full path of the file you will change, or the command you will run, and wait for my OK. 4) For EVALPAIR_API_KEY, leave "your-value" in place and tell me to paste the real value myself into the file or the app’s field. Never ask me to type a key in chat. 5) Do what we agreed, then check that the evalpair tools really show up.

This message has no key in it. If a key is needed, the AI will ask you to paste it yourself into the file or the app’s field.

1
Install uv (if you don’t have it)
Already installed? Check first · Open Terminal, type this command and press Enter
uv --version
- A version number such as uv … = installed, skip this step
- "command not found" (on Windows: "is not recognized") = not installed yet, or installed but Terminal needs to be closed and reopened
How to open Terminal
Mac: press ⌘ + Space, type Terminal, press Enter
Windows: open Start, type Windows PowerShell, then Open
Not installed: uv has no click-through installer. Copy the one-line command for your system (macOS or Windows) from uv’s official page into Terminal · no separate Python install needed; uv downloads it when required
Open the uv download page ↗
When it’s done, close Terminal, open it again and run the same command. You should see a version number: uv --version
Install once; it works for everything that needs uv
Done
Sign up and create a evalpair key
Open the maker’s site, sign up and create an API key for the next step. Check the maker’s site for pricing and free plans.
How to get a key is in the maker’s README below
Done
Click this button to open Cursor
Open in Cursor
Cursor asks you to confirm; click Install
Done
Fill in your key
In the evalpair settings, EVALPAIR_API_KEY is empty; add your key and save
Paste the key only into the app’s field, never into a chat or to anyone
Done
Open the AI chat in Cursor in Agent mode
Done
Try it
Use evalpair and tell me what you can do for me, then show me one thing
✓ If Cursor says it is using evalpair or shows a evalpair tool in the answer, you are connected

1 optional setting(s) (works without them)

The command above works without these. Add them only if you want to change something, following the maker’s description.

EVALPAIR_HOME Folder for the call cache, saved runs and your own suites (default ~/.evalpair).

Descriptions are as written by the maker.

Menu names checked against Cursor’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Works with VS Code, with some setup first

You need a evalpair key (create it on the maker’s site) · Install Python (uv) on your computer first (one time)

Let AI install it

Copy this message into a VS Code chat. The AI checks your computer, tells you which file it will change, and waits for your OK every time.

Please install the MCP server evalpair for VS Code on this computer, in this order: 1) Check uv with uv --version. If it is missing, tell me to install it from https://docs.astral.sh/uv/getting-started/installation/; do not install it yourself. 2) Use only this config (already checked by PopNx MCP): { "servers": { "evalpair": { "type": "stdio", "command": "uvx", "args": [ "evalpair", "mcp" ], "env": { "EVALPAIR_API_KEY": "your-value" } } } } 3) Tell me the full path of the file you will change, or the command you will run, and wait for my OK. 4) For EVALPAIR_API_KEY, leave "your-value" in place and tell me to paste the real value myself into the file or the app’s field. Never ask me to type a key in chat. 5) Do what we agreed, then check that the evalpair tools really show up.

This message has no key in it. If a key is needed, the AI will ask you to paste it yourself into the file or the app’s field.

1
Install uv (if you don’t have it)
Already installed? Check first · Open Terminal, type this command and press Enter
uv --version
- A version number such as uv … = installed, skip this step
- "command not found" (on Windows: "is not recognized") = not installed yet, or installed but Terminal needs to be closed and reopened
How to open Terminal
Mac: press ⌘ + Space, type Terminal, press Enter
Windows: open Start, type Windows PowerShell, then Open
Not installed: uv has no click-through installer. Copy the one-line command for your system (macOS or Windows) from uv’s official page into Terminal · no separate Python install needed; uv downloads it when required
Open the uv download page ↗
When it’s done, close Terminal, open it again and run the same command. You should see a version number: uv --version
Install once; it works for everything that needs uv
Done
Sign up and create a evalpair key
Open the maker’s site, sign up and create an API key for the next step. Check the maker’s site for pricing and free plans.
How to get a key is in the maker’s README below
Done
Click this button to open VS Code
Open in VS Code
VS Code asks you to confirm; click Install
Done
Fill in your key
In the evalpair settings, EVALPAIR_API_KEY is empty; add your key and save
Paste the key only into the app’s field, never into a chat or to anyone
Done
Open the AI chat in VS Code in Agent mode
Done
Try it
Use evalpair and tell me what you can do for me, then show me one thing
✓ If VS Code says it is using evalpair or shows a evalpair tool in the answer, you are connected

1 optional setting(s) (works without them)

The command above works without these. Add them only if you want to change something, following the maker’s description.

EVALPAIR_HOME Folder for the call cache, saved runs and your own suites (default ~/.evalpair).

Descriptions are as written by the maker.

Menu names checked against VS Code’s help page on Nov 11, 2026 · your screen may differ slightly by app version

Other apps that use MCP, such as Windsurf, Zed or Gemini CLI, each add servers differently. See how in the app list.

See every app that uses MCP →

JSON settings are under “For developers” below

Stuck? Ask people who use evalpair💬 Discussion →

For developers: Terminal commands and JSON settings

Installed on your computer

Claude Code

claude mcp add --scope user evalpair --env EVALPAIR_API_KEY=your-value -- uvx evalpair mcp

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair",
        "mcp"
      ],
      "env": {
        "EVALPAIR_API_KEY": "your-value"
      }
    }
  }
}

Cursor (~/.cursor/mcp.json)

{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair",
        "mcp"
      ],
      "env": {
        "EVALPAIR_API_KEY": "your-value"
      }
    }
  }
}

VS Code (mcp.json)

{
  "servers": {
    "evalpair": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalpair",
        "mcp"
      ],
      "env": {
        "EVALPAIR_API_KEY": "your-value"
      }
    }
  }
}

Values you fill in

EVALPAIR_HOME env
Folder for the call cache, saved runs and your own suites (default ~/.evalpair).
EVALPAIR_API_KEY secret required env
Bearer token for an OpenAI-compatible server, if it needs one (OPENAI_API_KEY also works).

Descriptions are as written by the maker.

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

At a glance

GitHub stars i: 0
Downloads per weeki: 252/wk
Licence i: MIT
Connection: Runs on your computer
Transport: stdio
Latest versioni: 0.1.1
Categoryi: Search & AI memory
Published to registryi: Nov 5, 2026
Code last changedi: Nov 9, 2026
Maker: Srikarmk
Packages: pypi: evalpair
Repo: Srikarmk/evalpair
Registry status: Active

Which apps can use it

Based on how this server connects and on each app’s own docs, checked Nov 10, 2026.

Run on your computer in

Claude Goose LM Studio Msty Studio Cherry Studio Cursor VS Code + GitHub Copilot Visual Studio Windsurf (Devin Desktop)Zed Kiro Cline Continue JetBrains AI Assistant Claude Code Codex CLI Gemini CLI OpenCode nodecode Warp LibreChat Raycast

All apps, with pros and cons →

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 16, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account Srikarmk, which owns the repo
✓ Pass
Still maintained
Code last changed 7 days ago
✓ Pass
Has a licence
Uses MIT
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Published vulnerabilities (OSV.dev)

No entries on OSV.dev for evalpair@0.1.1

Checks only this package version, not every dependency · checked Nov 15, 2026

What we check → · JSON

Makers: ask for a re-check or report wrong info

Discussion

Ask, share a tip or report that it does not work. Sorted by what people marked helpful.

No posts yet. Be the first to ask or share.

Posts here are written by users, not the PopNx team.

User reviews

No reviews yet

Compared with others in Search & AI memoryi

More stars than 0% of the other 6192 servers in this category
Published to the registry more recently than 79% of the other 6192 servers in this category
Basic checks equal to or better than 100% of the other 6192 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is evalpair safe?

It passed 3 of 3 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

You need to fill in: EVALPAIR_API_KEY

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed 7 days ago, and the latest version (0.1.1) reached the registry 11 days ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Python (uv) on your computer.

What can I use instead?

In the same category: OpenSEO, Exa, Mockserver, Microsoft Learn MCP, Docs

Similar alternatives

Same category, other makers, commands already checked. Official ones first, then by stars.

Reading a card:↓ Downloads per weeki★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

OpenSEO

Keyword research, site audits, rank tracking and AI visibility for your AI agent.

by every-appWidely used★ 27.8KUpdated 2 days ago

Runs in the cloud✓ everyapp.com

How to connect

Exa

Connect AI agents to Exa for web search, content fetching, and multi-step research.

by exa-labsWidely used★ 5.2KUpdated 2 months ago

Runs in the cloud✓ service owner · exa.aiTried by the PopNx team

How to connect

Mockserver

Mock, record/replay, verify and chaos-test any HTTP, REST, gRPC or LLM dependency over MCP.

by mock-serverGrowing★ 5KUpdated 20 days ago

Runs on your computer✓ service owner · mock-server.com

Install

Microsoft Learn MCP

Official Microsoft Learn MCP Server – real-time, trusted docs & code samples for AI and LLMs.

by MicrosoftDocsWidely used★ 2KUpdated 12 months ago

Runs in the cloud✓ service owner · microsoft.comTried by the PopNx team

How to connect

Docs

Search and read Empryo's documentation. Read-only, no auth, no local access.

by proxysoulGrowing★ 1.6KUpdated 2 months ago

Runs in the cloud✓ empryo.com

How to connect

OpenSEO — Runs in the cloud, nothing to install · No key needed
Exa — Runs in the cloud, nothing to install · No key needed · In Claude’s directory
Mockserver — Installs on your computer · No key needed
Microsoft Learn MCP — Runs in the cloud, nothing to install · No key needed · In Claude’s directory
Docs — Runs in the cloud, nothing to install · No key needed

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

evalpair

Run an LLM eval suite against a model, cache every call, and compare two runs item by item: did the new model, prompt or quantization actually change anything, or is the difference noise?

Most eval tools print two scores side by side. That reads a 2-point gap on 160 questions as a result, when it usually isn't one. evalpair pairs the two runs by item, puts a bootstrap interval on the difference, counts the items that got worse and better, and can measure a noise floor: the same model asked the same questions twice. In my own studies the same prompt sent twice at temperature 0 changed up to 3% of answers, so an answer-change rate only means something above that.

It's a CLI and an MCP server, so an agent can run an eval and read the verdict itself. It only calls the model server you point it at, and keeps everything in one local folder.

Quick start

uvx evalpair suites

uvx evalpair run gsm8k-160 -m llama3.1:8b --repeat --label llama uvx evalpair run gsm8k-160 -m qwen3:8b --think off --label qwen3 uvx evalpair compare llama qwen3

run prints the score with a 95% interval and the first failures, unparsed and truncated replies first (those are usually the parser's or the token budget's fault, not the model's). compare prints both scores, the paired difference with its interval, how many items got worse and better, the noise floor, and ends with a plain-language verdict: whether the score difference is distinguishable from zero, and whether the answers moved more than the same model asked twice.

A real comparison: Ollama's default 4-bit Llama 3.1 8B against its 2-bit build, on an M5 Pro laptop (--num-ctx 4096; about 40 minutes for both runs):

$ evalpair compare llama-q4 llama-q2

A llama-q4: 84.4% [79.4%, 90.0%] B llama-q2: 60.6% [53.1%, 68.1%] B − A -23.8 pp [-31.9 pp, -16.2 pp] changed 40.6% [33.1%, 48.1%] of 160 items (43 worse in B, 5 better) floor (A) 0.0% [0.0%, 0.0%]; excess +40.6 pp [+33.1 pp, +48.1 pp]

B scores lower than A: -23.8 pp [-31.9 pp, -16.2 pp], paired over 160 items. Answers changed on 40.6% of items (43 got worse, 5 better). The noise floor (the same model asked twice) is 0.0% for A, so the change is beyond noise. worse in B: gsm8k-0259 '168' → '52' ...

Both scores are the ones the quantization study measured for these builds on the same items.

Interrupt a run and run it again: every reply is cached, so it picks up where it stopped.

As an MCP server

claude mcp add evalpair -- uvx evalpair mcp

or in any MCP client's config: {"command": "uvx", "args": ["evalpair", "mcp"]}.

Tool	What it does
`list_suites`	Suites you can run: name, item count, checker, what it measures
`run_eval`	Run a suite against one model and save the run (calls the model; cached replies are reused)
`list_runs`	Saved runs, newest first
`show_run`	A run's score with its interval, its noise floor, and the end of failed replies
`compare_runs`	Two runs paired by item: difference with an interval, flips each way, change vs. noise floor, examples

Everything except run_eval only reads saved runs.

Backends

Ollama (default): --backend ollama, server at http://localhost:11434. The run records the model's digest, so you can tell later exactly which build was tested. --think on|off for models that think, --num-ctx for the context window.
Any OpenAI-compatible server: --backend openai --base-url http://localhost:8000/v1 for vLLM, llama.cpp's server or LM Studio, or a hosted API. A bearer token is read from EVALPAIR_API_KEY (or OPENAI_API_KEY) and never written to disk.

Temperature defaults to 0 and the seed to 7. Both are part of the cache key, along with the server URL, model, messages, token budget and thinking setting.

Suites

Three are bundled. They're the item sets from my Measuring LLM Systems studies, with those studies' prompts word for word, so a run can be checked against the published numbers:

Suite	Items	Checker	Source
`gsm8k-160`	160	`number`	GSM8K test (MIT)
`mmlu-hs-160`	160	`choice`, stratified by subject	MMLU, four high-school subjects (MIT)
`e2e-extract-150`	150	`json_fields`	E2E NLG restaurant records (CC BY-SA 4.0)

To make your own, create a folder with two files, in ~/.evalpair/suites/ / or anywhere you then pass by path:

# suite.toml

description = "Capitals, answered in one word" checker = "exact" max_tokens = 64 system = "Answer in one word." # optional user = "What is the capital of {country}?" group = "region" # optional: stratify the bootstrap by this item field

{"id": "fr", "country": "France", "region": "Europe", "gold": "Paris"}

{"id": "pe", "country": "Peru", "region": "Americas", "gold": ["Lima", "Ciudad de los Reyes"]}

{field} in a prompt is filled from the item; a list of strings becomes lettered options (A. …), anything else is shown as indented JSON. Use {{ and }} for literal braces.

Checker	Scores	Gold
`number`	the number after "Answer:", else `\boxed{}`, else the last number	a number
`choice`	the letter after "Answer:", with fallbacks (letters set by `checker_options.letters`)	a letter
`exact`	the text after "Answer:", or the last line, normalised	a string or list of strings
`contains`	whether a gold string appears as whole words	a string or list
`regex`	the last match of `checker_options.pattern` (first group)	a string or list
`json_fields`	the share of the gold object's fields the reply's JSON gets right (null must be null)	an object
`chrf`	character n-gram F-score (sacreBLEU's chrF) against a reference text, for free text such as commit messages or summaries; `checker_options.line = "first"` compares first lines only	a string or list

Checkers are deterministic code: no LLM grades another LLM here. The number, choice and json_fields parsers come from the studies, and re-scoring all 9,870 replies from the quantization study's main run with them gives the study's own scores on every one (scripts/check_against_studies.py).

How the numbers are computed

Unit of resampling: the item. 2,000 bootstrap draws, percentile intervals, seeded, so the same runs always give the same interval. With a group, items are resampled within each group.
Comparisons are paired over the items both runs answered, using the first pass of each. If the two runs used different versions of a suite (the hash of its spec and items differs), the comparison says so.
Noise floor: with --repeat, every item is sent a second time as a separate call (its own cache entry). The floor is the share of items whose parsed answer changed between the passes. A comparison calls a change "beyond noise" only if, per item, (changed between runs) minus (changed between the passes) has an interval above zero for every run that has a floor.
Items are taken in file order, so --limit 50 is the same 50 items every time and limited runs still pair up.

Where things live

$EVALPAIR_HOME, or ~/.evalpair:

cache.sqlite           every reply, keyed by request

runs/ /meta.json suite and its hash, model, digest, backend, options, counts, timings runs/ /rows.jsonl one line per item and pass: reply, parsed answer, score, tokens suites/ / your suites

Limits

One model per run, chat-style prompts, one turn. No tool use, multi-turn or constrained decoding yet.
Deterministic checkers only, so open-ended answers need a suite whose answer can be checked by code.
The noise floor measures run-to-run variation on the same server. It doesn't cover a different machine or a server update.

License

MIT for the code. The bundled suites keep their sources' licenses (MIT for GSM8K and MMLU, CC BY-SA 4.0 for E2E NLG); each suite.toml names its license and source.

evalpair

What it does

Install

Install uv (if you don’t have it)

Sign up and create a evalpair key

Open the Claude desktop app’s settings file

Paste these settings and save the file

Quit Claude and open it again

Try it

What you can do instead

Install uv (if you don’t have it)

Sign up and create a evalpair key

Open Terminal

Paste this command and press Enter

Type /mcp in Claude Code

Try it

Install uv (if you don’t have it)

Sign up and create a evalpair key

Click this button to open Cursor

Fill in your key

Open the AI chat in Cursor in Agent mode

Try it

Install uv (if you don’t have it)

Sign up and create a evalpair key

Click this button to open VS Code

Fill in your key

Open the AI chat in VS Code in Agent mode

Try it

Installed on your computer

Values you fill in

At a glance

Which apps can use it

Related guides

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Published vulnerabilities (OSV.dev)

Discussion

User reviews

Compared with others in Search & AI memoryi

Common questions

Similar alternatives

OpenSEO

Exa

Mockserver

Microsoft Learn MCP

Docs

README from the maker

evalpair

Quick start

As an MCP server

Backends

Suites

How the numbers are computed

Where things live

Limits

License