Evalview MCP

Name: Evalview MCP
Author: hidai25

by hidai25version 0.6.0i★ –iRuns on your computer

Registry nameio.github.hidai25/evalview-mcp

Install

Overview

Description from the maker (original English)

Regression testing for AI agents. Golden baselines, CI/CD, LangGraph, CrewAI, OpenAI, Claude.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs on your computer
Transport: stdio
Category: Search & AI memory
You need to fill in: OPENAI_API_KEY

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 1 of 1 passed
Runs on your computer with your user’s permissions (files, network, apps).
You fill in yourself: OPENAI_API_KEY

Values marked secret are keys to your account. Never share them or paste them into a chat.

You need

PyPIPython’s uv installed (for the uvx command)

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "evalview": {
      "command": "uvx",
      "args": [
        "evalview"
      ],
      "env": {
        "OPENAI_API_KEY": "your-value"
      }
    }
  }
}

Run this in your terminal

claude mcp add evalview --env OPENAI_API_KEY=your-value -- uvx evalview

Fill in the empty values before running it.

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "evalview": {
      "command": "uvx",
      "args": [
        "evalview"
      ],
      "env": {
        "OPENAI_API_KEY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "evalview": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalview"
      ],
      "env": {
        "OPENAI_API_KEY": "your-value"
      }
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.

Your choice stays on this device in a cookie; no account needed.

Values you fill in

OPENAI_API_KEY secret env
OpenAI API key for LLM-as-judge output quality scoring. Optional — deterministic tool/sequence evaluation works without it.

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 7, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account hidai25, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Compared with others in Search & AI memoryi

More stars than 0% of the other 7258 servers in this category
Published to the registry more recently than 7% of the other 7258 servers in this category
Basic checks equal to or better than 100% of the other 7258 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is Evalview MCP safe?

It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

You need to fill in: OPENAI_API_KEY

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed –, and the latest version (0.6.0) reached the registry 7 months ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Python (uv) on your computer.

What can I use instead?

In the same category: Context7, Codebase Memory, Exa

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Context7

by upstash

Feeds AI the latest docs for libraries and frameworks before it answers, so code it writes does not use outdated, removed functions.

Computer or cloud★ 64,224MITUpdated today

Install

Codebase Memory

by DeusData

Codebase knowledge graph for AI agents — 162 languages, sub-ms queries, 99% fewer tokens.

Runs on your computer★ 48,148MITUpdated 20 days ago

Install

Exa

by exa-labs

Connect AI agents to Exa for web search, content fetching, and multi-step research.

Runs in the cloud★ 5,213MITUpdated last month

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

Snapshot testing for AI agents.

Record what your agent does today. Get told when it silently changes.

Your agent returns 200 and looks fine. But a model update, a provider change, or a one-line prompt edit just made it skip a clarification, call the wrong tool, or quietly drop output quality. Your tests still pass. Your users notice before you do.

EvalView snapshots your agent's behavior — the tools it calls, in what order, with what output — and tells you the moment that behavior changes. Like Jest snapshots, but for tool-calling, multi-turn agents.

↑ 30-second live demo — no API key needed

Quick Start

OpenAI adapter migration: OpenAI shut down the Assistants API on August 26, 2026.

The latest published EvalView release, 0.8.1, still uses that API; the Responses API

migration is currently unreleased source. If you use openai-assistants, follow the

migration guide before running your tests. An assistant_id

alone cannot preserve your agent's configuration. Other adapters are unaffected.

pip install evalview

evalview snapshot    # Record your agent's current behavior as the baseline

evalview check # After any change, diff against the baseline

That's the whole loop. check returns one of:

  ✓ login-flow        PASSED          behavior matches baseline

⚠ refund-request TOOLS_CHANGED called a different tool, or in a different order ✗ billing-dispute REGRESSION score dropped — output quality fell

It diffs the whole trajectory — tool names, parameters, and order — not just the final string. The deterministic tool + sequence diff runs offline, with no API key. Add an LLM judge only when you want output-quality scoring.

Executing your agent can still incur backend API charges: --no-judge skips the judge, not those calls. Embedding-based semantic comparison is opt-in.

No agent yet? See it work in 30 seconds:

evalview demo

Why snapshot testing (and not assertions)?

Most eval tools ask you to *write down what "good" looks like* — assertions, metrics, rubrics. That's a lot of upfront work, and you can only catch the failures you thought to assert.

EvalView inverts it: it records what your agent actually does now, and flags any drift from that. You catch regressions you never anticipated, with zero assertions written. When the new behavior is correct, evalview snapshot accepts it as the new baseline — same as updating a snapshot in Jest.

	EvalView	Assertion-based eval tools
Setup	Record current behavior	Write assertions/metrics first
Catches	Any drift from baseline	Only what you asserted
Non-determinism	Multi-variant baselines (up to 5 valid paths)	You handle it
Unit of comparison	Full tool-call trajectory	Usually final output

This makes EvalView a merge-time regression gate, which is a different job from observability (Langfuse, LangSmith) or metric scoring (promptfoo, DeepEval, Braintrust). Many teams run one of those for visibility and EvalView as the gate. Honest comparisons →

EvalView tests itself in public, every day

Every day at 09:00 UTC, on pull requests, and on pushes to main, Core Dogfood exercises the non-live test suite, type checks, local mock-agent snapshot / check, evalview demo, end-to-end flows, and an evalview monitor smoke test. It uses no paid API credentials and makes no paid inference calls. GitHub runner usage is separate.

Live Provider Checks test the real evaluator and chat assistant only when a maintainer explicitly opts into paid API use on main. They have no automatic schedule. Their badge records the last manual run; a green core badge does not establish live-provider health or rule out provider drift.

Package CI, core dogfood, and live checks have separate badges. Failed or incomplete checks remain visible within their scope, with logs and reports preserved as artifacts. Rolling issues use separate dogfood-core and dogfood-live labels. A provider outage, exhausted quota, or missing credential means live health is unavailable; it does not prove an agent regression.

The historical incident #264 remains available for maintainer review of fresh evidence from both scopes. Neither workflow automatically closes it. Trust warnings are evidence to investigate, not proof of gaming or of a particular root cause.

Core runs → · Manual live runs → · Run and triage guide →

CI: block regressions in every PR

# .github/workflows/evalview.yml

name: EvalView on: [pull_request] jobs: agent-check: runs-on: ubuntu-latest permissions: { pull-requests: write } steps:

uses: actions/checkout@v4
uses: hidai25/eval-view@v0.8.1 with: openai-api-key: ${{ secrets.OPENAI_API_KEY }}

You get a PR comment with the diff, cost/latency deltas, and a pass/fail gate. CI/CD guide →

Works with your stack

LangGraph · CrewAI · OpenAI · Claude · Mistral · Ollama · MCP · any HTTP API.

evalview check --agent http://localhost:8000/invoke

Framework details →

Use it as a library

from evalview import gate

result = gate(test_dir="tests/") result.passed # bool result.diffs # per-test scores and tool diffs

Python API →

EvalView also does multi-turn testing, statistical/pass@k runs, record/replay cassettes, model-drift canaries, production monitoring with Slack alerts, and auto-generated regression tests from incidents. These are power-user features — start with snapshot and check, reach for the rest when you need them.

→ Full feature reference · Getting Started · FAQ

→ Documentation index · OpenAI migration · Release process

Why I built EvalView

An agent that looked successful kept pulling entire documents into its context and made one question cost $42.93. That experience led me to build EvalView. I wrote about it in “I Was Running an AI Casino. Then I Started Writing Tests for My Agents”. The December 2025 post is the origin story; use the current docs for setup and commands.

Contributing

This is a young project built mostly by one developer. Issues, PRs, and "I tried it and X was confusing" feedback are all genuinely valuable.

Open an issue · Discussions · CONTRIBUTING.md

License: Apache 2.0

Evalview MCP

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

Values you fill in

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Search & AI memoryi

Common questions

Alternatives

Context7

Codebase Memory

Exa

README from the maker

Quick Start

Why snapshot testing (and not assertions)?

EvalView tests itself in public, every day

CI: block regressions in every PR

Works with your stack

Use it as a library

More

Why I built EvalView

Contributing