MCP Benchmark Hygiene

Name: MCP Benchmark Hygiene
Author: sudo-ai-git

by sudo-ai-gitversion 1.0.1i★ –iRuns on your computer

Registry nameio.github.sudo-ai-git/mcp-benchmark-hygiene

Install

Overview

Description from the maker (original English)

Detect pytest config-leakage that corrupts agent-benchmark grading. Deterministic, no-LLM.

PopNx MCP’s three pillars

These three are never merged into one score: opinions, measurements and popularity are different things.

User voiceiNo reviews yetReviews open together with accounts in a later step
Our basic checksi1 of 1 passedChecked from public information; code not scanned yet
UsageiNot measured yet★ –

What you can do with it

We have not written a plain-language summary for this one yet. Below are the facts we read from the registry.

Connection: Runs on your computer
Transport: stdio
Category: Search & AI memory

Install

Pick the AI app you use, then copy the settings.

Before you install

Basic checks: 1 of 1 passed
Runs on your computer with your user’s permissions (files, network, apps).
The registry lists no key to fill in.

You need

PyPIPython’s uv installed (for the uvx command)

Which app do you use?

ClaudeClaude CodeCursorVS CodeChatGPT

Pick your app above. We remember it, so next time the button says “Install in …”.

Open Claude Desktop → Settings → Developer → Edit Config, and paste this into claude_desktop_config.json

{
  "mcpServers": {
    "benchmark-hygiene": {
      "command": "uvx",
      "args": [
        "mcp-benchmark-hygiene"
      ]
    }
  }
}

Run this in your terminal

claude mcp add benchmark-hygiene -- uvx mcp-benchmark-hygiene

Open in Cursor

Or copy the settings yourself

Paste into .cursor/mcp.json in your project, or ~/.cursor/mcp.json

{
  "mcpServers": {
    "benchmark-hygiene": {
      "command": "uvx",
      "args": [
        "mcp-benchmark-hygiene"
      ]
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

Open in VS Code

Or copy the settings yourself

Paste into .vscode/mcp.json in your project

{
  "servers": {
    "benchmark-hygiene": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "mcp-benchmark-hygiene"
      ]
    }
  }
}

Your browser asks to open the app, then the app shows the settings and asks you to confirm. Key values are left empty for you to fill in.

This one runs on your computer, and ChatGPT can only connect to servers on the internet. Pick another app.

Your choice stays on this device in a cookie; no account needed.

These settings are generated from the maker’s registry entry. If the README says otherwise, follow the README.

What it can do

Runs on your computer; the tool list appears in your AI app after you install it.

Basic checksi

We check information anyone can see; we have not read the code. Checks that do not apply are left out of the count. · Checked Nov 7, 2026

✓ Pass
Maker matches the repo
The registry name was verified with the GitHub account sudo-ai-git, which owns the repo
… Not checked yet
Still maintained
We have not read this repo from GitHub yet
… Not checked yet
Has a licence
We have not read this repo from GitHub yet
– Not applicable
Endpoint responds
This one runs on your computer; there is no internet endpoint
– Not applicable
Sign-in with OAuth
Runs on your computer and uses local settings instead
… Not checked yet
No hidden instructions in tool descriptions
Runs on your computer; we can only read tool lists from internet endpoints

Compared with others in Search & AI memoryi

More stars than 0% of the other 7251 servers in this category
Published to the registry more recently than 34% of the other 7251 servers in this category
Basic checks equal to or better than 100% of the other 7251 servers in this category

Compared with every active server in this category in the registry. Stars count only when the publisher owns the repo.

Common questions

Is MCP Benchmark Hygiene safe?

It passed 1 of 1 applicable basic checks. We have not read the code. Before adding a key or token, see what access the server asks for and use a key with only the permissions you need.

What keys or codes does it need?

The registry does not list a required key, though some servers ask you to sign in on first use.

Does it run on my computer or in the cloud?

It runs on your computer. Your AI app starts it when needed, and data does not pass through the maker’s servers (unless it calls an online service itself).

Is it still maintained?

The GitHub repo last changed –, and the latest version (1.0.1) reached the registry 2 months ago.

Which apps can use it?

Apps that support MCP, such as Claude Desktop, Claude Code, Cursor and VS Code. You also need Python (uv) on your computer.

What can I use instead?

In the same category: Context7, Codebase Memory, Exa

Alternatives

Servers in the same category from other makers, sorted by stars.

Reading a card:★ GitHub starsiThree pillars: reviews · checks · usageiUpdated = latest version published to the registryi

Context7

by upstash

Feeds AI the latest docs for libraries and frameworks before it answers, so code it writes does not use outdated, removed functions.

Computer or cloud★ 64,224MITUpdated today

Install

Codebase Memory

by DeusData

Codebase knowledge graph for AI agents — 162 languages, sub-ms queries, 99% fewer tokens.

Runs on your computer★ 48,148MITUpdated 20 days ago

Install

Exa

by exa-labs

Connect AI agents to Exa for web search, content fetching, and multi-step research.

Runs in the cloud★ 5,213MITUpdated last month

Install

README from the maker

Open the README

Copied from the maker’s repo as written. PopNx MCP has not reviewed or endorsed it. HTML, images and scripts are removed. Links lead off this site.

mcp-benchmark-hygiene

mcp-name: io.github.sudo-ai-git/mcp-benchmark-hygiene

Deterministic detection of pytest config-leakage that silently corrupts agent-benchmark / function grading.

No LLM. No network. One question, answered reliably:

*If I run python -m pytest inside this workspace, will it inherit a

host coverage/abort gate that mis-scores passing code as failed?*

The bug this catches

Automated agent-evaluation harnesses often run python -m pytest inside the target's workspace. If that workspace nests under a repo root carrying pytest addopts — e.g.:

[tool.pytest.ini_options]

addopts = "--cov=harness --cov-report=term-missing:skip-covered --cov-fail-under=80"

...pytest resolves that host pyproject.toml as its rootdir, inherits the addopts, and fails on the host's own coverage gate (harness collected at 0% → below threshold → non-zero exit). The harness then records functionally PASSING code as FAILED.

This is exactly the bug documented in [sudo-ai-git/vulcanbench-findings](https://github.com/sudo-ai-git/vulcanbench-findings): VulcanBench's declarative grader mis-scored every functional task as 0.0 for this reason; with -o addopts= neutralizing the leak, the same workspaces passed 10/10.

The fix it hands you

When a workspace is flagged CORRUPTED, the tool returns the corrected command:

python -m pytest -o addopts= <tests>

-o addopts= strips inherited coverage/abort gates. (Or run the grader from outside the repo root.)

Tools

tool	purpose
`inspect_workspace(path)`	full analysis: ini chain, effective addopts, CLEAN/CORRUPTED/UNKNOWN verdict + corrected command
`check_addopts(path)`	thin boolean: `corrupted` + reasons
`summarize(analysis)`	one-line actionable summary string

Deterministic core (no deps)

The analysis walks the workspace directory up to filesystem root, reading pyproject.toml / pytest.ini / tox.ini / setup.cfg in pytest's first-found order, and extracts addopts. Flags:

coverage gates — --cov, --cov-fail-under, --cov-report, --cov-config
abort/strict gates — --maxfail, -x, --strict, --strict-markers, --pdb, --ff

Only gates that change exit codes / abort grading are flagged. A harmless addopts is reported CLEAN with the exact string.

Install & run (MCP stdio)

One command (recommended) — installs from the repo, no PyPI token needed:

uv tool install git+https://github.com/sudo-ai-git/mcp-benchmark-hygiene

mcp-benchmark-hygiene # run stdio server mcp-benchmark-hygiene --http --port 8137 # or Streamable HTTP

Or with pipx: pipx install git+https://github.com/sudo-ai-git/mcp-benchmark-hygiene

Direct from source (fallback):

{ "mcpServers": {

"benchmark-hygiene": { "command": "python3", "args": ["/abs/path/to/mcp_server.py"] } }}

Requires the official mcp python package (pip install mcp). The deterministic core (inspect_workspace / check_addopts / summarize) imports and runs with zero dependencies — the mcp package is only needed for the stdio server.

Streamable HTTP (remote/Smithery-publishable)

python3 mcp_server.py --http --port 8137   # serves on http://<host>:8137/mcp/

Run with --http to serve over Streamable HTTP (a remote MCP endpoint) instead of stdio. This is the transport smithery mcp publish expects for URL-based publishing — so once a Smithery service token exists, the server deploys as-is.

Example

inspect_workspace(path="/home/runner/vulcanbench/workspace/task-1")

→ { "ok": true, "workspace": "/home/runner/vulcanbench/workspace/task-1", "ini_chain": [{"file": "/home/runner/vulcanbench/pyproject.toml", "addopts": "--cov=harness ... --cov-fail-under=80"}], "effective_addopts": "--cov=harness ... --cov-fail-under=80", "will_corrupt_grading": true, "verdict": "CORRUPTED", "fixed_command": ["python3", "-m", "pytest", "-o", "addopts=", " "], "reasons": ["coverage gate(s) present: ['--cov', '--cov-fail-under']"] }

Verification

python3 test_detector.py — 5/5 core detection checks (root gate, nested inheritance, clean, abort gate, pyproject-no-pytest)
python3 test_e2e.py — drives the real MCP stdio transport (initialize → tools/call) and asserts CORRUPTED / CLEAN thread through the wire

Part of a family

This is one of three deterministic, no-LLM agent-trust MCP servers by sudo-ai-git:

mcp-skill-sec — pre-install skill/security audit
mcp-verify-claim — evidence-gated, honestly-tiered claim reporting
mcp-benchmark-hygiene — pytest config-leakage / eval-honesty detection (this repo)

Sibling product: mcp-token-saver — token-cost proxy + analyzer for agent conversations (dedupes redundant tokens before they're billed; live-proven 74% cut). Discussion

Also in the family (a free CLI, not an MCP server): harness-audit — deterministic agent-eval / benchmark-grading hygiene audit that catches the same silent config-leakage mis-scoring class. Free lead-magnet; the same verification discipline, zero dependencies, auditable line-by-line.

License & provenance

MIT. Independently derived from the documented VulcanBench #79 finding; no endorsement by or affiliation with morganlinton/VulcanBench implied.

Hire a custom integration

Need this connected to *your* internal system (auth, logging, security-scan pass, hosted)? Open a custom-build request. MIT reference assets are free to use either way.

MCP Benchmark Hygiene

Overview

PopNx MCP’s three pillars

What you can do with it

Install

Before you install

What it can do

Basic checksi

Maker matches the repo

Still maintained

Has a licence

Endpoint responds

Sign-in with OAuth

No hidden instructions in tool descriptions

Compared with others in Search & AI memoryi

Common questions

Alternatives

Context7

Codebase Memory

Exa

README from the maker

mcp-benchmark-hygiene

The bug this catches

The fix it hands you

Tools

Deterministic core (no deps)

Install & run (MCP stdio)

Streamable HTTP (remote/Smithery-publishable)

Example

Verification

Part of a family

License & provenance

Hire a custom integration