MCP Compliance

@yawlabs/mcp-compliance

One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.

Test any MCP server for spec compliance. Two test suites — 88 tests for MCP 2025-11-25 and 103 tests for MCP 2026-07-28 — covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. The tool probes the server and grades the newest spec revision it speaks, or you pin one with --spec-version. Works against HTTP endpoints (https://my-server.com/mcp) and stdio servers (npx @modelcontextprotocol/server-filesystem /tmp) alike. CLI, MCP server, and programmatic API.

Built and maintained by Yaw Labs.

Why this tool?

MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.

This tool solves that:

• Two spec revisions, one tool — 88 tests for MCP 2025-11-25 (the initialize + session era) and 103 tests for MCP 2026-07-28 (stateless, per-request _meta, server/discover, caching hints, MRTR). Both cover the same 8 categories: transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. HTTP-specific tests (CORS, TLS, header validation, rate limiting) are gated out on stdio: the 2025-11-25 suite runs 85 tests on HTTP and 55 on stdio, the 2026-07-28 suite 99 and 75.
• Auto-detected revision — one server/discover probe tells the tool which era the server speaks; dual-era servers are graded against 2026-07-28 with a warning. Pin either revision with --spec-version. See Spec version.
• Capability-driven — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
• Graded scoring — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
• CI-ready — --strict mode exits with code 1 on required test failures. Drop it into any pipeline.
• Spec-referenced — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
• Three interfaces — CLI for humans, MCP server for AI assistants, programmatic API for integration.
• Published methodology — the testing methodology and rule catalog are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.

Quick start

Remote HTTP server:

npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp

Local stdio server (the vast majority of MCP servers on npm):

# Pass the command directly, Inspector-style

npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp

Or a local build

npx @yawlabs/mcp-compliance@latest test node ./dist/server.js

With env vars

npx @yawlabs/mcp-compliance@latest test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github

Install globally:

npm install -g @yawlabs/mcp-compliance

mcp-compliance test https://my-server.com/mcp

That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.

CLI usage

HTTP targets

# Terminal output with colors and grade

mcp-compliance test https://my-server.com/mcp

JSON / SARIF for scripting + GitHub Code Scanning

mcp-compliance test https://my-server.com/mcp --format json mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif

Strict mode for CI — exits 1 on required-test failure

mcp-compliance test https://my-server.com/mcp --strict

Auth (shorthand or full header)

mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123" mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"

Focus the run

mcp-compliance test https://my-server.com/mcp --only transport,lifecycle mcp-compliance test https://my-server.com/mcp --skip prompts,resources mcp-compliance test https://my-server.com/mcp --verbose

stdio targets

Pass the command and its args as positional arguments (MCP Inspector-style). Use -- to disambiguate when the target needs flags that collide with ours.

# npm-distributed stdio servers

mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp mcp-compliance test uvx mcp-server-git

Local build

mcp-compliance test node ./dist/server.js

With env vars (repeatable -E, or --env-file)

mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server mcp-compliance test --env-file .env -- node ./server.js

Set working directory

mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js

Target uses a flag that collides with ours — use -- to separate

mcp-compliance test --verbose -- node ./server.js --verbose

On Windows, npx and other .cmd shims are handled automatically by spawning through the shell.

Spec version

The tool ships one test catalog per MCP specification revision it supports — 2025-11-25 (88 tests: initialize handshake, sessions, ping) and 2026-07-28 (103 tests: stateless, per-request _meta, server/discover, caching hints, MRTR). A run grades exactly one revision, chosen by --spec-version:

# auto (default): probe the server, grade the newest revision it speaks

mcp-compliance test https://my-server.com/mcp

pin a revision — no probe, the named catalog runs regardless of what the server answers

mcp-compliance test https://my-server.com/mcp --spec-version 2026-07-28 mcp-compliance test https://my-server.com/mcp --spec-version 2025-11-25

--list never connects: under auto (the default) it prints every catalog in its own

section; --spec-version previews just that one

mcp-compliance test --list --spec-version 2026-07-28 --transport stdio

How auto decides, following the spec's own rules for dual-era clients: the tool sends one conformant 2026-07-28 server/discover (on HTTP it doubles as the preflight connectivity check, so detection costs no extra round-trip; on stdio it is the first exchange). A DiscoverResult, or a JSON-RPC error with a modern code (-32020, -32021, -32022), selects 2026-07-28. Anything else — -32601, -32000, a 400 "not initialized", a 404, an HTML page, or no reply within the timeout — selects 2025-11-25; a 401/403 with no modern error body also lands on 2025-11-25, the note says the era could not be determined, and the report's first warning (in every output format) says the grade below is not meaningful. With no Authorization header configured, a 401, or a 403 carrying a WWW-Authenticate: Bearer challenge, reads as authentication required (note authentication required -- pass --auth; the warning says to re-run with --auth). With one configured, a 401, or a 403 whose Bearer challenge carries an error parameter, reads as the credential rejected (note credential rejected -- check --auth; the warning names the reason from that error, e.g. invalid_token is an invalid or expired token and insufficient_scope a missing scope or permission). Any other 403 (the SDK's Host validation behind a tunnel hostname, Origin validation, a gateway) is worded neutrally with or without a header: the note says forbidden -- Host/Origin validation, a gateway, or missing credentials (or ... or token permissions with a header), and the warning quotes the body's JSON-RPC error message when there is one, points at Host/Origin validation and any gateway first, and without a header suggests --auth only if the server does require a credential. The fallback is deliberately not keyed to a single error code; a legacy server that ignores unknown methods is still classified correctly, just after the timeout. The report header names the reason (auto-detected from server/discover: supportedVersions [2026-07-28], ... JSON-RPC error -32601, legacy, ... no response, legacy) and the JSON warnings carry the same note (Spec version auto-detected as (server/discover -> ). Pin with --spec-version to override.). An unreachable server is graded as 2025-11-25 so every test that needs the server fails (the 2026-07-28 post-hoc scans fail too when nothing was received, rather than passing over an empty recording).

Two budgets bound the probe. On stdio it is bounded by --startup-timeout, and in terminal mode a dim status line appears on stderr after ~2 s naming the wait and the --spec-version 2025-11-25 skip, so a legacy server that stays silent on unknown methods does not look like a hang. If the child exits on the probe (a legacy server whose dispatcher throws on unknown pre-initialize methods), the report warns with the exit code and stderr tail, spawns a fresh instance and grades it as 2025-11-25. If the fresh instance exits at startup as well, before answering initialize, the warning says the server exits at startup regardless of the probe and points at the command, its arguments and its environment instead of suggesting a pin. benchmark re-spawns a probe-killed child the same way, bounds its probe by its own --startup-timeout, prints the same status line, and carries the warning in its output. On HTTP the probe is the preflight and is bounded by --preflight-timeout; a preflight that times out is re-probed once within --startup-timeout before the run defaults to 2025-11-25, so a slow cold start is graded in its real era instead of as unreachable.

A server that answers both eras (the default for servers built on the official SDK 2.0) is graded as 2026-07-28, and the report carries a warning naming the other era and how to pin it — keyed on supportedVersions listing 2025-11-25 *or* on the informational lifecycle-dual-era probe being served an InitializeResult (SDK 2.0 servers advertise only modern versions yet still serve initialize; on stdio that probe goes to a fresh process, since a dual-era server pins its era per process). To grade its legacy side too, run again with --spec-version 2025-11-25. A server that serves initialize while *rejecting* the conformant server/discover is not dual-era but legacy-only: lifecycle-dual-era says legacy-only: ... and the warning tells you to re-run with --spec-version 2025-11-25 (or auto). The report's specVersion is always the resolved revision, never auto.

A pinned run runs the named catalog whatever the server answers, with two honest side notes. On HTTP the preflight is still a modern server/discover, so a pinned run whose server plainly answered in the *other* era gets a warning (Server answered the 2026-07-28 server/discover probe with a DiscoverResult ...; this run is pinned to 2025-11-25. Re-run with --spec-version 2026-07-28 (or auto) to grade it). The warning needs an era signal -- a DiscoverResult, a JSON-RPC body or an HTTP 4xx; a 5xx or an HTML page says nothing about the era and d