Agent Reliability

Agent Reliability — MCP server

Testing, benchmarking and auditing autonomous AI agents — methods, harnesses, evidence

A remote MCP server over a curated knowledge graph. Every claim it returns is bound to a registered source: the tools hand back claims *with* their citations and a confidence value, so an agent can show its work instead of asserting.

Nothing to install. It is a hosted streamable-HTTP endpoint:

https://agentreliability.dev/mcp

Also listed on Smithery.

Add it to a client

Claude Code

claude mcp add --transport http agent-reliability https://agentreliability.dev/mcp

Claude Desktop / any client reading `mcpServers`

{

"mcpServers": { "agent-reliability": { "type": "streamable-http", "url": "https://agentreliability.dev/mcp" } } }

No API key, no account, no auth. Read-only.

Check it answers, without any client at all:

curl -s https://agentreliability.dev/mcp \

-H 'Content-Type: application/json' \ -H 'Accept: application/json, text/event-stream' \ -H 'mcp-protocol-version: 2025-06-18' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

Tools

Eight, each with an outputSchema, each returning structuredContent.

The intended path is get_overview → search or answer → get_entity → get_related. get_overview exists because an agent that has just arrived needs to know whether this corpus can help *before* it spends a call guessing.

What is in the corpus

Subject matter: evals and benchmarks (GAIA, AgentBench, Inspect), LLM-as-judge and its failure modes, Goodhart and benchmark contamination, fault injection and chaos testing, approval gates and autonomy levels, grounding and faithfulness.

Questions it is built to answer

• *How do I tell a real eval from a benchmark my agent has memorised?*
• *What does calibration mean for an LLM judge, and how is it measured?*
• *Which failure modes does fault injection actually catch?*

What an answer actually looks like

A real call against the live endpoint — answer with *"how do I tell a real eval from benchmark contamination"* — returns this structuredContent, trimmed:

{

"answered": true, "entity": { "id": "agent-reliability-glossary", "name": "Agent reliability glossary", "evidence_tier": "secondary", "confidence": 0.85, "last_verified": "2026-08-08", "canonical_url": "https://agentreliability.dev/k/agent-reliability-glossary" }, "claims": [ { "text": "An eval is a structured, repeatable test that measures an LLM or LLM-based system against a defined dimension; frameworks package evals as registries of reusable templates.", "sources": [{ "title": "openai/evals — framework for evaluating LLMs and LLM systems" }] } ] }

Note what travels with the answer: the evidence tier, a confidence, the date it was last verified, and the source behind the claim — not as prose an agent has to parse, but as fields it can act on. An agent can decline to use a weak claim, or cite the primary source directly.

When the corpus cannot answer, answered is false. It does not improvise, and the miss is recorded so the gap can be filled.

Machine-readable surfaces

The MCP endpoint is one of several. The same corpus is served as plain files an agent can read directly:

Each knowledge object has a human page and a machine twin at the same id, with a canonical URL that agrees across all of them.

Behaviour worth knowing before you integrate

• `POST` only. Every other method answers 405 with an Allow: POST, OPTIONS header.
• Rate limit: 120 requests per minute per client, counted in a shared store, published on every response as RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset (all three exposed via CORS). It fails open: if the store is unreachable the request is served.
• Malformed input gets a spec-correct JSON-RPC error — -32700 for unparseable bodies, -32602 for an unknown tool — never an HTML error page.
• Request bodies are capped and validated before transport.

Privacy

No accounts, no cookies, no ads. Usage is measured in aggregate with daily-rotating hashed identifiers and a 200-day retention; raw IPs are never stored. Full policy: PRIVACY.md.

Provenance and licence

Knowledge content is CC-BY-4.0: use it, cite it. The source registry is public precisely so a claim can be checked rather than trusted — get_sources returns what any given claim rests on.

Claims carry an evidence tier and a last_verified date. Where the evidence is weaker, the object says so rather than rounding up.

How it is built

Compiled and served by Citarium, a source-available framework for turning a knowledge graph into a website, an API, an MCP server and agent-readable files from a single source, under external evaluation.

The framework's code is licensed under the Business Source License 1.1 and its repository is not public. What is public — and what actually matters for trusting an answer — is this server, the corpus it serves, and the registered source behind every claim: get_sources returns what any given claim rests on, so it can be checked rather than trusted.

This repository is the server's public face: its manifest and its documentation. The corpus itself lives at agentreliability.dev.

Running it yourself

The endpoint above is the server. If you want to run a copy locally — or if a directory needs to start one in order to inspect it — the recipe is in CONTAINER.md: fetch-corpus.mjs downloads this instance's published corpus and @citarium/mcp serves it. Both files are GENERATED by citarium mirror; edit the instance, not them.

The container serves a snapshot taken when its image was built. This endpoint is the thing itself.