Agent Eval

Agent Evaluation

Statistical regression testing for LLM agents: run version A and version B 50x each and get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

pip install agent-regress-cli

(uv add and the npm/npx wrapper are covered in Install below.)

The test that every eval framework skips

You changed a prompt. Your evals still pass. But your agent's tool accuracy dropped from 84% to 70%.

Is that a real regression? Or is it LLM run-to-run noise?

Threshold testing cannot answer that question. agent-eval can.

Run your agent 50x on version A, 50x on version B. Get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

p=0.003, Cohen's d=-0.61 -> REGRESSED (deploy blocked)

p=0.410, Cohen's d=0.021 -> STABLE (safe to ship)

[!NOTE]

Promptfoo, one of the most widely used open-source LLM eval frameworks, was acquired by OpenAI in March 2026, staying open source but folding its team into OpenAI's Frontier platform. agent-eval is Apache 2.0-licensed, self-hostable, and has no commercial dependency. The statistical core (Mann-Whitney U, bootstrap CI, Cohen's d) will never be paywalled.

---

Install

pip install agent-regress-cli

or

uv add agent-regress-cli

or, from Node/npx (thin wrapper around the same Python CLI)

npx agent-regress-cli

---

Quickstart

In 30 seconds (CLI)

Already have per-run scores from your own harness? Point the CLI at two JSON arrays of scores, one per version:

pip install agent-regress-cli

agent-regress compare \ --version-a-results v1_scores.json \ --version-b-results v2_scores.json \ --metric tool_accuracy

============================================================

agent-regress Report -- tool_accuracy

============================================================

Verdict: REGRESSED

p-value: 0.0000

Cohen's d: -2.193

95% CI: [-0.213, -0.148]

#

Version A: 0.8470 +/- 0.0525 (n=50)

Version B: 0.6685 +/- 0.1025 (n=50)

Delta: -0.1786

============================================================

Add --json --fail-on-regression to get clean, parseable output and a non-zero exit code on REGRESSED, for wiring straight into CI.

All `agent-regress compare` flags:

The top-level agent-regress --version flag prints the installed version and exits.

Exit codes:

In your code (Python API)

Driving the agent yourself instead of pre-computing scores? Use the Python API:

from agent_regress import compare

Any callable that takes a test case dict and returns a score 0.0-1.0

def agent_v1(test_case: dict) -> float: ... # your existing agent

def agent_v2(test_case: dict) -> float: ... # your updated agent

test_suite = [ {"query": "find SKU for order 8823", "expected": "SKU-4492"},

... more test cases

]

report = compare( version_a=agent_v1, version_b=agent_v2, test_suite=test_suite, n_runs=50, metric="tool_accuracy", # use any name except "accuracy" when agents return floats )

print(report) # structured output with p-value, CI, effect size report.assert_stable() # raises AssertionError if behavior regressed

Agent returns text? Pass a scorer or use the built-ins:

from agent_regress import compare, exact_match_scorer, f1_scorer

exact_match_scorer: 1.0 if str(output).strip() == str(expected).strip()

f1_scorer: token-level F1 (multiset — handles repeated tokens correctly)

report = compare( version_a=agent_v1, version_b=agent_v2, test_suite=test_suite, n_runs=50, scorer=exact_match_scorer, # test_case must have an "expected" key )

Or write your own:

def my_scorer(output: str, test_case: dict) -> float:

return 1.0 if output.strip() == test_case["expected"] else 0.0

report = compare(..., scorer=my_scorer)

---

MCP Server

agent-eval ships a Model Context Protocol server so an AI agent (Claude, Cursor, or any MCP-compatible client) can run statistical regression tests directly, without a human invoking the CLI by hand.

Install the extra:

pip install "agent-regress-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop, claude_desktop_config.json):

{

"mcpServers": { "agent-eval": { "command": "uvx", "args": ["--from", "agent-regress-cli", "agent-regress-mcp"] } } }

The server exposes one tool, run, that shells out to the agent-regress CLI with the given subcommand and arguments plus --json, and returns the parsed JSON result:

run(["compare", "--version-a-results", "a.json", "--version-b-results", "b.json", "--metric", "accuracy"])

Transport is stdio, so there is nothing to host: the MCP client spawns the server as a local subprocess. Source: src/agent_regress/mcp_server.py.

---

Why not DeepEval, Promptfoo, or Braintrust?

†Promptfoo acquired by OpenAI, March 2026; remains open source under its current license.

DeepEval tests whether an individual agent response clears a quality bar. Agent Evaluation tests whether behavior changed significantly between two agent versions, a different statistical question that threshold testing cannot answer. The scipy Mann-Whitney U call at the core is one line, so any SaaS eval platform can add it. What accumulates over time through production use is version-specific regression history and a community-maintained benchmark leaderboard with independent result verification.

---

Real regressions statistical testing catches that threshold testing misses

LangGraph

• #5243: a new typed context= API replaced untyped config['configurable']. A single run on either invocation style still clears a threshold check; only a version-A-vs-B comparison shows whether the switch changed measured behavior.
• #4486: node/task-level result caching can silently mask repeated-sampling variance. Threshold checks don't care whether a result came from cache; a statistical comparison depends on genuinely independent samples, so agent-eval added cache-busting to protect that assumption.

OpenAI Agents SDK

• #2463: agent-as-tool calls were silently dropping the parent run's RunConfig. The nested call still returns a normal-looking response, so a single-response check clears; only inspecting config propagation across runs reveals the regression.
• #2214: image/audio/file tool outputs were silently downgraded to text-only. A text-only threshold scorer has no way to notice a dropped attachment.

CrewAI

• #6134: a security fix for file tools leaking absolute filesystem paths in responses. A quality scorer checks whether the answer is correct, not whether it also leaks a path, so the leak clears the bar.
• #6236: tools gained an optional Pydantic output_schema, moving from unstructured str() output to structured JSON. Both the old and new format can look "reasonable" to a threshold scorer even though the schema changed underneath.

These are the regressions that motivated this project. Full detail on all 14 individually-documented PRs (drawn from a 29-PR, 239-row validation campaign across LangGraph, CrewAI, and the OpenAI Agents SDK) is in docs/pr-analysis.md.

---

The probl