Flakyguard

flakyguard

"That test is flaky" is usually a feeling. flakyguard turns it into evidence. Feed it the per-test outcomes from a set of CI runs and it builds the record for every test:

• Per-test history — how many runs contained the test, and how many times it passed, failed, or was skipped
• Evidence runs — passes plus failures; a skip is recorded but never counts as evidence, because a test that did not run proves nothing either way
• Classification — a test needs a minimum amount of evidence (two evidence runs by default) before it earns a verdict: stable_pass if it never failed, stable_fail if it never passed, flaky if it did both, and insufficient_data until then
• Flakiness rate — the minority outcome's share of the evidence runs, so a test that splits its results evenly rates 0.5 and a mostly reliable one rates close to 0
• Quarantine list — the flaky tests whose rate meets the quarantine threshold (0.5 by default), worst first, each entry carrying the pass/fail counts behind its rate
• Verdict — quarantine_recommended when the list is non-empty, otherwise clean

Rates are rounded to four decimal places. Everything is computed locally from the runs you supply: no network, no configuration, no history kept between calls. Bad input — a run with no results, a status outside passed / failed / skipped, the same test twice in one run — is rejected with an error naming the offending path, and no partial analysis is produced.

CLI

npx @reneturcios/flakyguard analyze --input args.json

npx @reneturcios/flakyguard --version

args.json contains the full arguments object:

{

"runs": [ { "run_id": "run-101", "results": [ { "test": "checkout totals", "status": "passed" }, { "test": "search ranking", "status": "failed" } ] }, { "run_id": "run-102", "results": [ { "test": "checkout totals", "status": "failed" }, { "test": "search ranking", "status": "failed" } ] } ], "min_runs": 2, "quarantine_threshold": 0.5 }

min_runs and quarantine_threshold are optional. The analysis is printed as JSON.

Exit codes: 0 on success, 1 on a runtime error (including an unreadable or invalid input file, or invalid input), 2 on a usage error.

MCP server

npx -y @reneturcios/flakyguard mcp

MCP client config:

{

"mcpServers": { "flakyguard": { "command": "npx", "args": ["-y", "@reneturcios/flakyguard", "mcp"] } } }

Tools

• analyze_test_flakiness — Analyze test flakiness across CI runs.
• runs (array, required, at least one) — runs as {run_id, results} objects, where each result is {test, status} and status is one of passed, failed, skipped; a test may not appear twice in one run
• min_runs (integer, optional, >= 1, default 2) — minimum evidence runs before a test is classified as stable or flaky
• quarantine_threshold (number, optional, 0 to 1, default 0.5) — flakiness-rate threshold for the quarantine list

Returns {run_count, test_count, tests, quarantine, summary, verdict}.

Docker

docker run -i ghcr.io/cloudcorprecords/flakyguard

docker run --rm -v "$PWD:/w" ghcr.io/cloudcorprecords/flakyguard analyze --input /w/args.json

Development

npm install

npm run build # tsc -> dist/ npm test # tsx --test test/*.test.ts (pure local, no network) npm run dev # run the CLI from source

License

MIT — Copyright (c) 2026 Rene Turcios