AdversaryGate

AdversaryGate

High AI usage ≠ high confidence. The cost of an agentic coding pipeline is

not the model's intelligence — it is the pipeline's self-deception.

A fail-closed verification gate for patches written by AI coding agents. It runs your tests on the baseline and on the patch, mutation-tests only the lines the patch wrote, and answers MERGE, BLOCK or INCONCLUSIVE — where uncertainty is a first-class result instead of a silent approval. CLI, GitHub Action and MCP server.

Python only, for now. It measures Python/pytest repositories; on other stacks it runs your suite via --test-command but answers INCONCLUSIVE, never MERGE — see Language support.

pip install adversary-gate

🎬 Watch it decide (60 seconds)

B and C are the same patch. Same code, same tests, same execution. The only difference is that C brought evidence (--diff + --coverage-json); B brought none.

That middle row is the whole product. An unproduced measurement is not a measurement, so it cannot clear a floor — and a number nobody can re-derive is an opinion with a false precision label.

python3 demo/demo.py          # runs the three scenarios; exits non-zero if one regresses

python3 demo/render_gif.py # rebuilds demo/demo.gif from the captured frames

demo/demo.py is an acceptance test, not a screenshot: it builds the fixtures, runs the real CLI as a subprocess, reads the real exit codes, and fails if any of the three decisions changes. Its evidence artefacts are checked in under demo/out/.

`INCONCLUSIVE` is not an error. It is the gate saying *"I did not measure that"*, and it is exit 2 so CI treats it as "do not merge yet", not as "pass". See How diff_coverage gets its value for the one-step path from INCONCLUSIVE to MERGE.

🚀 Quick Start (CLI & GitHub Action)

First run, in one minute

From the root of a git repository whose agent's change sits on top of main (new files tracked: git add -N them), with pip install adversary-gate coverage:

BASE=main                                         # the commit before the agent's change

W=$(mktemp -d) git worktree add -q "$W/before" "$BASE" # the baseline, exactly as committed git diff --no-ext-diff "$BASE" > "$W/changes.diff" # baseline -> your working tree coverage run -m pytest -q && coverage json -q -o "$W/coverage.json" adversary-gate --baseline "$W/before" --patch . \ --claim tests/test_shop.py::test_discount_never_negative \ --diff "$W/changes.diff" --coverage-json "$W/coverage.json" > "$W/answer.json" echo "exit $?" # 0 MERGE, 1 BLOCK, 2 INCONCLUSIVE python -c "import json,sys; a=json.load(open(sys.argv[1])); print(a['decision'], '-', a['reason'])" "$W/answer.json" git worktree remove --force "$W/before"

Replace the --claim with the test that proves your fix (path::test). On a correct one-line fix, expect this first answer:

exit 2

inconclusive - every claim held and the fix is proven, but suite strength's lower bound is 0.38, below the 0.75 floor (1/1 mutants killed): the change is too small to measure at this floor, which takes at least 5 killed mutants; see next_steps

That is not a failure: the fix *is* proven, and the gate says plainly that it could not measure how strong the tests are on a change this small. See Small changes.

Command-Line Usage

adversary-gate \

--baseline /path/to/before \ --patch /path/to/after \ --test-path tests/test_auth.py \ --test-id test_token_expiry \ --diff changes.diff \ --coverage-json coverage.json \ --model "${AGENT_MODEL_NAME}" \ --evidence-log evidence.jsonl \ --report

Produce the two artefacts in your own pipeline, for example:

git diff --no-ext-diff baseline...patch > changes.diff

coverage json -o coverage.json # after: coverage run -m pytest

Which tests? Name several, or let coverage say

One run verifies any number of claims, and the decision is over all of them (one REFUTED blocks; one UNVERIFIED makes it INCONCLUSIVE):

adversary-gate ... --test-path tests/test_auth.py --test-id test_expiry --test-id test_refresh

adversary-gate ... --claim tests/test_auth.py::test_expiry --claim tests/test_api.py::TestLogin::test_ok

Or name none, and let the gate verify the tests that executed a changed source line — read from coverage.py's per-test contexts, not guessed:

printf '[run]\ndynamic_context = test_function\n' > contexts.coveragerc

coverage run --rcfile=contexts.coveragerc -m pytest coverage json --show-contexts -o coverage.json adversary-gate --baseline before --patch after \ --diff changes.diff --coverage-json coverage.json --discover-claims

A test file the patch rewrote is still picked, and judged with the baseline's copy of it (the baseline oracle, above); it is named under discovery (rewritten_test_files_judged_by_baseline), next to anything that could not be resolved to a node id, and the cap (--max-claims, default 10) with how many it cut. From 3.0.9 a cut test is not left unjudged: with the full suite off (--full-suite-path ""), a patch whose kept claims would merge is INCONCLUSIVE instead, and the reason says to raise the cap or run the suite. A report recorded without contexts is a usage error (exit 3), not "no tests".

Your project's interpreter, its limits, its environment

adversary-gate ... --python .venv/bin/python \

--timeout 120 --cpu-seconds 60 --memory 2G \ --pass-env PATH --env PYTEST_DISABLE_PLUGIN_AUTOLOAD=1

The same choices apply to every run — claims, mutants and the collateral suite — except time and memory: the whole suite gets its own limits, because ones sized for a single test killed every suite of real size on both sides (out of time, or pytest dying with MemoryError under 512M of address space): never a MERGE (two failing sides are unverifiable), but the regression the suite would have caught came back INCONCLUSIVE instead of BLOCK (AG-033). The output records them under execution, with the names of the variables and never their values. A run the memory limit killed (MemoryError, Fatal process out of memory, Could not reserve enough space…) is reported as the harness dying, not as a failing test: it says nothing about the patch.

Not a pytest suite, or running a patch you have not seen?

# 1. run your own command instead of pytest (any language, any runner)

adversary-gate --baseline before --patch after \ --test-path run_tests.sh --test-command "./run_tests.sh"

2. wrap every run in bubblewrap: no network, private PID//tmp,

read-only system, only the repo writable

adversary-gate ... --sandbox bwrap

3. both together -- the usual shape for a patch from someone else

adversary-gate ... --test-command "make check" --sandbox bwrap \ --require-network-isolation

--test-command needs no --test-id (there are no node IDs to name), and it feeds the collateral full-suite run too unless you pass --full-suite-command. --sandbox bwrap needs bubblewrap installed and refuses to start without it — see Execution boundary for exactly what it does and does not contain.

Exit Codes for CI Integration:

• 0 — `MERGE`: Every claim executed cleanly and cleared coverage & suite strength floors.
• 1 — `BLOCK`: Regressions or collateral suite failures detected.
• 2 — `INCONCLUSIVE`: Infrastructure error, missing test, missing coverage evidence, or weak suite (blocks merge).
• 3 — `USAGE_ERROR`: Bad arguments, malformed JSON, or a coverage claim that was not declared as untrusted.

GitHub Action Integration (action.yml)

One ref in. base-sha derives the baseline, the diff and the coverage report itself, so the first thing you install is not a INCONCLUSIVE:

name: Verification Gate

on: [pull_request]

jobs: verify-agent-patch: runs-on: ubuntu-latest steps:

base-sha is a commit in history: this is not optional.

• uses: actions/checkout@v4 with: fetch-depth: 0

Your Python, your dependencies -- the tests run here.

• uses: actions/setup-python@v5 with: python-version: '3.11'
• run: pip install -r requirements.txt pytest coverage

• name: Run AdversaryGate uses: Sanflow10/adversary-gate@v3 with: base-sha: ${{ github.event.pull_request.base.sha }} evidence-log: 'evidence.jsonl'

No test is named: the Action records coverage per test and verifies the tests that executed the lines the pull request changed. Name them yourself with claims (one path::test per line) or test-path + test-id (one id per line); either one turns discovery off unless discover-claims: 'true'.

Any non-zero exit fails the step, and that is the safe default. BLOCK (1), INCONCLUSIVE (2) and a usage or harness error (3/4) all turn the check red. Do not add continue-on-error and stop there: that lets a BLOCK through along with everything else. If some pull requests are INCONCLUSIVE by nature (a change to C++ or SQL, which the gate runs but cannot measure), keep the step from failing and branch on the exit-code output instead (2.13.0 and later):

      - name: Run AdversaryGate

id: gate uses: Sanflow10/adversary-gate@v3 continue-on-error: true with: base-sha: ${{ github.event.pull_request.base.sh