Autopia

Autopia

Autopia empirically tests HTTP APIs that ship an OpenAPI spec, discovers how they actually behave, verifies every finding by reproducing it, and serves the result as a versioned, queryable knowledge base that AI agents consult over MCP, and that a Claude Code plugin injects into an agent's context at the moment it is about to call one of those APIs.

It is the MVP of the "Discrimen Compendium" described in docs/compendium-spec.txt (the blueprint) and docs/compendium-stack.txt (stack choices). Read the spec's glossary first; the code uses those exact terms. HANDOFF.txt holds the running state and decisions.

Live: the hosted server is at https://autopia-mcp.onrender.com/mcp (per-user keys, see Identity); the database is a managed Postgres on Neon; a GitHub Actions schedule refreshes every target nightly (rules only) and weekly (with the model pass). The catalogue currently holds 16 APIs, including GitHub REST, the npm registry, PyPI, the Anthropic models endpoints, the Open-Meteo family, weather.gov, Binance, Frankfurter, Open Library, PokeAPI and Nager.Date.

What an agent gets

A behavior sheet per API: Markdown generated from verified claims only, ordered by value. Where the live API contradicts its own spec comes first, then rules the spec never states, then behavior common to every endpoint, then per endpoint: what responses look like, error-body shapes, pagination, validation, auth. Every line ends with the trial ids that back it. Examples of what the sheets record, all reproduced on the live APIs:

• GitHub silently applies since=not-a-date on commits as a filter that matches nothing (200, empty list), while issues rejects the same value with a 422 that spells out the format.
• GitHub accepts a missing token on public reads but serves the request from the 60/hour tier instead of 5000/hour; per_page is clamped to 100 (proved by the Link header).
• npm search accepts size=2500 against a documented maximum of 250, and does not type-check its score-weight parameters.
• The Anthropic /v1/models cursors must name an existing model id; an unknown id is a 400 as a cursor but a 404 as a path.

The pipeline

discover -> exercise -> aggregate -> cross-ref -> verify -> write (+ diff)

spec TRIALS rules + spec + evidence CLAIMS fetch, (no LLM) shape rules siblings + check + + DIFFS hash, + one LLM self- reproduce trim pass consistency

• Trials are pinned requests generated by the battery (src/worker/battery, version 1.3.0), a deterministic catalogue of edge cases: omit required, wrong type, enum outside the documented set, null and empty values, numeric boundaries (zero, negative, max int, below the minimum, just above and far above the maximum, 2048-character strings), duplicate parameters, unknown fields, bad and missing credentials (scheme-preserving), page zero, a huge page size, a page beyond the last, four date formats for date-time parameters (date only, no timezone, epoch, garbage), wrong content type, trailing slash, HEAD. No randomness anywhere, so two runs produce identical trial inputs. Every stored response keeps the lengths of its arrays before truncation, so list semantics survive storage.
• Claims come from three sources: deterministic rules over the trials (free), shape rules over successful responses (content type, timestamp formats, null-valued fields, pagination fields, rate-limit and cache headers), and one batched LLM pass for open-ended discovery. The LLM output is validated with ajv against src/schemas/claim.schema.json; invalid objects are discarded. Rules classify what came back, not just the status: a 200 on a bad date value is "applied as an impossible filter" when the list is empty where the comparison request had rows, "apparently ignored" when the counts match, "interpreted" otherwise. A page-size cap is claimed only when more items provably exist (a Link rel="next", a total field, or the documented maximum reached). Error-shape claims count only probes that sent a bad value, not auth, HEAD or content-type probes.
• Verification is the product: every claim cites trial ids and the status each showed. The evidence check is mechanical; reproduction re-runs the exact inputs and compares a normalized response signature (status, body kind, JSON shape, error message), never raw bytes, so live data can change without breaking reproduction. Non-reproducing claims are discarded. Stored claims are always verified = true.
• Divergences are claims that contradict a documented fact (required flag, type, enum, range). The money view. Silence in the spec is never a divergence: a rule the API enforces that the spec never states (parameters that must be sent together, implicit defaults) is stored under its own kind, undocumented_constraint.
• Claim kinds: required, type, auth, pagination, error_behavior, response_shape, undocumented_constraint, undocumented, other.
• Sibling comparison. Tools that share a category are compared probe by probe: the same rule on the same parameter name with the opposite verdict marks the claim divergence_type = sibling with a note naming the other tool.
• Carry-forward. Rules-only runs re-admit the previous run's LLM claims as candidates, remapped to tonight's trials by endpoint and probe, and send them through the same evidence check and reproduction. A carried claim is dropped when a fresh LLM claim on the same endpoint cites the same probes (the model restates findings in new words), and carried claims that duplicate each other keep the more confident one.
• Diffs compare claim sets across tool versions by stable claim_key. A tool gets a new version whenever its spec hash changes; large specs are stored trimmed to the exercised paths, with the hash still over the full document.

The scenario battery: how an API behaves in use

The probe battery changes one parameter of one request and judges the answer against the spec. That finds where the docs are wrong, but only along lines the docs already drew. The scenario battery uses an API the way a program does, several requests with state flowing between them, and judges the outcome against an invariant that must hold for any sane API, so it needs no documentation to compare against and can learn what the docs never say.

Scenario 1, pagination completeness, runs on every endpoint whose baseline answered with a list and that takes a page number. Two walks of a few small pages, one by page number and one by following the Link header's rel="next", plus one page read twice:

• each item appears exactly once across pages (or which ids repeat, and on which pages);
• the Link chain visits the same pages as counting up (or where it diverges, or that there is no Link header at all and paging must be driven by number);
• a page read twice a few seconds apart is identical (or the list is live-ordered and shifts);
• the requested page size is what comes back (or it is silently clamped);
• how the list ends: a short page, an empty page, no rel="next".

Every request is a trial; every claim cites all of them and is reproduced by VERIFY like any other. A finding such as duplicates is claimed only when both walks show it, so the second walk reproduces the invariant itself. Budget: two walks of 4 pages of 5 (scenarios.pagination: { pages, page_size } in the target, or false to skip), ~8 requests per endpoint. The sheet shows these under How it behaves in use; the brief sheet carries them as pagination watch-outs.

Scenarios 3 and 6, conditional requests and rate-limit accounting, run on every endpoint with a working baseline, at most four requests each: a fresh GET, the same GET with If-None-Match set to the ETag it returned, the same with a stale tag, and (when Last-Modified was sent) one with If-Modified-Since. From that sequence:

• whether If-None-Match yields 304 with an empty body, and that a stale tag still yields 200 (so the 304 was really conditional), or that the ETag is ignored, or that there are no validators at all and every poll is a full request;
• whether a 304 costs a request against the rate limit or is free;
• whether each full request decrements the remaining counter by exactly one with the reset unchanged (a fixed window), what format the reset value is in, and what the limit is.

Statements name no volatile numbers, so they stay stable across nightly runs. Disable with scenarios.conditional: false in the target. On GitHub this found that /rate_limit is not counted, that search has its own pool of 30 and no validators, and that every other endpoint honors ETags with free 304s.

Scenario 4, filter composition, runs on list endpoints that have filter parameters with known-good values in the target: one unfiltered page, one page per filter, one per pair. A filter is "honored" when every returned item matches the field of the same name; a mismatch is never read as "not honored" (GitHub's commits sha is a start ref, not an item predicate) but as "no visible effect" or "does not select by that field"; combinations are checked for containment against complete single results and for field agreement only on filters whose own check passed. Scenario 2, list vs detail, finds the detail endpoint in the spec (list path plus one template segment), reads two items' details and says which fields the detail adds, which the list alone carries, and whether shared scalar values agree, so a program knows when a second request is needed. Disable with scenarios.filters: false / scenarios.list_detail: false.

Scenario 5, sort contract, runs on list endpoints with a sort parameter: one unsorted page, then one page per sort key (the target's value plus the spec's enum, at most three) per direction, plus one repeat. It says whether each page is actually ordered by the field the key names, in the stated direction; which sorted page the default order equals; whether the two directions differ; and whether the same sort returns the same order twice. On GitHub issues it found that sort=comments&direction=asc does not return the page in ascending order of the items' comments field, while every other key and direction does. Disable with scenarios.sort: false.

Next scenarios: encoding and boundaries. All read-only, none needing an account; the write battery comes after.

Health (for the maintainer)

npm run autopia -- tool-delete --yes removes a dead tool and everything under it (versions, runs, trials, claims, diffs) in one transaction; use it for an API that turned out to be unreachable or was run once by hand and never scheduled.

npm run autopia -- health prints one line per tool that needs attention: red when a scheduled tool's last claim-producing run is older than 3 days, when the latest run kept fewer than half the claims of the best of the previous runs (or none at all), when the last three runs all failed, or when a scheduled tool has had three runs and never stored a claim; warn when only the latest run failed (one bad night is routine), when a tool has no target file and is never refreshed, or when an unscheduled tool never stored a claim. The nightly run-all prints the same block at the end and exits non-zero only on red, so a single transient failure no longer fails the workflow and a quietly decaying tool does.

Freshness

Every claim records when its evidence last reproduced. A full sheet says on which date its claims were last re-executed against the live API; a claim not re-run in the last 3 nights says so after its evidence ids; a sheet nobody has re-run for 3 nights carries a Stale banner; a run that kept fewer than half the previous successful run's clai