ABS Data (observed)

ABS Data — Australian Bureau of Statistics data for your AI assistant

Ask the question you actually have. Add one web address to Claude, ChatGPT or Copilot, say what you care about, and your assistant finds the right ABS data, fetches it live, and explains it — with its caveats. Free, anonymous and read-only.

Start here: [https://abs-data-front-door.aicolab.workers.dev](https://abs-data-front-door.aicolab.workers.dev) — setup guides for each AI tool, things to ask, and a feedback form.

An independent, unofficial connector — not an ABS product, and not endorsed by the Australian Bureau of Statistics. Data: ABS, CC BY 4.0.

How it was built

Before offering anything, this project measured what the ABS Data API actually serves — every dataflow, by retrieval — so the connector only ever offers tables and options that have data, and a selection that doesn't exist comes back with the ones that do.

• 621,536,714 series exist across 1,227 dataflows. ABS's metadata implies 14,761,557,818 — a density of 4.21%, overstated 23.8×.
• The per-dimension metadata is exact: every advertised option exists, and nothing real is unadvertised. The entire overstatement is combinatorial.
• 81.0% of series sit in flows whose sparsity is principled and verified in the keys — almost entirely hierarchical geography encoded as separate dimensions (STATE is a function of REGION in 664 flows). The remaining 19% is not reducible to a single-dimension dependency.
• detail=serieskeysonly — the endpoint that should answer "what exists" — is broken (HTTP 200, malformed JSON). Eight of fifteen documented behaviours do not conform. See reports/.

DESIGN.md is the full record: verified API findings, every decision and why, and the corrections made along the way.

Layout

packages/schema      Drizzle schema (declared vs observed, never joined), shared CSV/period parsing

packages/crawler structural crawl, observed probe, conformance suite, sizing packages/catalogue delta report, sparsity analysis, contract generator, OpenAPI emitter, artifact packages/contract the corrected contract: stable Zod verbs + generated tables/options (per crawl) packages/worker the front door: MCP door (SDK v2) + HTTP door (oRPC + Scalar) + RPC door (Cap'n Web) on one Worker; src/ops/ holds new operations in @aicolab/app-kit's shape (DESIGN.md decision 34) packages/site the pages (landing, /admin) and the MCP App views — Solid 2 + @aicolab/ui-solid, built by Vite packages/ui-solid git submodule: the AI CoLab component system (public); packages/solid is its Solid helpers packages/sdk @abs/sdk — typed TypeScript client over the RPC door (connect(), invalidSelectionFromError()) tooling/tsconfig copy of the portal's shared tsconfigs, for the submodules (deleted at the move to app-kit) reports/ delta.md/json, sparsity.md/json, openapi.json, abs-report-draft.md data/ local SQLite catalogue (~400GB), raw archive, D1 import SQL — not committed

Running it

git submodule update --init           # packages/ui-solid and packages/solid

pnpm install pnpm db:migrate # local SQLite at data/abs-catalogue.sqlite

pnpm crawl:structural # 7 bulk requests, ~1 minute pnpm crawl:probe -- '--key-values=*' # every dataflow by retrieval; ~24h, 20GB heap for the giants pnpm report:delta && pnpm report:sparsity pnpm generate:contract # -> packages/contract/src/generated, data/d1/*.sql pnpm --filter @abs/catalogue emit:openapi

cd packages/worker npx wrangler d1 execute abs-catalogue --local --file=../../data/d1/00-schema.sql # then each data file npx wrangler d1 execute abs-catalogue --local --file=../schema/migrations/0003_public_micromacro.sql # feedback table printf 'FEEDBACK_ADMIN_TOKEN=local-dev-token\nFEEDBACK_SALT=local-dev-salt\n' > .dev.vars # gitignored pnpm dev # builds packages/site, then wrangler dev ABS_MCP_URL=http://localhost:8787/mcp FEEDBACK_ADMIN_TOKEN=local-dev-token npx tsx test/protocol.ts # ~80 checks, all doors pnpm --filter @abs/sdk example # the SDK against production ABS_URL=http://localhost:8787 npx tsx test/limits.ts # rate limits (spends a minute's budget)

The in-chat views in a dev-only MCP Apps host, against the local Worker:

pnpm --filter @abs/site dev # then http://localhost:5180/harness.html

Deploying (pnpm run deploy in packages/worker builds the site first — not pnpm deploy, which is pnpm's own command) needs, once: the feedback migration on the remote D1 (wrangler d1 execute abs-catalogue --remote --file=../schema/migrations/0003_public_micromacro.sql) and two secrets (wrangler secret put FEEDBACK_ADMIN_TOKEN, wrangler secret put FEEDBACK_SALT).

Rate limits, per client address: 600 requests/min overall; 120/min for data pulls (each is a live ABS call); 20/min for Code Mode execute; 5/min for feedback (and 20 per day, in D1). Over the limit, HTTP doors answer 429 with Retry-After; an MCP tool call gets a tool error.

Using the SDK from TypeScript:

import { connect, invalidSelectionFromError } from "@abs/sdk";

const abs = connect(); // https://abs-data-front-door.aicolab.workers.dev/rpc const [hits, cpi] = await Promise.all([ // one HTTP batch abs.searchTables({ query: "cpi rent", limit: 3 }), abs.describeTable("CPI"), ]); try { const data = await abs.getData({ table: "CPI", select: { REGION: "Sydney", INDEX: "10001" }, lastN: 4 }); } catch (err) { const invalid = invalidSelectionFromError(err); // typed: reason, dimension, validOptions from observation }

The probe is resumable: rerunning skips flows already recorded as complete. For the largest census flows run with NODE_OPTIONS=--max-old-space-size=20480.

Gotcha: wrangler keys the *local* D1 state by database_id. Changing the id in wrangler.toml (e.g. after d1 create) gives wrangler dev a fresh, empty local database — re-run the local import.

Principles

• Observed-only. Declared metadata steers the crawler and feeds the defect report; it never enters what a caller sees. Availability is empirical: observed options from the catalogue, exact combination checks live against ABS's availableconstraint (which agrees with the probe on every dimension of every flow).
• One contract, derived doors. Every crawl regenerates one contract; the MCP verbs, the HTTP routes, the OpenAPI document and the docs all read or render it. Nothing hardcodes the surface, so nothing drifts.
• Only calls that return data. A selection is verified before it is forwarded; an invalid one comes back with the observed valid options — as an MRTR input_required on MCP, a typed 422 on HTTP.