Dataparity

Dataparity

Cross-database data diff and reconciliation for AI agents, with a portable row hash.

Compare two tables — in the same database or in two different ones — and get back what changed, which columns changed, and which slice of your data the changes concentrate in. Runs as a CLI, an MCP server, and a library. MIT licensed.

What it does · Why the hash is portable · Install · CLI · MCP · Library · How it works · Status

---

What it does

Give it two tables and a key, and it tells you what changed:

HIGH  3 difference(s) found across 5 row(s) (60.00% of the left side).

left pg :: public.orders_prod (5 rows) right mysql :: analytics.orders_staging (5 rows) strategy checksum keys id

added 1 removed 1 modified 1 unchanged 3

where the changes are concentrated: country = DE 4 change(s) 80.0% of changes 6.7x over-represented

sample changes: removed key=2 modified key=3 amount added key=6

signals:

• 1 row(s) exist only on the left
• 1 row(s) exist only on the right
• 1 row(s) differ in at least one column

Three things make it different from a SELECT EXCEPT you write yourself:

1. It works across two different databases. Postgres ↔ MySQL, Postgres ↔ DuckDB, Parquet ↔ Postgres. Both sides compute the same canonical row hash *inside their own engine*, so only hashes cross the wire — never row data.

1. It says where the changes are, not just that there are changes. A diff that reports "432 rows differ" is nearly useless. A diff that reports "98.6% of the modified rows are German orders" tells you the VAT rule landed and nothing else did.

1. It is built for agents. An MCP server with four read-only tools, a stable JSON result contract, and a risk verdict (none / low / medium / high / critical) that a coding agent can act on.

Why the hash is portable

Comparing rows across engines means agreeing on what a value *is*. 0.1 in Postgres and 0.1 in MySQL are both the double closest to one tenth, and 0.1 + 0.2 is not 0.3. A boolean is true in Postgres and 1 in MySQL. A timestamp has microsecond precision in one engine and second precision in another. A NULL is not an empty string.

Dataparity solves this with two rules, applied identically in every engine:

• Every value is tagged. NULL renders as N; every other value renders as V followed by its normalized text. NULL and '' can never be confused, and a real value can never be mistaken for NULL.
• Every value is length-prefixed before it is joined into a row hash. A text value containing the field separator cannot shift a column boundary and forge a match.

Floats are scaled to a fixed number of decimals *in exact decimal arithmetic* before they are rendered, so 0.1 + 0.2 and 0.3 agree. Timestamps are normalized to UTC and truncated to seconds. Booleans render as true/false everywhere.

The result is an MD5 row hash that is byte-identical across engines. This is not a claim in a README — it is asserted by test/integration/canonical.test.ts, which writes the same awkward values into Postgres, MySQL and DuckDB and requires the hashes to match:

docker run -d --name pgdiff -e POSTGRES_PASSWORD=test -e POSTGRES_DB=difftest -p 55432:5432 postgres:16-alpine

docker run -d --name mydiff -e MYSQL_ROOT_PASSWORD=test -e MYSQL_DATABASE=difftest -p 53306:3306 mysql:8 npm run test:integration

Those tests need the databases. When one is unreachable they report themselves as *skipped* rather than passed, so a green run never means "nothing was tested" — and CI fails the job if anything was skipped.

Install

npm install -g dataparity

Or run it without installing:

npx dataparity --help

Requires Node.js 22.16 or newer.

CLI

Configure your sources once, in ~/.dataparity/config.json (or point DATAPARITY_CONFIG at another path):

{

"sources": [ { "name": "prod", "type": "postgres", "connectionString": "postgres://..." }, { "name": "staging", "type": "postgres", "connectionString": "postgres://..." }, { "name": "mysql", "type": "mysql", "host": "127.0.0.1", "port": 3306, "user": "readonly", "password": "...", "database": "analytics" }, { "name": "local", "type": "sqlite", "path": "data/app.db" }, { "name": "files", "type": "duckdb", "files": ["data/orders.parquet"] } ] }

Then diff:

# Two tables in the same database

dataparity diff \ --left prod --right staging \ --left-table public.orders_prod --right-table public.orders_staging \ --key id

Two tables in two different databases

dataparity diff \ --left prod --right mysql \ --left-table public.orders_prod --right-table orders_staging \ --key id --json

Explore first

dataparity tables --source prod dataparity describe --source prod --table public.orders_prod

Exit codes follow the GNU diff convention: 0 identical, 1 differences found, 2 error. That makes it drop straight into a CI check.

Credentials can come from the environment instead of the file, so they never have to be written to disk:

export DATAPARITY_PROD_CONNECTION_STRING='postgres://...'

export DATAPARITY_MYSQL_PASSWORD='...'

MCP

claude mcp add --scope user dataparity -- npx -y dataparity mcp

Or in .cursor/mcp.json:

{

"mcpServers": { "dataparity": { "command": "npx", "args": ["-y", "dataparity", "mcp"] } } }

Four read-only tools:

Tools take source *names*, never connection strings — credentials never pass through the model. Every tool is read-only; dataparity never writes to your database.

Library

import { diffTables, PostgresSource, MysqlSource } from 'dataparity';

const prod = await PostgresSource.connect({ connectionString: process.env.PROD_URL! }); const staging = await MysqlSource.connect({ host: '127.0.0.1', port: 3306, user: 'ro', password: '...', database: 'analytics', });

const report = await diffTables(prod, 'public.orders', staging, 'orders_staging', { keyColumns: ['id'], });

if (report.risk.level === 'critical') { console.error(report.risk.signals.join('\n')); process.exit(1); }

How it works

Same database → a single in-database join. The three-way classification (added, removed, modified) is one anti-join plus one outer join, entirely inside the database. Row data never leaves it, and the result is exact.

Different databases → canonical hashes, compared in buckets. Each side computes a per-bucket count and two independent checksums inside its own engine. Only buckets whose aggregates disagree are expanded down to the row level, so a billion-row table that differs in a few rows transfers a few hashes, not a billion rows.

Attribution. For each candidate column, the distribution of that column among the changed rows is compared to its distribution across the whole table. A value that is much more common among the changed rows than expected points at the affected slice. The score is descriptive ("over-represented"), never causal.

Risk. The verdict weighs the size of the change against the schema difference. A removed or retyped column alongside added or removed rows is critical; a small percentage of changed rows with no schema change is low.

Supported engines

A boolean and a 0/1 integer are treated as the same logical type, so a boolean column in Postgres compares correctly against the integer column SQLite has no choice but to use.

Status

Early. The core is solid and tested against live Postgres, MySQL and DuckDB, but the surface is small and the edges are sharp.

What works today

• Same-database diffs with exact column-level detail
• Cross-database diffs between Postgres, MySQL, SQLite and DuckDB
• Schema comparison, change attribution, risk verdicts
• CLI, MCP server and a typed library API
• Read-only by construction; credentials never reach the model
• Cross-database diffs localise *which columns* changed, by comparing a per-column hash computed inside each engine — so the column detail survives the hash comparison without any value crossing the wire

Known limits

• Cross-database diffs report *which columns* changed, but not the before/after *values*: only hashes cross the wire, so the value itself never leaves either engine. A same-database diff carries the values too.
• JSON columns are compared as text, so key order differences read as changes. Compare them structurally instead.
• Floats and decimals are compared to six decimal places. Beyond that, two engines may disagree on the last bits.
• The where clause is a raw SQL fragment. It is passed to your engine, not to the model, but treat it as trusted input.
• No monitors, no lineage graph, no dbt integration, no UI.

Contributing

docs/design.md is the source of truth for the scope and the key decisions behind them. Read it before changing behaviour — it records the reasoning, not just the outcome.

The workflow is:

npm ci

npm test # unit tests npm run test:integration # needs live Postgres + MySQL; skips if absent npm run test:audit # every engine x every option x every error path npm run lint npm run typecheck npm run build

The integration suite needs Postgres 16 on :55432 and MySQL 8 on :53306:

docker run -d --name pgdiff -e POSTGRES_PASSWORD=test -e POSTGRES_DB=difftest -p 55432:5432 postgres:16-alpine

docker run -d --name mydiff -e MYSQL_ROOT_PASSWORD=test -e MYSQL_DATABASE=difftest -p 53306:3306 mysql:8

License

MIT. See LICENSE.