Misata: synthetic data library

Misata

Declare what must be true. Get data that makes it true. Prove it.

Misata is a declarative synthetic data generator for Python. You write down the facts a dataset has to satisfy, from a monthly revenue curve to a balanced ledger to a foreign key that must never dangle, and the engine builds multi-table data that satisfies all of them at once. Then DuckDB checks the files on disk and writes a certificate anyone can re-run.

Docs · Use cases · The language · Limitations · Studio · Changelog

---

Sixty seconds

pip install "misata[duckdb]"

# misata.yaml

seed: 7 tables: customers: rows: 2000 columns: customer_id: {type: int, unique: true, min: 1, max: 2000} name: {type: text, text_type: name} signed_up: {type: date, start: "2024-01-01", end: "2024-12-31"} orders: rows: 40000 columns: order_id: {type: int, unique: true, min: 1, max: 1000000} customer_id: {type: foreign_key} channel: {type: categorical, choices: [web, app, store]} amount: {type: float, min: 5, max: 900} ordered_at: {type: date, start: "2025-01-01", end: "2025-12-31"} relationships:

• customers.customer_id -> orders.customer_id

outcome_curves:

• table: orders column: amount time_column: ordered_at start_date: "2025-01-01" curve_points:
• {month: 1, target_value: 180000}
• {month: 6, target_value: 240000}
• {month: 12, target_value: 310000}

group_shares:

• table: orders measure: amount group_column: channel shares: {web: 0.5, app: 0.3, store: 0.2}

misata generate -c misata.yaml -o data --attest

python data/verify.py

Attestation: data

result: PASSED files pinned: 2 invariants held: 69 of 69

ok 2 file(s) match their attested sha256 ok 69 of 69 invariant(s) hold

The 69 are derived from the YAML before a row exists: twelve monthly revenue totals (180,000.00 in January, 240,000.00 in June, 310,000.00 in December, straight lines between), the channel split inside every one of those months, the planned row count per month, zero orphaned customer_ids and unique keys. verify.py needs only the standard library and duckdb, and it shares no code with the generator.

The YAML above, generated and summed by month and channel. Each bar is a month of generated revenue split by channel; the cap is the total the YAML declares. Drawn by readme_charts.py from a fresh run of this README's own YAML.

Change one cent in one row and it says so:

FAIL  orders.csv: sha256 301165f9dc4e does not match the attested 670f89195826

FAIL orders.amount.month:1.outcome_curve_period: expected 180000.00, observed 180000.01 FAIL orders.amount.outcome_curve_total: expected 2945000.00, observed 2945000.01 FAIL orders.amount.web.2025-01.group_share_period: expected 90000.00, observed 90000.01 FAIL orders.amount.web.group_share_total: expected 1472500.00, observed 1472500.01 FAIL 65 of 69 invariant(s) hold

---

Why declare instead of generate

Every test fixture has an oracle problem. To check a dbt model, a Spark job, a dashboard tile or a data agent you need input data and an answer you know independently of the code under test. Copying production gives you data without an answer. Hand-written fixtures give you an answer that rots. Random generators give you data whose answer you have to compute with the code you are testing.

Misata turns it around. The specification is the ground truth and the rows are derived from it. "January revenue is 180,000" is true before a single row exists, so your pipeline either returns 180,000.00 or it has a bug.

flowchart LR

A["Declare curves, shares, ledgers, keys, lifecycles"] --> B{"Can they all hold?"} B -- "no" --> R["Refused, with the arithmetic and a fix"] B -- "yes" --> C["Build rows closed form where it can, by construction elsewhere"] C --> D["Write CSV, Parquet, JSONL, DuckDB, Postgres"] D --> E["Attest DuckDB re-runs every derived invariant"] E --> F["Certificate in-toto Statement plus verify.py"]

Three properties make this different from a generator with options:

• A declaration is honoured or refused, never quietly replaced. Declarations that cannot all hold stop generation with an error that names each one, shows the arithmetic and suggests the change.
• The certificate is checked by code that did not write the data. A bug in Misata's arithmetic shows up as a failed invariant, not a self-consistent wrong answer.
• The same spec, seed and version give the same bytes on Linux, macOS and Windows, so a fixture can live in git with its fingerprint.

---

Where this is going

Misata's aim is that nobody writes a Python script to make a dataset, of any kind. You say what has to be true; the engine works out the rows, refuses what cannot hold, and proves what it built.

It is not there yet, and it is getting there brick by brick. Each release adds things you can declare instead of programming, and a brick is laid only when it passes three tests: it can be declared in a few lines of YAML, it is refused before generation when it contradicts another declaration, and it is re-checked on the rows afterwards by code that did not write them.

Every brick is something you declare instead of code. Drawn from misata/registry.py , the list a contract test holds the code to, so the wall cannot claim a brick the engine does not have.

One fact, two ways. What a script has to do to make a fact true, and what you write instead:

What still takes code, and where each one stands:

LIMITATIONS.md keeps the full list. When a line leaves this table, it is because the brick that replaces it has passed all three tests.

---

The evidence

The Gauntlet: 13 tables and 136 SQL assertions run by DuckDB. The Faker script ties wherever a fact is about one column at a time and loses wherever columns, rows or tables have to agree. Source , rerun with python -m benchmarks.gauntlet .

<img src="https://raw.githubusercontent.com/rasinmuhammed/misata/main/docs/assets/readme/detection-light.svg" width="680" alt="Detection AUC, lower is better, 0.5 means a classifier cannot tell generated rows from real ones. Olist orders: Misata from a declared schema 0.63, Misata from a sentence 0.73, Faker script 0.78, SDV fitted to the real data 0.71, Misata mimic 0.54. NYC taxi trips: 0.