GoldenFlow

Moved. This repo has moved into the benzsevern/goldenmatch monorepo at packages/python/goldenflow (and packages/typescript/goldenflow)/. This repo is archived; new development happens in the monorepo.

GoldenFlow

Data transformation toolkit — standardize, reshape, and normalize messy data before it hits your pipeline. Built by Ben Severn.

Works on files (CSV, Excel, Parquet), cloud storage (S3, GCS), or live databases. Zero-config mode auto-detects what needs fixing. One command to clean what GoldenCheck found and prep what GoldenMatch needs. Available in Python and TypeScript with full feature parity.

# Python

pip install goldenflow goldenflow transform data.csv

TypeScript / Node.js

npm install goldenflow npx goldenflow-js transform data.csv

---

The Problem

Your data arrives broken in predictable ways:

• Phone numbers come in 15 different formats
• Dates are mixed between MM/DD/YYYY and YYYY-MM-DD
• Addresses have inconsistent abbreviations
• Column names don't match between systems ("fname" vs "first_name" vs "given_name")
• Values have leading whitespace, unicode garbage, smart quotes
• Categoricals are inconsistent ("USA", "US", "United States")

Every data engineer writes throwaway scripts to fix these. Every script is slightly different. None of them are reusable.

GoldenFlow makes the transforms reusable, composable, and automatic.

---

Quick Start

Python

pip install goldenflow

Auto-transform (zero-config)

goldenflow transform messy_data.csv

Try the demo first

goldenflow demo goldenflow transform demo_data.csv -c demo_config.yaml

With config

goldenflow learn messy_data.csv -o config.yaml goldenflow transform messy_data.csv -c config.yaml

Schema mapping

goldenflow map --source system_a.csv --target system_b.csv

Full pipeline

goldencheck scan data.csv goldenflow transform data.csv goldenmatch dedupe data_transformed.csv

TypeScript / Node.js

npm install goldenflow

Auto-transform (zero-config)

npx goldenflow-js transform messy_data.csv

Try the demo first

npx goldenflow-js demo npx goldenflow-js transform demo_data.csv -c demo_config.yaml

With config

npx goldenflow-js learn messy_data.csv -o config.yaml npx goldenflow-js transform messy_data.csv -c config.yaml

Schema mapping

npx goldenflow-js map -s system_a.csv -t system_b.csv

Programmatic (TypeScript)

import { TransformEngine } from "goldenflow";

const result = new TransformEngine().transformDf([ { name: " JOHN ", phone: "(555) 123-4567", email: "John@Example.COM" }, ]); console.log(result.rows[0]); // { name: "JOHN", phone: "+15551234567", email: "john@example.com" }

---

Zero-Config Mode

goldenflow transform customers.csv

or just:

goldenflow customers.csv

GoldenFlow profiles every column and applies safe transforms automatically:

• Strips whitespace and normalizes unicode
• Standardizes phone numbers to E.164 format
• Normalizes email casing
• Parses and standardizes date formats to ISO 8601
• Normalizes zip codes (zero-padding, strip +4)
• Replaces smart/curly quotes with straight quotes
• Auto-corrects categorical misspellings via fuzzy matching

Output: a clean CSV with a sidecar manifest showing every transform applied.

customers.csv           -> customers_transformed.csv

-> customers_manifest.json

The manifest is an audit trail — what changed, why, and which rows were affected.

---

CLI Commands

GoldenFlow has 14 commands. The most common ones:

# Core transforms

goldenflow transform data.csv # Zero-config: auto-detect and fix goldenflow transform data.csv -c config.yaml # Apply saved config goldenflow transform data.csv --domain healthcare # Use a domain pack goldenflow transform data.csv --strict # Fail on any transform error goldenflow transform data.csv --llm # Enable LLM-enhanced corrections goldenflow data.csv # Shorthand: auto-routes to transform

Schema & profiling

goldenflow map -s a.csv -t b.csv # Auto-map schemas between files goldenflow profile data.csv # Show column profiles goldenflow learn data.csv -o config.yaml # Generate config from data patterns goldenflow validate data.csv # Dry-run: show what would change goldenflow diff before.csv after.csv # Compare pre/post transform

Continuous & scheduled

goldenflow watch ./data/ # Auto-transform new/changed files goldenflow schedule data.csv --every 1h # Run on a schedule (5m, 1h, 30s...) goldenflow stream large_file.csv --chunk-size 50000 # Stream-process in batches

Discovery & history

goldenflow init data.csv # Interactive setup wizard goldenflow demo # Generate sample data to try goldenflow history # Show recent transform runs goldenflow history -n 50 # Last 50 runs

Integrations

goldenflow interactive data.csv # Launch TUI goldenflow serve # REST API for real-time transforms goldenflow mcp-serve # MCP server for Claude Desktop

Default Routing

Running goldenflow without a subcommand auto-routes to transform:

goldenflow customers.csv       # equivalent to: goldenflow transform customers.csv

goldenflow - # read from stdin, write to stdout

---

Streaming

For files too large to load into memory, use StreamProcessor or the stream command:

goldenflow stream large_file.csv --chunk-size 50000

from goldenflow.streaming import StreamProcessor

processor = StreamProcessor(config=config)

Process a single record

result = processor.transform_one({"name": " John ", "phone": "(555) 123-4567"})

Process a batch

result = processor.transform_batch(df_batch)

Stream a large file in chunks

for result in processor.stream_file("large_data.csv", chunk_size=10_000): write_to_output(result.df)

print(f"Processed {processor.batches_processed} batches")

---

Cloud Connectors

GoldenFlow reads from and writes to S3 and Google Cloud Storage transparently:

# S3

goldenflow transform s3://my-bucket/raw/customers.csv -o s3://my-bucket/clean/

GCS

goldenflow transform gs://my-bucket/data/records.csv

from goldenflow.connectors.s3 import read_s3, write_s3

from goldenflow.connectors.gcs import read_gcs, write_gcs

df = read_s3("s3://my-bucket/raw/customers.csv") df = read_gcs("gs://my-bucket/data/records.csv")

Cloud paths are detected automatically — no extra flags needed.

---

Watch Mode

Auto-transform files as they arrive in a directory:

goldenflow watch ./data/

goldenflow watch ./incoming/ -c config.yaml -o ./processed/

GoldenFlow polls the directory and applies transforms to any new or changed files.

---

Scheduling

Run transforms on a repeating schedule:

goldenflow schedule data.csv --every 1h

goldenflow schedule data.csv --every 30m -c config.yaml -o ./output/

Supported intervals: 30s, 5m, 1h, 2h, etc.

---

Setup Wizard

Generate a YAML config interactively:

goldenflow init data.csv

The wizard profiles your data, suggests transforms, and saves a goldenflow.yaml ready to use.

---

History

GoldenFlow tracks every transform run in ~/.goldenflow/history/:

goldenflow history         # Last 20 runs

goldenflow history -n 50 # Last 50 runs

Each run record captures: source file, row count, transforms applied, errors, and duration.

---

Schema Mapping

When you need to merge data from different systems:

goldenflow map --source crm_export.csv --target warehouse_schema.csv

GoldenFlow auto-maps columns between schemas using name similarity and data profiling:

crm_export.csv              warehouse schema

---

email_address -> email (rename) phone_number -> phone (rename) fname -> first_name (alias match) st -> state (alias match)

Ambiguous mappings get flagged for human review. Confident mappings apply automatically.

---

Domain Packs

Pre-configured transform sets for common industries. All 5 are now implemented:

goldenflow transform patients.csv --domain healthcare

goldenflow transform employees.csv --domain people_hr goldenflow transform transactions.csv --domain finance goldenflow transform orders.csv --domain ecommerce goldenflow transform listings.csv --domain real_estate

---

Transform Library (76 transforms)

Text Transforms (18)

Phone Transforms (5)

| phone_country_code | Extract country call