datoon

datoon

smart structured-data→TOON gateway — converts only when it actually saves tokens

Before/After • Install • What You Get • How It Works • Benchmarks • Full install guide

---

Raw structured data is often verbose in LLM prompts. TOON can save tokens — but blind conversion can also make payloads *worse*. datoon adds a decision layer: convert when structure and savings justify it, skip when they don't, and always explain why.

Supports JSON, CSV, JSONL, YAML, XML, Parquet, Avro, ORC, Excel, and Apple Numbers — auto-detected from file extension.

Before / After

JSON in the prompt (43 tokens)

{"users":[

{"id":1,"name":"Ada","role":"admin"}, {"id":2,"name":"Lin","role":"analyst"}, {"id":3,"name":"Grace","role":"viewer"} ]}

datoon converts → TOON (24 tokens)

users[3]{id,name,role}:

1,Ada,admin 2,Lin,analyst 3,Grace,viewer

{"decision":"convert","reason":"Estimated savings 44.19% (threshold 15.00%)."}

CSV from a data pipeline (111 tokens as JSON)

id,name,role

1,Ada,admin 2,Lin,analyst 3,Grace,viewer

datoon auto-converts → TOON (24 tokens)

datoon data.csv --report-stdout

Same result. Zero JSON serialization in your code.

Non-uniform payload (26 tokens)

{"config":{"debug":true},"tags":["a","b"]}

datoon skips → keeps JSON

{"decision":"skip","reason":"No uniform object arrays found with at least 3 rows."}

No Node.js call. No silent corruption.

Same data. Right format. Always explained.

┌──────────────────────────────────────────────────┐

│ PAYLOAD SAVINGS (auto avg) ████░░░░░░ 28% │ │ PAYLOAD SAVINGS (agent skill) ████████░░ 62% │ │ DECISION ACCURACY ██████████ 100% │ │ HARMFUL CONVERSIONS BLOCKED ██████████ 100% │ └──────────────────────────────────────────────────┘

[!IMPORTANT]

datoon saves payload tokens — the structured data portion of your prompt. Token savings depend on payload shape: uniform tabular data converts well; deeply nested or non-uniform structures are skipped. Every decision includes a reason so pipelines can log, debug, and trust the outcome.

Install

# core (JSON, CSV, JSONL, XML — no extra deps)

uv add datoon pip install datoon

with YAML support

pip install "datoon[yaml]"

with Excel support

pip install "datoon[excel]"

with Parquet / ORC / Avro support

pip install "datoon[columnar]"

with Apple Numbers support

pip install "datoon[numbers]"

with tiktoken-based token counting

pip install "datoon[tokens]"

with MCP server

pip install "datoon[mcp]"

everything

pip install "datoon[all]"

Requires Python 3.12+. TOON conversion requires Node.js with npx in PATH — analysis and format reading work without it.

For Claude Code plugin, Codex, and MCP config → INSTALL.md.

What You Get

Supported input formats

How It Works

1. Detect format — from --format flag, file extension, or default to JSON for stdin
2. Read + normalize — parse source into list of row dicts; serialize to compact JSON
3. Analyze structure — uniform object arrays? acceptable depth? minimum rows?
4. Gate early — non-candidates skip before any CLI call; no Node.js overhead
5. Convert + estimate — TOON CLI runs, token savings calculated
6. Gate savings — below threshold → return JSON; above → return TOON with report

Every path returns a ConversionReport with decision, reason, and token estimates. Pipelines never get silent surprises.

---

Quick Start

JSON (stdin):

echo '{"users":[{"id":1,"name":"Ada"},{"id":2,"name":"Lin"},{"id":3,"name":"Grace"}]}' | datoon --report-stdout

CSV (auto-detected from extension):

datoon data.csv --report-stdout

JSONL:

datoon data.jsonl -o output.toon

YAML (requires `datoon[yaml]`):

datoon data.yaml --report-stdout

Parquet (requires `datoon[columnar]`):

datoon data.parquet --report ./report.json

Explicit format override:

datoon --format csv < data.csv --report-stdout

Force conversion (bypass gating — for experiments):

datoon data.json --force --report-stdout

---

Python API

JSON conversion:

from datoon import convert_json_for_llm, ConversionConfig, DatoonError

config = ConversionConfig(min_savings_ratio=0.15, max_depth=6, min_uniform_rows=3)

try: outcome = convert_json_for_llm(raw_json, config) except DatoonError as exc: raise

outcome.payload_text — TOON or original JSON

outcome.report.decision — "convert" | "skip"

outcome.report.reason — human-readable explanation

send_to_model(outcome.payload_text)

Any format via `read_tabular`:

import json

from pathlib import Path from datoon import read_tabular, convert_json_for_llm, ConversionConfig

text formats: csv, jsonl, yaml, xml

rows = read_tabular("csv", text=csv_string)

binary formats: excel, parquet, orc, avro, numbers

rows = read_tabular("parquet", path=Path("data.parquet"))

json_text = json.dumps(rows, separators=(",", ":")) outcome = convert_json_for_llm(json_text, ConversionConfig()) send_to_model(outcome.payload_text)

Structure-only analysis (no Node.js required):

from datoon.analyzer import analyze_payload

from datoon.models import ConversionConfig

analysis = analyze_payload(parsed_data, ConversionConfig()) print(analysis.is_candidate, analysis.reason)

---

MCP Server

datoon ships an MCP server with three tools:

Claude Desktop / Cursor / Windsurf config:

{

"mcpServers": { "datoon": { "command": "uvx", "args": ["--from", "datoon[mcp]", "datoon", "mcp"] } } }

Run locally:

datoon mcp     # or the standalone script: datoon-mcp

Listed on the MCP Registry, Smithery, and Glama. See MARKETPLACES.md.

---

Claude Code Plugin

Install directly from GitHub:

claude plugin marketplace add andrii-su/datoon

claude plugin install datoon@datoon

Trigger in-session:

/datoon

convert this JSON to TOON if it saves tokens use datoon mode for structured data

---

CLI Reference

Format is auto-detected from file extension. Use --format to override or when reading from stdin.

---

Benchmarks

PYTHONPATH=src python benchmarks/run.py --dry-run

PYTHONPATH=src python benchmarks/run.py PYTHONPATH=src python benchmarks/run.py --update-readme

Why auto mode outperforms forced conversion

Auto mode avoids low-benefit and high-risk payloads (orders-nested, mixed-non-uniform) while matching forced TOON's average token count on suitable ones. Every decision comes with a reasoned report.

*Forced conversion succeeded for 5/5 payloads.*

Format conversion benchmark

Token savings when converting from common structured formats (CSV, JSONL, XML, YAML). Baseline is the JSON representation of the same data — what an LLM would receive without datoon.

| Dataset | Format | JSON Tokens | TOON (forced) | Auto | Auto Tokens | Auto Saved | |