SonicCraft

SonicCraft

A declarative, deterministic audio engine for AI agents. Describe a production as a YAML (or JSON) timeline: who says what, in which order, over which music, at what loudness. SonicCraft compiles it into TTS speech, music beds and sound effects, mixed with sidechain ducking, mastered to an EBU R128 loudness target, plus WebVTT/SRT subtitles that match the audio it actually rendered. It ships as a CLI, an MCP server and an Agent Skill.

[Website](https://user-github-me.github.io/soniccraft/) · Quickstart · Use it from an AI agent · MCP server · Agent Skill

Use it from an AI agent

Give an agent (Claude Code, Codex, Cursor, Gemini CLI, ...) the link to this repository and ask for audio, for example *"Use https://github.com/user-github-me/soniccraft to make a 30-second voiceover with background music for my product video."* Everything it needs is below and in AGENTS.md.

1. Install the CLI (prebuilt and checksum-verified, no Rust needed). Homebrew, Scoop, Docker and Cargo work too: see Quickstart.

   curl -fsSL https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.sh | sh   # macOS, Linux

   irm https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.ps1 | iex        # Windows

1. Write a timeline. soniccraft init my-audio starts one that renders offline: macOS voices on a Mac, elsewhere a mock test voice (not real speech). Real voices anywhere: Kokoro or Piper (free, local) or an ElevenLabs, OpenAI or Cartesia key, see Providers. soniccraft init --list shows the templates, soniccraft providers the voices usable here, and soniccraft schema the JSON Schema; the Agent Skill explains the format.
2. Check it, see the cost, render it:

   soniccraft validate my-audio/timeline.yaml           # errors with a code, line, column and fix hint

soniccraft plan my-audio/timeline.yaml # free dry run: timing, ducking, characters to synthesize soniccraft render my-audio/timeline.yaml --preview # mastered audio, subtitles, manifest, preview page

To keep SonicCraft in an agent for good, add the skill and the MCP server. Both use the CLI from step 1; with Node.js 20 or newer the MCP server also runs with nothing installed: use npx -y soniccraft-mcp in place of soniccraft mcp below.

The server is also in the official MCP Registry as io.github.user-github-me/soniccraft (container image). Details: MCP server and Agent Skill.

Why

Ask an agent for a two-voice explainer with background music and it will usually hand-write an ffmpeg filter graph: adelay offsets computed from guessed speech lengths, an amix, perhaps a half-remembered sidechaincompress. That fails in predictable ways:

• Timing drift and overlaps. Synthesized speech is never exactly as long as the agent guessed, so lines collide or leave dead air, and every script edit shifts every later offset.
• No ducking. Sidechain compression is fiddly to get right in a filter graph, so the music either drowns the voice or sits uniformly too low.
• Loudness disparity. Different providers and voices come out at different levels. Nothing measures integrated loudness or true peak, so the result is uneven and can clip.
• Drifting subtitles. Captions are written from estimated times instead of from the audio that was rendered.
• Opaque, expensive iteration. A broken graph is a cryptic ffmpeg error, and every retry re-bills every TTS call.

How SonicCraft works

HyperFrames showed that agents make better video when they write declarative HTML and a deterministic renderer does the frame-accurate work. SonicCraft applies the same idea to audio. The agent writes intent as data; a compiler validates it, measures the real durations, resolves all timing in integer sample frames, and renders natively in Rust. The same timeline, TTS cache and binary always produce byte-identical audio, subtitles and manifests.

version: "1.0"

output: format: "mp3" sample_rate: 48000 bitrate: "320k" target_lufs: -16.0 # integrated loudness target, measured per EBU R128 / ITU-R BS.1770 export_subtitles: true # Emits matching .vtt and .srt

tracks:

• id: ambient_bed type: "music" source: "assets/audio/synthwave_loop.wav" loop: true volume: 0.65 fade_in_ms: 1200 fade_out_ms: 2500 ducking: trigger_tracks: ["voice_primary", "voice_secondary"] reduction_db: -18.0 attack_ms: 80 release_ms: 350

• id: voice_primary type: "speech" pan: -0.15 # Subtle left stereo placement sequence:
• id: cue_01 speaker: "narrator" provider: "elevenlabs" voice_id: "21m00Tcm4TlvDq8ikWAM" text: "Autonomous software development requires deterministic compilation targets."
• pause_duration: 350ms
• id: cue_03 speaker: "narrator" provider: "elevenlabs" voice_id: "21m00Tcm4TlvDq8ikWAM" text: "By separating high-level planning from digital signal processing, systems eliminate runtime hallucinations."

• id: voice_secondary type: "speech" pan: 0.15 # Subtle right stereo placement sequence:
• id: cue_02 start_relative: after_cue: "cue_01" offset_ms: 150 speaker: "guest" provider: "local_kokoro" voice_id: "am_adam" text: "Exactly. SonicCraft treats audio composition the same way Hyperframes treated web-based video."

• id: sfx_impact type: "sfx" source: "assets/sfx/whoosh.wav" volume: 0.8 start_at_ms: 0

This is examples/tech-briefing/timeline.yaml. Here is what soniccraft render does with it:

1. Validates everything at once. Unknown keys, misspelled enum values, dangling anchors, missing voices, missing files and missing API keys are all reported together, each with a stable code, a path, a line and column, and a fix hint.
2. Synthesizes and caches. cue_01 and cue_03 go to ElevenLabs (with word timestamps); cue_02 goes to a local Kokoro-FastAPI server (local_kokoro is an alias of the built-in kokoro provider). Each result is cached by a content hash, so later renders only pay for lines that changed.
3. Levels the voices. Each cue is trimmed of leading and trailing silence and normalized to -20 LUFS, so the cloud voice and the local voice sit at the same level.
4. Lays out the timeline in sample frames. cue_01 starts at 0. cue_02 starts 150 ms after cue_01 ends. cue_03 is sequential, 350 ms after cue_01, but that slot collides with cue_02, so the default overlap policy (push) automatically moves `cue_03` to 350 ms after `cue_02` ends, keeping its pause as the gap, and reports an info cue_pushed. The result is a natural three-turn exchange with no timing arithmetic by the agent.
5. Places the music. The bed is a looping clip without a length, so it runs to the end of the timeline with a 1.2 s fade-in; the timeline ends 2.5 s after the last word (the bed's fade_out_ms) and the bed fades out over exactly that tail. The whoosh fires at 0.
6. Ducks the bed by 18 dB whenever either voice speaks. Detection runs on the voices' own signal with an 80 ms lookahead (it defaults to the attack time), so the bed is already down when the first syllable lands; it recovers over 350 ms, and the default 250 ms hold bridges the 150 ms turn gap.
7. Pans the two voices to -0.15 and +0.15 (constant power).
8. Masters the mix to -16 LUFS integrated with a -1.0 dBTP true-peak ceiling (from the default podcast preset), iterating loudness gain and a 4× oversampled true-peak limiter until within 0.2 LU (the closest pass is kept and loudness_target_missed says by how much when no pass gets there; an unreachable target gives the loudest master within 6 dB of average limiting, whatever the target).
9. Encodes a 48 kHz, 320 kbps MP3 natively, decodes and measures the delivered file (re-limiting if the codec pushed a peak over the ceiling), and writes timeline.vtt and timeline.srt from the placed word timings, labelled narrator and guest.

Before spending anything, soniccraft plan (alias dry-run) shows the same layout from estimated durations without calling a TTS API (abridged JSON, estimated numbers):

{ "valid": true, "duration_ms": 20616.9, "estimated": true,

"items": [ { "id": "cue_01", "kind": "cue", "start_ms": 0.0, "end_ms": 4838.7, "duration_source": "estimate" }, { "id": "cue_02", "kind": "cue", "start_ms": 4988.7, "end_ms": 10863.7, "duration_source": "estimate" }, { "id": "cue_03", "kind": "cue", "start_ms": 11213.7, "end_ms": 18116.9, "duration_source": "estimate", "pushed_from_ms": 5188.7 } ], "synthesis": { "cues": 3, "cached": 0, "characters_to_synthesize": 276 } }

Features

• Timeline DSL v1.0 in YAML or JSON with a published JSON Schema. Strict validation: unknown keys are errors with did-you-mean hints; every problem is reported at once with a stable code, path, line/column and a mechanical fix.
• Timing without arithmetic: sequential cues and pauses, absolute starts, and relative anchors (after_cue, with_cue, before_cue, with signed offsets) on cues, clips or whole tracks. Speech overlaps are pushed apart automatically (or warned about, rejected, or allowed per cue for crosstalk).
• Eight TTS providers: ElevenLabs, any OpenAI-compatible /audio/speech server, Cartesia, Kokoro (Kokoro-FastAPI), Piper, macOS say, any local command, and a deterministic mock voice. Mix them freely in one timeline. Speakers registry with layered voice settings.
• Cheap iteration: a content-addressed TTS cache, dry runs that never call paid APIs (with estimated durations and per-provider character counts), bounded concurrency and retries with backoff.
• Music and sound effects: loops (seamless, or with loop crossfades), trims, fades with three curves, crossfades between clips, open-ended beds that end with an automatic fade-out tail.
• Sidechain ducking driven by the actual speech signal (or by the timeline), with attack, rele