Clipswarm

clipswarm

Open-source, agent-native video clipping. Point any number of AI agents at YouTube videos *or live streams*, and get ready-to-post viral shorts back: 9:16, a hook header, and word-by-word captions. Runs on your machine, with no API keys and no subscription.

Made with one command: clipswarm clip https://youtu.be/75-H-i9ctkE 6 14.4 --viral --title "Astronauts celebrate July 4th from 250 miles up" Footage: NASA (public domain). No endorsement implied.

"Get me the 3 best moments from each of these 10 podcasts as shorts"

│ ├─ clipper agent 1 ─┐ ├─ clipper agent 2 ─┼─► clipswarm MCP ─► yt-dlp ─► ffmpeg ─► clips/*_viral.mp4 └─ clipper agent N ─┘ shared queue, cached transcripts, retries

Why clipswarm

OpusClip-style apps decide for you what's "viral". clipswarm gives your agent the tools to decide, then does the production work:

• Viral format built in: 1080×1920, a hook header card, the whole video frame (never cropped or zoomed), and animated captions that highlight each word as it's spoken. The layout adapts to the source's shape.
• **YouTube videos *and* live streams**: clip what just happened on a live stream (start: -60, end: "now"), or what aired at 8:41pm.
• Captions from real word timings: YouTube's auto-captions for regular videos, and local whisper.cpp for live streams. Free and offline.
• Runs on autopilot: give it a category and a schedule ("netflix stock, every 6h"), and it finds new videos, has AI pick the moments and write the hooks, and renders the clips in the background.
• Built for swarms: dozens of parallel jobs across many videos and many simultaneous agents, all through one shared queue. Transcripts are fetched once per video, and one bad job never sinks a batch.
• Fast and frugal: downloads only the seconds you clip (not the whole 3-hour stream), and uses hardware encoding on macOS.
• Works with any ffmpeg: text is rendered by clipswarm itself, so stock builds without drawtext/libass (e.g. Homebrew's) work fine.

Install

1. Dependencies (Node 20+):

brew install yt-dlp ffmpeg whisper-cpp     # macOS

Linux: pip install -U yt-dlp && sudo apt install ffmpeg (+ whisper.cpp for live captions)

npx -y clipswarm setup # checks everything; downloads the caption model (~140MB)

whisper-cpp is optional. Without it, everything works, but live-stream clips render without captions.

Keep yt-dlp up to date (yt-dlp -U / brew upgrade yt-dlp). YouTube regularly breaks older versions. This is the #1 cause of failures.

2. Add it to your AI app:

{ "mcpServers": { "clipswarm": { "command": "npx", "args": ["-y", "clipswarm", "mcp", "--out", "/Users/you/clips"] } } }

Why not claude.ai or ChatGPT on the web? Those only connect to *hosted* servers. Hosting a YouTube downloader means datacenter IP blocks, Terms of Service exposure, and shipping 50MB videos through a chat window. clipswarm runs locally, where all three problems disappear.

Use it

In Claude Code with the plugin installed:

Spawn a clipper agent for each of these videos in parallel and get me the 3 strongest moments from each as viral shorts:

Clip the last 60 seconds of https://www.youtube.com/@aljazeeraenglish/live as a viral short.

Each clipper agent reads its video's transcript, picks self-contained moments, writes a hook for each, and calls create_clips. They all run at once.

The viral format

The full frame is always shown. clipswarm never crops or zooms, because what matters in a video (a spreadsheet, a chart, a second speaker) can be anywhere in the frame. The layout follows the source:

Captions come from YouTube's auto-captions (exact per-word timing) for regular videos, and from local whisper.cpp for live streams or videos without captions. If neither is available, the clip still renders and the result includes a warnings entry explaining why.

Live streams

Paste a link to a stream that's live right now (a watch?v= or a channel's /live link). Times are relative to the live edge:

• About the last hour is available (YouTube's DVR window).
• Plain live clips are stream-copied for speed and start on the previous keyframe (≤5s early). live.from / live.to in the result give the exact wall-clock range. Use precise: true for frame-accurate cuts. Viral clips are always frame-accurate.
• end can't be in the future. Retry once it has aired.

Watches: clips that find themselves

Give clipswarm a category and a schedule. It searches YouTube for new videos, picks the best moments, and renders them as viral clips while you're away. A Mac notification appears when new clips land.

clipswarm watch add "netflix stock analysis" --every 6h          # what to look for, how often

clipswarm watch install # background job (launchd on macOS) clipswarm watch run --force # or run it right now clipswarm watch inbox # what it found and clipped

Or just ask your agent: *"Watch for new AI agent news videos every 12 hours and clip the best moments."* (MCP tools: create_watch, list_watches, run_watch, get_inbox, update_inbox_item, delete_watch, watch_scheduler.)

How it picks moments, best first:

1. AI. If the Claude Code CLI is installed, Claude reads the transcript, picks self-contained moments and writes a hook for each header (e.g. *"Netflix at $67, but this model says $127"*). This runs on your own Claude account.
2. Most replayed. YouTube's replay peaks, on videos old enough to have them (usually several days).
3. Transcript heuristic. Free and offline: it scores stretches for questions, numbers, strong claims and energy.

If YouTube's captions are missing or rate-limited, the watcher downloads just the audio and transcribes it with whisper.cpp. Anything it can't clip lands in the inbox as needs_agent for your agent to handle.

State lives in ~/.clipswarm/ (CLIPSWARM_HOME to move it). Each video is only processed once. On Linux, watch install prints a crontab line, and clipswarm watch daemon runs a foreground loop anywhere.

How it works

1. Find the moment. The agent calls get_video_info, search_transcript and get_transcript. Transcripts come from YouTube's captions via yt-dlp and are cached.
2. Cut it.

• Regular videos: yt-dlp downloads only the requested section.
• Live streams: clipswarm reads YouTube's rolling HLS playlist (5-second segments with wall-clock stamps) and fetches just the segments covering your range.

1. Make it viral. clipswarm gets word timings and draws the header and caption frames (font → vector paths → PNG). ffmpeg composites them with the full, uncropped video on a 9:16 canvas.
2. Share the machine. Every job from every agent goes through one concurrency limiter, with retries for YouTube's intermittent 403s.

Contributors: see docs/ARCHITECTURE.md for a full walkthrough of the code.

MCP tools

CLI

clipswarm clip <url> 13:53 14:28 --viral --title "His best advice in 30 seconds"

clipswarm clip -60 now --viral --title "This just happened live" clipswarm clip 1:23 1:58 --label raw-cut # plain clip clipswarm batch jobs.json --out clips # many jobs at once, writes clips/manifest.json clipswarm search stay hungry clipswarm transcript clipswarm setup

Scheduled watches

Watches search YouTube for recently uploaded videos in a category and can clip their strongest moments. Video searches and clipping happen only when you run a watch or install the background scheduler; adding a watch only saves its settings.

clipswarm watch add "AI research" --every 6h

clipswarm watch list clipswarm watch run ai-research clipswarm watch inbox clipswarm watch remove ai-research

Automatic clipping is on by default. T