Agentic Video

agentic-video

agentic-video lets an AI agent that reads text and images, but not video, work with video files on your own machine. It is a Python library plus a stdio MCP server. The agent asks small, targeted questions ("show me 12 frames from 0:40 to 1:10", "when is 'launch' spoken?") and gets back measurements, transcripts and contact sheets: grids of frames, each with its timestamp burned into the pixels. Everything runs locally with FFmpeg, Whisper and optional local models, except two optional cloud tools (watch_for, label_sounds), which send frames or audio to Cloudflare Workers AI only when you set credentials and call them. An opt-in shadow transcription trial (off by default, owner machines only) also sends audio to Cloudflare when its switch and credentials are set.

See it work: https://agentic-video-gallery.infra-1b7.workers.dev (the gallery: real tasks, including the misses).

Try it in 2 minutes

Pick one route and follow its steps.

• Claude Code: brew install ffmpeg uv, then claude mcp add -s user agentic-video -- uvx agentic-video.
• Claude Desktop: download the .mcpb from the latest release and double-click it (macOS; run brew install ffmpeg first).
• Claude Code plugin: brew install ffmpeg uv first (the plugin's server runs uvx), then claude plugin marketplace add tlq5l/agentic-video, then claude plugin install agentic-video@agentic-video (docs/claude-code-plugin.md).
• Other clients: Run the MCP server has the JSON block.

Then paste this first prompt: "Use agentic-video: make a contact sheet of /path/to/video.mp4 and list its cuts"

• Six task recipes with real output: each has a prompt and the tool result.
• Used it for something? Tell us in Discussions.
• Found a miss? Open an issue.

Install: brew install ffmpeg on macOS, or sudo apt-get install -y ffmpeg on Debian or Ubuntu (see Install for the uv line). Then add it to Claude Code with claude mcp add -s user agentic-video -- uvx agentic-video (or pipx install agentic-video). Other clients: Run the MCP server.

Claude Desktop (one click): download agentic-video- .mcpb from the latest GitHub release and double-click it. It still needs brew install ffmpeg. The first start takes about a minute while Claude Desktop installs the Python packages. macOS only. The bundle does not include local visual search or on-screen text counting; for those, use the git install in the "Optional: visual search (CLIP) and on-screen text" section below. Details: docs/features/desktop-bundle.md.

*One contact_sheet call on a 5-second clip: six frames, each stamped with its time, as a single image the agent can read.*

In 2 minutes (macOS) your agent can answer "what happens in this video?" and "when is X said?" for a local video file. You get the length and size, a contact sheet image with timestamps, the cuts, a loudness summary and searchable speech. Nothing leaves your machine unless you use the optional cloud tools below or set the shadow transcription trial switch (off by default, owner machines only). Setup for Claude Code:

brew install ffmpeg uv

git clone https://github.com/tlq5l/agentic-video.git && cd agentic-video uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements.txt claude mcp add -s user agentic-video -e PYTHONPATH="$PWD" -- "$PWD/.venv/bin/python" -m src.executor.mcp_server

On Debian or Ubuntu, run sudo apt-get install -y ffmpeg git and curl -LsSf https://astral.sh/uv/install.sh | sh in place of the first line, then open a new shell so uv is on your PATH.

Then ask: "Use agentic-video to review /path/to/video.mp4." Other clients and details: Run the MCP server.

Status: experimental. The author uses it every day; support is best-effort. Issues and pull requests are welcome, but expect slow replies.

What it gives an agent

The MCP server exposes 25 tools (full schemas in src/adapters/mcp_schemas.json):

• Look: review_video (first look in one call: metadata, contact sheet, cuts, near-black stretches and a loudness summary), probe_video (duration, fps, size, codecs, audio), contact_sheet, overview_chunks (consecutive sheets across a long video), inspect_window / inspect_windows (dense frames for short ranges; read_text on inspect_window adds each frame's on-screen text), frame_at (one exact frame at full size, optionally cropped and enlarged to read small text; read_text: true also returns the on-screen text it reads, Apple Vision on macOS), inspect_moments (frames around given instants), refine_temporal_beam (zoom into candidate windows).
• Listen: transcribe (timestamped speech, cached; a video with no cached transcript starts a background job, so the first call returns status: in_progress and retry_after_seconds: call the same tool again after that many seconds), search_transcript (find when words are spoken: exact words first, then passages that match by meaning when the optional on-device EmbeddingGemma 2 model is set up; keyword-only without it).
• Find: detect_cuts (hard cuts and animated scene changes), check_frame_cadence (stretches where frames repeat so the picture updates less often than the frame rate, such as 30 fps padded to 60), build_visual_index + search_visual (local CLIP search for things that are visible, optional; the interval holding the top hit grows from its best frames only over neighbouring frames that score close to the best, up to 8 seconds, so it can be shorter; a span already longer than 8 seconds, such as a held frame or hits merged together, is kept whole, so it can be longer and can include lower-scoring hits). watch_for (cloud: wakes when yes/no questions about the picture start to hold, from sampled frames).
• Count and track: count_occurrences (separate appearances of an object or on-screen text; direction counts those moving one way across the frame, plus spans too brief to show motion), count_repetitions (cycles of a repeated action), track_region_changes / track_region_events (changes inside one fixed rectangle).
• Audio and sync: audio_onsets (merges hits closer than min_gap_sec, default 0.035 s), loudness_timeline, spectrogram_window, sync_check (cuts and sound cues against measured audio; each cue gives candidate_onsets_count; verbose: true gives each cue its full candidate_onsets list instead of the count). label_sounds (cloud: names the sounds in a stretch of audio).

Which tools need a Mac, Apple Vision, torch with the CLIP model or Cloudflare credentials, and where each one ran: docs/capabilities.md. Agents may pass inspect_moments moments as bare numbers or {"time": 12.5}, and region as [x, y, width, height]; both are read as the documented form. See docs/features/argument-shapes.md.

There is also a command-line checker for rendered videos: python -m src.loop check measures sync, loudness, onsets, cuts and some picture faults, then writes report.md plus evidence images. python -m src.loop cadence prints the check_frame_cadence result as JSON.

AGENTS.md is a guide for agents on which tool to use when. FEATURES.md maps each feature to its code and tests.

Install (macOS or Linux)

You need FFmpeg, Git and uv. Tested with Python 3.12 on Apple silicon (macOS) and on Debian 12 (x86_64).

macOS (Homebrew):

brew install ffmpeg uv

git clone https://github.com/tlq5l/agentic-video.git cd agentic-video uv venv --python 3.12 .venv uv pip install --python .venv/bin/python -r requirements.txt pytest jsonschema .venv/bin/python -m pytest -q # optional: about 3 minutes

Debian or Ubuntu (apt and the uv installer; open a new shell after the uv line so uv is on your PATH):

sudo apt-get install -y ffmpeg git

curl -LsSf https://astral.sh/uv/install.sh | sh git clone https://github.com/tlq5l/agentic-video.git cd agentic-video uv venv --python 3.12 .venv uv pip install --python .venv/bin/python -r requirements.txt pytest jsonschema .venv/bin/python -m pytest -q # optional: about 3 minutes

The test run is optional for users.

To lint before every commit (uvx ruff check ., the core rules in ruff.toml), turn the hook on once per clone: git config core.hooksPath .githooks.

What is macOS-only:

• On-screen text matching in count_occurrences (text=) uses Apple Vision through pyobjc.
• The bench-v1 harness uses sandbox-exec; some tests and generators use say. Tests that need a missing tool skip themselves.

The core tools run on Linux. A fresh install of 0.3.1 on Debian 12 (x86_64, ffmpeg 5.1) passed the probe, frame and speech-to-text checks; other distributions and CPUs are untested.

Optional: faster, more accurate speech (whisper.cpp turbo)

By default transcribe uses faster-whisper on the CPU (tiny, base or small; weights are downloaded on first use). On Apple silicon, whisper.cpp with the large-v3-turbo model is much faster and more accurate:

brew install whisper-cpp

mkdir -p ~/.cache/whisper.cpp curl -L -o ~/.cache/whisper.cpp/ggml-large-v3-turbo-q5_0.bin \ https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo-q5_0.bin

Set AGENTIC_VIDEO_WHISPER_CPP_MODEL if you keep the model somewhere else. When it is installed, turbo becomes the default model.

Optional: visual search (CLIP) and on-screen text

uv pip install --python .venv/bin/python -r requirements-visual.txt

.venv/bin/python -c 'from huggingface_hub import snapshot_download; snapshot_download(repo_id="openai/clip-vit-base-patch32", revision="3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268", allow_patterns=["*.json", "*.txt", "pytorch_model.bin"])'

This installs PyTorch, Transformers and (on macOS) the Apple Vision bindings, and downloads about 605 MB of CLIP weights. The visual search only loads local files and never calls a paid API. Without these, search_visual and build_visual_index return an error that explains the setup.

Optional: cloud tools (watch_for, label_sounds)

watch_for and label_sounds send video frames or audio to Cloudflare Workers AI (the Clef models). They are optional: the other tools never call Cloudflare, except the off-by-default shadow transcription trial (see the trial switch below). Do not use them on recordings you need to keep private. Set these in the server's environment:

• CLOUDFLARE_ACCOUNT_ID: your account id.
• CLOUDFLARE_API_TOKEN: a token with Workers AI access. Or set AGENTIC_VIDEO_CF_TOKEN_COMMAND to a command whose last printed line is the token (for example, wrangler auth token); the command runs at most once every 50 minutes per process, and each detached shadow child keeps its own token cache.

Without them the two tools fail with a message naming what is missing, and nothing is sent.

Trial switch (maintainer only): AGENTIC_VIDEO_SHADOW_CF_ASR set to this machine's hostname SENDS AUDIO TO CLOUDFLARE after every fresh local transcription, to compare it with the local result in a log. The user's transcript never changes. It is off unless that variable names the machine, and it stops by itself after 14 days or 200 uses. Details: