Marey

Marey

Give AI agents eyes for motion.

A screenshot tells an agent what the screen *looks like*. Marey tells it what *changed*.

Marey is a Model Context Protocol server that turns a short screen interaction into a timestamped, agent-readable contact sheet — a grid of still frames an agent can inspect as a temporal sequence, with any frame retrievable at full resolution.

claude mcp add marey -- npx -y @anilyesilkaya/marey

That's the whole install. Zero runtime dependencies, nothing to clone.

---

See it work

You: "When I click More, the project menu opens but most of its text gets cut off. Can you find out why?"

Agent: "Let me record the interaction with Marey." *(records while you reproduce the click)*

The bug: clicking More opens a readable menu, but a layout update makes it so narrow that most of each action label is cut off.

The contact sheet preserves the failure in three frames: before the click → readable menu → clipped text. The red outline highlights the broken menu width.

An illustrative agent response:

Agent: "At 0.36 s, I can read all three actions. By 0.48 s, the menu has shrunk to roughly the More button's width and the labels are clipped. It stays open throughout. I'd inspect the post-open sizing code: it may be using the trigger's width instead of the menu's content width."

The sequence shows the menu opens at a readable width, then narrows while it remains open. This directs the agent to the post-open layout measurement; source inspection confirms the cause.

Staged demonstration: a browser fixture rendered at fixed timestamps and composed with Marey's own contact-sheet code. The headings, outline, and callout are explanatory annotations added for the README; the full contact sheet contains the raw frames. The fixture reproduces a sizing pass that incorrectly uses the trigger's width for the menu. Regenerate both images with node demo/make-contactsheet.mjs. Requires Chrome, Chromium, or Edge; set CHROME_BIN if it isn't found automatically.

For containers without browser sandbox support, use node demo/make-contactsheet.mjs --no-sandbox.

---

Why Marey?

AI coding agents are good at understanding screenshots, but motion is still awkward. A bug such as:

• a control point jumping while it is dragged,
• a menu flashing and immediately closing,
• a canvas updating in the wrong order,
• an animation stuttering between states,

cannot be understood from a single screenshot. Marey bridges that gap: because it speaks MCP, the agent requests the recording itself and receives the resulting image directly in context.

No GIF inspection. No manually extracting frames. No dragging a dozen screenshots into chat.

It is named after Étienne-Jules Marey, a pioneer of chronophotography — the study of motion through sequences of images. Marey applies the same idea to AI agents: instead of handing a model a video it cannot reliably inspect frame by frame, it converts motion into a visual sequence the model can reason about.

---

How it works

1. An MCP client asks Marey to record the screen.
2. Marey captures frames at a chosen frame rate.
3. Each frame is numbered and timestamped.
4. Marey composes the frames into a contact sheet.
5. The contact sheet is returned to the agent as MCP image content.
6. Raw frames remain available on disk for closer inspection or re-stitching.

The idea is deliberately simple:

motion becomes one image containing time.

---

Tools

Marey exposes ten MCP tools:

record

The result contains:

• the contact sheet as MCP image content,
• a short text summary with capture metadata and the full-resolution path of every frame, so an agent can open the exact frame it needs,
• raw frames saved locally for later inspection.

Resolution and the detail preset

A single returned image has a fixed resolution budget, and a contact sheet splits that budget across its columns. So legibility comes from fewer, wider cells — not from thumbWidth alone (past a point, a large sheet is just downscaled again by the client). The detail preset picks a sensible columns/width pair:

Two more levers when detail still falls short:

• Capture a `window` instead of the full screen. A 2560px desktop shrunk into a 480px thumbnail loses ~5× of its detail; an 800px window barely shrinks at all.
• Open the raw frame. Every frame is saved at full resolution under captures/ /, and the record response lists each one's path. Reading a single raw frame is better than re-recording.

start_recording / stop_recording

record is fixed-duration — the agent decides how long. When you control the timing (you will drag something, open a menu, trigger an animation and the duration is unpredictable), use the open-ended pair instead:

1. The agent calls start_recording on your cue (same parameters as record except seconds: fps, region, title/windowId, rect, delay, detail, cols, thumbWidth).
2. You perform the interaction.
3. The agent calls stop_recording, which composes and returns the contact sheet — identical output to record.

Only one recording may be active at a time. Frames are written to disk as they are captured, and a safety cap stops a forgotten session before it grows without bound.

capture

Captures one frame immediately using the same region-selection semantics as record.

get_frame

Returns a single frame from a prior recording at full resolution, as image content over MCP. The contact sheet is a downscaled overview; when it is too small to read fine detail, call get_frame with the recording directory and the frame number (both listed in the record / stop_recording response), or a direct frame path. Because the frame is returned through the protocol, this works even for clients with no filesystem access.

list_windows

Returns visible window titles, and geometry where available, so an agent can choose a target for window capture.

---

Example

Once Marey is connected to an MCP client, interaction can be as simple as:

Use Marey to record 6 seconds of my editor window at 4 fps with high detail while I drag an element, then tell me what changes between frames.

The agent receives the complete sequence as a single image and can reason about the transition rather than only the initial state. If any frame needs a closer look, the full-resolution originals are listed in the response and saved under captures/.

---

Command-line use

Marey runs standalone from the repo — the easiest way to try the whole workflow, including the interactive observe and replay modes, without wiring up an MCP client. Clone it first (see From source); every command is node src/cli.mjs .

Diagnose capture — confirm the backend actually works on this machine before anything else:

node src/cli.mjs doctor                  # probe the backend with a REAL one-frame capture (pass/fail)

node src/cli.mjs backend # just name the detected backend node src/cli.mjs windows # list targetable windows (hidden ones are flagged)

Record a fixed clip — you choose the duration up front:

node src/cli.mjs capture --region primary                                   # one frame

node src/cli.mjs record --seconds 4 --fps 4 --detail high # 4s of the screen node src/cli.mjs record --region window --title "Chrome" --fps 4 --detail max # just one window

Capture an unpredictable moment — when *you* control the timing:

# observe: a browser control page opens; reproduce the issue, then click Finish

(or run marey finish from any terminal) and the same call returns the sheet.

node src/cli.mjs observe --fps 3 --detail high node src/cli.mjs finish # ...from a second terminal, to end it

replay: a rolling buffer always holds the last few seconds (a dashcam). Press

Mark the instant something glitches; each mark becomes its own clip.

node src/cli.mjs replay --window-seconds 10 --fps 4 node src/cli.mjs mark # ...from a second terminal, to pin the look-back window node src/cli.mjs finish # ...then end it — one contact sheet per mark

Inspect a frame — zoom into any frame at full resolution without re-recording:

node src/cli.mjs get-frame --dir captures/<timestamp> --index 7

node src/cli.mjs get-frame --dir captures/ --index 7 \ --normalized --crop-x 0.5 --crop-y 0 --crop-w 0.5 --crop-h 0.5 # top-right quadrant

Window capture grabs the window's own surface (via PrintWindow), so it works even when the target is behind other windows. A minimized or off-screen window has nothing to render, so Marey reports that rather than capturing junk — restore the window (node src/cli.mjs windows flags which are hidden).

Inspecting evidence quality and transitions

Recording results include requested and achieved FPS, mean spacing, maximum frame gap, completion reason, warnings, errors, and the resolved target ID and bounds. New MCP clients (protocol 2025-06-18 or later) also receive these as structuredContent; older clients receive the same quality information as text. Partial recordings remain inspectable and failed recordings return isError:true alongside any available images. stop_recording retrieves evidence even if the backend already stopped automatically.

A contact sheet selects neighborhoods around the strongest transitions: the immediate before, changed, and following frames. Larger budgets also retain th