Ollos MCP

ollos-mcp

Eyes and ears for AI agents. Local, offline transcription, keyframes, on-screen text and a pre-publish review of any audio, video or image — as an MCP server, a CLI and a Node library. No Python, no cloud, no API key.

Docs: Tool reference · Design doc — decisions and measurements · Evaluation results · Contributing · Security · Changelog

11 minutes of screencast become 14 contact sheets and 3 KB of text. And it tells you if your API key is visible at 2:50.

npx ollos-mcp            # MCP server on stdio

npx ollos review talk.mp4

*Ollos* is Galician for *eyes*.

Contents: Why · Install · Tools · Sources · CLI · Library · How it works · Evaluation · Requirements & performance · Privacy & security · Configuration · Troubleshooting · Known limits · Roadmap

Why

Agents can't hear or watch. Today you either pay a transcription API, install a Python pipeline, or paste frames by hand. Ollos runs Whisper, speaker segmentation, perceptual-hash keyframing and OCR in Node, through ONNX Runtime, on your machine. The file never leaves it.

It was built for one workflow first — reviewing a screen recording before publishing — and grew into the general case: meetings, lessons, podcasts, downloaded videos.

Install

Node 20+. npm install brings its own ffmpeg (ffmpeg-static); a system ffmpeg is used if present.

Claude Code

claude mcp add ollos -- npx -y ollos-mcp

or in the project's .mcp.json (the same JSON works for Claude Desktop's claude_desktop_config.json, Cursor's .cursor/mcp.json and Windsurf's mcp_config.json):

{

"mcpServers": { "ollos": { "command": "npx", "args": ["-y", "ollos-mcp"] } } }

VS Code — .vscode/mcp.json uses servers instead of mcpServers:

{

"servers": { "ollos": { "type": "stdio", "command": "npx", "args": ["-y", "ollos-mcp"] } } }

Claude Desktop reads ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows. Environment variables (OLLOS_HOME, OLLOS_YTDLP, …) go in an env object next to args; use absolute paths, ~ is not expanded.

CLI

npm i -g ollos-mcp

ollos warmup # download the default ASR model and VAD (2.75 GB) ollos warmup --all # every model: fast ASR, speakers, search, OCR data (3.3 GB total) ollos doctor # check ffmpeg, models, free memory, cores

Models download on first use into ~/.ollos/models. Set OLLOS_OFFLINE=1 afterwards to forbid all network access.

Tools

Ten tools, one per distinct contract. Long work never blocks: it returns a jobId you poll.

Every parameter is documented in docs/TOOLS.md (one anchor per tool, e.g. ollos_transcribe); the tool descriptions the agent sees carry the same information.

Sources

source accepts a local path, a file:// URL, a Zoom local-recording folder (one audio track per participant), a direct https:// media URL, a video-site URL (YouTube, Instagram, TikTok, Vimeo, X, Loom… through yt-dlp) and a data: URI. URLs are downloaded once into the cache; the download runs inside the job and can be cancelled. Refused: private, loopback and link-local addresses on any redirect hop (OLLOS_ALLOW_PRIVATE=1 to allow), downloads over OLLOS_MAX_DOWNLOAD_MB, media over OLLOS_MAX_DURATION_SEC from any origin.

ollos transcribe "https://www.youtube.com/watch?v=eur8dUO9mvE" --lang en --from 0 --to 30

Tested commands, yt-dlp setup and proxy notes: examples/url-sources.md.

Results are concise by default and point to MCP resources (ollos://jobs/ /transcript, /ocr, /report, /sheet/ ) for the full artifacts, so a 2-hour meeting doesn't flood the context window. Pass format: "detailed" when you want it all.

CLI

ollos probe recording.mp4

ollos transcribe meeting.mp4 --lang pt --vocab "Claude Code,n8n,webhook" ollos keyframes lesson.mp4 --sensitivity normal --max-frames 120 ollos read-screen demo.mp4 ollos review episode.mp4 --platform youtube # exit 3 = block, 1 = warn or failure, 2 = usage error, 0 = ok ollos jobs · ollos job · ollos events · ollos cancel

Add --json for machine output.

Library

import { createOllos, type TranscribeResult } from 'ollos-mcp'

const ollos = createOllos() const { job, result } = await ollos.transcribe({ source: 'talk.mp4', language: 'pt', vocabulary: ['MCP'] }) const transcript = result ?? (await ollos.wait (job.id)).result // inline when small, a job otherwise console.log(transcript?.segments[0])

More: examples/library.ts (every capability) and examples/library-url.ts (a YouTube URL as the source).

ollos-mcp/core has no MCP dependency: use it from n8n, a script, a Lambda.

How it works, and what was measured

Numbers below were measured on an 11:37 screencast (1890×1080, webcam overlay) on a 16-core laptop. They are why the design is what it is.

Transcription. Silero VAD marks speech; Whisper only sees speech (fewer hallucinations, 20–40% less work on meetings). whisper-large-v3-turbo at 1.7× real time got "MCP servers", "n8n", "VS Code" right where whisper-base (5.4×) got all three wrong. The one phonetic miss left ("Cloud Code") is fixed by vocabulary. Two Whisper sessions in parallel measured slower than one (0.6–1.0×), so ASR concurrency is 1 and speed comes from VAD and from running vision in parallel instead.

Anti-hallucination. Whisper doesn't go quiet on silence — it invents "Obrigado." and "Subtitles by the Amara.org community". Four filters, from production experience shared by the Vexa project: exact blocklist per language, repetition-loop collapse, no-speech gate, impossible speaking rate.

Keyframes. ffmpeg scene detection at 0.3 kept 4 frames in 11 minutes of screencast; mpdecimate removed 0% (the cursor and streaming text change every pixel). A 64-bit perceptual hash (dHash) at Hamming ≥ 6 kept 20% — one frame every 5–8 s — and that is the default. Hard cuts, transcript anchors and a 20-second floor fill the gaps.

OCR. Tesseract on a full 1890-px frame missed an on-screen URL entirely; on a 3× upscaled tile it read it whole at 90% confidence in 2.8 s. So OCR runs per tile, and URL-like text is re-joined when OCR splits it ("up. railway .app").

Secrets. Three signals, because OCR garbles the secret more often than the words around it. On a real "API Key Created" modal the plain JWT regex missed (OCR read eyJ as eyl), the entropy detector caught the 157-char token, and the UI context read at 66%. With an OCR-tolerant JWT pattern, native-resolution frames and a centre tile, the end-to-end run now reports it as high · jwt · near "API Key" → block. The first version also produced 58 false positives by running the entropy test on whitespace-stripped text; that is a regression test now. Values are always masked; the tool that warns about a leak must not be the leak.

Change-detected keyframes are the wrong tool for this specific check: a credential left on an unchanging screen for a minute contributes one dHash candidate, and a frame cap's prune step used to drop that one candidate on the same terms as a genuinely redundant frame — real miss, filed as #3. ollos_review's secrets check now reads the screen at a fixed 4-second cadence regardless of whether the picture changed, protects that guarantee through the frame cap (preserveFloor), and states the coverage it actually achieved in the finding — Read the full 11:37 at a 4s cadence on a normal-length recording, or an explicit Partial coverage: … note bounded by OLLOS_MAX_FRAMES on a very long one, instead of a clean verdict that silently checked less than it implied.

Speakers. Segmentation alone labelled three speakers on a one-person video (its ids are local to each 10-second window). Embedding every turn ≥ 1.5 s, average-linkage clustering at cosine 0.35, and absorbing tiny clusters brought it to one. The evaluation then showed the real failure mode: the same voice scores 0.58–0.86 against itself across positions and lengths, but 0.06–0.16 once background music is under it, so a jingle or an outro becomes its own "speaker" at any threshold. The tool stays experimental and says so in its output.

Jobs. Client timeouts are short (Messages API ~60 s). Every long tool returns a job handle; state lives in ~/.ollos/jobs/ /job.json, written atomically, with a 5-second heartbeat. On restart, orphaned jobs become interrupted instead of hanging forever. Small work (< 8 s estimated) runs inline and returns directly.

Evaluation

npm run eval runs ollos against public videos and writes eval/RESULTS.md. Reference transcripts are YouTube captions, so on auto-captioned fixtures WER is an agreement rate between two recognisers, not an absolute error.

| Karine Lago, n8n assistant