Macos Vision MCP

macos-vision-mcp

Your documents and your screen stay on the machine. Only extracted text and structured JSON ever reach the model.

Local, private, offline OCR and UI testing for any MCP client — no API keys, no uploads. Cut document token costs by ~97%, and let an agent see and click your Mac's UI without a single screenshot leaving the machine.

Pre-extracts text and image data locally before your AI ever sees it — cutting token usage by ~97% on real documents and returning structured paragraphs, lines, and bounding boxes so the model can reconstruct the document into Markdown, HTML, DOCX, or any other format. Files never leave your Mac: no cloud API, no API keys, no network requests.

How the ~97% is measured: a 44-page scanned PDF sent as page images costs ~73,500 tokens; the same file run through analyze_document returns ~2,400 tokens of extracted text and structure (raw page-image tokens vs. extracted-text tokens). Your numbers vary with page density and tokenizer — treat 97% as the order of magnitude, not a guarantee.

Contents: Quick Start · What you get · What agents use this for · UI testing · Why it's different · Available Tools · Usage · Example workflows · Configuration · Privacy layer

What you get

• OCR for images and PDFs (JPG, PNG, HEIC, TIFF, multi-page PDF) via Apple Vision Framework.
• ~97% token reduction: a 44-page PDF costs ~2,400 tokens instead of ~73,500.
• Reading-order paragraphs + raw text blocks with bounding boxes — rich structure for the model to reconstruct the document into any output format (Markdown, HTML, DOCX, JSON), not a lossy plain-text dump.
• Face detection, barcode/QR reading, and image classification — all on-device.
• Full document pipeline: OCR + faces + barcodes + rectangles in a single tool call.
• Works with Claude Code, Claude Desktop, and Cursor — any MCP-compatible client.
• No files uploaded to any server — processing stays entirely on your Mac.
• UI testing for agents: screenshot a window locally, find an element by its visible text, get back click coordinates, and assert what's on screen — all without uploading the screenshot.
• 100% offline after npm install — powered by Apple Vision Framework, same engine as Live Text in Photos.app.

❌ Without / ✅ With

❌ Without macos-vision-mcp:

• Sending a 44-page PDF costs ~73,500 tokens
• Every image, invoice, or contract goes through a cloud API
• Sensitive documents leave your machine on every request

✅ With macos-vision-mcp:

• Local Apple Vision pre-extracts text before Claude ever sees it
• ~2,400 tokens for the same 44-page PDF — 97% fewer
• Files never leave your Mac

What agents use this for

Most work an agent does on a Mac needs no deep understanding of a layout. It needs to see what is on screen, find the thing it is looking for, act on it, and confirm what happened. That loop — list_windows → find_element → click (via any input driver) → assert_text — covers a lot, and every step of it runs on the machine.

• Drive an app that has no API. Native tools, Electron apps, internal software, anything with a GUI and no scripting interface. find_element("Export") returns the point to click.
• Read what is on screen right now. A dialog, an error banner, a notification, a progress state — including a window sitting behind others, without bringing it to the front.
• Confirm an action actually worked. assert_text is string matching after unicode normalisation, so it answers pass/fail the same way every time instead of asking a model to judge a picture.
• Pull data out of software that will not export it. OCR a window, get the text, move on.
• Audit accessibility. ui_snapshot reports every piece of visible text the accessibility tree does not account for — unlabelled controls and custom-drawn text, with coordinates.
• Test a UI for regressions. The case this started as, and still a good one — see the chapter below.
• Work through documents. The original job: invoices, contracts, scans, PDFs.

Why doing it locally is better, not just different

Cheaper. A verdict costs ~240 tokens against ~6,900 for the screenshot it replaces — about 29×. Over a twenty-step task that is ~4,800 tokens instead of ~138,000. It is the difference between an agent that can afford to check its work after every step and one that cannot.

More private, and this is the part that never shows up on a bill. A screenshot is not a neat crop of the button you cared about. It carries whatever else was on screen: another window, a password manager, an open inbox, a customer's record. Sending one to a third party is a disclosure you cannot withdraw, and it repeats on every single step. Here the image is written to a temp file, read by a model on the Neural Engine, and never serialised into the conversation. That invariant is enforced in the code, not promised in this README: no tool returns image bytes.

Faster in practice, and predictable, which matters more. find_element takes 1.17–1.25 s end to end on an M1 Pro — capture 0.31–0.41 s, OCR ~1.04 s, matching under a millisecond. There is no network term at all: no ~750 KB upload before inference can start, no rate limit, no provider under load, no failure when the Wi-Fi drops. The same call costs the same on a plane as it does at a desk.

What this does not do: click. It is eyes, not hands, and therefore never asks for control of

your machine. Pair it with an input driver — cliclick, a macos-mcp-style automation server,

or anything that accepts screen coordinates — and hand it the clickPoint that find_element

returns. The split is deliberate: seeing and acting are different permissions, and this server

only ever asks for the first.

UI testing without sending screenshots anywhere

The usual way to let an agent work with a GUI is to screenshot the screen and upload it to a vision model. That is one network round trip, one image-token bill, and one copy of whatever was on screen — per step. A ten-step flow means ten uploads of your desktop.

This server does the seeing locally. Apple's Vision framework runs on the Neural Engine, so the screenshot stays on disk and only text, geometry, and verdicts reach the model.

What the agent actually receives. Element boxes, roles and labels — measured, not guessed — and the exact point to click. The screenshot itself never leaves the Mac.

find_element(query: "Save", app: "MyApp")

→ { found: true, matches: [{ text: "Save", method: "exact", clickPoint: { x: 812, y: 556 }, bbox: {...} }] }

hand clickPoint to any input driver — macos-mcp, cliclick, CGEvent

then verify, again locally:

assert_text(expect: "Saved", app: "MyApp") → { pass: true, ... }

clickPoint is in global screen points with a top-left origin — the same space click drivers use, so it goes straight to a driver with no conversion. This server deliberately does not click: it is eyes, not hands, and therefore never asks for control of your machine.

Is it actually cheaper, safer, and faster?

Measured on an Apple M1 Pro (2021, 16 GB) against a 2992×1734 Retina window capture of a real, text-dense app — median of five runs each.

Image tokens are estimated with Anthropic's width × height / 750 rule; other providers

tile differently, so the exact figure moves but the order of magnitude does not. Local token

counts are the actual JSON payloads the tools returned, at ~4 characters per token.

Same question, same answer, ~29× the price. Measured on an M1 Pro against a 2992×1734 window capture.

The three claims behind that table — cheaper, safer, faster — are argued in What agents use this for above. What this section adds is the measurement: the numbers are a median of five runs on an Apple M1 Pro (2021, 16 GB) against a 2992×1734 Retina capture of a real, text-dense window.

We have not benchmarked any specific vision provider, so treat the right-hand column as structure rather than a measured figure. What can be stated is that the local path has no variance from bandwidth, rate limits or provider load, and does not fail when the network does.

Two honest caveats. Targeting a single region instead of a whole window cuts the OCR term sharply, since cost scales with pixels searched. And the first call after install spends ~2 s compiling a small Swift helper; every call after that is warm.

What it is good at — and what it is not

Good at: **native macOS apps, Electron apps with poor accessibility