Noracu

noracu

A local MCP server that lets Claude, Cursor, or any MCP client operate macOS apps and a Playwright browser by `element_index` — and, experimentally, a real iPhone through Apple's iPhone Mirroring (OCR on the Mac, normalized taps). No API key; no telemetry from the server.

中文说明见 README.zh-CN.md。

One stdio MCP server. The agent snapshots a tree, gets an indexed element list, then clicks or types by element_index instead of raw pixels — the same observe/act split on the Mac and in the browser. The experimental iPhone layer runs the same observe → act loop over OCR text and normalized coordinates.

*browser_snapshot returns an indexed ARIA tree; browser_click acts by element_index. The Mac layer runs the same loop (mac_ax_state → mac_click). Highlight boxes are overlays from demo/record-browser.mjs, not a product feature. The iPhone layer is not in this demo.*

What it does

One MCP server, three control surfaces:

The iPhone layer is experimental and ships with its driver (imctl, in imctl/). It needs Apple's iPhone Mirroring with the iPhone locked and near the Mac, and a python3 with pyobjc; it observes with OCR and acts on normalized coordinates rather than element_index. Without pyobjc, phone_* tools return a one-line imctl unavailable error with the install command, and the Mac and browser layers are unaffected. The observe/act vocabulary follows the action/observation split popularized by ChatGPT's computer-use, re-implemented on open local tooling.

Why noracu

noracu is the local hands, not the model. Your MCP client (Claude Desktop, Claude Code, Cursor) decides what to do; this server only observes and acts on this machine.

• The server runs entirely on your machine. No cloud service, no API key, no telemetry, no postinstall scripts. Sole runtime dependency: Playwright. noracu makes no network calls of its own; the only automatic one is the one-time Chromium download on the first browser_launch (NORACU_DISABLE_AUTO_INSTALL=1 turns that off). With NORACU_ALLOW_CDP=1 it also talks to a DevTools port on 127.0.0.1 that it opened itself (NORACU_ALLOW_CDP=any for other loopback ports). The browser still fetches the pages you point it at, and your MCP client still sends tool results — screenshots, OCR text — to whatever model it uses.
• Acts on semantics, not pixels. Observation returns the accessibility tree (Mac, read directly through the macOS Accessibility API — about 0.15 s for a 500-element window on our test Mac) or ARIA tree (browser) as an indexed list — text, not screenshots. Actions reference element_index; indices expire after 30 s. On the Mac an index is backed by identity evidence (the window's ID, the child path, role, identifier, name) and the element is re-identified before every action; in the browser, tag/name/href/position are re-checked. A gone or changed element returns element_no_longer_valid instead of clicking the wrong thing. Screenshots are there when the tree is not enough.
• Says what happened. A Mac action uses the element's own accessibility action when it has one (press, select, set value, insert text — no pointer, no focus change, IME-safe text); a physical click is sent only after a hit test shows that the point reaches the element. Every input action result (Mac, browser and iPhone) states whether it was delivered (delivered_verified when a change was observed, delivered_unverified, not_delivered, or unknown when a step timed out or failed mid-action) and whether a retry is safe; Mac actions also name any sheet, dialog or menu they opened.
• One loop, three surfaces. Native Mac apps and a persistent-profile browser share the element_index protocol (observe a tree, act by index). The experimental iPhone layer runs the same observe → act loop on Apple's iPhone Mirroring, with OCR text and normalized coordinates, because the phone exposes no accessibility tree to the Mac.
• Uses your real sessions. Your logged-in browser profile, your Mac apps — not a hosted VM. Compared with cloud computer-use: neither is universally more reliable; they fail differently. noracu keeps the screen on your machine.
• Off switches that are actually off. NORACU_READ_ONLY=1 (observation-only, default-deny), NORACU_DISABLE_EVAL=1, and file:// blocked by default. See Security.
• Verifiable. End-to-end tests (npm test; the hardware-free ones run in CI on macos-latest), MIT; the server is src/server.mjs plus the Mac accessibility engine src/mac-ax.js.

Not a good fit if you need multiple browser tabs or sessions (single session for now), you're not on macOS (the Mac and iPhone layers are macOS-only; the browser layer is cross-platform), or you need a finished iPhone layer today (it ships, but experimental — and Apple does not offer iPhone Mirroring in the EU). If you only need browser automation, playwright-mcp has tab management and selector APIs that noracu lacks. If you need desktop apps on Windows or Linux, agent-desktop (Vercel Labs) covers macOS, Windows and Linux through each OS's accessibility API; noracu's desktop layer is macOS-only. noracu is for when one agent drives Mac apps, a browser and an iPhone from one server.

Quickstart

First success is mac_ax_state (or a browser snapshot) — not a click, and not a phone_* tool. The biggest day-one failure is granting macOS permissions to node instead of to the host app; grants apply only after a full quit and reopen.

1. Install (Node ≥ 20; no postinstall script):

npm install noracu          # server at node_modules/noracu/src/server.mjs

brew install cliclick # needed for physical Mac input (mac_key, mac_paste, mac_drag, and physical mac_click/mac_type)

Or from source: git clone https://github.com/sgInnora/noracu.git && cd noracu && npm install && npm test (add NORACU_CI=1 to skip hardware-gated checks).

2. Add to your MCP client — use an absolute path to `src/server.mjs`. GUI apps launched from Finder/Dock do not inherit your shell PATH, so give an absolute path to the node binary as well (which node) — a bare node, npx, or noracu needs Node on the GUI PATH and otherwise fails with ENOENT.

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json) / Cursor (~/.cursor/mcp.json):

{

"mcpServers": { "noracu": { "command": "/opt/homebrew/bin/node", "args": ["/ABSOLUTE/PATH/TO/noracu/src/server.mjs"] } } }

Claude Code: claude mcp add noracu -- /ABSOLUTE/PATH/TO/node /ABSOLUTE/PATH/TO/noracu/src/server.mjs

Find your node path with which node (Apple Silicon: /opt/homebrew/bin/node; Intel: /usr/local/bin/node; nvm: the versioned path). Convenience, only if Node is already on the GUI PATH: "command": "npx", "args": ["-y", "noracu"].

Then quit the host app completely (including any menu-bar icon) and reopen it. A loaded tool list means the transport works — it does not need any permission yet.

3. Grant macOS permissions to the host app, not to `node`. System Settings → Privacy & Security → Accessibility and Screen Recording (macOS 15: "Screen & System Audio Recording") → enable the host (Claude Desktop / Cursor / your terminal). noracu never appears here. Quit and reopen the host after granting.

4. Verify. Call mac_status — it reports whether the host has Accessibility and Screen Recording, and whether cliclick is found. Then, with Finder in front, call mac_ax_state (no args) — it returns an indexed accessibility tree; indices last 30 s. Browser smoke test (no Accessibility needed): browser_launch (first run downloads Chromium unless NORACU_DISABLE_AUTO_INSTALL=1) → browser_navigate → browser_snapshot → browser_click by index.

Troubleshooting

| element_no_longer_valid | the snapshot is older than 30 s, or the element is