WebKitUI MCP

WebKitUI MCP

Native WebKit automation for Apple Silicon, exposed through MCP and designed for an LLM rather than a human.

This repository is a Swift rewrite. The retained TypeScript/Playwright files are prior art only and are not the implementation being extended.

What exists

• Native WKWebView runtime using the persistent system WebKit data store for real authenticated sessions.
• MCP 2026-07-28 stdio server with legacy initialization compatibility.
• Thirteen bounded tools, including authenticated native downloads, explicit page/element scrolling, and bounded text extraction for rendered virtualized logs.
• Observation-scoped element symbols backed by semantic locator recipes and fresh action-time resolution.
• Five separate addressing counters: address_resolution_failed, address_now_ambiguous, logical_target_changed, node_replaced_but_semantic_locator_recovered, and coordinate_invalidated_by_layout_change.
• Provenance attached to every serialized page string.
• Observation-time privacy minimization: non-rendered controls are removed before bounding, sensitive/token-like fields never serialize values, and a sensitive select publishes neither its option labels, nor how many it has, nor the one currently chosen in it — a selected label is that control's value. The control is still reported, with its role and accessible name, so it can be handed to a human.
• Checkpoint-plus-delta observation history and Minimal Failure Set coverage metrics.
• Optional local Ollama ranker with think: false, strict budgets, and deterministic fallback.
• Transaction ledger with preconditions, exact post-conditions, idempotency keys, indeterminate outcomes, receipts, and reconciliation without replay.
• Human confirmation before every exposed click and open-world navigation. Navigation defaults to a server-owned native macOS dialog so a client cannot silently decline the round trip; MCP multi-round navigation remains opt-in.
• browser_session can set a bounded 320×240–3840×2160 CSS viewport and can navigate back, forward, or reload using WebKit's own history list. History movement is confirmed against the destination WebKit reports; reloading a page produced by a form submission is refused before confirmation to prevent replay.
• Camera, microphone, and geolocation requests are always denied by the native WKUIDelegate. The observation names the requesting origin and permission, with repeated requests counted and distinct records bounded; there is no MCP operation that can grant one.
• The confirmation names the origin a submit control's data would reach, computed server-side from the freshly re-resolved element rather than from its accessible name, and says so explicitly when that origin differs from the page the operator is on. A submit button's formaction overrides its form's action, so a control whose accessible name reads harmlessly can post elsewhere.
• Confirmations are counted in a rolling 60-second window. From the fifth, the dialog states how many have been requested; the twentieth is refused instead of presented, because a dialog a human clicks twenty times in a row has stopped being a gate.
• Legacy MCP clients receive the same exact-action authority boundary through a server-owned native macOS confirmation dialog; the model cannot supply or forge the approval value.
• Local human handoff: handoff_start immediately returns an opaque session-bound resume token while the actual WebKit session becomes a visible window. handoff_status is non-blocking; handoff_resume consumes the token only after local confirmation and returns a fresh observation. The agent remains locked out throughout human control.

Handoff replies include the localized return_control_button_label. Returning control confirms the human step, not remote saving. Arguments are specific to each operation: handoff_start accepts only operation/session ID; status adds the resume token; resume additionally accepts compact/maximum_elements. Timing arguments belong to back/forward/reload, not handoff. Discovery remains composition-free for clients that cannot safely parse conditional schemas.

• An active human-control window can be brought forward again with handoff_focus without changing who controls the page. Session status says whether that surface is currently presented, and restricted authentication status exposes only bounded rendering booleans—never page text, field values, private URL components, or network payloads.
• The packaged handoff app installs a native Edit menu, so standard first-responder shortcuts such as Command-X/C/V/A work inside WebKit form controls.
• MCP sessions use a loopback SOCKS5 boundary, one per profile and shared by its sessions, with failover disabled: hostnames are resolved once, public addresses are pinned, and private/reserved destinations plus non-TCP SOCKS commands fail closed.
• One process at a time controls the browser for the macOS account: the host lease belongs to that process and is released with its last session or on process death. The production CLI holds one session.
• Private remote clients can use the app-owned broker plus a forced-command SSH relay; WebKit and authenticated profile data remain on the logged-in Mac.
• Multiple relay clients may use the same long-lived broker concurrently, each in its own session and window, up to three, on the shared default profile. A session belongs to one client: another client gets session_in_use with wait_only=true and cannot observe, navigate, act, invalidate addresses, or request duplicate user control in it. Ownership transfers after the owning client disconnects, or through a human-confirmed client_handoff. Native confirmations are shown one at a time across the app, and each names the requesting agent as a quoted, self-reported name.
• Each independent task must call browser_session(operation: "open") and keep its returned session_id for all page operations. On a shared transport, a full broker refuses an open that would return an already-owned page. Status without an ID selects the caller's sole session; when it owns several, it asks for an explicit ID. A new client with spare capacity is directed to open its own page. Host lease metadata reports the broker's remaining session slots.
• While a person holds the human control window, a pop-up the page opens (a sign-in with Google or Apple, say) is a real window sharing the page's profile, protected proxy and window.opener, at most four; all close when the agent resumes. Under agent control ordinary new windows stay refused and reported. An explicitly approved download can retain one request-scoped export view, with the original configuration/profile, without replacing the page. Only same-origin GET attachment responses or Blob downloads are accepted there; it is closed at the request's terminal outcome and is never an agent-visible tab.
• The app broker owns one live browser across MCP client reconnects. Every reconnect invalidates observations, pending approvals, capabilities, and transaction coordinators before returning the preserved session handle.
• The native host starts the AppKit event loop before MCP work and keeps the live WKWebView attached to one opaque window. Agent control orders that window out; human handoff presents the same view without reparenting it. A packaged visual smoke mode checks the actual WindowServer pixels.
• The stdio relay stays alive across broker restarts. It reconnects before the next undispatched request and never silently replays a request whose outcome became unknown during a restart.
• Durable multi-client status works without a session ID and reports only privacy-safe holder metadata (client name/version, PID, age, inactivity and policy). open(wait_timeout_ms: ...) can wait up to 60 seconds for the local host lease without stealing it; dead transports release their session owner. browser_session(operation: "client_handoff") can transfer an idle session only after local human confirmation and a final no-active-call check.
• A secretless SiliconPass fill can return credential_not_found; WebkitUIMCP then offers a native human handoff for manual sign-in and addition/update in SiliconPass. No credential value crosses MCP, JSON, logs, or the clipboard.
• Web-content termination invalidates every observation immediately; recovery reloads the host-owned last URL without replaying an action. A forced-crash fixture verifies that an HttpOnly authenticated cookie plus localStorage and sessionStorage survive in the same view/data-store lifetime.
• The macOS companion is a regular Dock app with a local activity window. It records only allowlisted MCP tool names, outcome, timestamp, duration and a bounded error type. Parameters, URLs, page content, credentials, cookies, keystrokes and response bodies are structurally absent. Owner-only JSONL files rotate automatically and can be exported or cleared from the app.

Deliberate limits

• browser_act exposes click, native submit-control click, bounded non-sensitive input/textarea fill, Enter/Tab/Escape, blur, explicit input commit, and answering the one JavaScript alert/confirm/prompt dialog a page is suspended on (dialog_accept, dialog_dismiss, dialog_accept_value). A pending dialog is reported in the observation as untrusted site content, blocks every other operation until it is answered, is never answered automatically, and resolves to an explicit indeterminate outcome if nobody answers it. beforeunload is out of scope: the public macOS SDK exposes no before-unload panel to an embedder. Native approval also routes public text fills through AppKit insertion with a measured trusted input receipt. Fill verifies both the freshly re-resolved semantic target's exact value and that its live validation state is not invalid. Other actions support exact URL or URL prefix, title, heading, exact/contains semantic text, checked, selected, enabled, semantic target absence (target_absent with value="true"), value, validation state, character count, bounded state attributes, dialog, named panel, or selected-option postconditions. A same-URL SPA mutation returns same_url_page_state_changed guidance instead of implying success. UI state never proves backend commit.
• browser_download converts authenticated attachment responses to WKDownload from either a fresh observed control or a protected same-origin URL fallback. It requires exact native confirmation plus a save-panel destination, never overwrites an existing file, and succeeds only after an on-disk receipt reports the HTTP status, suggested/final filename, MIME type, byte count, SHA-256, and decoded provisioning-profile UUID when available. Cookies, headers, and the absolute local destination path stay outside MCP.
• macOS file inputs use WebKit's native open-panel delegate. Selected regular files are bounded to 10 files and 50 MiB each; receipts may expose filenames, byte counts and SHA-256 values, but never local paths or file contents. A file-selection receipt is not provider acceptance: the action's independent postcondition must still verify the uploaded preview or saved state.
• Call browser_upload directly on the freshly observed file input or chooser-opening label/button, including a menu's upload button. Opening its file panel with browser_act first leaves a pending chooser that browser_upload refuses without another click. Use human handoff to finish or cancel that chooser, return control and reconcile before a new upload. Missing gesture evidence remains uncertain even when a separate receipt proves file selection; never replay automatically.
• Fill dispatches normal input/change events, so site handlers may autosave or cause server effects. It requires human authorization (exact approval or an explicitly remembered interact