hypruse

hypruse

Computer use for [Hyprland](https://hypr.land). An MCP server that gives AI agents native hands on your Wayland desktop: workspaces, windows, mouse, keyboard, screenshots.

No ydotool daemon. No root. No portals. No X11.

Why

Computer use exists on macOS and Windows, and the vendors' Linux apps still ship without it: Claude Desktop's Linux docs say computer use is not available there, and ChatGPT's computer use covers macOS and Windows only. Anthropic's reference implementation is an X11 container. Other Wayland tools lean on ydotool and uinput, on GNOME-only portals, or on a compositor plugin built for one exact Hyprland version.

Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:

• Semantic first. desktop returns the real window/workspace tree (addresses, classes, titles, geometry) in one call. The agent switches workspaces and focuses windows the way you do (instantly, over IPC), not by squinting at pixels.
• Vision when it matters. Screenshots of a monitor, one window (with grim 1.5+ on Hyprland 0.54+, captured as itself even on a hidden workspace or under another window, though a hidden app may show how it last looked), or a zoomed region, with the geometry/scale metadata to map any pixel back to a clickable coordinate: a coarse-to-fine loop grounded in the GUI-agents research.
• Native input. Clicks and scrolls are spoken directly over the Wayland wire (zwlr_virtual_pointer_v1); typing goes through wtype's virtual keyboard with a proper XKB keymap, unicode-safe on any layout.
• Works beside you. The agent can work in apps on its own workspaces while you keep working on yours. It presses controls through the accessibility tree and types through it or with Hyprland's sendshortcut, so your cursor, keyboard focus and workspace stay where you left them. Stock Hyprland, no compositor plugin: see Background mode.

How it works

agent (Claude Code, or any MCP client)

│ stdio ▼ hypruse ├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers ├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor, │ and sendshortcut for keys to a hidden window ├── grim ··············▶ screenshots: monitor, window, region ├── D-Bus (AT-SPI) ····▶ accessibility tree: named controls, current │ values, exact coords (ui / marks / click_ui), │ and presses and text in the background ├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap) └── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)

Optional binaries gate two more tools: imagemagick draws the numbered overlay for marks, and wl-clipboard backs the opt-in clipboard tool.

Design decisions:

• No ydotool / uinput. That path needs a daemon, udev rules or root, and types US scancodes that break on other layouts. hypruse is just another Wayland client of your compositor, same standing as wlrctl.
• No portals. xdg-desktop-portal-hyprland does not implement the RemoteDesktop portal (InputCapture is capture, not injection), so anything built on libei/portals silently degrades on Hyprland. hypruse doesn't try.
• Cursor positioning through the compositor's own cursor dispatcher (global logical coordinates, exact on any monitor layout), with only button/axis events on the virtual pointer, sidestepping the known multi-monitor bugs of absolute virtual-pointer motion (hyprwm/Hyprland#6749).

Tools

The acting tools (pointer, keyboard, click_ui, hypr, use_bind, sequence) take an optional then argument that appends the result to the same call, so the agent sees the effect without a second round-trip: then='desktop' adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes), then='screenshot' a stable capture (best for visual changes), then='ui' the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling; click_ui reads the window it clicked even if the click handed focus to a dialog, the others read the focused window), then='none' nothing (the default everywhere except sequence, which defaults to 'desktop').

Every tool is also a shell verb (hypruse desktop, hypruse click_ui Save --window 0x...) for agents that run commands instead of MCP, with a skill that teaches them: see Shell verbs and the agent skill.

Features

The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.

1. Semantic desktop control (start here)

desktop returns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces the compositor is tracking (launchers, bars, notification popups). hypr and launch then act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.

Use it well: never take a screenshot to find or arrange windows. Read desktop, act on the address you want. launch blocks on the real openwindow event and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.

2. Click controls by name, no pixels

When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel. ui lists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state. click_ui resolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply), or, in a window you are not in, through the control's own accessibility action without touching your cursor. marks draws numbered marks over a screenshot with a legend, for when you want to see the options first and then click_ui(mark=N).

Use it well: reach for click_ui name="Save" before estimating any pixel, since it is exact and spends no image. Read a form's state with ui (did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without --force-renderer-accessibility) the tool says so, and you fall back to vision.

3. Vision when it matters: the zoom loop

For everything the accessibility tree cannot name, screenshot (monitor, window, or region) and zoom (a native-resolution re-capture around a point) carry a strict coordinate contract, global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.

Use it well: don't guess a small control from a full-screen image. Work coarse-to-fine: screenshot the window, estimate the target, zoom there, re-estimate on