Deskhand

Deskhand — let AI agents use your KDE desktop, safely

*Türkçe: README.tr.md*

Deskhand lets AI agents (Claude Code and other MCP clients) use your KDE Plasma desktop safely — on your screen, or on an invisible second desktop while you watch a movie.

An agent can read windows, click on visible text, type and open apps. Every input is checked against what is actually on screen at that moment. Buy/Pay buttons and payment pages wait for your answer in a dialog the agent cannot reach. It stops the moment you touch the mouse, and one click on STOP halts everything. When you are busy, the work moves to a hidden desktop that can reuse some of your logged-in sessions, so your film or game is never interrupted.

It is a safety layer for well-meaning agents, not a sandbox (see Threat model).

Screenshots are rendered from sample data. The panel's language is Turkish for now; an English UI is planned.

Features

Safety layers

These are enforced in the tool itself, on both desktops, whatever the agent or its permission mode says:

• Forbidden windows. Windows whose title or class matches a pattern in ~/.config/deskhand/yasak-pencereler.txt never receive input, and their apps are never started (for example an online game whose rules forbid automation, or a password manager). Over MCP they are also blacked out in screenshots and left out of OCR.
• Sensitive buttons and payment pages need a human. Buy, Pay, Place order, Authorize, Allow, Delete account (Turkish and English wording) and any click or Enter on a page whose title looks like checkout/payment trigger an approval dialog in the Deskhand panel: *Allow once* or *Deny*, with a small picture of the target. Only the panel can write the answer; the dialog itself is a forbidden window, so the agent cannot click it. No answer, timeout or panel not running = denied.
• Window/tab title lock. With -b PATTERN (CLI) or a window pattern (MCP) the title is re-checked right before every input. If a browser switched tabs, typing stops mid-word.
• Human priority. If you are using the mouse or keyboard (Wayland idle notification, no extra permissions), Deskhand waits for 1.5 s of quiet, up to 8 s, then gives up. Keys it is holding are released as soon as you touch anything.
• STOP switch. Panel button, tray menu or an optional global shortcut. While it is on, nothing sends input and held keys are released. An agent may set STOP itself, but it cannot lift a STOP that you set; only the panel can.
• Activity log. Every input is logged (~/.local/state/deskhand, kept 7 days) and shown live in the panel.

Background desktop

deskhand desktop start starts an invisible desktop: Xvfb + openbox, 1920×1080 by default, separate from your session.

• Nothing reaches you. No input goes to your screen (the background backend can only talk to its own X display). Apps there get a private D-Bus, so their notifications are captured to a log instead of popping up, and their audio goes to a silent sink. Wayland variables are removed so an app cannot escape to your screen.
• Light. Low CPU/IO priority and a memory cap (systemd scope, 4 GB by default). Starting it takes about 0.2 s; an empty desktop uses about 30 MiB.
• Per-app session policy. Some apps can open there already logged in:
• Firefox: a consistent, one-way snapshot copy of your profile (cookies, logins, site storage for the task's sites). Nothing is ever written back. Sync identity is left out; cookies and saved passwords of banking/payment/government sites (including Turkish banks and e-Devlet) and central login pages (Google, Microsoft) are not copied, so Google/YouTube are not signed in there. Camera, microphone, location and notification permissions are removed from the copy.
• Flatpak apps that do not use the keyring (for example Discord, Spotify): snapshot copy.
• Apps that need the keyring (Chromium family, VS Code) and apps that write to real files (Obsidian vaults) run only on your screen.
• Apps that break with two clients (Telegram, Signal, WhatsApp wrappers, Element, Steam, Thunderbird and similar) are never cloned and never started in the background: the agent is told to wait for you. Mail/sync daemons (KMail/Akonadi, Nextcloud) are never started either.
• The policy is matched on the resolved app (desktop entry, executable, Flatpak id), so a different name or path cannot bypass it; the strictest match wins. Password managers are never started.
• Unknown apps start normally only when they are certainly not running for you (otherwise their single-instance lock could open the window on your screen); if that cannot be known, the agent waits. Policy file: ~/.config/deskhand/oturum.toml.
• Watch and take over. The panel shows a live preview. *Take over* sends your mouse and keyboard into the background desktop while the agent waits.

Busy detection: "Background / My screen / Later"

deskhand busy checks, read-only, whether you are busy: a fullscreen window, a playing media player (MPRIS), recent keyboard/mouse input, a game or forbidden window, a locked screen.

deskhand decide "task summary" turns that into a decision. If you are not busy, the answer is *my screen*. If you are, by default a notification asks Background / My screen / Later (no answer in 45 s = background). Over a fullscreen film or game it does not interrupt you at all and goes straight to the background. All of this is configurable in the panel.

Clicking by text (OCR first)

Agents click on what they can read, not on guessed coordinates: click-text "Save". Tesseract reads Turkish and English. With a graphics card, a second reader (EasyOCR on the GPU) joins in by itself: it finds coloured web buttons and faint or small text that tesseract misses (measured: 55/55 test phrases instead of 49/55, ~0.15 s per read). It only adds lines; tesseract's reading is kept. It is never started while you play a game or watch something full screen, and it frees the graphics memory after 5 idle minutes. Turkish letters are matched loosely (ş/s, ı/i…) and an ambiguous match stops instead of guessing. Also available: the accessibility tree (AT-SPI, click by button name), saved icon templates, colour areas, and screen-recognition *flows* for repetitive tasks.

MCP server (12 tools)

deskhand mcp is a stdio MCP server: get_status, choose_desktop, manage_desktop, screenshot, read_text, find, click, type_text, press_key, scroll, open_app, run_flow. Screenshots are scaled to 1280 px on the long edge. Coordinates are checked against a shot_id, so a click on a stale screenshot is refused. There is no "approved" flag on the MCP surface: sensitive actions always go through the panel.

Monitoring panel and tray

Activity (live log, filter), background desktop (preview, stop, take over), approvals (pending + history), settings (busy behaviour, forbidden windows, STOP shortcut), saved icons and a health check of all dependencies. The tray icon shows idle / on your screen / in the background / STOP, with a badge when an approval is waiting.

How it compares

There are good projects in this space; pick what fits you.

• agent-sh/agent-workspace-linux is the closest: a hidden desktop on six backends (including a native KWin Wayland virtual compositor), keyring and Chrome profile import, app allow-list, a live viewer, 82 tools. It is broader and more portable than Deskhand.
• ahmetadakli/kde-desktop-mcp has the closest safety model on KDE: app allow-list with levels, confirmation list, password-field protection, user-takeover pause, stop shortcut. It works through the accessibility tree.
• agent-sh/computer-use-linux (GNOME-first, many backends), Samthesurf/kwin-mcp (KDE Wayland), alex-oleshkevich/wayland-mcp (niri/wlroots, nested sandbox, local OCR) are other options.

What Deskhand puts together, which we have not seen combined elsewhere:

1. Checks based on what is on screen, not only on which app: forbidden windows, sensitive buttons, payment-page detection, title/class lock.
2. Busy detection and the *Background / My screen / Later* loop.
3. Session policy per app for the background desktop (copy / wait / never), including the apps that must not run twice.
4. Click by visible text with Turkish OCR, also where there is no accessibility tree.
5. A Plasma panel: tray, STOP, live preview, take over, human-only approvals.

It is also smaller in scope: KDE Plasma only, one background desktop, no GPU there.

Requirements

• KDE Plasma 6 on Wayland for the foreground (your own screen). The background desktop is an X server (Xvfb), so it runs apps that still support X11 (most do).
• Python 3.11+
• Foreground: kdotool, dotool, spectacle, access to /dev/uinput
• Both: tesseract with Turkish and English data
• Background: Xvfb, openbox, python-xlib; recommended xdotool, pactl (silent audio sink; without it background apps get no audio at all), systemd-run (memory cap)
• Python modules: PyQt6, python-xlib, numpy, scipy, pillow, jeepney, mcp ≥ 2.3
• To build the small keyboard-activity helper: a C compiler, make, pkg-config, wayland (without it, human priority watches only the mouse)
• With an AMD or NVIDIA graphics card, for text recognition on the GPU: PyTorch for your card, torchvision, OpenCV, scikit-image, shapely, pyclipper, and EasyOCR in a small venv with its models (~100 MB). Without a GPU nothing extra is needed. install.sh checks all of it and prints the commands
• To build the screenshot helper (KWin's screenshot interface, about 8 ms a shot): the same tools and libsystemd (part of systemd). install.sh builds it and registers its desktop entry

Arch / CachyOS:

sudo pacman -S --needed spectacle tesseract tesseract-data-tur tesseract-data-eng \

xorg-server-xvfb openbox xdotool libpulse python-pyqt6 python-xlib python-numpy \ python-scipy python-pillow python-jeepney python-mcp base-devel wayland paru -S kdotool-bin dotool # AUR (or yay, or by hand)

with a graphics card (NVIDIA: python-pytorch-opt-cuda instead of -rocm)

sudo pacman -S --needed python-pytorch-opt-rocm python-torchvision python-opencv \ python-scikit-image python-shapely python-pyclipper python3 -m venv --system-site-packages ~/.local/share/deskhand/ocr-venv ~/.local/share/deskhand/ocr-venv/bin/python -m pip install --no-deps easyocr python-bidi cd ~/.local/src/deskhand && ~/.local/share/deskhand/ocr-venv/bin/python -m deskhand.ocr_gpu indir # models

dotool needs to write to /dev/uinput. Its package installs a udev rule for the input group, so:

sudo usermod -aG input "$USER"                        # then log out and back in

sudo modprobe uinput # if /dev/uinput does not exist echo uinput | sudo tee /etc/modules