Glovebox MCP

glovebox-mcp

A sandboxed computer-use MCP server — let an AI agent drive a real browser and desktop apps (mouse, keyboard, screenshots, clipboard, vision grounding), confined to a nested X11 window so its input and vision never touch your real screen or the windows of your other apps.

Like a lab glovebox: the agent reaches in and manipulates real applications, sealed off from your desktop. Bring the sandbox up, log into whatever sites or apps you want to automate *inside that window*, and the agent operates only there — you can watch it live and close it instantly.

Speaks the Model Context Protocol, so it works with MCP clients

like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.

An agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (Nadja Kovačič), typing, and submitting. All confined to a nested X11 window.

Why a nested X11 sandbox?

• Most desktop automation (xdotool, PyAutoGUI) is X11-only, but many modern desktops run Wayland.
• Xephyr provides a real X server *inside a single window* (DISPLAY :1). Everything the agent does — clicks, typing, screenshots — is confined to that window, not your real desktop.
• You stay in control: watch it live, pkill Xephyr to close everything.

The isolation is display-level (see Safety): the apps themselves run as your user with your filesystem. For OS-level isolation run glovebox inside a VM or container.

Requirements

• Linux — the sandbox nests a real X server (Xephyr), so it works even on Wayland hosts (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.
• Python 3.10+ and uv (used for the virtualenv).
• System packages — xserver-xephyr (Xephyr), openbox, scrot, x11-utils, xdotool, wmctrl, xclip (+ tesseract-ocr for basic). On Debian/Ubuntu the installer auto-installs them via apt (sudo); on Fedora/Arch it prints the matching dnf/pacman command. The MCP server itself is distro-agnostic — any Linux with these tools works. openbox is not optional in practice: without a window manager Xephyr ignores synthetic pointer motion, so clicks would land in the screen center (launch_app warns when it is missing).
• A browser in the sandbox (Chromium, Chrome, Brave, Edge, Vivaldi or Opera).
• NVIDIA GPU (≥6 GB VRAM) — only for the local vision mode. The server checks this itself (status() → vision.gpu) and auto falls back to basic/none when it is not met.

Install

Pick a vision backend and run its one-liner (clone → install). Each one installs the system packages (auto via apt on Debian/Ubuntu) and the Python deps for that mode, and writes a ready-to-paste mcp-config.json with your paths.

`none` — no local models; your agent reads screenshots itself (lightest, instant):

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh none

`basic` — Tesseract OCR grounding (parse_screen → text lines + coordinates, CPU-only):

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh basic

`local` — OmniParser on an NVIDIA GPU (parse_screen → text + icons, pixel-precise; ~4 GB weights, ≥6 GB VRAM):

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh local

Your choice is written to .vision-mode (override per run with the GLOVEBOX_VISION env var). Without either, the server runs in `auto` and picks the best backend that is actually usable. python vision.py check prints the same diagnosis status() returns (GPU, weights, tesseract).

Works with any MCP client / harness

Claude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host. Two compatibility notes: `basic`/`local` return element coordinates as text, so they work even with text-only agents; `none` relies on the client passing the tool's screenshots to a multimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients). In any mode, clipboard() gets exact text out of an app without vision at all.

Quickstart

1. Start the sandbox (leave it running):

   ./start-display.sh              # 1440×900 Xephyr window with a browser

./start-display.sh 1920x1080 # …or pass a screen size (or set $RES)

   Log into any sites or apps you want to automate **in that window**.

1. Register the server with your MCP client. install.sh already wrote `mcp-config.json` with your real install path — copy its glovebox block into your client's MCP config:

   { "mcpServers": { "glovebox": {

"command": "/abs/path/to/glovebox-mcp/.venv/bin/python", // filled in by install.sh "args": ["/abs/path/to/glovebox-mcp/server.py"], "env": { "DISPLAY": ":1" } } } }

   Restart the client so it loads the server.

1. Ask the agent to screenshot / click / type — it operates only on the :1 window.

Driving it with an AI agent? Paste AGENTS.md into the agent's system prompt — it

teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.

Tools (23)

Every control tool takes `instance=N` and optional `observe` / `settle_ms` (see below). In local mode OmniParser is lazy-loaded on first parse_screen (~6 s once, then ~2 s/parse), after a GPU preflight (VRAM present and free).

Structured, honest responses

Every tool returns JSON: {"ok": true, "action": "click", "instance": 1, "detail": "clicked (10,20) button 1", …}. Failures come back with the MCP isError flag set and the same JSON shape embedded in the error text — {"ok": false, "error": "…", "fix": "…"} — where `fix` names the call that unblocks you (e.g. a click on a dead instance says to run list_instances() / launch_app(); an unknown element id says to re-run parse_screen(); a GPU that is too small says which mode to switch to). Silent no-ops are treated as failures too: an invalid keysym, a zero scroll, an off-screen click, closing an instance that isn't running, an app that exits on launch, or non-ASCII typing without xclip all error instead of pretending success. Clamped waits are flagged (clamped / settle_clamped). Image-returning calls (screenshot(), observe="screenshot"|"parse") return [json, image]. No subprocess can hang a call: xdotool/xdpyinfo/xclip run under timeouts and a frozen display is reported as such.

Vision backend (selectable, self-diagnosing)

GLOVEBOX_VISION env var, or the .vision-mode file, or default `auto`:

In local mode the models are loaded by one shared vision-service process per machine (vision_service.py), not by every agent's server: the first parse_screen starts it, parallel agents queue on it (queued_ms in the result), and it exits after 15 idle minutes so a forgotten service never pins the GPU. If it can't be reached the server loads the models in-process instead — the service is an optimization, never a new failure mode. status() shows it under vision.omniparser.service; python vision_service.py status|stop from the shell.

An explicitly requested basic/local that is not usable fails `parse_screen` with the fix named (OmniParser not installed, no/too-small GPU, GPU out of memory, missing tesseract language data) — it never silently falls back; only auto does, and it says so. Tesseract languages default to slv+eng filtered to what is installed (GLOVEBOX_TESS_LANG overrides). Switch anytime with ./install.sh (installs only what that mode needs).

Multi-instance (a fleet of app windows)

Every control tool takes `instance=N` (default 1 = the start-display.sh sandbox). Sp