XuLan ScreenPilot

XuLan ScreenPilot

Turn the real Windows screen into a structured element table your AI agent can operate.

English | 简体中文

XuLan ScreenPilot solves the two classic pain points of GUI automation for AI agents — *not seeing the elements* and *missing the click* — by giving the agent a numbered element table instead of a screenshot to guess coordinates from. The agent scans once, reads the table, and drives the screen by element id (click / type), in a closed loop.

Pure local CPU. No cloud model, no GPU, no API key.

How it works

Three engines run in parallel and fuse into one element table (priority UIA > OCR > visual):

Fusion dedupes by center-containment or IoU ≥ 0.3, numbers elements top-to-bottom / left-to-right, and produces:

• elements.json — the element table cache (queried by id for click/type)
• annotated_scan.png — annotated screenshot (red = UIA, blue = OCR, green = visual, id printed on each box)

On top of the table:

• Frame-to-frame id stability — rescan keeps the same element on the same id; new elements get new ids (meta.tracker reports kept/new/lost).
• Background capture — grab a covered window's real pixels via PrintWindow (no need to raise it).
• Drifted clicks — physical clicks land with Gaussian jitter and move trails, so automation doesn't look robotic; precise=True restores pixel accuracy.
• Stacking awareness — every coordinate click reports which top-level window it actually hit (hit_window), and can refuse clicks on occluded targets (top_must).
• Two-level vision + text channel — crop any element at native resolution to read fine text/icons; a SigLIP vector book lets you find UI elements by name or by image without rescanning.
• Time dimension — burst watch for toasts/transient feedback, show-verify for typing, ESC probe for hangs, and static/dynamic separation (UI base layer vs. moving entities) across frames.
• Business-level waits — expect {find, mode: appear|disappear} makes a click "successful" only when the business evidence shows up on screen.

Install

Requirements: Windows 10/11, Python 3.11+, CPU only.

git clone <repo-url> screenpilot && cd screenpilot

pip install uiautomation rapidocr-onnxruntime onnxruntime opencv-python numpy pillow pyperclip pyautogui mcp python screenpilot.py scan :: full-screen scan, prints the element table

Tested on Python 3.13. The optional element-book vector channel and the benchmark matcher additionally need tokenizers and the SigLIP int8 ONNX weights under SCREENPILOT_MODELS_DIR.

Optional: the semantic matcher and the element-book vector channel use SigLIP int8 ONNX models. Set SCREENPILOT_MODELS_DIR to a local folder holding the model directories (see README.zh-CN.md §两级视觉与文本通道). Datasets for the public benchmark go under SCREENPILOT_DATASETS_DIR.

Quickstart: the agent closed loop

# MCP tools (stdio server) or the CLI expose the same verbs:

scan_screen(window="记事本") # 1. scan -> numbered element table click_element(element_id=7) # 2. act by id (UIA control-level when possible) type_text(element_id=3, text="hi") # 3. type into an edit field click_safe(element_id=12, expect={"find": "设置", "mode": "appear"}) # 4. act only when evidence shows watch_burst(window="记事本", seconds=3) # 5. catch transient feedback frames

MCP config (e.g. Claude Desktop / ZCode):

{"mcpServers": {"screenpilot": {

"command": "python", "args": [" /screenpilot/mcp_server.py"], "env": {"PYTHONIOENCODING": "utf-8"} }}}

Benchmarks

Public GUI-grounding benchmarks, frozen matcher, official scoring. Pure CPU on a desktop with a GTX 1070 Ti (GPU unused):

\* oracle = the GT target has a recognizable candidate in the element table — an upper bound assuming a perfect matcher. It separates the *recognition engine's* account from the *matcher's* account: on ScreenSpot-Pro the gap is almost entirely matcher-side (match_error 88% of misses), which is where the roadmap points. The acc numbers are with a frozen heuristic matcher standing in for the agent; in real use the agent reads the table itself, so live experience tracks the oracle line more than acc. We report both, no cherry-picking.

Reproduce: python bench/public_sspot.py --dataset v2 / --dataset pro (datasets not included; see the docstring for the two HF sources).

MCP server

18 tools over stdio (scan_screen, scan_image, click_element, click_safe, click_xy, type_text, press_key, scroll_screen, snap_screen, crop_screen, crop_xy, book_find, watch_burst, verify_typing, probe_hang, split_static_dynamic, window_stack, activate_window). scan_image runs the same engines offline on a static file; book_find queries the persistent element book (two-level: text match + SigLIP vector search) so recurrent UIs are found without a full rescan.

Documentation

Full docs (in Chinese, with internals, known limitations, and the test-suite map): README.zh-CN.md.

Status & roadmap

Working system in daily use as the studio's screen channel. Known gaps are documented, not hidden: dark-theme professional apps (deep-background small text) hurt OCR recall, 4K toolbar icons stress the matcher, and the Pro benchmark score is capped by the frozen matcher — see README.zh-CN.md §已知限制与坑.

License

Open core: the core engine in this repository is released under the Apache License 2.0. Studio value-added integrations (closed source) are not part of this repository.

Acknowledgements

Built on RapidOCR, uiautomation, OpenCV, and onnxruntime.