Screen Control

🖥️ Computer use for all agents

A local remote-control system for AI agents: watch your computer's screen

live and send mouse/keyboard commands to it. Everything runs on your

own machine — no data ever leaves it, no cloud middleman.

Every keystroke and every screenshot in this GIF went through the API — the agent never touched a physical keyboard.

---

If You Are an AI Agent

Read [AGENT_GUIDE.md](AGENT_GUIDE.md) first — it is written for you:

endpoints, safety gates and working recipes. This README is the human-facing

reference.

First ask the server what it is — GET /api/capabilities → platform. Windows: follow the whole guide as written. Linux: check echo $XDG_SESSION_TYPE — X11 is supported (guide → Linux (X11) Usage); Wayland is unsupported: the server refuses to start, HTTP 501 UNSUPPORTED_DISPLAY_SERVER (switch to an X11 session). Full routing lives in AGENT_GUIDE.md → Find Your Environment First.

---

Why Screen Control?

AI agents today can write code and call APIs — but they can't *see* or *touch* your desktop. Screen Control gives any agent general-purpose computer use over a clean, safety-gated HTTP/MCP interface:

• Perceive — OCR for text, single frames or a live MJPEG stream for vision-capable models, and a text-only diff endpoint for models that can't consume images at all.
• Act — absolute and relative mouse, Unicode-safe keyboard, window management, background (focus-free) control, virtual desktops.
• Stay safe — token auth, blocked deadly shortcuts, focus guard, a stuck-input watchdog and an emergency failsafe are all enforced server-side, no matter how confused the agent gets.

One process, zero configuration, works with any language that can speak HTTP — or natively through MCP in Claude Desktop, Cursor, VS Code and cloud agents.

Performance Is Agent-Bound

Screen Control is the perception and actuation layer — the eyes and hands. The effective speed and capability of any agent using it are bounded by that agent itself and by the environment it runs in:

• Thinking speed — one action per agent "turn": the perceive → plan → act → verify loop lives in the agent, so model inference latency and reasoning depth directly set the pace. The API itself adds only milliseconds per call.
• Context capacity — screen readings (OCR text, frames, diffs) consume the agent's context window; a larger window means more situational awareness before verification degrades.
• Runtime environment — network latency, MCP/HTTP round-trip overhead, tool-call limits and hosting constraints all stack on top of the loop.

In practice this means: the same repo makes a fast reasoning model fast and capable, and makes a slow model slow — the toolchain is not the bottleneck. Real-time or action-heavy tasks need an agent with fast inference and tight tool-loop latency; slower agents should prefer deliberate, verification-heavy tasks.

---

Table of Contents

• If You Are an AI Agent
• Why Screen Control?
• Features
• Architecture
• Installation
• Linux (X11)
• Quick Start
• API Reference
• Authentication
• Screen Capture
• Mouse Control
• Keyboard Control
• OCR (Screen Reading)
• Vision Access (Image Models)
• Window Management
• Focus-Free (Background) Control
• Virtual Desktops
• Game Mode
• Safety Endpoints
• 🤖 For AI Agents
• MCP Support (One-Click Cloud Agents)
• Security Model
• Game Mode Guide
• Vision Access Guide
• Troubleshooting
• Testing
• Contributing
• License

---

Features

---

Architecture

┌─────────────────────────────────────────────────────────┐

│ Browser (Web UI) │ │ ┌──────────┐ ┌──────────┐ ┌────────────────────────┐ │ │ │ Live │ │ Control │ │ Windows / Game Mode │ │ │ │ View │ │ Panel │ │ Panel │ │ │ └────┬─────┘ └────┬─────┘ └───────────┬────────────┘ │ │ │ │ │ │ └───────┼──────────────┼─────────────────────┼──────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────┐ │ HTTP API (Flask) │ │ 127.0.0.1:8745 │ │ │ │ /api/screenshot /api/mouse /api/key │ │ /api/vision/* /api/ocr /api/window │ │ /api/game /api/held /api/release_all │ │ /api/windows /api/desktops /api/desktop │ │ │ │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Auth Layer │ │ Watchdog │ │ OCR Engine │ │ │ │ (token) │ │ (30s auto) │ │ (RapidOCR) │ │ │ └─────────────┘ └──────────────┘ └──────────────┘ │ └─────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────┐ │ control.py (Core) │ │ │ │ Screen: mss (fast capture), PIL (processing) │ │ Mouse: pyautogui (absolute), SendInput (relative) │ │ Keyboard: pyautogui + SendInput+KEYEVENTF_UNICODE │ │ Windows: Win32 API (EnumWindows, SetForegroundWindow) │ │ Background: PrintWindow (capture), PostMessage (input) │ │ Virtual Desktops: pyvda │ │ Game Mode: ClipCursor + MOUSE_MOVE_RELATIVE │ └─────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────┐ │ backends/ (pluggable) │ │ WindowsBackend │ LinuxBackend │ MacOSBackend│ │ (full) │ (X11) │ (stub) │ └──────────────────────────────────────────────┘

Coordinates & Concurrency

Per-Monitor DPI awareness. control.py calls SetProcessDpiAwarenessContext(PER_MONITOR_AWARE_V2) at import time — before the pyautogui import, because pyautogui touches coordinate APIs during import and would otherwise lock the process to the interpreter manifest's default (system-aware). With PMv2 active, every coordinate in the system is a physical pixel end to end: mss capture, OCR bounding boxes, pyautogui/SendInput clicks, ClipCursor. On High-DPI displays (125%/150% scaling) nothing drifts between what OCR reports and where the mouse clicks.

Lock architecture. The server uses two independent locks instead of one global lock:

A slow OCR (3–5 s on a busy screen) no longer freezes concurrent screenshot or vision reads — reads queue behind reads, inputs behind inputs.

Live-Loop Working Principle

This system is designed for a live perceive-act loop, not pre-written command chains:

1. READ — OCR or vision reads the screen before and after every action
2. ONE ACTION — each round sends a single command
3. VERIFY — acceptance is "it appeared on screen", not "I sent it"
4. ADAPT — if verification fails, the next step changes based on what is actually seen

This is enforced by the expect_hwnd guard: typing is refused (409) if the foreground window doesn't match the target.

---

Installation

Pick the version that fits how your agent works:

Skill version

The portable agent skill lives in skills/computer-use/ (SKILL.md plus a full API reference). It teaches any skills-capable agent the perceive-act-verify loop, the safety rules and the endpoint catalogue — the agent then drives a running server.

# Claude Code — user-level (available in every project)

cp -r skills/computer-use ~/.claude/skills/

Claude Code — this project only

mkdir -p .claude/skills && cp -r skills/computer-use .claude/skills/

For other skill-capable agents, copy the folder in