droidsight

droidsight

An MCP server that drives a real Android device over ADB — and hands the agent back the screen its action produced.

One native binary. The server has no Python, Appium, scrcpy, or ffmpeg dependency and runs no Node at all — npx below is only a convenient installer — so the runtime is the binary plus adb. A background H.264 screenrecord stream is decoded in process, so the screen that follows an action is attached to the tool result automatically instead of costing a second round trip to ask what happened.

Every image carries the coordinate space it was produced in, so a model can tap what it just looked at without guessing a scale factor. When the accessibility tree comes back empty — Flutter, React Native, canvas, games — OCR and template matching still find the target.

This is a powerful local controller, not a sandbox. Run it only against a device and an MCP client you trust.

Install

npx -y @edgecasehuman/droidsight

That downloads a prebuilt binary for your platform and needs no build tools. The server is also listed in the official [MCP Registry] as io.github.edgecasehuman/droidsight.

To compile it instead, from [crates.io]:

cargo install droidsight

Or from source:

cargo build --locked --release --bin droidsight

Both compiling routes need NASM on the PATH. The H.264 decoder is built from C++ by openh264-sys2, which assembles its hot paths with NASM, and no platform ships it by default -- without it the build fails inside a transitive dependency rather than in this crate. Install it with apt install nasm, brew install nasm, or winget install NASM.NASM. The npx route above needs none of this.

The server is target/release/droidsight. It speaks newline-delimited JSON-RPC 2.0 over stdin and stdout.

Client configuration

{

"mcpServers": { "android": { "command": "npx", "args": ["-y", "@edgecasehuman/droidsight"], "env": { "DROIDSIGHT_DEVICE_SERIAL": " " } } } }

ADB must be reachable through PATH, ANDROID_SDK_ROOT, ANDROID_HOME, or an explicit DROIDSIGHT_ADB_PATH. If more than one authorized device is visible the server refuses to guess; set DROIDSIGHT_DEVICE_SERIAL.

Tools

Thirty-one tools are published by default. Two more appear only when DROIDSIGHT_ALLOW_SHELL=1 is set, and are listed last.

Seeing the screen

Driving the device

Applications

Device and system state

Diagnostics and data

Capture and presence

Only with DROIDSIGHT_ALLOW_SHELL=1

Coordinate space

Screenshots are downscaled to 720 pixels wide by default, so a coordinate read off the returned pixels is usually not a device coordinate. Every tool that returns an image also returns a metadata.image object describing how to convert one:

{

"image": { "width": 720, "height": 1600, "device_width": 1080, "device_height": 2400, "origin_x": 0, "origin_y": 0, "scale": 1.5, "coordinate_space": "image", "note": "To convert a coordinate (x, y) read from this image into a device coordinate: device_x = 0 + x * 1.5, device_y = 0 + y * 1.5." } }

When coordinate_space is device, the image is already 1:1 and coordinates can be used directly. Request max_width: 1440 to reduce or remove downscaling. Crops additionally report a non-zero origin_x/origin_y that must be added after scaling.

Taps, uiautomator hierarchy bounds, OCR boxes, and template matches are all in device coordinates. The background stream runs screenrecord without --size, so decoded frames are native resolution and share that one space.

You can avoid the arithmetic entirely. The mcp_android_vision_query tool's elements action returns an indexed snapshot with a precomputed center per element, and its tap_element action takes that snapshot_id and an index.

Environment variables

Continuous vision cache

One background H.264 screenrecord stream starts at process startup and the latest decoded RGB frame is kept in memory. Screenshot tools and input observations read that cache, so a snapshot is available immediately once the first frame decodes. A static screen's last frame stays valid while the stream runs; it does not expire merely because the encoder stopped emitting unchanged pixels.

The stream allows 20 seconds for the first decodable frame before reconnecting. After that, silence is treated as a static screen, while EOF and read errors still reconnect. Starting or stopping a stream synchronously clears its frame cache so a previous session's image cannot be reused. The mcp_android_vision_stream stop action suspends capture until an explicit start; shutdown drops the owned ADB stream and leaves no screenrecord process behind.

Continuous capture has privacy, battery, CPU, and wireless-bandwidth costs. The device screen may contain sensitive information even though cached frames stay in process memory until a tool returns one. Use the explicit stop action when capture should be suspended.

MCP transport

Stdout carries one JSON-RPC value per line; diagnostics go to stderr. The server supports initialize, tools/list, and tools/call, plus the initialized notification and legacy mcp.* aliases. Batch arrays are handled member by member and produce one response-array line; notification-only batches produce no output and empty batches are rejected. Clients must complete the ordered initialize request and notifications/initialized notification handshake before listing or calling tools; premature calls return JSON-RPC error -32002. Input frames are limited to 16 MiB; oversized or invalid UTF-8 frames are rejected without preventing the next valid request from being processed.

Published text from logs, events, notifications, device files, hierarchy and OCR results, crash and forensic reports, instrumentation, shell/macro output, and aggregate flows is limited to 256 KiB. Truncated responses carry metadata.truncation with the strategy, original size, returned size, and configured limit; chronological feeds retain their newest tail while file-like output retains its head.

Finite ADB subprocesses drain stdout and stderr concurrently and retain at most 4 MiB from each stream while counting all observed bytes, which prevents pipe deadlocks and unbounded capture. The opti