ElfDeckAI

ElfDeckAI

Let an AI play games on your Steam Deck.

ElfDeckAI gives an AI assistant such as Claude a virtual controller, keyboard and mouse on your Steam Deck, plus the ability to see the screen. The AI looks at the game, decides what to do, and presses the buttons itself. It works in both Gaming Mode and Desktop Mode, and on other Linux machines too.

Start here: Steam Deck (about 5 minutes)

You need a Steam Deck, and a PC on the same Wi-Fi with an AI app that supports MCP, such as Claude Desktop.

1. Install on the Deck. Switch the Deck to Desktop Mode (Steam button → Power → Switch to Desktop), open Konsole, and paste:

curl -fsSL https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/install-on-deck.sh | sh

At the end it prints the Deck's IP address. Keep it for step 3.

2. Turn on SSH on the Deck (once), so your PC can reach it. In the same Konsole window:

passwd                            # choose a password for the Deck, if you haven't

sudo systemctl enable --now sshd

3. Connect your PC. On the PC, paste one line. It asks for the Deck's IP address and its password (once), then sets everything up, including Claude Desktop.

• Windows (in PowerShell):

  irm https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/setup-pc.ps1 | iex

• Mac or Linux (in Terminal):

  curl -fsSL https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/setup-pc.sh | sh

4. Try it. Restart Claude Desktop and ask: *"Take a screenshot of my Steam Deck."* When you see your Deck's screen, start a game and ask the AI to play.

Tips

• Turn-based games, RPGs, puzzles and menus work best. The AI needs a second or more per move, so fast action games are out of reach.
• In Gaming Mode, if a game ignores the AI's controller, ask the AI to make its controller Player 1. It knows how, using Steam's Reorder Controllers menu while the game runs.
• Something not working? Give AI_SETUP_GUIDE.md to your AI and ask it to help. It covers setup and troubleshooting step by step.

How it works

Three small programs on the Deck create virtual input devices through the Linux kernel's /dev/uinput, so games and Steam see them as real hardware:

• ElfKey: a virtual keyboard
• ElfPad: a virtual Xbox 360-class gamepad (buttons, analog triggers, two sticks, D-pad)
• ElfMouse: a virtual mouse, which can also click an exact point on the screen

A fourth program, an MCP server, exposes them, together with a screenshot tool, as tools any MCP client can call. Your PC's AI app starts that server on the Deck over SSH. Nothing depends on a particular game or emulator: ElfDeckAI has played Pokémon FireRed in RetroArch on a physical Steam Deck, from the intro through the first rival battle, with the agent seeing only screenshots. See notes/driving.md for what was learned.

Everything is single-file Python using only the standard library, and the Deck install needs no sudo.

Other Linux machines

pip install elfdeckai

elfkey & elfpad & elfmouse &

echo "key x 150" > /tmp/elfkey.fifo # tap X for 150 ms echo "press a" > /tmp/elfpad.fifo # tap A echo "move 300 -100" > /tmp/elfmouse.fifo # move the pointer echo "click left" > /tmp/elfmouse.fifo # left click

grep -E "ElfKey|ElfPad|ElfMouse" /proc/bus/input/devices # confirm the devices exist

Without pip, clone the repository and run python3 elfkey.py (and so on) directly. Runtime state lives in /tmp and is cleared on reboot.

/dev/uinput is user-writable on the Steam Deck by default. On other distributions, install the udev rule first (see Installation).

MCP server

elfdeckai-mcp is a Model Context Protocol server (stdio transport) that exposes 22 tools:

screenshot captures the screen of the machine the server runs on and returns a PNG, so an agent can work in a look, act, look again loop without a separate video stream. It supports the Steam Deck in both Gaming Mode (through gamescope) and Desktop Mode (KDE Plasma), wlroots compositors and X11. Screens larger than 1280 pixels on their long side are scaled down before sending (set ELFDECKAI_SCREENSHOT_MAX to change the limit, or 0 to turn it off), and the result includes the image's size.

mouse_click_at and mouse_move_to take coordinates in the pixels of the most recent screenshot, so an agent can click exactly what it sees, whatever the screen's real resolution or the scaling applied. They assume a single display.

Running on the same machine as the daemons:

{"mcpServers": {"elfdeckai": {"command": "elfdeckai-mcp"}}}

Running on a Steam Deck from an AI client on your PC, over SSH (the setup-pc scripts in Start here write this for you):

{"mcpServers": {"elfdeckai": {

"command": "ssh", "args": ["-i", "~/.ssh/elfdeckai_ed25519", "-o", "IdentitiesOnly=yes", "-o", "BatchMode=yes", "deck@DECK-IP-ADDRESS", ".local/bin/elfdeckai-mcp"]}}}

It logs in with an SSH key, because the MCP client cannot answer a password prompt. Without pip, use "command": "python3", "args": ["/path/to/elfdeckai_mcp.py"].

Every tool call is validated against the tool's schema and the daemon's own parser before anything is written to a FIFO. If a daemon is not running, the call returns an error immediately instead of blocking.

Command reference

ElfKey (/tmp/elfkey.fifo)

Keys: a–z, 0–9, arrows, enter, esc, space, tab, backspace, shift, rshift, ctrl, rctrl, alt, ralt, capslock, f1–f10, home, end, pageup, pagedown, insert, delete and common punctuation. The full table is KEYS in elfkey.py.

ElfPad (/tmp/elfpad.fifo)

Buttons: a b x y lb rb l2 r2 select start guide l3 r3, plus D-pad directions up down left right.

ElfPad identifies as an Xbox 360 controller, so SDL, Steam Input and RetroArch map it without configuration. l2 and r2 drive the analog trigger axes, as on the real controller; there are intentionally no digital trigger buttons, because declaring them shifts Select, Start and Guide off the standard button indices.

ElfMouse (/tmp/elfmouse.fifo)

move is relative: negative dy moves up. Its units are motion counts, not guaranteed screen pixels, because pointer acceleration applies. Each command is limited to ±10,000 counts and ±120 wheel ticks.

moveto uses a second virtual device, ElfMouse Absolute, which compositors treat like a virtual machine's tablet pointer, so the position is exact. Buttons are pressed on whichever device moved last.

Safety model

Because the caller may be a language model, the daemons are built so that bad input cannot leave the machine in a bad state:

• Malformed input is refused, not fatal. Every line passes a strict parser; invalid commands are logged and ignored.
• Taps are bounded. A timed hold lasts at most 5 seconds. Longer holds use down/up.
• Nothing stays stuck. Timed actions restore in a finally block, held inputs are tracked, and on quit, SIGTERM, SIGINT or SIGHUP every input is released before the device is removed. If the MCP client disconnects mid-hold, the server releases every held input as it exits. key_release, mouse_release and pad_reset release everything on demand.
• Owner-only access. Each FIFO is created with mode 0600.
• Audit log. Every accepted command is written to stderr (the journal, when run as a service). That includes every key of anything typed — passwords included, one line per keystroke. Do not have an agent type secrets you are not willing to see in the journal.
• Robust protocol handling. The MCP server returns proper JSON-RPC errors for malformed or oversized requests and never blocks on a stopped daemon.

Scope: these are real OS-level input devices. They act on whichever window has focus, including terminals and browsers. ElfDeckAI keeps input bounded, validated and releasable; deciding where an agent is allowed to act is up to the operator. Only run agents you trust, in sessions you can supervise.

Manipulated callers: the hardening above covers a caller that hallucinates or sends malformed input. It does not cover a caller that is manipulated — and the agent's eyes are a screenshot, so anything on screen (a web page, a game dialog, a chat message) is untrusted input that could try to talk it into typing into a terminal. There is no key or window allowlist and no physical kill switch; the mitigations are the operator's: supervise the session, keep terminals and browsers you care about out of focus, and stop the daemons (systemctl --user stop elfkey elfpad elfmouse) when the agent is not driving. If the MCP client disconnects, the server now releases every held input on its way out — but a live, manipulated agent is an operator problem, not one this validation layer can solve.

Comparison with other approaches

Installation

Requirements: 64-bit Linux with /dev/uinput and Python