ios-agent

ios-agent

Give an AI agent an iPhone. It checks every step it takes.

ios-agent lets an AI agent use an iOS Simulator or a real iPhone, over a cable or Wi-Fi. Every action returns the screen it produced, so the agent can tell when a tap did nothing or typed text did not land. Screens arrive as a few hundred tokens rather than tens of thousands. Anything that sends, pays, deletes or reaches another person asks first. It uses only Apple's public APIs, so an Xcode update does not break it. Use it from the terminal, from Claude Code or any MCP client, or as a Python library.

One goal, start to finish, at 2.5x. The agent deep-links into Maps for the driving time, then taps through to walking and transit and scrolls to read the detail: 4 actions, 1 observation, 2,767 device tokens, 49.8s of real time. The terminal is the agent's own transcript; the phone is an iOS Simulator being driven by it.

uv sync && uv run ios-agent quickstart

quickstart checks the toolchain, offers the repairs that are cheap enough to be worth offering, builds WebDriverAgent if it is missing (about 20 seconds, once), and drops you into manual mode, which drives the device by hand and needs no API key. When you want the agent itself:

uv run ios-agent "turn on bold text"

Contents

• What you can do with it · Features · Who it is for
• Requirements · Setup
• The terminal app · Connecting your own agent over MCP
• What the model sees · Safety
• Measured on real hardware
• Why it is built this way · Why only public APIs
• Tracing · Development · Contributing

What you can do with it

Typical goals: check that a change you just made works in the running app, change a setting, read an answer out of an app that has no API, or walk a flow on a phone you cannot hand to a test suite. docs/use-cases.md lists what works today, with the evidence for each and what is not supported yet.

Features

It knows when an action did not work

The most common complaint about agent tools for phones is an action that reports success while nothing happened. Here:

• Every action returns the screen it produced, with screen_changed, so a tap that did nothing is visible at once. A real no-op reports screen_changed=False on a physical iPhone too, not only on a simulator.
• Typed text is read back from the field. Text that did not land returns ok: false with what the field shows, rather than being reported as typed.
• The agent names elements, never coordinates. It passes a ref like e2; if the screen moved, the host re-finds the same element by identity through six tiers, and refuses when a ref now points at something else.
• Switches are set, not toggled. Asking for on when a switch is already on does nothing, instead of turning it off.
• Retries are safe. Idempotency keys mean an agent framework that replays a step does not tap Send twice.

It works on a real iPhone, over a cable or Wi-Fi

• Verified on an iPhone 17 Pro Max on iOS 26.6, over USB and over Wi-Fi, at the same action count as the simulator.
• ios-agent doctor checks the toolchain, the runtimes, the tunnel and the runner's signing expiry, and gives a remedy for each failure.
• The device runner stops with the process that started it, a SIGKILL included, so a crashed run does not hold the phone. ios-mcp reset clears anything else.
• The device picker never pre-selects a physical phone, and /device switches devices mid-session.

It is cheap on tokens

• Screens arrive as a compact digest, not accessibility XML: 251 raw nodes to 12 elements on a real third-party screen, 50 to 474 tokens per step across thirteen golden flows.
• When the next screen is similar, an action returns only what changed.
• A 300-row Contacts list comes back in one observation of 438 tokens.
• ios_find searches the full tree for anything the digest left out, so compaction never hides a control for good.
• The bundled agent looks at the screen once per run, because every action already hands back the screen it produced.

It asks before anything risky

• Send, Pay, Buy, Delete, Confirm, Sign Out, and anything that reaches another person (Like, Follow, Share, Message) need approval before they happen. With no one to ask, they are refused.
• Passwords come from the Mac's keychain and go straight to the device. They never enter a prompt or the audit trail, and once typed, every screen the session returns shows [secret] in their place.
• Card numbers and email addresses are redacted before any client sees the screen, and every action is recorded in an exportable audit trail.

It keeps working when Xcode updates

Everything goes through XCTest, the framework Apple ships for UI testing. Faster routes through private frameworks were measured and turned down, and Xcode 27 broke several tools that took them. See Why only Apple's public APIs.

Three ways in, and any model

• A terminal app that streams the model's reasoning beside the screen it is reading, with actions, tokens and cost on screen as they climb.
• An MCP server: 31 tools, 5 resources and an ios_operator prompt, over stdio or HTTP. See the tool reference.
• A Python library, IosSession, that the server and the agent both sit on, so your own agent can skip the protocol.
• Anthropic, OpenAI, Azure OpenAI, Gemini, Vertex AI, Bedrock, Groq, Mistral or a local Ollama model, switched by two environment variables.

It is measured, and you can rerun the numbers

39 of 39 agent runs succeeded at 1.14x the actions a hand-written oracle needs, for $1.50 in total. Golden flows track tokens, time and how each element was found, and CI fails if the free series moves without a recorded reason. See Measured on real hardware.

What it does not do

• No Android, no other platforms, no builds or profiling.
• Flutter canvases and WebViews have no tree to read. The digest says so and points at a screenshot, rather than returning a screen that looks empty. See ADR 0007.
• The approval gate is not a defence against instructions planted in a screen. See ADR 0013.
• A physical iPhone needs a signed runner, and a free Apple ID's profile lasts seven days.

Who it is for

• People building agents that need iOS hands, from Claude Code, Cursor or any MCP client.
• iOS developers who want their coding agent to check a change on a simulator or on their own phone.
• Anyone studying agent engineering. The eval harness and the decision records show what was measured, and what was turned down because the numbers said no.

If you need Android, React Native profiling, Xcode builds, or many devices in parallel, another tool will suit you better. The documentation site has an honest comparison.

Requirements

Xcode ships without a simulator runtime. If xcrun simctl list runtimes is empty, xcodebuild -downloadPlatform iOS fetches one (around 8 GB). If a runtime is installed but no simulator has been created, ios-agent offers to create one for you: that part takes about a second.

Setup

uv sync

./scripts/prepare_wda.sh simulator # builds WebDriverAgent, once uv run ios-agent doctor # says exactly what is still missing

doctor is the first thing to run whenever anything misbehaves. It checks the toolchain, the simulator runtimes, the tunnel, WebDriverAgent's signing expiry and the model, and returns a remedy for each failure rather than letting it surface later as a connection error.

The app runs those same checks before it touches a device, so a machine that is not set up is told so in about a second rather than after a simulator has booted.

The terminal app

uv run ios-agent                             # open it, decide later

uv run ios-agent "turn on bold text" # give it a goal uv run ios-agent --pick "turn wi-fi off" # choose the device from a list uv run ios-agent manual # drive it by hand, no model needed uv run ios-agent devices # what is reachable

It streams the model's reasoning as it arrives, shows the digest the model is reading beside it, and keeps the numbers on screen while they climb: actions, observations, device tokens, cost.

`manual` mode needs no API key. It drives the agent's nine verbs by hand, which is the fastest way to debug perception on an app nobody has pointed this at before.

The front end is held to one rule, asserted rather than argued: watching a run may not change what it costs. tests/tui/test_cost.py runs the same task wrapped and unwrapped and compares every counter by equality.

Choosing a model

The provider is configuration, not a dependency. The loop builds through LangChain's init_chat_model, so switching is two environment variables and an extra:

uv sync --extra openai

I