A $7 ESP32 Board Lets an AI Tap Through Any iPhone App by Posing as an AirPlay Screen and a USB Mouse

Chris, seen from above at a workbench, watches a tiny circuit board wired to a phone while a cartoon robot hand taps the phone's screen on its own — Illustration: LockerGnome

“Let an AI model use a real iPhone with a $7 ESP32,” Henry Yuan’s README opens. “Just the phone and the board: no computer, no jailbreak.”

The first version, tagged v0.1.0 on October 14, still needed one. A Mac or Linux machine ran UxPlay to catch the iPhone’s screen and a Python server to think. The next day Yuan moved the agent into an iPhone app on the phone itself, and the computer dropped out. The old version lives on as the legacy branch.

The board does two jobs. For the agent’s eyes, it “opens its own Wi-Fi hotspot and appears in the phone’s Screen Mirroring list as ESP32 Agent.” It doesn’t decode the video. It keeps the H.264 stream and forwards it over Bluetooth Low Energy to the app. For the hands, the board is a standard USB mouse and keyboard. “With AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.”

The app is the brain. It decodes the stream, sends screenshots to the model, and runs a planner and an actor that can tap, swipe, draw, type, press keys and go home. The pointer is relative, so the app tracks it, re-homes it in a corner, and wiggles it before precise taps to find it on screen and correct the miss.

The demo task is the kind of thing no app exposes an API for: “Set an alarm for 8 AM that repeats every Monday and Tuesday, then save it.” The agent leaves its own app, opens Clock, types the time into the wheel, ticks the days and saves.

Play the video on YouTube

The parts list is short. The tested board is a Seeed Studio XIAO ESP32S3, the plain 8 MB PSRAM version, “About $7.” The iPhone needs iOS 18 or later and cellular data, because the board’s hotspot takes the Wi-Fi. USB-C iPhones plug in directly, and Lightning models need Apple’s camera adapter. The model can be a cloud API or your own server; the README lists vLLM, SGLang, llama.cpp and Ollama. “A free Apple ID can sign it (re-sign every 7 days).”

The limits are in the README too. Bluetooth carries “about 300-500 kbps,” so the mirror is 480×1040 at five frames a second or less, and “each step takes a few seconds, most of it the model.” iOS won’t mirror passcode entry, Apple Pay or DRM video, so the agent can’t see them. Typing is ASCII only, and landscape apps aren’t supported yet.

Why it matters

No API means no automation, until now. Shortcuts reaches only what an app chooses to expose. This works at the level of fingers and eyes, so anything a person can tap is in reach, from Clock to an app that never shipped an integration.

Apple’s own accessibility feature does the tapping. AssistiveTouch exists so people who can’t touch the screen can use a pointer. The board just presents itself as the mouse that feature expects, which is why nothing has to be jailbroken.

The phone watches itself. Screen Mirroring, also a stock feature, is the camera. Earlier projects like VersaiOS needed a computer to receive the mirror. The README says the closest one, phone-agent, already ran as an iPhone app, and this one “also brings the eyes.”

Every run can be replayed. “Every run is kept on the phone: replay it step by step with the taps and swipes drawn on top.” Traces export for another agent to review.

Phone agents so far have mostly meant Android, a desktop in the loop, or a developer setting turned on. Yuan’s build uses three features every iPhone already ships with, Screen Mirroring, AssistiveTouch and a USB port, and puts a few dollars of hardware between them. Apple can’t easily close that without hurting the people AssistiveTouch was built for. What the app does still lean on is a private iOS API, suspend, to leave itself and hand the screen to another app. The README calls that “fine for a self-built app, to be replaced before the App Store,” and doesn’t say what replaces it.

Steal this

When an app has no API, look for the accessibility layer before you look for an exploit. Pointer, keyboard and screen-reader paths are built to be driven from outside. This is like using the service elevator when the front door is locked: it’s there on purpose and nobody has to pick a lock.
Give an agent a way to check its own hands. Yuan’s app wiggles the pointer, finds it on screen and corrects before a precise tap, then tells the model whether the screen changed. Kind of like a carpenter who measures again after the first cut instead of trusting the pencil mark.
Keep every run as a replayable trace with the actions drawn on top. When an agent does something odd, you want the tape, not the agent’s own account of it. This is like a dashcam: you rarely watch it, and you’re glad it’s there when you need it.

The README’s last word on safety is plain: “The agent acts with your hands: watch it, and keep the stop button within reach.” Yuan, who posts demos as @iamhenryyuan, developed it against nvidia’s Qwen3.8-Flash-Next-NVFP4 on two DGX Sparks, and the to-do list still has non-ASCII input, like Chinese through the pinyin keyboard, unchecked.

Search

A $7 ESP32 Board Lets an AI Tap Through Any iPhone App by Posing as an AirPlay Screen and a USB Mouse

More AI