CLI & AI agents
Your agent already works on your computer. Give it a way to use your iPhone, too. Handsy CLI (esp32agent) connects over Bluetooth, or over your Wi-Fi, to the small board plugged into your phone. Your agent can request a screenshot, tap, swipe, and type, then look again to check the result.
Use Claude Code or Codex through MCP, or connect another agent that can run local commands or use a local stdio MCP server. The agent runs on your computer; the board receives the iPhone’s Screen Mirroring and acts as its mouse and keyboard.
Handsy CLI is open source under the MIT license: github.com/onehumanbeing/esp32-iphone-agent, in cli/ on the release-1.0 branch.
Watch a real task
Section titled “Watch a real task”In this example, Claude Code uses an iPhone to open Chrome’s incognito mode, visit Google Maps, and search for coffee near East Broadway station in New York. It reads the actual screen and works through the task, rather than calling a Maps API.
The same plugin works in DeepSeek Harness. Here it opens Settings, reads which Wi-Fi the phone is on and goes back Home, with the phone (left) and dsh (right) time-aligned:
Try the same task with your agent once connected:
Use esp32agent to open Chrome in incognito mode on my iPhone, visit the Google Maps website, and find coffee near East Broadway station in New York. Check a fresh screenshot after each action and stop when the map and café results are visible.
Setup walkthrough
Section titled “Setup walkthrough”Flash the board first (Prepare the board), then follow this video to install the CLI and connect your agent.
Install once, then hand it to your agent
Section titled “Install once, then hand it to your agent”Handsy CLI 0.1.2 is available for macOS, Linux, and Windows. It needs board firmware v0.2.2 or later (v0.2.3 for the Wi-Fi link), available on the installer’s TestFlight channel; the official 0.2.1 firmware does not include the CLI connection.
macOS or Linux
Section titled “macOS or Linux”curl -fsSL https://handsy.tech/install.sh | shWindows PowerShell
Section titled “Windows PowerShell”irm https://handsy.tech/install.ps1 | iexnpm install -g esp32agentThe npm package needs Node.js 18 or later and downloads the same release for your system.
Installer options
Section titled “Installer options”| Variable | What it does |
|---|---|
ESP32AGENT_VERSION |
Install a specific version, for example 0.1.0. Default: the latest. |
ESP32AGENT_BIN_DIR |
Where to put esp32agent. Default: ~/.local/bin (Windows: ~\.local\bin). |
curl -fsSL https://handsy.tech/install.sh | ESP32AGENT_VERSION=0.1.0 ESP32AGENT_BIN_DIR=/usr/local/bin shThe installers check each download’s SHA-256 before installing it.
Upgrade
Section titled “Upgrade”esp32agent upgrade --check # current and latest versionesp32agent upgrade # install the latest releaseThe CLI needs Bluetooth on the computer. Screenshots also need ffmpeg; the installer can offer a package-manager command for your system. Follow its prompts and reopen your terminal if esp32agent is not found. The shell installer uses ~/.local/bin; make sure that directory is on your PATH.
1. Find and pair your board
Section titled “1. Find and pair your board”Choose the supported board and attach its antenna using the hardware guide. Flash firmware v0.2.3 from the installer’s TestFlight channel; Handsy CLI cannot find a board running 0.2.1.
Pairing does not need the iPhone app. Plug the board into your iPhone, enable Bluetooth on your computer, and keep them nearby. Your computer can stay on its normal internet connection.
An agent with local terminal access can handle discovery and pairing. Give it this request:
Use your local terminal to set up esp32agent for my iPhone. My board runs firmware v0.2.3. Discover my board and pair if needed. Help me find the board’s hotspot password (https://handsy.tech/docs/faq#how-do-i-find-the-boards-wi-fi-password), then guide me to join the board’s Wi-Fi and enable Screen Mirroring and AssistiveTouch on the phone. Verify a fresh screenshot and calibrate the pointer. Ask which board is mine if several are found.
For manual setup, run:
esp32agent devicesesp32agent pairpair discovers and pairs with a single nearby board itself; you do not need to pair it in the computer’s Bluetooth settings first. If more than one board is found, use the identifier returned by devices: esp32agent pair --device <id>.
Run the first pair within three minutes of powering the board. If that window has closed, unplug and reconnect it, then retry.
2. Connect Wi-Fi and mirror your screen
Section titled “2. Connect Wi-Fi and mirror your screen”Find the board’s hotspot name (ESP32Agent-…) and password in the ESP32 Agent app, or read them over USB serial: see how to find the hotspot password. With firmware v0.2.3, esp32agent net shows them too.
Then, on your iPhone:
- Open Settings → Wi-Fi and join the board’s hotspot with its name and password.
- Turn on Control Center → Screen Mirroring → ESP32 Agent.
- Enable Settings → Accessibility → Touch → AssistiveTouch. Keep the pointer visible, and the phone unlocked and in portrait.
The phone uses the board’s Wi-Fi for mirroring; the computer reaches the board over Bluetooth. If you restart the board, reconnect the phone’s Wi-Fi and start Screen Mirroring again.
Optional: put the board on your home Wi-Fi
Section titled “Optional: put the board on your home Wi-Fi”With the board and the computer on the same Wi-Fi, the CLI sends commands and video over Wi-Fi: screenshots are close to live and taps don’t queue behind video. Bluetooth stays connected as the way back.
esp32agent net joinIt asks for your Wi-Fi’s name and password (the password is not shown), sends them to the board over the paired Bluetooth session, and tells you each step: join the same Wi-Fi on the iPhone, then start Screen Mirroring to ESP32 Agent again. esp32agent net shows which link is in use (via: wifi or ble); esp32agent net hotspot puts the board back on its own hotspot. A computer that joins the board’s hotspot also gets the Wi-Fi link.
This needs CLI 0.1.2 and board firmware v0.2.3 (installer’s TestFlight channel). With v0.2.2 the CLI keeps working over Bluetooth.
Check the current screen and calibrate:
esp32agent device infoesp32agent screenshotesp32agent calibrateThe screenshot command saves a PNG on your computer and reports its filePath. Open that file and check that it shows the phone’s current screen. The CLI measures the pointer itself; the first pointer action also calibrates automatically if needed.
If the picture is stale, disconnect and restart Screen Mirroring. If pairing fails, check the antenna and move the computer closer.
3. Connect your agent
Section titled “3. Connect your agent”Complete the initial pairing in a terminal first, or ask an agent with local terminal access to do it using the request above. The MCP server provides phone controls; its tools do not perform the initial device discovery and pairing.
Claude Code
Section titled “Claude Code”Install the Handsy plugin. It adds the MCP server and the skill in one step. Run these inside Claude Code:
/plugin marketplace add onehumanbeing/esp32-iphone-agent/plugin install esp32agent@handsyOr set it up by hand: register the local MCP server, then install the CLI’s Claude skill.
claude mcp add esp32agent -- esp32agent mcpesp32agent skill installThe skill goes under ~/.claude/skills/esp32agent/. It gives Claude the phone workflow and command guidance; MCP provides the tools.
Register the same MCP server with Codex:
codex mcp add esp32agent -- esp32agent mcpDeepSeek Harness
Section titled “DeepSeek Harness”Install the plugin; it adds the same MCP server and the handsy skill:
dsh plugin --profile web add dsh-esp32agentRestart dsh web, then ask for a small task. Full steps: DeepSeek Harness. Plugin source: integrations/dsh for details.
Another agent
Section titled “Another agent”Use esp32agent mcp as a local stdio MCP server, or let your agent call the CLI directly. It needs access to the computer’s Bluetooth and the saved screenshots. A remote agent without access to this computer cannot reach the board by itself.
The skill is also on ClawHub as handsy, and the MCP server is in the MCP Registry as io.github.onehumanbeing/esp32agent (npm package esp32agent). The skill files are MIT-0; the rest of the repository keeps its own license.
For a setup request you can paste into an agent, use the homepage’s CLI entry. The agent setup guide contains the installation and verification steps in plain Markdown.
4. Give it a small first task
Section titled “4. Give it a small first task”Try this request in your agent:
Use esp32agent to take a screenshot of my iPhone. Open Settings and go to Wi-Fi. Take a fresh screenshot after each action to check the result. Stop when the Wi-Fi page is visible.
The agent should look → take one action → look again. Watch the first task and interrupt your agent if it goes off track. For a longer task, describe the destination and what counts as finished.
The CLI provides the controls; your agent supplies the reasoning and AI service. The free AI service in the phone app is a separate way to use Handsy.
Useful commands
Section titled “Useful commands”These are examples of individual actions. Choose coordinates from a fresh screenshot before running a tap or swipe; coordinates are pixels in the original screenshot, not percentages.
esp32agent screenshotesp32agent io tap 200,400esp32agent io swipe 200,800,200,300esp32agent io text 'hello'esp32agent io keys enteresp32agent io button HOMEChinese typing uses the phone’s Pinyin keyboard and requires selecting the intended candidate. Enable that keyboard first and have the agent check the text on screen.
All commands
Section titled “All commands”Every command prints JSON: {"status":"ok","data":...}, or {"status":"error","error":"..."} with exit code 1.
| Command | What it does |
|---|---|
devices |
Boards nearby. |
pair, unpair |
Pair this computer (within 3 minutes of plugging the board in), or forget the board here. |
screenshot [--format png|jpeg] [--quality N] [--scale F] [--max-size N] [--output PATH|-] |
The screen after the last action settled. Saves a file in the current folder by default. |
io tap X,Y [--aim] [--count N] |
Tap at screenshot pixels. --aim finds the pointer first for a precise tap; --count 2 double-taps. |
io longpress X,Y [--duration MS] |
Press and hold. |
io swipe X1,Y1,X2,Y2 |
Swipe. |
io text 'TEXT' |
Type ASCII text. Chinese is typed as Pinyin. |
io keys COMBO |
enter, esc, tab, backspace, space, arrows, cmd+a, shift+tab, … |
io button HOME |
Go to the Home Screen. HOME is the only button. |
screencapture --format h264|mjpeg |
Live video to stdout, for example | ffplay -. |
device info |
Board status: mirroring, frozen video, whether the app is using the phone, USB, Bluetooth. |
calibrate |
Measure the pointer. Runs by itself on the first tap. |
net |
The CLI’s link (via: ble or wifi) and the board’s Wi-Fi. |
net join [SSID [PASSWORD]] |
Put the board on your Wi-Fi; asks for the name and password if not given. |
net hotspot |
Put the board back on its own hotspot. |
net via auto|ble |
ble keeps the CLI on Bluetooth (also ESP32AGENT_TRANSPORT=ble). |
history list, history show RUN|latest, history new |
What was done on the phone: recent runs, one run’s steps, or start a new run. |
intent list, intent show NAME |
Saved intents. |
intent save --name N --prompt P [--hint H ...] [--from-run RUN [--steps 1,3,4]] |
Save an intent, optionally with a run’s replayable steps. |
intent delete NAME |
Delete an intent. |
intent export NAME... [--screenshots] [--output FILE] |
Write an .esp32intent file the iPhone app can import. |
intent import FILE |
Import an .esp32intent file from the app or export. |
report --intent NAME [--screenshots] |
Prepare a mail with one redacted intent for the developer. Interactive; you send it yourself. |
logs [--board] [--save] [--output F] |
The CLI’s log; --board the board’s own log; --save one file for support. See below. |
upgrade [--check] [--force] |
Install the latest release. |
mcp |
MCP server on stdio. |
skill install |
Install the agent skill to ~/.claude/skills/esp32agent/. |
daemon [stop], version |
The background process (started automatically), and the CLI version. |
With several boards nearby, pass --device <id> or set ESP32AGENT_DEVICE.
MCP tools
Section titled “MCP tools”esp32agent mcp offers these tools: screenshot, tap, longpress, swipe, type_text, press_key, press_button, device_info, history, intent_list, intent_show, and intent_save. Pairing and report are terminal-only.
The first command starts a background esp32agent daemon that holds the Bluetooth link; it writes daemon.log in the CLI’s settings folder and rotates it as it grows.
esp32agent logs # last lines of the CLI's logesp32agent logs --board # the board's own log (firmware 0.2.3 and later)esp32agent logs --save # CLI log, board log and board state in one fileesp32agent logs --save --output handsy-logs.txt--save writes a text file and prints its path; send it to [email protected]. Nothing is sent by the CLI. For more detail in daemon.log, set ESP32AGENT_DEBUG=1: it records each command’s name, result, and timing, but no typed text, screenshots, or keys.
Limits
Section titled “Limits”- The app has priority. While the iPhone app drives the phone, and for about 20 seconds after, pointer and keyboard commands fail with
busy: the app is using the phone. Wait and retry. - Over Bluetooth, video is slow. Screenshots can be about a second behind, and animations are choppy. Use the Wi-Fi link when you can.
- No app launching or installing. The CLI can’t open apps directly; have the agent go through the Home Screen or search.
- No volume or power keys. HOME is the only hardware button.
- Typing is ASCII, one key per character. There is no clipboard access.
Switch between the app and CLI
Section titled “Switch between the app and CLI”Use them in turns. An active app operation has priority and can interrupt the CLI connection. Stop the app task before starting a CLI task; if the CLI reports that the board is busy, wait before retrying. If the picture stops updating after a switch, restart Screen Mirroring and check a fresh screenshot.
CLI history is stored on your computer. Inspect the latest run with esp32agent history show latest. Intent files (.esp32intent) can be exported and moved between the computer and app with Files or AirDrop; this is a file handoff, rather than automatic history synchronization.
Screenshots may be sent to the model used by your computer agent. Check that agent’s permissions and provider’s data policy, as well as Handsy’s privacy policy, before using personal accounts or screens.
Need help with mirroring, the antenna, or the pointer? See the FAQ. Use only devices and accounts you own or are authorized to control; see the disclaimer.
