Skip to content

CLI & AI agents

Your agent already works on your computer. Give it a way to use your iPhone, too. Manina AI CLI (esp32agent) connects over Bluetooth, or over your Wi-Fi, to the small board plugged into your phone. Your agent can request a screenshot, tap, swipe, and type, then look again to check the result.

Use Claude Code or Codex through MCP, or connect another agent that can run local commands or use a local stdio MCP server. The agent runs on your computer; the board receives the iPhone’s Screen Mirroring and acts as its mouse and keyboard.

Manina AI CLI is open source under the MIT license: github.com/onehumanbeing/esp32-iphone-agent, in cli/ on the release-1.0 branch.

In this example, Claude Code uses an iPhone to open Chrome’s incognito mode, visit Google Maps, and search for coffee near East Broadway station in New York. It reads the actual screen and works through the task, rather than calling a Maps API.

Claude Code uses the CLI to operate a real iPhone, open Google Maps in Chrome, and find cafés near East Broadway station in New York. Some parts are sped up or cut.

The same plugin works in DeepSeek Harness. Here it opens Settings, reads which Wi-Fi the phone is on and goes back Home, with the phone (left) and dsh (right) time-aligned:

Try the same task with your agent once connected:

Use esp32agent to open Chrome in incognito mode on my iPhone, visit the Google Maps website, and find coffee near East Broadway station in New York. Check a fresh screenshot after each action and stop when the map and café results are visible.

Flash the board first (Prepare the board), then follow this video to install the CLI and connect your agent.

Manina AI CLI 0.1.2 is available for macOS, Linux, and Windows. It needs board firmware v0.2.2 or later (v0.2.3 for the Wi-Fi link), available on the installer’s TestFlight channel; the official 0.2.1 firmware does not include the CLI connection.

Terminal window
curl -fsSL https://manina.ai/install.sh | sh
Terminal window
irm https://manina.ai/install.ps1 | iex
Terminal window
npm install -g esp32agent

The npm package needs Node.js 18 or later and downloads the same release for your system.

Variable What it does
ESP32AGENT_VERSION Install a specific version, for example 0.1.0. Default: the latest.
ESP32AGENT_BIN_DIR Where to put esp32agent. Default: ~/.local/bin (Windows: ~\.local\bin).
Terminal window
curl -fsSL https://manina.ai/install.sh | ESP32AGENT_VERSION=0.1.0 ESP32AGENT_BIN_DIR=/usr/local/bin sh

The installers check each download’s SHA-256 before installing it.

Terminal window
esp32agent upgrade --check # current and latest version
esp32agent upgrade # install the latest release

The CLI needs Bluetooth on the computer. Screenshots also need ffmpeg; the installer can offer a package-manager command for your system. Follow its prompts and reopen your terminal if esp32agent is not found. The shell installer uses ~/.local/bin; make sure that directory is on your PATH.

Choose the supported board and attach its antenna using the hardware guide. Flash firmware v0.2.3 from the installer’s TestFlight channel; Manina AI CLI cannot find a board running 0.2.1.

Pairing does not need the iPhone app. Plug the board into your iPhone, enable Bluetooth on your computer, and keep them nearby. Your computer can stay on its normal internet connection.

An agent with local terminal access can handle discovery and pairing. Give it this request:

Use your local terminal to set up esp32agent for my iPhone. My board runs firmware v0.2.3. Discover my board and pair if needed. Help me find the board’s hotspot password (https://manina.ai/docs/faq#how-do-i-find-the-boards-wi-fi-password), then guide me to join the board’s Wi-Fi and enable Screen Mirroring and AssistiveTouch on the phone. Verify a fresh screenshot and calibrate the pointer. Ask which board is mine if several are found.

For manual setup, run:

Terminal window
esp32agent devices
esp32agent pair

pair discovers and pairs with a single nearby board itself; you do not need to pair it in the computer’s Bluetooth settings first. If more than one board is found, use the identifier returned by devices: esp32agent pair --device <id>.

Run the first pair within three minutes of powering the board. If that window has closed, unplug and reconnect it, then retry.

Find the board’s hotspot name (ESP32Agent-…) and password in the ESP32 Agent app, or read them over USB serial: see how to find the hotspot password. With firmware v0.2.3, esp32agent net shows them too.

Then, on your iPhone:

  1. Open Settings → Wi-Fi and join the board’s hotspot with its name and password.
  2. Turn on Control Center → Screen Mirroring → ESP32 Agent.
  3. Enable Settings → Accessibility → Touch → AssistiveTouch. Keep the pointer visible, and the phone unlocked and in portrait.

The phone uses the board’s Wi-Fi for mirroring; the computer reaches the board over Bluetooth. If you restart the board, reconnect the phone’s Wi-Fi and start Screen Mirroring again.

Optional: put the board on your home Wi-Fi

Section titled “Optional: put the board on your home Wi-Fi”

With the board and the computer on the same Wi-Fi, the CLI sends commands and video over Wi-Fi: screenshots are close to live and taps don’t queue behind video. Bluetooth stays connected as the way back.

Terminal window
esp32agent net join

It asks for your Wi-Fi’s name and password (the password is not shown), sends them to the board over the paired Bluetooth session, and tells you each step: join the same Wi-Fi on the iPhone, then start Screen Mirroring to ESP32 Agent again. esp32agent net shows which link is in use (via: wifi or ble); esp32agent net hotspot puts the board back on its own hotspot. A computer that joins the board’s hotspot also gets the Wi-Fi link.

This needs CLI 0.1.2 and board firmware v0.2.3 (installer’s TestFlight channel). With v0.2.2 the CLI keeps working over Bluetooth.

Check the current screen and calibrate:

Terminal window
esp32agent device info
esp32agent screenshot
esp32agent calibrate

The screenshot command saves a PNG on your computer and reports its filePath. Open that file and check that it shows the phone’s current screen. The CLI measures the pointer itself; the first pointer action also calibrates automatically if needed.

If the picture is stale, disconnect and restart Screen Mirroring. If pairing fails, check the antenna and move the computer closer.

Complete the initial pairing in a terminal first, or ask an agent with local terminal access to do it using the request above. The MCP server provides phone controls; its tools do not perform the initial device discovery and pairing.

Install the Manina AI plugin. It adds the MCP server and the skill in one step. Run these inside Claude Code:

/plugin marketplace add onehumanbeing/esp32-iphone-agent
/plugin install esp32agent@handsy

Or set it up by hand: register the local MCP server, then install the CLI’s Claude skill.

Terminal window
claude mcp add esp32agent -- esp32agent mcp
esp32agent skill install

The skill goes under ~/.claude/skills/esp32agent/. It gives Claude the phone workflow and command guidance; MCP provides the tools.

Register the same MCP server with Codex:

Terminal window
codex mcp add esp32agent -- esp32agent mcp

Install the plugin; it adds the same MCP server and the handsy skill:

Terminal window
dsh plugin --profile web add dsh-esp32agent

Restart dsh web, then ask for a small task. Full steps: DeepSeek Harness. Plugin source: integrations/dsh for details.

Use esp32agent mcp as a local stdio MCP server, or let your agent call the CLI directly. It needs access to the computer’s Bluetooth and the saved screenshots. A remote agent without access to this computer cannot reach the board by itself.

The skill is also on ClawHub as handsy, and the MCP server is in the MCP Registry as io.github.onehumanbeing/esp32agent (npm package esp32agent). The skill files are MIT-0; the rest of the repository keeps its own license.

For a setup request you can paste into an agent, use the homepage’s CLI entry. The agent setup guide contains the installation and verification steps in plain Markdown.

Try this request in your agent:

Use esp32agent to take a screenshot of my iPhone. Open Settings and go to Wi-Fi. Take a fresh screenshot after each action to check the result. Stop when the Wi-Fi page is visible.

The agent should look → take one action → look again. Watch the first task and interrupt your agent if it goes off track. For a longer task, describe the destination and what counts as finished.

The CLI provides the controls; your agent supplies the reasoning and AI service. The free AI service in the phone app is a separate way to use Manina AI.

These are examples of individual actions. Choose coordinates from a fresh screenshot before running a tap or swipe; coordinates are pixels in the original screenshot, not percentages.

Terminal window
esp32agent screenshot
esp32agent io tap 200,400
esp32agent io swipe 200,800,200,300
esp32agent io text 'hello'
esp32agent io keys enter
esp32agent io button HOME

Chinese typing uses the phone’s Pinyin keyboard and requires selecting the intended candidate. Enable that keyboard first and have the agent check the text on screen.

Every command prints JSON: {"status":"ok","data":...}, or {"status":"error","error":"..."} with exit code 1.

Command What it does
devices Boards nearby.
pair, unpair Pair this computer (within 3 minutes of plugging the board in), or forget the board here.
screenshot [--format png|jpeg] [--quality N] [--scale F] [--max-size N] [--output PATH|-] The screen after the last action settled. Saves a file in the current folder by default.
io tap X,Y [--aim] [--count N] Tap at screenshot pixels. --aim finds the pointer first for a precise tap; --count 2 double-taps.
io longpress X,Y [--duration MS] Press and hold.
io swipe X1,Y1,X2,Y2 Swipe.
io text 'TEXT' Type ASCII text. Chinese is typed as Pinyin.
io keys COMBO enter, esc, tab, backspace, space, arrows, cmd+a, shift+tab, …
io button HOME Go to the Home Screen. HOME is the only button.
screencapture --format h264|mjpeg Live video to stdout, for example | ffplay -.
device info Board status: mirroring, frozen video, whether the app is using the phone, USB, Bluetooth.
calibrate Measure the pointer. Runs by itself on the first tap.
net The CLI’s link (via: ble or wifi) and the board’s Wi-Fi.
net join [SSID [PASSWORD]] Put the board on your Wi-Fi; asks for the name and password if not given.
net hotspot Put the board back on its own hotspot.
net via auto|ble ble keeps the CLI on Bluetooth (also ESP32AGENT_TRANSPORT=ble).
history list, history show RUN|latest, history new What was done on the phone: recent runs, one run’s steps, or start a new run.
intent list, intent show NAME Saved intents.
intent save --name N --prompt P [--hint H ...] [--from-run RUN [--steps 1,3,4]] Save an intent, optionally with a run’s replayable steps.
intent delete NAME Delete an intent.
intent export NAME... [--screenshots] [--output FILE] Write an .esp32intent file the iPhone app can import.
intent import FILE Import an .esp32intent file from the app or export.
report --intent NAME [--screenshots] Prepare a mail with one redacted intent for the developer. Interactive; you send it yourself.
logs [--board] [--save] [--output F] The CLI’s log; --board the board’s own log; --save one file for support. See below.
upgrade [--check] [--force] Install the latest release.
mcp MCP server on stdio.
skill install Install the agent skill to ~/.claude/skills/esp32agent/.
daemon [stop], version The background process (started automatically), and the CLI version.

With several boards nearby, pass --device <id> or set ESP32AGENT_DEVICE.

esp32agent mcp offers these tools: screenshot, tap, longpress, swipe, type_text, press_key, press_button, device_info, history, intent_list, intent_show, and intent_save. Pairing and report are terminal-only.

The first command starts a background esp32agent daemon that holds the Bluetooth link; it writes daemon.log in the CLI’s settings folder and rotates it as it grows.

Terminal window
esp32agent logs # last lines of the CLI's log
esp32agent logs --board # the board's own log (firmware 0.2.3 and later)
esp32agent logs --save # CLI log, board log and board state in one file
esp32agent logs --save --output manina-logs.txt

--save writes a text file and prints its path; send it to [email protected]. Nothing is sent by the CLI. For more detail in daemon.log, set ESP32AGENT_DEBUG=1: it records each command’s name, result, and timing, but no typed text, screenshots, or keys.

  • The app has priority. While the iPhone app drives the phone, and for about 20 seconds after, pointer and keyboard commands fail with busy: the app is using the phone. Wait and retry.
  • Over Bluetooth, video is slow. Screenshots can be about a second behind, and animations are choppy. Use the Wi-Fi link when you can.
  • No app launching or installing. The CLI can’t open apps directly; have the agent go through the Home Screen or search.
  • No volume or power keys. HOME is the only hardware button.
  • Typing is ASCII, one key per character. There is no clipboard access.

Use them in turns. An active app operation has priority and can interrupt the CLI connection. Stop the app task before starting a CLI task; if the CLI reports that the board is busy, wait before retrying. If the picture stops updating after a switch, restart Screen Mirroring and check a fresh screenshot.

CLI history is stored on your computer. Inspect the latest run with esp32agent history show latest. Intent files (.esp32intent) can be exported and moved between the computer and app with Files or AirDrop; this is a file handoff, rather than automatic history synchronization.

Screenshots may be sent to the model used by your computer agent. Check that agent’s permissions and provider’s data policy, as well as Manina AI’s privacy policy, before using personal accounts or screens.

Need help with mirroring, the antenna, or the pointer? See the FAQ. Use only devices and accounts you own or are authorized to control; see the disclaimer.