What this is. This is a capability map, not a ranking. It describes open-source tools that give an LLM agent eyes and hands on a real operating system: desktop (macOS, Windows, Linux) and mobile (Android, with an evidence-backed note on iOS/iPadOS). It profiles mechanisms — how each tool sees, aims, and acts — so an engineer can match a tool to a constraint set. Browser-only automation is out of scope.
Data date. All facts (stars, licenses, commit dates, code-level mechanisms, issue states) were measured from the GitHub API or project source on 2026-08-06. Assume every claim is measured against the linked primary source unless a label marks otherwise: [vendor] the project's own claim, unchecked · [derived] an inference from measured facts · [lead] found but not independently verified · [approx] a sweep-stage approximation. Nothing was executed; "works" claims rest on code reading and adoption evidence.
Coverage. ~31 projects were refutation-verified across three research passes (desktop, Windows/Linux, mobile) — each adversarially checked against five refutations — plus ~10 iOS candidates evidence-profiled at lower depth (a mechanism survey, not a full refutation pass); 45 verification agents were dispatched in total. Dozens more were swept and set aside; the notable exclusions are named in section 9. A handful of rows are candidate-level (evidence-profiled or API-checked, not five-refutation-verified) and are marked [cand]. Every tool name links to its repository — click it to investigate directly.
- Functional anatomy — the five questions every tool answers
- Ecosystem role map
- Capability matrix — perception, grounding, actuation by tool
- Platform matrix — desktop
- Platform matrix — mobile, and the iOS reality
- Per-tool profiles
- Composition recipes
- Platform mechanisms
- Refuted and excluded
- Method and coverage
In a hurry? The OS-support quick reference table below lists every tool (with repo links) against macOS, Windows, Linux, Android, and iOS. Sections 4 and 5 break those cells down by mechanism.
Every profiled tool against every operating system. Each name links to its repo. Values name the execution surface, not just yes/no:
- Host = controls that OS's real desktop · VM = only inside a sandbox / VM / container · Eyes = perception only, no input · Lib = a library, not a runnable agent · ✓ = mobile device supported · — = not supported.
- Linux notes X11 vs Wayland (Wl) because they are genuinely different answers (section 8).
| Tool | Role | macOS | Windows | Linux | Android | iOS |
|---|---|---|---|---|---|---|
| cua-driver (trycua/cua) | desktop driver | Host | Host | Host (X11+Wl¹) | via cua sandbox | — |
| Peekaboo | desktop driver | Host | — | — | — | — |
| Windows-MCP | desktop driver | — | Host | — | — | — |
| Terminator | desktop driver | — | Host | — | — | — |
| computer-use-mcp | desktop driver | Host | Host | Host (X11) | — | — |
| computer-use-linux | desktop driver | — | — | Host (X11+Wl) | — | — |
| nut.js (@nut-tree-fork) | library | Lib | Lib | Lib (X11) | — | — |
| Hermes Agent | integrated stack | Host | Host | Host (X11+Wl¹) | — | — |
| UI-TARS Desktop | integrated stack | Host | Host | SDK only | via ADB operator | — |
| Agent S v3 | integrated stack | Host | Host | Host (X11) | — | — |
| UFO / UFO2 | integrated stack | — | Host | shell only² | via ADB | — |
| TuriX-CUA | integrated stack | Host | Host | Host (X11) | — | — |
| OmniParser | perception | Eyes | Eyes | Eyes | Eyes | Eyes |
| screenpipe (proprietary) | perception | Eyes | Eyes | Eyes | — | — |
| computer-use-demo (Anthropic) | surface | Host³ | — | VM (Docker) | — | — |
| Lume (in cua) | surface | VM | VM | VM | — | — |
| AIO Sandbox | surface | VM⁴ | — | VM | — | — |
| E2B Desktop | surface | — | — | VM (cloud) | — | — |
| Bytebot (archived) | surface | VM⁴ | — | VM | — | — |
| scrcpy | mobile driver | — | — | — | ✓ | — |
| Appium (+ appium-mcp) | mobile driver | — | — | — | ✓ | ✓ |
| Maestro | mobile driver | — | — | — | ✓ | ✓ (sim only) |
| mobile-mcp (Mobile Next) | mobile driver | — | — | — | ✓ | ✓ |
| uiautomator2 | mobile library | — | — | — | ✓ | — |
| Open-AutoGLM | mobile stack | — | — | — | ✓ | ✓⁵ |
| Mobile-Agent / GUI-Owl | mobile stack | — | — | — | ✓ | —⁵ |
| Airtest | mobile library | Host | Host | — | ✓ | ✓ |
| OSWorld (benchmark) | yardstick | via Fusion | VM | VM | — | — |
| WindowsAgentArena (benchmark) | yardstick | — | VM | — | — | — |
| AndroidWorld (benchmark) | yardstick | — | — | — | ✓ (emulator) | — |
¹ cua-driver / Hermes on Wayland is per-compositor: Sway full, GNOME needs a bundled Shell extension, KDE preflight only. Background input to occluded windows is a permanent Wayland refusal by design.
² UFO's "Linux agent" is an allowlisted shell executor — no GUI, no screen, no input. macOS is entirely absent from its code.
³ Anthropic's macOS support is the computer-use-best-practices native variant; the Docker demo controls a Linux desktop inside the container.
⁴ Runs on a Mac only via Docker (itself a Linux VM); it controls the container's Linux desktop, never macOS.
⁵ Open-AutoGLM and Mobile-Agent also drive HarmonyOS via HDC; Open-AutoGLM adds iOS via a self-signed WebDriverAgent.
Every computer-use tool supplies some slice of five functions. Read a tool by which slices it owns and which it leaves to you. (Tool names in this tree are plain text; every one is a clickable link in the tables above and below.)
A computer-use stack — five separable functions
│
├── 1. PERCEPTION (eyes) ──── how the agent sees the screen
│ • screenshot (raw pixels)
│ • accessibility tree — macOS AX · Windows UIA · Linux AT-SPI · Android UiAutomator · iOS XCTest
│ • vision parsing (screenshot → labeled elements, "set-of-marks" / SOM)
│ • continuous recording (a searchable history of the screen)
│ WHY it matters: a tree gives a text-only model exact element names and bounds;
│ raw pixels force a vision model into the loop.
│
├── 2. GROUNDING (aim) ────── how intent becomes a concrete target
│ • element ID from a tree (deterministic, no model needed)
│ • pixel coordinate from a VLM (needs a vision model that emits x,y)
│ • OCR / template match (classical CV, brittle to theme changes)
│ WHY it matters: this is the single biggest source of "it clicked the wrong thing".
│
├── 3. ACTUATION (hands) ──── how input reaches the OS
│ • OS event synthesis — CGEvent · SendInput · XTest/XSendEvent · adb input · XCTest
│ • accessibility action — press a tree node directly (AXPress, UIA Invoke, do_action)
│ • hardware HID — a physical USB device the OS cannot tell from a real keyboard
│ WHY it matters: determines drag, hotkeys, non-ASCII text, background operation,
│ and whether Wayland or a vendor lock blocks input.
│
├── 4. EXECUTION SURFACE ───── where the acting happens
│ • your host desktop (drives your real, logged-in apps)
│ • local VM / container (isolated; safe to run unattended)
│ • cloud VM (isolated + remote; metered)
│ • phone/tablet over ADB or a tunnel
│ WHY it matters: host = maximum reach + maximum risk; sandbox = safe but cannot
│ touch a native app you already have open.
│
└── 5. BRAIN COUPLING ──────── who does the thinking
• MCP tool for an external agent (tool-not-brain: Claude Code reasons; ideal here)
• scriptable CLI / SDK (an agent shells out per task)
• built-in loop, pluggable models (the tool reasons; you pick the model)
• hard-wired dedicated VLM (the tool reasons only with its own model)
WHY it matters: only the first two let an external host like Claude Code be the reasoner.
Four operational concerns cut across all five functions. They decide production-readiness more often than raw capability:
- Permission model — what the OS makes you grant (macOS TCC, Windows UAC/UIPI, Android USB debugging, iOS signing).
- Isolation — host control versus a sandbox; the safety/reach trade.
- Telemetry — several tools phone home by default.
- License and maintenance — an OSI license on the runtime you actually run, plus a live maintainer.
A tool's role is which functions it supplies. Five roles emerge. This is the map the rest of the atlas fills in. (Tree names are plain; the same tools are linked in every table.)
Open-source computer use (Aug 2026)
│
├── DRIVERS ── supply eyes+hands (functions 1–3) on a surface; you bring the brain
│ │ → the "tool-not-brain" shape: an MCP host like Claude Code reasons
│ ├── Desktop host
│ │ ├── cua-driver (trycua/cua) ──── mac + win + linux, 54 MCP tools, background
│ │ ├── Peekaboo ─────────────────── macOS, MCP + CLI, model-free
│ │ ├── Windows-MCP (CursorTouch) ── Windows, ~20 MCP tools
│ │ ├── Terminator (mediar-ai) ───── Windows, 35 MCP tools, deterministic UIA
│ │ ├── computer-use-mcp (domdomegg) mac + win + linux/X11, nut.js-fork
│ │ └── computer-use-linux (agent-sh) linux incl. Wayland portals
│ ├── Mobile
│ │ ├── scrcpy ──────────────────── Android eyes+hands primitive (pixels only)
│ │ ├── Appium (+ appium-mcp) ────── Android + iOS substrate, first-party MCP
│ │ ├── Maestro ─────────────────── Android + iOS-sim, first-party MCP
│ │ ├── mobile-mcp (Mobile Next) ── Android + iOS, MCP
│ │ └── uiautomator2 · Airtest ──── Android libraries (tree / vision)
│ └── Library primitive (not an agent; the layer drivers are built on)
│ └── nut.js (@nut-tree-fork) ─── mac + win + linux/X11 native automation
│
├── INTEGRATED STACKS ── own brain + eyes+hands; an external host can only delegate whole tasks
│ ├── Live
│ │ ├── Hermes Agent (NousResearch) ─ mac+win+linux; its hands ARE cua-driver
│ │ ├── UI-TARS Desktop (ByteDance) ─ mac+win; locked to UI-TARS VLM
│ │ ├── Agent S v3 (Simular) ──────── mac+win+linux/X11; needs a grounding model
│ │ ├── UFO / UFO2 (Microsoft) ────── Windows GUI; Linux = shell only
│ │ ├── Open-AutoGLM (Zhipu) ──────── Android/HarmonyOS/iOS; open VLM (dormant since 2026-03)
│ │ ├── Mobile-Agent / GUI-Owl ────── Android/HarmonyOS; open-weight VLM
│ │ └── TuriX-CUA ─────────────────── mac+win+linux/X11; coarse MCP
│ └── Pivoted or dormant (see section 9)
│ ├── pivoted: Open Interpreter (→ coding agent)
│ └── dormant/slow: Self-Operating Computer · Open Interface · Cradle · Computer Use OOTB
│
├── COMPONENTS ── one function each; you compose them with a driver
│ ├── Perception: OmniParser (screenshot→labeled elements) · screenpipe (recorder; now proprietary)
│ └── Grounding models: UI-TARS-1.5-7B · GUI-Owl (open weights that emit coordinates)
│
├── SURFACES ── supply an isolated desktop for a driver/agent to act in
│ ├── computer-use-demo (Anthropic) ── Docker Ubuntu+X11 (arm64 image)
│ ├── Lume (in cua) ────────────────── real macOS VMs on Apple Silicon
│ ├── AIO Sandbox (ByteDance) ──────── Linux container + MCP hub
│ ├── E2B Desktop ──────────────────── cloud Linux VM, SDK
│ └── Bytebot ──────────────────────── Linux container (archived)
│
└── YARDSTICKS ── measure agents; not tools you drive
├── OSWorld ─────────── 369 Ubuntu-desktop tasks (the desktop standard)
├── WindowsAgentArena ─ Windows-desktop tasks
└── AndroidWorld ────── 116+ Android tasks on a pinned emulator
How each driver and live stack sees, aims, and acts. Cells name the mechanism, not a Yes/No. "Background" = can act without stealing your cursor or window focus. Desktop drivers and live stacks are listed first; the mobile rows are repeated from section 5 for cross-comparison — section 5 is authoritative for mobile. "SOM" = set-of-marks (numbered overlay on a screenshot).
| Tool | Perception | Grounding | Actuation | Background? | Brain coupling |
|---|---|---|---|---|---|
| cua-driver | AX/UIA/AT-SPI tree + screenshot (SOM-numbered) | element ID or pixel | CGEvent / SendInput / XSendEvent + AX actions | Yes (per-OS, documented gaps) | MCP (tool-not-brain) |
| Peekaboo | AX tree + screenshot | element ID | CGEvent + AXPress | Yes (when target process resolvable) [vendor] |
MCP + CLI |
| Windows-MCP | UIA tree + screenshot | element ID or pixel | SendInput | no | MCP |
| Terminator | UIA tree (screenshot optional) | element ID (UIA selector) | UIA element actions + input synthesis | no [derived] |
MCP + CLI |
| computer-use-mcp | screenshot | pixel (VLM) | nut.js (SendInput/CGEvent/XTest) | no | MCP |
computer-use-linux [cand] |
AT-SPI tree + screenshot | element ID or pixel | ydotool / portal / GNOME D-Bus | no [vendor] |
MCP |
| TuriX-CUA | AX tree + screenshot | element ID or pixel | Quartz CGEvent / pywin32 / xdotool | no | coarse MCP (run_task) |
| Hermes Agent | AX/UIA/AT-SPI tree + SOM screenshot | element ID or pixel | via cua-driver | Yes (via cua-driver) | own loop (~40 providers) |
| UI-TARS Desktop | screenshot | pixel (UI-TARS VLM) | nut.js | no | own loop, UI-TARS VLM only |
| Agent S v3 | screenshot (dropped its tree in v3) | pixel (separate grounding model, required) | pyautogui | no | own loop, pluggable + grounding model |
| UFO / UFO2 | UIA tree + Win32 + app COM + screenshot | element ID | pywinauto / pyautogui | no | own loop, pluggable |
| OmniParser (eyes only) | screenshot → labeled boxes + OCR | produces element IDs for a text model | — (no hands) | — | HTTP API component |
| screenpipe (eyes only) | 24/7 recording: AX tree + OCR + audio | search over history | — (no hands) | — | MCP (perception only) |
| scrcpy (Android) | H.264/H.265/AV1 video stream (pixels) | pixel only | reflected InputManager.injectInputEvent | — | primitive; MCP via wrapper |
| Appium (Android+iOS) | UiAutomator / XCTest tree + screenshot | element handle (locators) | UiAutomator2 / XCTest injection | — | first-party MCP |
| Maestro (Android+iOS-sim) | UiAutomator / XCTest tree + screenshot | selector (hierarchy) | gRPC UiAutomator + adb / XCTest | — | first-party MCP (flow-granular) |
| Open-AutoGLM (Android) | screenshot only | pixel (AutoGLM-Phone VLM, 0–1000 space) | adb input + ADBKeyBoard IME | — | own loop, open-weight VLM |
The four cross-cutting concerns from section 1, tabulated for the desktop drivers and components where they bite. "Telemetry" = phones home by default.
| Tool | Telemetry default | License flag | Maintenance |
|---|---|---|---|
| cua-driver | on — Rust: CUA_DRIVER_RS_TELEMETRY_ENABLED=0; Python SDK: CUA_TELEMETRY_ENABLED=false |
MIT | very active, pre-1.0 |
| Peekaboo | none noted | MIT | active; bus factor ≈ 1 |
| Windows-MCP | on — PostHog; ANONYMIZED_TELEMETRY=false |
MIT | active; bus factor ≈ 1 |
| Terminator | on — Sentry; SENTRY_DISABLED=true |
MIT | 2-month gap; open shell-injection #479 |
| Hermes Agent | via cua-driver | MIT | very active |
| UI-TARS Desktop | none noted | Apache-2.0 | maintenance-only |
| Agent S v3 | none noted | Apache-2.0 | slowing; exec() injection #199 |
| OmniParser | none noted | CC-BY-4.0 code + AGPL ultralytics dep |
research cadence |
| screenpipe | on — Sentry | proprietary since 2026-06 | very active |
Each cell is the strongest execution surface the tool offers on that platform, plus the mechanism. Host = your real desktop. VM = an isolated desktop. Cloud client = the SDK runs on this OS but the desktop it controls is remote. Eyes = perception only. Lib = a library, not a runnable agent. No = unsupported. Linux is split because X11 and Wayland are genuinely different answers (section 8).
Read the two halves differently. For driver / stack rows, a cell answers "can this tool control that platform's real UI?". For surface / benchmark rows (Lume, computer-use-demo, AIO, E2B, OSWorld, WindowsAgentArena), the cell answers "can this run on that platform to provide a guest desktop?" — e.g. computer-use-demo's macOS "VM (Docker)" means the Ubuntu container runs on a Mac; OSWorld's macOS cell is the host (via Fusion) while its Linux/Windows cells are the guests. The Brain column reads "—/infra/ref agent/benchmark" for those rows because they supply no reasoning.
| Tool | macOS | Windows | Linux / X11 | Linux / Wayland | Brain |
|---|---|---|---|---|---|
| cua-driver | Host (AX+CGEvent) | Host (UIA+SendInput) | Host (AT-SPI+XSendEvent) | Host, per-compositor¹ | MCP |
| Peekaboo | Host (AX+CGEvent) | No² | No² | No² | MCP+CLI |
| Windows-MCP | No | Host (UIA+SendInput) | No | No | MCP |
| Terminator | No³ | Host (UIA) | No³ | No³ | MCP |
| computer-use-mcp | Host (nut.js) | Host (nut.js) | Host (nut.js/XTest) | No (X11 only)⁴ | MCP |
| computer-use-linux | No | No | Host (AT-SPI) | Host (portals/ydotool) [vendor] |
MCP |
| Hermes Agent | Host | Host | Host | Host (XWayland capture)¹ | own loop |
| UI-TARS Desktop | Host (nut.js) | Host (nut.js) | SDK-level only⁵ | SDK-level only⁵ | UI-TARS VLM |
| Agent S v3 | Host (pyautogui) | Host (pyautogui, second-class) | Host (pyautogui+wmctrl) | No⁴ | own loop + grounding model |
| UFO / UFO2 | No⁶ | Host (UIA+COM) | Shell only⁶ | Shell only⁶ | own loop |
| TuriX-CUA | Host (Quartz) | Host (pywin32) | Host (xdotool) | No (X11 only) | coarse MCP |
| nut.js (library) | Lib | Lib | Lib (XTest) | No (X11 only)⁴ | — |
| computer-use-demo (Anthropic) | VM (Docker) | No | VM (Docker) | — | ref agent |
| Lume (in cua) | VM (real macOS) | No⁷ | VM (Linux) | — | infra |
| AIO Sandbox | VM (Docker) | No | VM (container) | — | MCP |
| E2B Desktop | Cloud client | Cloud client | VM (cloud) | — | SDK |
| Bytebot (archived) | No | No | VM (Docker) | — | MCP (unmaintained) |
| OmniParser | Eyes (CPU, slow) | Eyes | Eyes | Eyes | component |
| screenpipe | Eyes | Eyes [vendor] |
Eyes [vendor] |
Eyes [vendor] |
MCP |
| OSWorld (benchmark) | Host→VM (Fusion) | VM (49 tasks) | VM (369 tasks) | — | Python API |
WindowsAgentArena [cand] |
No | VM (Win11) | — | — | benchmark |
¹ cua-driver Wayland is per-compositor: Sway/wlroots fully validated (116/116 CI), GNOME needs a bundled Shell extension, KDE compiles but has no accepted test matrix; background input to occluded windows is a permanent Wayland refusal by design.
² Peekaboo Windows/Linux "rewrites" are separate, unaffiliated community projects [lead].
³ Terminator removed its macOS/Linux engines in Dec 2025; a compile_error! blocks any non-Windows build.
⁴ pyautogui/nut.js use Xlib/XTest — they raise on Wayland; XWayland is the only workaround.
⁵ UI-TARS Desktop ships a Windows/macOS app only; its nut.js layer supports Linux at the SDK level, not as a shipped app.
⁶ UFO has zero macOS code; its "Linux agent" is an allowlisted shell executor with no GUI, no screen, no input.
⁷ Lume manages macOS and Linux guests only (Apple Virtualization.Framework). Windows/Android guests in the cua ecosystem come from the separate sandbox SDK's QEMU/cloud path [vendor], not Lume.
Android exposes three sanctioned control channels, which is why the ecosystem is deep: (a) adb shell input + screencap, (b) Google's UiAutomator accessibility tree, (c) an on-device accessibility-service APK (used by the swept Mobilerun/OpenDroid agents; no fully-verified tool in this atlas uses channel (c), listed for completeness). No root is needed for any of them.
| Tool | Perception | Grounding | Actuation | Physical / Emulator | Brain / MCP |
|---|---|---|---|---|---|
| scrcpy | H.264/H.265/AV1 video stream | pixel only | reflected injectInputEvent (shell uid) |
both; USB or Wi-Fi | primitive; scrcpy-mcp wrapper |
| Appium | UiAutomator tree + screenshot | element handle | on-device UiAutomator2 instrumentation | both; no root | first-party appium-mcp |
| Maestro | UiAutomator tree + screenshot | selector | gRPC UiAutomator + adb input |
both; no root | first-party maestro mcp |
mobile-mcp [cand] |
accessibility snapshot + screenshot (adb/devicekit) | element or pixel | adb | both | MCP |
uiautomator2 [cand] |
UiAutomator tree | element | on-device HTTP server (jar) | both | Python lib / community MCP |
| Open-AutoGLM | screenshot only | pixel (VLM, 0–1000) | adb input + ADBKeyBoard IME |
physical-first | own loop, open VLM; CLI |
| Mobile-Agent / GUI-Owl | screenshot only | pixel (GUI-Owl VLM) | adb input + ADBKeyBoard IME |
physical-first | own loop, open VLM; CLI |
| Airtest | minicap video / screencap | OpenCV template match | minitouch/maxtouch (no root) | both | Python lib / CLI |
| AndroidWorld | screenshot + a11y forest | element index or pixel | adb via android_env | emulator (pinned Pixel 6) | benchmark; Python API |
Recurring Android friction, verified across tools: non-ASCII text usually needs the third-party ADBKeyBoard IME (Open-AutoGLM, Mobile-Agent) [measured]; FLAG_SECURE screens (banking, payments) screenshot as black across tools (an OS behavior) — Open-AutoGLM specifically turns this into a forced human-takeover flow; some vendors (Xiaomi) require an extra "USB debugging (Security Settings)" toggle before input injection works.
Verdict: full computer use on iOS is possible but expensive and Apple-gated, not open like Android. Each fact below was checked against a primary source on 2026-08-06.
- No public input-injection API exists for third-party processes. Every non-jailbreak tool converges on one choke point: Apple's XCTest / XCUITest, driven through WebDriverAgent (WDA) — automation must ship as a signed test bundle. There is no iOS equivalent of
adb shell input.[measured: WDA README, 40+ repos surveyed] - The accessibility tree is not externally readable (unlike macOS). It is reachable only through XCTest inside a WDA session, through private frameworks against the simulator (Meta's idb, AXe), or on a jailbroken device.
[measured] - The Mac + signing tax. Real-device automation needs Developer Mode, a provisioning profile, and a Mac to build WDA.
[measured: Appium XCUITest device-setup docs]Free-account WDA signatures expire every 7 days.[lead: widely-documented ecosystem knowledge, not a scraped page]Non-macOS hosts can drive only iOS 18+ real devices over RemoteXPC (via go-ios) with a pre-installed WDA.[measured]
What exists as a result:
| Layer | Tools | Reality |
|---|---|---|
| Simulator MCPs + CLI (highest quality) | XcodeBuildMCP (6.2k★, MCP), ios-simulator-mcp (2.1k★, MCP), AXe (2.1k★, CLI engine bundled by XcodeBuildMCP), idb (Meta; simulator-first) | simulator only; use Xcode private frameworks |
| Real-device MCP / substrate | mobile-mcp, Appium + WebDriverAgent, go-ios, Open-AutoGLM (self-signed WDA, Mac + free dev account) | production-grade but tethered, signed, per-device setup |
| Untethered, Mac-free | witchan/ios-mcp (594★) | jailbreak only — an existence proof that the gap is Apple policy, not hardware |
System-wide control is possible via WDA (XCUITest can target Springboard, Settings, any app), so iOS "computer use" is not absent — it is signing-gated and simulator-biased. [derived]
Profiles are grouped by role. Each gives mechanism, platforms, integration, and the load-bearing limits. Star counts and dates are measured 2026-08-06.
cua-driver — trycua/cua · ⭐20,972 · MIT · pushed 2026-08-06
The most complete cross-platform driver, and the most rigorously evidenced. A Rust daemon exposes 54 MCP tools over stdio with a "no-foreground contract": it drives native apps without moving your cursor, raising windows, or stealing focus, and returns structured background_unavailable refusals with an explicit foreground-escalation rung.
- Perception/grounding: AX (macOS), UIA (Windows), AT-SPI (Linux) trees, plus SOM-numbered screenshots; act by element or by pixel.
- Actuation depth: per-OS backends with CI-verified behavior baselines — Windows 122/122 (UIA Invoke + SendInput/PostMessage + Windows.Graphics.Capture for occluded windows), Linux X11 116/116 (background
XSendEvent, not XTest, so it does not steal focus), Wayland per-compositor (Sway 116/116; GNOME via a bundled Shell extension; KDE preflight only). CI run IDs verified against real GitHub Actions runs. - Integration:
claude mcp add --transport stdio cua-driver -- cua-driver mcp; a--claude-code-computer-use-compatmode scopes the screenshot for Claude Code's vision grounding. - Limits: pre-1.0 and fast-moving (v0.17→v0.19 in 4 days, all releases marked pre-release); Windows background mode has documented refusal classes (Chromium DOM, GTK buttons, elevated apps); stock Wayland structurally cannot deliver background input to occluded surfaces; telemetry on by default — opt out with
CUA_DRIVER_RS_TELEMETRY_ENABLED=0for the Rust driver andCUA_TELEMETRY_ENABLED=falsefor the Python SDK.
Peekaboo — openclaw/Peekaboo · ⭐4,969 · MIT · pushed 2026-08-05
macOS-native, deterministic, model-free. Swift 6.2 CLI + menu-bar app by Peter Steinberger. see --json returns an AX-derived UI map with element IDs; click/type/scroll act via CGEvent + AXPress.
- Distinctive: the observe-act core needs no AI provider at all
[derived]— grounding is by element ID from the tree. Two integration paths: MCP (npx -y @steipete/peekaboo mcp) or plain CLI from Bash. - Limits: macOS 15+ only; bus factor ≈ 1; MCP transport is stdio only; needs Screen Recording + Accessibility + synthetic-input TCC grants.
Windows-MCP — CursorTouch/Windows-MCP · ⭐6,647 · MIT · pushed 2026-08-05
The most-adopted dedicated Windows driver. ~20 MCP tools over the Windows UIA tree with SendInput injection; documented claude mcp add. Windows-only by construction (pywin32/comtypes). Telemetry on by default (PostHog). From a Mac, reachable only via a Windows VM + network transport, and that path has an open UIA-empty bug under emulated Python (#301).
Terminator — mediar-ai/terminator · ⭐1,605 · MIT · pushed 2026-06-02
"Playwright for Windows." 35 MCP tools; deterministic via UIA selectors with no screenshots or VLM required, optional OCR/OmniParser/Gemini vision fallbacks. First-class Claude Code one-liner. Windows-only is compiler-enforced (compile_error!); macOS/Linux engines were deliberately removed Dec 2025. Caveats: 2-month commit gap as the company pushes a commercial IDE; Sentry telemetry on by default; open shell-injection report (#479).
computer-use-mcp — domdomegg/computer-use-mcp · ⭐343 · MIT · pushed 2026-08-05
The safe cross-platform pick. Wraps @nut-tree-fork/nut-js in an MCP server modeled on Anthropic's official computer-use tool shape: claude mcp add computer-use -- npx -y computer-use-mcp. Screenshot + pixel grounding on macOS/Windows/Linux. Linux is X11-only (inherited from nut.js).
computer-use-linux — agent-sh/computer-use-linux · ⭐361 · MIT · pushed 2026-08-04
The most credible Linux-first MCP server. Controls the Linux desktop over AT-SPI, a GNOME Shell extension, Wayland portals, and ydotool — the one general tool that addresses Wayland head-on [vendor]. Young and small, but it is the closest thing to a Wayland answer outside cua-driver.
Hermes Agent — NousResearch/hermes-agent · ⭐226,510 · MIT · pushed 2026-08-06
The highest-profile agent host with three-platform desktop control — and a key structural finding: its hands are literally cua-driver, spawned as a cua-driver mcp child process. It reads AX/UIA/AT-SPI trees, returns SOM screenshots, posts pid-scoped background input; ~40 model providers. Because the desktop capability is cua-driver, the reusable piece for an external host is cua-driver itself, not Hermes. Independent confirmation that cua-driver's cross-platform story is real.
UI-TARS Desktop — bytedance/UI-TARS-desktop · ⭐38,472 · Apache-2.0 · pushed 2026-07-01 The largest dedicated desktop-agent project. Native macOS/Windows Electron app; OS-level control via nut.js; the UI-TARS vision-language model (Apache-2.0 weights, self-hostable) is the brain. The desktop app hard-requires UI-TARS-family VLMs — an external host cannot be its brain, and no MCP exposes the desktop operator. 2026 activity is maintenance-only; an open macOS Tahoe click-offset bug (#1876) sits unanswered.
Agent S v3 — simular-ai/Agent-S · ⭐12,134 · Apache-2.0 · pushed 2026-08-01 (code 2026-05-13)
Research-grade, Linux/OSWorld-centered. Screenshot in, pyautogui code out, with a required separate visual-grounding model endpoint (UI-TARS-1.5-7B recommended). Linux is the tuned flagship (X11 only; wmctrl; OSWorld-shaped, with a hardcoded OSWorld sudo password baked into generated actions). Windows is a real but second-class path (a Win+D "show desktop and type" app-switch heuristic; set_cell_values stripped). macOS runs but has zero benchmark evidence. Executes model code via exec() (prompt-injection #199), plus unescaped model text reaching os.system in permission dialogs (#196).
UFO / UFO2 — microsoft/UFO · ⭐9,428 · MIT · pushed 2026-07-08 Microsoft's flagship Windows agent. On Windows it is deep: a HostAgent orchestrates per-app AppAgents using the UIA tree + Win32 + application COM APIs (native Word/Excel/PowerPoint control) + screenshots. But full GUI control is Windows-only: the "Linux agent" is an allowlisted shell executor with no GUI or screen, and macOS is entirely absent from the code. Its own agent loop (pluggable models incl. a native Claude adapter); no MCP for external hosts, but a documented REST dispatch API and headless CLI. The advertised isolated Picture-in-Picture desktop is not shipped.
TuriX-CUA — TurixAI/TuriX-CUA · ⭐3,148 · MIT · pushed 2026-07-30
Real host code for all three desktop OSes across 7 branches, with stdio MCP servers on macOS and Windows exposing a single coarse run_task tool. Honest-labeling caution: its "3rd on the OSWorld leaderboard, 64.2%" is a self-reported entry (OSWorld's own data shows 57.99 on the self-reported track; the vendor's Linux branch cites a third number), and the SOTA default models have no published weights.
OmniParser — microsoft/OmniParser · ⭐25,224 · CC-BY-4.0 · pushed 2026-07-20
Pure-vision perception. Converts any screenshot into labeled, ID'd interactable elements via a local FastAPI /parse/ endpoint — grounding for when accessibility trees fail (canvas apps, games). No hands. No MPS support, so Apple Silicon inference is CPU-only and slow (#187). License is CC-BY-4.0 code + AGPL ultralytics in the default install (README badge wrongly says MIT).
screenpipe — screenpipe/screenpipe · ⭐20,780 · proprietary since 2026-06-10 · pushed 2026-08-06
24/7 recording of screen + audio + input into searchable SQLite; first-class claude mcp add screenpipe. Two cautions: it was relicensed from MIT to a proprietary commercial license (free for personal/non-commercial use only), and it has zero actuation — perception/memory only, verified by the absence of input-injection code in its engine routes.
computer-use-demo — anthropics/claude-quickstarts · ⭐17,374 (repo-wide) · MIT · pushed 2026-05-28
Anthropic's reference Docker Ubuntu + X11 desktop; multi-arch image (arm64) runs natively on Apple Silicon Docker. The canonical pattern most containerized projects copy. A sibling computer-use-best-practices directory is a macOS-native variant (pyautogui + Quartz) that Anthropic advises running inside a disposable VM. No MCP — shell out or reuse the MIT code.
Lume (in trycua/cua) — macOS and Linux VMs on Apple Silicon via Apple's Virtualization.Framework; the only way to sandbox native macOS automation (Windows/Android guests come from cua's separate QEMU/cloud sandbox path, not Lume). AIO Sandbox — agent-infra/sandbox (⭐5,617, Apache-2.0) — a Linux desktop container + aggregated MCP hub (claude mcp add --transport http aio); the runtime image is closed-source freeware (no published Dockerfile) and needs seccomp=unconfined [vendor]. E2B Desktop — e2b-dev/desktop (⭐1,443, Apache-2.0) — clean SDK over a metered cloud Linux VM; no MCP. Its companion reference agent e2b-dev/open-computer-use (⭐2,188) runs an own-loop agent over that sandbox — cloud-only, dormant. Bytebot — bytebot-ai/bytebot (⭐11,089) — archived 2026-03-07; still deployable, unmaintained privileged container.
scrcpy — Genymobile/scrcpy · ⭐147,084 · Apache-2.0 · pushed 2026-07-12
The highest-quality Android eyes+hands primitive. Pushes a Java server over adb (runs as shell uid, no root, no install), streams the screen as H.264/H.265/AV1 (~35–70ms [vendor]), and injects multi-touch/key/text/scroll via reflection on the hidden InputManager.injectInputEvent. Pixels only — no tree, no OCR — so an external VLM must do all grounding. Not an agent; driven via the community scrcpy-mcp (65★, claude mcp add android -- npx scrcpy-mcp) or subprocess. Its protocol has no version compatibility, so wrappers bundle their own server jar and drift.
Appium — appium/appium · ⭐21,839 · Apache-2.0 · pushed 2026-08-06
The industry-standard mobile substrate: a W3C WebDriver server with UiAutomator2 (Android) and XCUITest/WDA (iOS) drivers. Real input injection through on-device instrumentation, full accessibility-tree page source, both platforms, physical + emulator, fully local. Ships a first-party MCP since 2025 (appium-mcp: claude mcp add appium-mcp -- npx -y appium-mcp@latest) with embedded drivers; its VLM-grounding tool is opt-in, so an external host is the brain. Heavy toolchain (Node 22+, Android SDK / Xcode).
Maestro — mobile-dev-inc/Maestro · ⭐15,245 · Apache-2.0 · pushed 2026-08-04
Production E2E automation with a first-party MCP bundled in the CLI (claude mcp add maestro -- maestro mcp). Its own UiAutomator-gRPC driver (Android physical + emulator, no root) and XCUITest runner (iOS simulators only upstream). Selector-based grounding over the native hierarchy; no VLM required. MCP actuation is flow-granular (runs YAML flows, not per-gesture tools).
mobile-mcp — mobile-next/mobile-mcp · ⭐5,824 · Apache-2.0 · pushed 2026-08-06 · [cand]
The broadest mobile MCP for an agent host: Android over adb (accessibility snapshots as primary perception, screenshots as fallback, via the mobile-next/devicekit-android helper), iOS simulators via Xcode, iOS real devices via go-ios + WDA + tunnel. The most complete single MCP for cross-platform mobile. Limit: iOS real-device use inherits the full signed-WDA + tunnel setup. Candidate-level: evidence-profiled and API-measured, not run through the five-refutation pass.
Open-AutoGLM — zai-org/Open-AutoGLM · ⭐25,960 · Apache-2.0 (framework), MIT (weights) · pushed 2026-03-06 Zhipu's open phone-agent + open-weight AutoGLM-Phone-9B VLM. Pure screenshot→coordinates loop over adb; also HarmonyOS (HDC) and iOS (self-signed WDA). No tree, no OCR — grounding is trained into the model, so swapping in a generic VLM (a user tried Claude 4 Sonnet, #383) mis-taps. CLI/Python only; the README ships a deployment guide addressed to Claude Code as orchestrator. Dormant since March 2026 with 260 open issues.
Mobile-Agent / GUI-Owl — X-PLUG/MobileAgent · ⭐9,051 · MIT · pushed 2026-07-07 Alibaba's multi-agent GUI lineage around the open-weight GUI-Owl VLM (2B–32B, MIT). Android + HarmonyOS, screenshot-based, adb + ADBKeyBoard hands; iOS explicitly unsupported. Mobile-Agent-E natively accepts Claude as the backbone. No MCP — CLI scripts.
Airtest — AirtestProject/Airtest · ⭐9,496 · Apache-2.0 · pushed 2026-03-23
NetEase's classical vision-automation stack: OpenCV template matching (no OCR, no VLM), minicap capture, minitouch/maxtouch injection, no root. Companion Poco adds UI trees two ways — the poco-sdk compiled into game engines, and the UiAutomation hierarchy from pocoservice.apk for native Android apps (it kills com.github.uiautomator, so it conflicts with uiautomator2, with crash-loop reports) — but Poco is unmaintained since Jan 2024. Android-deep, iOS via WDA, also Windows desktop. Python API / CLI; no official MCP. Vision grounding needs user-supplied crop images — an LLM agent must generate them from live screenshots.
uiautomator2 — openatx/uiautomator2 · ⭐8,252 · MIT · [cand]
The workhorse Python library for Android UiAutomator control: pushes an on-device UiAutomator HTTP server (jar), exposes the accessibility tree and input injection over it. Not an agent and has no first-party MCP (community MCP wrappers exist, e.g. mcp-android-server-python). Candidate-level: swept and API-measured, not run through the five-refutation pass.
OSWorld — xlang-ai/OSWorld (⭐3,066) — 369 Ubuntu-desktop tasks (+49 Windows) with executable checkers; the desktop standard, used by Anthropic/OpenAI/ByteDance/Simular; runs on Apple Silicon via VMware Fusion. WindowsAgentArena — microsoft/WindowsAgentArena (⭐885, MIT) — the Windows-desktop equivalent, 154 tasks [vendor] in Windows 11 VMs. [cand] — its metadata was API-confirmed but it was not run through the five-refutation pass. AndroidWorld — google-research/android_world (⭐841) — 116+ tasks on a pinned Pixel 6 emulator with deterministic device-state rewards; the Android standard, built on google-deepmind/android_env.
The point of the anatomy is that functions compose. Common stacks an engineer can assemble:
| Goal | Perception | Grounding | Actuation / surface | Brain |
|---|---|---|---|---|
| Drive my real Mac, deterministic | Peekaboo AX tree | element ID | Peekaboo CGEvent, host | Claude Code (MCP) |
| Drive my real Mac/Win/Linux, one tool | cua-driver tree | element or pixel | cua-driver, host | Claude Code (MCP) |
| Safe unattended Linux tasks | demo screenshot | pixel | computer-use-demo, Docker VM | demo's own loop (Claude API); Claude Code delegates via shell — nested-agent cost |
| Native Mac apps, isolated | cua-driver | element | cua-driver inside Lume macOS VM | Claude Code |
| Canvas/game UI with no tree | OmniParser boxes | OmniParser box → pixel center | any driver's pixel click | Claude Code |
| Windows, deterministic, no VLM | Terminator UIA | UIA selector | Terminator, host | Claude Code (MCP) |
| Android real device | Appium/Maestro tree | selector | first-party MCP, adb | Claude Code (MCP) |
| Android, cheapest pixels-only | scrcpy stream | pixel (your VLM) | scrcpy-mcp | Claude Code (MCP) |
| iOS real device | Appium/WDA tree | element | mobile-mcp (go-ios+WDA) | Claude Code (MCP) — Mac + signing required |
Two structural lessons from the map:
- Perception and grounding are the swappable layer. When a tree-based driver mis-clicks on a canvas app, drop OmniParser in front of it; the actuation and brain stay put.
- The brain coupling decides whether Claude Code fits at all. Drivers (MCP/CLI) let Claude Code reason. Integrated stacks with a hard-wired VLM (UI-TARS Desktop) reduce Claude Code to an orchestrator that delegates whole tasks.
The same OS-level facts recur across every tool, so they belong in one place.
macOS — AX + CGEvent, gated by TCC. External processes read the Accessibility tree and post CGEvents. The OS gates both behind TCC grants (Screen Recording + Accessibility). This is why every macOS host driver has the same first-run permission ritual.
Windows — UIA + SendInput, gated by UAC/UIPI. The UI Automation tree is rich (it even exposes app COM APIs, which UFO exploits for native Office control). UIPI blocks a normal-integrity process from driving an elevated window — a UIAccess worker is the emerging workaround (cua #1706). A separate problem: SSH/Session-0 logins have no interactive desktop, so a logon Scheduled Task is needed to get one.
Linux — X11 is trivial, Wayland is the wall.
- X11: xdotool, XTest, wmctrl, scrot, pyautogui all work. This is why nearly every generic tool is "X11-only."
- Wayland: blocks external screenshot and input injection by design. The sanctioned path is XDG portals with interactive consent — awkward for an agent. Workarounds observed across tools: compositor-private D-Bus APIs (GNOME Shell, KWin), running code inside the compositor (a GNOME Shell extension — cua-driver and computer-use-linux both do this), kernel
uinputvia ydotool (needs root/udev, breaks non-US layouts), wlroots virtual-pointer protocols (Sway/Hyprland only), or an XWayland capture bridge. There is no universal Wayland answer; support is per-compositor.
Android — three open channels, no root. adb input, UiAutomator tree, and accessibility-service APKs. The friction is text (non-ASCII needs the ADBKeyBoard IME) and secure screens (black screenshots).
iOS — one gated channel. XCTest/WebDriverAgent only, signed and Mac-built (section 5). Everything else is simulator-bound or jailbreak.
Claims that died under verification, plus credible-looking tools deliberately set aside. Re-importing these unexamined would corrupt a future decision.
| Claim / candidate | What is actually true |
|---|---|
| "Open Interpreter does computer use" | Pivoted to a Rust Codex-fork coding agent; its "Computer Use" skill just curl-installs cua-driver. Old URL 404s. |
| "Bytebot is a live option" | Archived 2026-03-07 after acquisition [lead: PitchBook]; unmaintained privileged container. |
| "screenpipe is open source" | Relicensed to proprietary 2026-06-10; also eyes-only, no hands. |
| "apple-mcp is Mac computer use" | Archived 2026-01; only 7 Apple apps via AppleScript, no screenshot/mouse/keyboard. Maintained AppleScript alternative: peakmojo/applescript-mcp. |
| "Terminator / MCPControl are cross-platform" | Both are Windows-only now; Terminator compile_error!s off-Windows, MCPControl stale since Dec 2025. |
| "nut.js is a safe foundation" | Upstream dead (maintainer's "I give up" post, 2024-05; npm packages unpublished). Live lineage is the community @nut-tree-fork scope; UI-TARS Desktop and Bytebot build on nut.js forks. |
| "UFO / Agent S give real Linux (or Mac) desktop control" | UFO Linux = shell only, no macOS at all. Agent S Linux is X11-only and OSWorld-shaped; macOS unbenchmarked. |
| "Self-Operating Computer / Cradle / Computer Use OOTB are current" | All dormant (last functional code 2025-05 / 2024-11 / 2025-05) and their pinned Claude models are retired, so their Claude paths fail against the API today. |
| "Open Interface can be driven by an agent" | Alive but slow (GPL-3.0 GUI app, last substantive code 2026-02); zero integration surface — no MCP, no CLI flags, no API, input only via its Tkinter textbox. Cannot be driven by an external host. |
| "TuriX is 3rd on the OSWorld leaderboard" | Self-reported entry (57.99 on the self-reported track); three inconsistent numbers; SOTA models have no open weights. |
| "AndroidWorld/OSWorld let you control a device" | Benchmarks — the agent acts on a sandboxed VM/emulator, never your own machine. |
| Excluded: askui/vision-agent (now askui/python-sdk) | Cross-platform incl. Android, but needs a proprietary AskUI Controller backend — fails the open-runtime test. |
| Excluded: mikechambers/adb-mcp | "adb" here is Adobe, not Android — a search trap. |
Method. Three research passes (original desktop; Windows + Linux supplement; Android + iOS supplement), 45 verification agents dispatched, ~31 projects taken through a full five-refutation pass. Each such candidate got one adversarial verification agent briefed to refute five claims against primary sources: browser-only? abandoned? cloud/VLM-locked? per-OS support inflated (checked separately per platform)? no agent-host path? Per-platform completeness critics then surfaced misses (UFO, Terminator, Airtest, AndroidWorld) which were verified the same way; WindowsAgentArena was API-spot-checked only (marked [cand]), not five-refutation-verified. The ~10 iOS candidates were evidence-profiled for mechanism reality rather than run through the full refutation pass. cua-driver's and Agent S's non-macOS support were re-verified in depth because the first pass left them at [vendor]. Rows marked [cand] throughout are candidate-level: evidence-profiled or API-checked, not five-refutation-verified.
Declared coverage limits.
- Nothing was executed. Every capability claim is code-level plus adoption evidence.
- Windows was the weakest desktop platform in the sweep before Terminator + WindowsAgentArena were added; it is now representative.
- Linux host-control is a young, per-compositor niche — outside cua-driver and computer-use-linux, most agents get Linux via containers, not host control.
- iOS was surveyed for mechanism reality, not exhaustively for every simulator MCP.
- Browser-only heavyweights seen and excluded: Browser Use (~52k), Midscene.js (~14.5k), Microsoft Fara (~6.1k)
[approx].
The full verification record — every project's mechanism breakdown, five refutation verdicts, and per-claim evidence labels — is retained with the research archive and exceeds what is summarized here.
Atlas compiled 2026-08-06. ~31 projects refutation-verified plus ~10 iOS candidates evidence-profiled, across desktop and mobile; 45 verification agents dispatched. Every load-bearing claim traces to the linked primary source under the evidence convention in the header; rows marked [cand] are candidate-level.