A closed loop set of deterministic scripts `claude` made itself to drive iOS simulator based testing using xcrun and ffmpeg

!ios-osascript

A closed loop set of deterministic scripts `claude` made itself to drive iOS simulator based testing using xcrun and ffmpeg

README.md

Simulator driving toolkit

Deterministic scripts to drive + capture the booted iOS Simulator from the shell — built while fixing the tux soft-keyboard bug (#184). The point: reliably tap the sim and see what a tap triggers, including UI that flashes for a few frames.

One-time setup

1. Boot a sim with an installed runtime:

   xcrun simctl list runtimes                     # need e.g. iOS 26.5
   xcrun simctl create washe-test "iPhone 17 Pro" com.apple.CoreSimulator.SimRuntime.iOS-26-5
   xcrun simctl boot washe-test && open -a Simulator

2. Install deps: brew bundle --file=osascript/Brewfile (cliclick + ffmpeg)
3. Grant Accessibility to BOTH the controlling terminal AND the cliclick binary (System Settings → Privacy & Security → Accessibility → + → ⌘⇧G → /opt/homebrew/Cellar/cliclick/<ver>/bin/ → cliclick).
   • The terminal grant covers osascript/System Events (window reads, menu clicks). It does not cover cliclick — CGEventPost button events fail silently without cliclick's own grant (cursor moves still work, so a moving cursor is NOT proof clicks land).
4. Move the window to the main display: ./sim-move-to-main.sh cliclick reads a leading - as a relative move, so a left-hand display (negative X) can't be tapped.

Coordinate space

Everything takes device-pixel coords — the pixels of xcrun simctl io booted screenshot. Check your device's size once with sips -g pixelWidth -g pixelHeight shot.png and export IMAGE_W/IMAGE_H if it isn't the iPhone-17-Pro default (1206×2622). Mapping to screen coords assumes the device glass is rendered height-constrained inside the window content area (below a ~28pt title bar) and horizontally centered.

Scripts

script	what
sim-window-geometry.sh	print live window frame X Y W H
sim-move-to-main.sh [X Y]	move sim window to main display (default 80,60)
sim-tap.sh <dx> <dy>	real HID tap at device px (cliclick)
sim-swipe.sh <x1> <y1> <x2> <y2>	real HID drag (washe page-turn etc.)
sim-shot.sh [out.png]	screenshot
sim-capture-tap.sh <dx> <dy> [base]	record → tap → montage (the main loop)
sim-hardware-keyboard.sh	toggle Connect Hardware Keyboard (suppresses the soft keyboard when on)
ffmpeg-montage.sh <in.mov> …	tile a recording into one grid image (catch ~150ms flashes)
ffmpeg-frames.sh <in.mov> …	extract individual stills from a recording

Soft keyboard gotchas (iOS Simulator)

• I/O → Keyboard → Connect Hardware Keyboard (✓ by default): your Mac keyboard types into the sim. This does NOT reliably suppress the on-screen keyboard on modern iOS — the soft keyboard can still show on focus.
• I/O → Keyboard → Toggle Software Keyboard (⌘K): force-show/hide the on-screen keyboard. It's a persistent toggle — if you hide it to read the screen, later focus() calls won't show it until you toggle back. Reset the sim (Device → Erase All Content and Settings) if the state gets confusing.
• First time the soft keyboard appears you get iOS's QuickPath "slide to type" intro with a Continue button — dismiss it once (a physical tap in the window is most reliable; synthetic taps on that modal can be flaky).
• To read the REPL prompt while typing, hide the soft keyboard with ⌘K (the textarea stays focused) and type via osascript -e 'tell application "System Events" to keystroke "..."' — the full screen is then unobscured.
• sim-hardware-keyboard.sh clicks the Connect Hardware Keyboard item; menu clicks can occasionally no-op, so verify the mark with the AX query in that script.

Why not System Events click at {x,y}?

It does an AX-press on the element under the point (returns group 1 of window …), which web content ignores. Only real HID events become iOS touches — hence cliclick.

Brewfile

# Homebrew deps for the iOS Simulator driving/capture toolkit in this dir.
# Install:  brew bundle --file=osascript/Brewfile
#
# cliclick posts the real HID clicks the Simulator turns into touches. AFTER
# installing you must grant the cliclick BINARY Accessibility (System Settings >
# Privacy & Security > Accessibility) — the terminal's grant does NOT cover it,
# and CGEventPost button events fail silently without it. See README.md.
brew "cliclick"

# ffmpeg tiles recordings into montages / extracts stills (ffmpeg-montage.sh,
# ffmpeg-frames.sh) so a soft keyboard that flashes for ~150ms is visible.
brew "ffmpeg"

# Not brew-installable, but required by the toolkit (documented for reference):
#   - Xcode + an installed iOS runtime  (xcrun simctl, Simulator.app)
#       xcrun xcodebuild -downloadPlatform iOS
#   - macOS built-ins: osascript, sips, python3

ffmpeg-frames.sh

#!/bin/bash
# Extract individual PNG frames from a recording, for close inspection of a
# moment the montage flagged. Usage:
#   frames.sh <in.mov> [out_dir] [fps] [start_s] [count]
# Defaults: out_dir=frames fps=30 start=0 count=all
# Frames are written as <out_dir>/frame-0001.png ...
set -euo pipefail
IN="$1"; DIR="${2:-frames}"; FPS="${3:-30}"; START="${4:-0}"; COUNT="${5:-}"
mkdir -p "$DIR"; rm -f "$DIR"/frame-*.png
ARGS=(-ss "$START" -i "$IN" -vf "fps=$FPS")
[ -n "$COUNT" ] && ARGS+=(-frames:v "$COUNT")
ffmpeg "${ARGS[@]}" "$DIR/frame-%04d.png" -y >/dev/null 2>&1
echo "wrote $(ls "$DIR"/frame-*.png | wc -l | tr -d ' ') frames to $DIR/ (@${FPS}fps from ${START}s)"

ffmpeg-montage.sh

#!/bin/bash
# Tile frames of a screen recording into ONE image so transient UI (e.g. a soft
# keyboard that flashes for ~150ms) is visible at a glance without paging through
# dozens of stills. High fps + a grid is the reliable way to catch sub-frame UI.
#
# Usage: montage.sh <in.mov> [out.png] [fps] [cols] [rows] [start_s] [thumb_w]
# Defaults: out=montage.png fps=30 cols=8 rows=6 start=0 thumb_w=150
#   fps    = sampling rate (30 catches ~33ms events; raise for briefer ones)
#   start  = seconds to skip (jump near the action to keep the grid dense)
#   cols*rows = number of tiles shown (8x6 = 48 frames)
set -euo pipefail
IN="$1"; OUT="${2:-montage.png}"; FPS="${3:-30}"; COLS="${4:-8}"; ROWS="${5:-6}"
START="${6:-0}"; TW="${7:-150}"
ffmpeg -ss "$START" -i "$IN" \
  -vf "fps=$FPS,scale=$TW:-1,tile=${COLS}x${ROWS}" \
  -frames:v 1 "$OUT" -y >/dev/null 2>&1
echo "wrote $OUT  (${COLS}x${ROWS} @ ${FPS}fps from ${START}s of $IN)"

sim-capture-tap.sh

#!/bin/bash
# Deterministic "tap and capture" loop: start a screen recording, tap the sim at
# device-pixel coords, stop, and tile the result into a montage so you can SEE
# transient UI the tap triggers (e.g. a soft keyboard that flashes for ~150ms).
#
# Usage: sim-capture-tap.sh <device_px_x> <device_px_y> [out_basename]
# Produces: <base>.mov and <base>-montage.png in the current dir.
#
# Timeline: 1.0s settle -> tap -> 2.5s observe. The montage samples 30fps from
# 1.0s so the tap moment lands in a dense grid. Same GOTCHAS as sim-tap.sh.
set -euo pipefail
DX="$1"; DY="$2"; BASE="${3:-capture}"
HERE="$(dirname "$0")"
MOV="$BASE.mov"; MON="$BASE-montage.png"
rm -f "$MOV"

xcrun simctl io booted recordVideo --codec=h264 --force "$MOV" >/dev/null 2>&1 &
REC=$!
sleep 1.0
echo "tapping ($DX,$DY)..." >&2
"$HERE/sim-tap.sh" "$DX" "$DY"
sleep 2.5
kill -INT "$REC" 2>/dev/null || true
wait "$REC" 2>/dev/null || true

"$HERE/ffmpeg-montage.sh" "$MOV" "$MON" 30 8 6 1.0 150
echo "wrote $MOV and $MON"

sim-hardware-keyboard.sh

#!/bin/bash
# Toggle "I/O > Keyboard > Connect Hardware Keyboard" in the Simulator menu bar.
# When the hardware keyboard is CONNECTED, iOS SUPPRESSES the on-screen software
# keyboard on focus() — so disconnect it to test soft-keyboard behavior.
# Uses AX menu actions (authorized via the terminal's Accessibility grant).
# Usage: sim-hardware-keyboard.sh   (toggles; check the menu to see resulting state)
osascript <<'EOF'
tell application "Simulator" to activate
tell application "System Events" to tell process "Simulator"
  click menu item "Connect Hardware Keyboard" of menu "Keyboard" of menu item "Keyboard" of menu "I/O" of menu bar item "I/O" of menu bar 1
end tell
EOF

sim-move-to-main.sh

#!/bin/bash
# Move the booted Simulator window onto the MAIN display at positive coords.
# REQUIRED before using sim-tap.sh: cliclick parses a leading '-' as a RELATIVE
# move, so a window on a left-hand display (negative X) can't be tapped.
# Usage: sim-move-to-main.sh [X] [Y]   (default 80 60)
X="${1:-80}"; Y="${2:-60}"
osascript -e 'tell application "Simulator" to activate'
osascript -e "tell application \"System Events\" to tell process \"Simulator\" to set position of first window to {$X, $Y}"

sim-shot.sh

#!/bin/bash
# Screenshot the booted Simulator (device-pixel PNG). Usage: sim-shot.sh [out.png]
set -euo pipefail
OUT="${1:-shot.png}"
xcrun simctl io booted screenshot "$OUT" >/dev/null 2>&1
echo "wrote $OUT ($(sips -g pixelWidth -g pixelHeight "$OUT" 2>/dev/null | awk '/pixel/{printf $2" "}'))"

sim-swipe.sh

#!/bin/bash
# Swipe/drag on the booted Simulator between two DEVICE-PIXEL points (same
# coordinate space as sim-tap.sh). Posts a real HID drag via cliclick, which the
# Simulator turns into a touch drag (washe uses vertical drags to page-turn).
#
# Usage: sim-swipe.sh <x1> <y1> <x2> <y2>
# Same REQUIREMENTS/GOTCHAS as sim-tap.sh (cliclick a11y, main display, mapping).
set -euo pipefail
X1="$1"; Y1="$2"; X2="$3"; Y2="$4"
IMAGE_W="${IMAGE_W:-1206}"; IMAGE_H="${IMAGE_H:-2622}"; TITLEBAR="${TITLEBAR:-28}"
read WX WY WW WH < <("$(dirname "$0")/sim-window-geometry.sh")

map() { # device px -> screen px
  python3 - "$1" "$2" "$WX" "$WY" "$WW" "$WH" "$IMAGE_W" "$IMAGE_H" "$TITLEBAR" <<'PY'
import sys
dx,dy,wx,wy,ww,wh,iw,ih,tb = map(float, sys.argv[1:10])
gh = wh - tb; gw = gh*iw/ih
print(round(wx+(ww-gw)/2+(dx/iw)*gw), round(wy+tb+(dy/ih)*gh))
PY
}
read AX AY < <(map "$X1" "$Y1")
read BX BY < <(map "$X2" "$Y2")
echo "swipe device($X1,$Y1)->($X2,$Y2)  screen($AX,$AY)->($BX,$BY)" >&2
osascript -e 'tell application "Simulator" to activate' >/dev/null; sleep 0.25
cliclick m:"$AX","$AY" w:60 dd:"$AX","$AY" w:40 m:"$BX","$BY" w:40 du:"$BX","$BY"

sim-tap.sh

#!/bin/bash
# Tap the booted Simulator at DEVICE-PIXEL coords (the coordinate space of a
# `xcrun simctl io booted screenshot`). Deterministic: re-reads the live window
# frame every call and maps device px -> macOS screen coords, then posts a REAL
# HID click via cliclick (which the Simulator translates to an iOS touch).
#
# Usage:   sim-tap.sh <device_px_x> <device_px_y>
# Example: sim-tap.sh 603 2500     # Safari URL bar on a 1206x2622 screen
#
# REQUIREMENTS / GOTCHAS (learned the hard way):
#   - cliclick must be granted Accessibility (System Settings > Privacy &
#     Security > Accessibility). The controlling terminal's grant does NOT
#     cover the cliclick binary; CGEventPost button events fail SILENTLY
#     without it (cursor *moves* still work, so "the mouse moved" is not proof).
#   - The Simulator window must be on the MAIN display (positive coords): run
#     sim-move-to-main.sh first. cliclick reads a leading '-' as a relative move.
#   - System Events `click at {x,y}` does NOT work here: it does an AX-press on
#     the element under the point (returns "group 1 of window ...") which the
#     web content ignores. Only real HID events become touches. Hence cliclick.
#   - IMAGE_W/IMAGE_H must match your device's screenshot pixel size
#     (`sips -g pixelWidth -g pixelHeight shot.png`). Defaults = iPhone 17 Pro.
#   - TITLEBAR is the Simulator window's title bar height in points (~28).
set -euo pipefail
DX="$1"; DY="$2"
IMAGE_W="${IMAGE_W:-1206}"
IMAGE_H="${IMAGE_H:-2622}"
TITLEBAR="${TITLEBAR:-28}"

read WX WY WW WH < <("$(dirname "$0")/sim-window-geometry.sh")

# The device screen is rendered height-constrained inside the window content
# area (below the title bar) and centered horizontally; compute that glass rect.
read SX SY < <(python3 - "$DX" "$DY" "$WX" "$WY" "$WW" "$WH" "$IMAGE_W" "$IMAGE_H" "$TITLEBAR" <<'PY'
import sys
dx,dy,wx,wy,ww,wh,iw,ih,tb = map(float, sys.argv[1:10])
content_h = wh - tb
glass_h = content_h
glass_w = glass_h * iw / ih
x0 = wx + (ww - glass_w) / 2
y0 = wy + tb
sx = round(x0 + (dx/iw) * glass_w)
sy = round(y0 + (dy/ih) * glass_h)
print(sx, sy)
PY
)

echo "device($DX,$DY) -> screen($SX,$SY)  [win $WX,$WY ${WW}x${WH}]" >&2
osascript -e 'tell application "Simulator" to activate' >/dev/null
sleep 0.25
# move, dwell, explicit press/release — steadier than a bare click for the sim
cliclick m:"$SX","$SY" w:80 dd:"$SX","$SY" w:90 du:"$SX","$SY"

sim-window-geometry.sh

#!/bin/bash
# Print the booted Simulator window frame as: X Y W H
# (top-left origin, macOS global screen coords; needs Accessibility for the
# controlling terminal). Used by sim-tap.sh to map device px -> screen coords.
osascript -e 'tell application "System Events" to tell process "Simulator"
set w to first window
set {p, s} to {position, size} of w
return (item 1 of p as string) & " " & (item 2 of p as string) & " " & (item 1 of s as string) & " " & (item 2 of s as string)
end tell'