From session-orchestrator
Drives native-UI AX-tree snapshots and screenshots on macOS 15+ via steipete/peekaboo. Dispatched by test-runner to capture deterministic JSON output.
How this skill is triggered — by the user, by Claude, or both
Slash command
/session-orchestrator:peekaboo-driverhaikuThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Before anything else, read and internalize `soul.md` in this skill directory. It defines WHO you are — a thin executor, not an orchestrator. Every action in this session should reflect that identity.
Before anything else, read and internalize soul.md in this skill directory. It defines WHO you are — a thin executor, not an orchestrator. Every action in this session should reflect that identity.
Read skills/_shared/bootstrap-gate.md and execute the gate check. If GATE_CLOSED, invoke skills/bootstrap/SKILL.md and wait for completion. If GATE_OPEN, continue.
Both install paths surface the same peekaboo binary at the same version:
| Install path | Command | Notes |
|---|---|---|
| Homebrew (preferred) | brew install steipete/tap/peekaboo | Installs to /opt/homebrew/bin/peekaboo |
| npm / npx (CI-friendly) | npx -y @steipete/peekaboo | Requires Node 22+. No global install needed. |
The GitHub repository is github.com/steipete/peekaboo. Older links may surface openclaw/Peekaboo — both redirect to the same canonical repo and binary.
Canonical name verification: Always verify a package name via brew info or npm view before documenting. The playwright-driver PRD originally referenced @playwright/cli@0.1.13 (an unrelated stub) instead of canonical playwright@1.60.0. Verify @steipete/peekaboo against any look-alike before trusting an install path.
brew install steipete/tap/peekaboo # Option A — Homebrew
npx -y @steipete/peekaboo --version # Option B — npx (CI)
peekaboo --version # Verify: expect 3.1.x
Baseline: v3.1.0. Latest at time of writing: v3.1.2 (May 11 2026).
Mandatory pre-flight. Run before any other peekaboo command.
[ "$(uname -s)" != "Darwin" ] && { echo "peekaboo-driver: non-darwin, skipping." >&2; exit 0; }
MACOS_MAJOR="$(sw_vers -productVersion | cut -d. -f1)"
[ "$MACOS_MAJOR" -lt 15 ] && { echo "peekaboo-driver: requires macOS 15+, skipping." >&2; exit 0; }
command -v peekaboo >/dev/null 2>&1 || { echo "peekaboo-driver: binary not found — install first." >&2; exit 2; }
Exit 0 (non-fatal skip) on non-darwin or macOS < 15.0 (Sequoia). Exit 2 (fatal) only when the binary is missing on an otherwise-compatible system.
PERMS_JSON="$(peekaboo permissions status --json)"
MISSING="$(echo "$PERMS_JSON" | jq -r '.data.permissions[] | select(.isRequired == true and .isGranted == false) | .name')"
[ -n "$MISSING" ] && echo "peekaboo-driver: missing required permissions: ${MISSING}" >&2
The permissions status --json schema (verified 2026-05-14):
{
"success": true,
"data": {
"permissions": [
{ "name": "Screen Recording", "isRequired": true, "isGranted": false, "grantInstructions": "System Settings > Privacy & Security > Screen Recording" },
{ "name": "Accessibility", "isRequired": true, "isGranted": false, "grantInstructions": "System Settings > Privacy & Security > Accessibility" },
{ "name": "Event Synthesizing", "isRequired": false, "isGranted": false, "grantInstructions": "System Settings > Privacy & Security > Accessibility" }
],
"source": "local"
}
}
Required: Screen Recording (hard), Accessibility (hard). Optional: Event Synthesizing. Permission failures are never silent. Route to Phase 3 when any required permission is not granted.
Surface missing required permissions via AskUserQuestion (per ask-via-tool.md AUQ-003). Do not auto-grant — macOS requires manual user action in System Settings.
If $MISSING from Phase 2 contains more than one permission, present one AUQ per missing permission in the order they appear. The user can grant or skip each independently; the driver re-checks after each grant (one full Phase 2 sweep per round).
The loop executes once per permission in $MISSING, exiting after Phase 3 permission re-probe shows all required permissions granted or the user selects Skip for the current permission.
For each ${PERM_NAME} in $MISSING:
AskUserQuestion({
questions: [{
question: `${PERM_NAME} permission is required but not granted. Open System Settings > Privacy & Security > ${PERM_NAME}, enable the terminal entry, then confirm here.`,
header: `Missing Permission: ${PERM_NAME}`,
options: [
{ label: "Granted — continue (Recommended)", description: `I have enabled ${PERM_NAME} in System Settings.` },
{ label: "Skip this run", description: "Abort peekaboo-driver. Test-runner will record a framework-error finding." }
],
multiSelect: false
}]
})
After confirmation, re-probe ($MISSING Phase 2 sweep). If still not granted or user selects Skip for any permission, exit 2 (fatal). Do NOT attempt any capture without required permissions — the binary will fail ungracefully.
The orchestrator (skills/test-runner/) dispatches via Bash. Inputs via environment variables: RUN_ID, RUN_DIR, TARGET (app name), PROFILE.
# Validate TARGET (app name) — alphanumeric, spaces, dots, hyphens only
# (Prevents shell injection per SEC-PD-MED-1 + path traversal per SEC-PD-LOW-1)
[[ "${TARGET}" =~ ^[A-Za-z0-9\ \.\-]+$ ]] || { echo "peekaboo-driver: invalid TARGET (must match ^[A-Za-z0-9 .-]+$)" >&2; exit 2; }
TARGET_SAFE="${TARGET//[^A-Za-z0-9_.-]/_}" # filename-safe variant for artifact paths
RUN_DIR=".orchestrator/metrics/test-runs/${RUN_ID}"
mkdir -p "${RUN_DIR}/ax-snapshots" "${RUN_DIR}/screenshots"
peekaboo see --app "${TARGET}" --json > "${RUN_DIR}/ax-snapshots/main-${TARGET_SAFE}.json"
peekaboo image --app "${TARGET}" --format png --path "${RUN_DIR}/screenshots/main-$(date +%s%3N).png"
.orchestrator/metrics/test-runs/<run-id>/
results.json # driver summary {run_id, exit_code, scenarios_*}
exit_code # plain integer: 0, 1, or 2
ax-snapshots/
<scenario>.json # peekaboo see --json output (one per scenario)
glass-modifiers-<ts>.json # Liquid Glass conformance artifact (Check 4)
screenshots/
<step>-<timestamp>.png # peekaboo image output
console.ndjson # driver log events (NDJSON)
Token discipline: NEVER inline AX-tree dumps into the coordinator context. Always write to disk. The ux-evaluator agent reads from disk, not from prompt context.
peekaboo see --app "${APP_NAME}" --json > "${RUN_DIR}/ax-snapshots/${SCENARIO}.json"
Key fields in the see --json output the ux-evaluator reads: snapshot_id (capture ID), ui_elements (flat AX element list — primary consumer input), ui_map (hierarchical tree), interactable_count, capture_mode ("screen" or "window"). Each ui_elements entry carries role, role_description, bounds ({x, y, width, height}), identifier, and children. ux-evaluator Check 4 scans ui_elements for frame identifiers and cross-references the glass-modifiers artifact.
For SwiftUI 26+ targets (projects with Package.swift declaring .iOS("26") or .macOS("26")+), emit a conformance artifact alongside the AX snapshot:
# Gate: glass-modifiers emit is opt-in for v2 rubric forward-compat (v1 rubric does not consume).
if [ "${RUBRIC_GLASS_V2:-0}" = "1" ]; then
cat > "${RUN_DIR}/ax-snapshots/glass-modifiers-$(date +%s%3N).json" <<EOF
{
"schema_version": "v1",
"status": "stub",
"run_id": "${RUN_ID}",
"captured_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"glassEffect_frames": [],
"legacy_material_frames": [],
"blur_modifier_frames": []
}
EOF
fi
The ux-evaluator Check 4 reads this file. glassEffect_frames = compliant (uses .glassEffect()). legacy_material_frames = non-compliant (uses .background(.thinMaterial) etc.). blur_modifier_frames = non-compliant (uses .blur(radius:) as background). In v1 the arrays are always empty — the evaluator falls back to screenshot-only analysis. Do NOT emit a bare {} — use the full schema structure with empty arrays.
RUN_DIR="${RUN_DIR}" TARGET="${APP_NAME}" PROFILE="${PROFILE}" bash skills/peekaboo-driver/SKILL.md
All inputs via environment variables. No positional arguments accepted.
| Code | Meaning | Orchestrator Action |
|---|---|---|
| 0 | All captures succeeded (or platform skip) | Record pass or skip, continue |
| 1 | At least one capture failed | Failures become findings (non-fatal) |
| 2 | Framework error (missing binary, permission denied, OS mismatch) | Surface as driver error, halt peekaboo portion of run |
${RUN_DIR}/exit_code — plain integer file written by driver before exit${RUN_DIR}/results.json — driver summary with exit_code, scenarios_attempted, scenarios_passed, scenarios_failed${RUN_DIR}/ax-snapshots/<scenario>.json — peekaboo AX-tree output per scenario${RUN_DIR}/ax-snapshots/glass-modifiers-<ts>.json — Liquid Glass conformance artifact (consumed by ux-evaluator Check 4; only emitted when RUBRIC_GLASS_V2=1 env var is set — see Phase 4 emission block)${RUN_DIR}/screenshots/<step>-<ts>.png — per-step screenshots (evidence for ux-evaluator findings)${RUN_DIR}/console.ndjson — driver log events as NDJSONBaseline: v3.1.0 (2026-05-14). Latest: v3.1.2 (May 11 2026). Re-validate on v3.2+. Key surfaces: permissions status --json schema, see --json schema (ui_elements, ui_map), image --path flag behavior.
| Subcommand | Purpose |
|---|---|
peekaboo permissions status [--json] | Check Screen Recording / Accessibility / Event Synthesizing grant status |
peekaboo see --app <name> [--json] [--mode screen|window] | Capture AX-tree snapshot for a running application |
peekaboo image --app <name> [--format jpg|png] [--path <output>] | Capture screenshot of a running application |
peekaboo click [query|--on <id>|--coords x,y] | Synthesize a click event (requires Event Synthesizing) |
peekaboo type [text] [--delay ms] [--wpm 80-220] | Synthesize keyboard input |
peekaboo scroll --direction up|down|left|right [--amount n] | Synthesize a scroll gesture |
peekaboo run <scriptPath> [--output file] [--no-fail-fast] [--json] | Execute a .peekaboo.json script in scripted mode |
peekaboo agent [task] [--max-steps n] [--dry-run] | Autonomous agent mode (non-deterministic — not for CI) |
tccutil or similar workarounds.${RUN_DIR}/ax-snapshots/. Trees for complex apps exceed 50K tokens.peekaboo agent mode in automated CI runs — non-deterministic and expensive. Reserve for manual exploratory sessions.{} for the glass-modifiers artifact — use the full schema with empty arrays. The ux-evaluator validates schema fields.| File | Purpose |
|---|---|
soul.md | Driver identity: thin wrapper around native macOS UI capture, not an orchestrator |
npx claudepluginhub kanevry/session-orchestrator --plugin session-orchestratorVerifies UI changes visually for mobile (Maestro), web (Playwright), and macOS (Peekaboo). Includes availability probes, path selection, screenshot capture, and dev-server preflight.
Captures screenshots and videos of running macOS app windows via osascript, screencapture, and ffmpeg for UI verification, mockups, and visual comparisons in agent workflows.
Controls macOS GUI applications via mouse automation, keyboard input, screenshots, image recognition, and AppleScript execution.