android-screen

android-screen

Three primitives, used in two main loops:

1. Annotate-then-tap — screen capture --annotate produces a labeled image; screen resolve translates label numbers into (x, y); adb shell input tap performs the action.
2. Layout diff — android layout produces a structured JSON snapshot of the on-screen view tree; --diff shows only what changed since the last snapshot.

This skill assumes a device is already connected (use android-emulator or android-deploy to get there).

Workflow A — annotated tap automation

android screen capture --annotate -o ui.png
   ↓ (user/agent picks a label)
android screen resolve --screenshot=ui.png --string="input tap #5"
   ↓ (resolved to e.g. "input tap 540 1283")
adb shell <resolved>

Concrete sequence:

# 1. Capture an annotated screenshot
android screen capture --annotate -o ui.png

# 2. Show the user (or yourself) the image to identify which numbered label
#    corresponds to the target UI element.

# 3. Resolve label → coordinates
android screen resolve --screenshot=ui.png --string="input tap #5"
# Output: input tap 540 1283

# 4. Execute the tap
adb shell input tap 540 1283

The --string argument can interleave labels with literal text: "input swipe #3 #7" resolves both #3 and #7.

Critical rule: re-capture invalidates labels

The annotation numbering is generated at capture time. After any UI change (a tap that opens a new screen, a layout reflow, a snackbar appearing), the previous annotated image's label numbers no longer correspond to the same elements. Always re-capture before resolving if anything has changed since the last capture.

Workflow B — layout diff debugging

# 1. Snapshot baseline
android layout --pretty -o before.json

# 2. Trigger the change (user action, server response, etc.)

# 3. See only what changed
android layout --diff

Use cases:

• Tracking which views actually re-rendered after a state change.
• Confirming a Compose recomposition is or isn't reaching a target node.
• Investigating "why did this view appear / disappear".

The internal snapshot is updated each time android layout runs, so two consecutive calls without intervening UI changes will show no diff.

Operating rules

1. Capture is cheap, re-capture before each resolve. Label numbers are not stable across captures.
2. Resolve, then tap — don't paste coordinates from the screenshot inspector. Hardcoded pixels break across devices and orientations.
3. For repetitive automation, script the chain, don't run interactively. Example:

   android screen capture --annotate -o /tmp/ui.png
   adb shell $(android screen resolve --screenshot=/tmp/ui.png --string="input tap #2")
   

4. --annotate is required for resolve. A plain capture (without --annotate) has no labels to resolve against.
5. Use --pretty only for human reading. For diffs and automated parsing, the compact form is more efficient.
6. adb shell input for taps, not monkey. Monkey is for stress testing, not deterministic automation.
7. Don't chain captures into git. Annotated PNGs are large and noisy; write them to /tmp/ or build/ (gitignored) by default.

Decision matrix — which tool for what

Goal	Use	Don't use	Why
Find tap coordinates without measuring pixels	screen capture --annotate + resolve	manual inspection	Labels survive DPI / orientation differences
Detect what changed in the UI	layout --diff	screenshot diff	Structured, ignores anti-aliasing noise
Full view hierarchy for offline analysis	layout --pretty -o	adb shell uiautomator dump	JSON is structured and stable; uiautomator XML is verbose
Scripted button clicks in tests	Espresso / Compose UI test	this skill	Tests should not depend on pixel coordinates
Visual regression	Paparazzi / Roborazzi	this skill	Headless, hermetic, runs in CI
Pixel-perfect screenshot for marketing	Studio Device Screenshot	screen capture	This skill is for automation, not asset creation

Anti-patterns

• ❌ Resolving against a stale ui.png after taps have changed the screen.
• ❌ Hardcoding input tap 540 1283 in a script — different device → wrong button.
• ❌ Capturing without --annotate then trying to resolve — fails silently with bad output.
• ❌ Using android layout as a substitute for proper UI tests.
• ❌ Committing annotated screenshots to the repo.
• ❌ Calling this skill when no device is connected — produces empty/error output. Pre-flight adb devices.

Error recovery

Error	Cause	Fix
screen capture produces empty PNG	No connected device or screen off	adb devices; adb shell input keyevent KEYCODE_WAKEUP
resolve returns the input unchanged	Label #N doesn't exist in screenshot	Re-run screen capture --annotate, look at fresh labels
layout returns {}	App not in foreground	adb shell am start -n <pkg>/<activity> first
Coordinates feel "off" by a constant	Status bar / cutout offsets in fullscreen mode	Re-capture; coordinates account for current insets