Computer Use & GUI Automation

Computer Use & GUI Automation

Computer use lets Claude interact with GUIs: click buttons, fill forms, take screenshots, and navigate native apps. This is powerful but expensive and slow — use it only when a more precise tool doesn't exist.

Tool Selection Priority

Before reaching for computer use, exhaust these options first:

Task	Prefer This	Over Computer Use
API endpoint testing	Bash + curl	Clicking through UI
Database inspection	MCP postgres/sqlite	Navigating admin UI
File operations	Read/Write/Edit	Drag-and-drop UI
Web scraping	Firecrawl MCP	Screenshot + parse
Browser automation	Playwright MCP	Computer use click
CI status	GitHub API / gh CLI	Browser navigation
Log inspection	Bash + grep	Terminal screenshot

Rule: If you can express the task as a shell command or API call, do that. Computer use is the fallback for GUI-only workflows.

---

When Computer Use Is the Right Choice

1. Native App Validation

Testing a desktop app that has no API or CLI interface.

# Example: Validate Electron app UI after a build
Take a screenshot of the app after launch.
Click the "New Project" button.
Verify the dialog opens with the correct fields.
Fill in project name: "Test Project 2026"
Click Create and verify the project appears in the list.

2. Visual Regression Checks

Detecting layout regressions that unit tests can't catch.

# Workflow:
1. Take baseline screenshot of the current UI state
2. Apply the change
3. Take comparison screenshot
4. Highlight pixel differences > 1%
5. Human reviews diff

3. GUI-Only Admin Tools

Admin panels, legacy enterprise software, and embedded UIs with no API.

# Example: Generate a report from a legacy admin panel
Navigate to: http://admin.internal/reports
Click: "Export" → "CSV" → "Last 30 days"
Wait for download
Move file to: /tmp/report-{date}.csv

4. Local Simulator Flows

Mobile simulator or desktop app testing that requires visual interaction.

# Example: iOS simulator validation
Launch: xcrun simctl launch booted com.example.MyApp
Take screenshot
Verify: "Welcome" text is visible in the header
Tap: "Get Started" button (coordinates or element description)
Verify: onboarding screen loads

---

Result Verification

Computer use output is inherently visual and unstructured. Always verify results with a structured check after GUI actions:

Verification Pattern

After each GUI action:
1. Take a screenshot
2. Verify the expected visual state (specific text, element position, color)
3. If verification fails: log "FAIL: {what was expected vs. what was seen}"
4. If unsure: take another screenshot from a wider viewport

At the end:
- List each action and its verification result
- Count: {N} actions taken, {M} verified OK, {K} failed

Confidence Levels

Confidence	Verification	Action
HIGH	Text matches exactly / element found by ID	Proceed
MEDIUM	Visual match but element found by position	Log and proceed
LOW	Can't find element / ambiguous screenshot	Stop, report to human

---

Safety Guardrails

Computer use can cause irreversible actions (delete files, send emails, submit forms). Apply these guardrails:

Never Without Confirmation

• Form submissions in production environments
• Delete or "Archive" actions
• Payment or billing interactions
• Sending emails or messages
• Anything involving real user data

Screenshot Audit Trail

Keep screenshots of:

• State before any action
• State after each major action
• Final state

Dry-Run First

For complex GUI flows, describe the steps and ask for confirmation before executing:

Before I click "Submit", here's what will happen:
- Form data: {summary}
- This action cannot be undone
- Proceeding? (yes/no)

---

Computer Use vs. Playwright MCP

For web UIs, Playwright MCP is almost always better than computer use:

	Playwright MCP	Computer Use
Reliability	High (DOM-based)	Medium (pixel-based)
Speed	Fast	Slow (screenshot per action)
Testability	Scriptable, repeatable	Hard to reproduce exactly
Cost	Low	High (vision model per screenshot)
Works on	Web browsers	Any visual surface

Use Playwright MCP for: Web app testing, scraping, form automation on websites.

Use Computer Use for: Native desktop apps, embedded UIs, legacy apps with no API.

---

Cost Awareness

Computer use is expensive:

• Each screenshot = vision model inference (high token cost)
• A 10-step GUI flow = 10+ vision inferences
• Compare: a 10-step shell script = near-zero cost

Estimate before using: If a GUI flow has N steps, expect N × (screenshot tokens + generation tokens). For flows > 20 steps, consider whether a shell/API approach exists.

---

Claude Desktop Requirement

Computer use requires the Claude Desktop app (not CLI or Web). The Desktop app has the screen capture and input simulation capabilities that CLI lacks.

CLI:     ❌ Computer use not available
Web:     ❌ Computer use not available
Desktop: ✅ Computer use available