desktop-act

desktop-act

This plugin exposes computer-use primitives via the mcp__desktop-act__* tools plus a one-shot act(goal) that runs an autonomous loop via OAuth (no API key).

Architecture

• Parent Claude session is the "brain" — calls primitives, Reads screenshots, decides next action, calls next primitive. Natural streaming reasoning.
• MCP server provides primitives: screenshot, click, type_text, keypress, scroll, list_windows, launch_app.
• High-level act(goal) spawns a sub-session via claude-agent-sdk for hands-off goals — but the streaming-friendly path is calling primitives directly.

Watching the desktop

Connect via noVNC at: http://:6081 That bridges to Xvnc :1. Whatever the primitives do is visible there in real time.

When to use

Task	Approach
Browser dashboard with auth (Stripe/Vercel/AWS/etc)	Prefer mcp__kapture__* (Sam's real Chrome)
Local app you need to click/type into	desktop-act primitives
Generic public site automation	mcp__playwright__*
Perf/console/Lighthouse	mcp__chrome-devtools__*
"Just do it" hands-off	mcp__desktop-act__act(goal)

See ~/.claude/skills-library/browser-automation.md for the full decision tree.