desktop-act

desktop-act

Computer-use primitives + multi-desktop pool + autonomous act() loop for Claude Code. One MCP server, no API key.

desktop-act lets Claude (or any MCP client) drive a desktop: screenshot it, click, type, scroll, launch apps, run a goal-driven loop. It is OS-agnostic — a backend is selected at startup from platform.system():

• Linux (X11Backend): python-xlib primitives + a pooled Xvnc/websockify backend so you can spin up isolated desktops on demand, each streamed over noVNC for live viewing.
• macOS (MacBackend): screencapture for frames, cliclick/osascript for input, and the single real logged-in session (no Xvnc pool). Built-in Screen Sharing (VNC :5900) is bridged to noVNC via websockify when enabled.

All tool signatures and return shapes are identical across backends, so callers never need to know which platform they're on.

Auth path is Claude CLI OAuth via claude-agent-sdk — costs ride on a Claude Max subscription, no ANTHROPIC_API_KEY needed.

---

Install

As a Claude Code plugin

/plugin marketplace add Lifecycle-Innovations-Limited/desktop-act
/plugin install desktop-act@desktop-act

Via the claude-ops launcher

If you already have claude-ops, its bundled desktop-act-launcher.py auto-clones this repo into your per-user cache ($XDG_CACHE_HOME/desktop-act-mcp on Linux) the first time the /ops:desktop skill fires.

Manual

git clone https://github.com/Lifecycle-Innovations-Limited/desktop-act.git
# Point your MCP host at desktop-act/mcp-server/run.sh (stdio transport)

The first invocation auto-bootstraps a Python venv at ${CLAUDE_PLUGIN_ROOT}/.venv (or $DESKTOP_ACT_VENV if set) from requirements.txt.

System dependencies

Linux (X11Backend)

You need an X11 stack and a VNC server reachable as Xvnc (TigerVNC):

# Amazon Linux 2023 / RHEL / Fedora
sudo dnf install -y tigervnc-server websockify metacity xterm python3

# Ubuntu / Debian
sudo apt install -y tigervnc-standalone-server websockify metacity xterm python3-venv

Optional but recommended: openbox, xdotool, firefox, nautilus, libreoffice (whatever GUI apps you want to drive).

macOS (MacBackend)

Input is driven by cliclick; everything else (screencapture, osascript, open) is built into macOS.

brew install cliclick
# Optional — for a live noVNC view bridged off built-in Screen Sharing:
brew install websockify   # (or: pip install websockify)

Required macOS permissions (System Settings → Privacy & Security):

• Screen Recording — for screenshot/observe/act_step (the screencapture capture path). Without it, captures fail with could not create image from display.
• Accessibility — for click/type_text/keypress/scroll (the cliclick and System Events input path).

Grant both to the host process driving the MCP server (your terminal, Claude Code, etc.). For a live view, optionally enable Screen Sharing (System Settings → General → Sharing → Screen Sharing); acquire_desktop still returns a session without it and surfaces a vnc_hint.

---

Tools exposed by the MCP server

Tool	Purpose
acquire_desktop	Spawn a fresh Xvnc + websockify on the next free display
release_desktop	Kill that desktop's processes, free ports
list_desktops	Inspect the pool
screenshot	Grab the X11 desktop (JPEG default, SHA-deduped)
observe	Screenshot + window list in one round-trip
list_windows	Enumerate top-level X11 windows + geometry
launch_app	Spawn a GUI app on the display
click	Absolute-coord mouse click (buttons 1–5)
keypress	Press a key chord (e.g. Return, Control+L)
type_text	Type a string into the focused window
scroll	Scroll wheel up/down
batch	Execute many primitives in one MCP round-trip
act	Autonomous goal-driven loop via claude-agent-sdk
act_step	Single-step variant for transparent in-prompt control
status	Pool + binary inventory

A /desktop:act <goal> slash command is also registered.

---

Configuration

All knobs are environment variables — no config file required.