screen-mcp

screen-mcp is an MCP server for Claude Code that lets a model see and operate your GNOME/Wayland desktop. It captures any monitor through PipeWire, drives the pointer and keyboard through the xdg-desktop-portal RemoteDesktop portal, and optionally reads the screen with OCR and grounds icons with an OmniParser ONNX model. It is for developers who want an agent that can use real desktop apps, not just a browser. It is pure Python and runs CPU-only.

Quickstart

Install the plugin in Claude Code:

/plugin marketplace add 88plug/screen-mcp
/plugin install screen-mcp@screen-mcp

Then run the one-time dependency setup (the server has system and Python deps the manifest cannot install for you):

# in the installed plugin dir (or a clone)
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt

On first use the desktop portal pops a consent dialog asking which monitor(s) to share. Pick one, and ask the model to take a screenshot:

Take a screenshot of my desktop and tell me which window is focused.

You should get back a labeled capture plus the focused-window name within a few seconds. The portal returns a restore token cached at ~/.config/mcp-screen so later runs are silent.

[!IMPORTANT] screen-mcp runs on Linux + Wayland + GNOME only, and grounding is CPU-only by design. See Requirements before installing.

What it does

The server turns one capture-and-act loop into MCP tools a model can call directly:

• Screenshot any monitor or region, with numbered Set-of-Marks overlays and click coordinates.
• Click, type, scroll, and drag in any visible app, including native Wayland apps that xdotool and XTEST cannot reach.
• Read on-screen text with OCR (RapidOCR) and ground icons with an OmniParser ONNX model (both optional).
• Sense changes: an ambient layer diffs frames so the agent knows when something opened or when an action was a no-op.
• Cache learned screens: a write-through world model lets a recognized screen skip OCR on the next visit.
• Gate destructive actions: an opt-in ack guard blocks close-combos and destructive-keyword clicks until the caller passes a confirmation token.

It also ships a drive-screen skill that encodes the locate → ground → act → confirm loop, so the model knows how to use the tools well out of the box.

Principles — The Agent Oath

screen-mcp is a reference enforcer of The Agent Oath: the user-takeover guard yields control the instant a human moves the mouse (a STOPPED result), keeping the human in charge of their own desktop. That's §2 (human agency) and §11 (human oversight) made executable — don't fight the human for the mouse. The opt-in ack gate (§7, don't bypass safety) and the on-screen visibility of every action (§5, transparency) round it out.

MCP tools

These are the tools the server exposes. Every action also accepts space (view / desktop / norm, default view — coords as seen in the last screenshot), shot: true to return a screenshot after, verify: true to warn on no-change misclicks, force: true to bypass the user-takeover guard, and element: <id> to click an element id from the last annotated shot.