gridhand

gridhand

_{Formerly published as gui-tool / agent-desktop-interface.}

Let an AI agent click any desktop app by naming a grid cell — not guessing pixels.

A cross-platform Rust CLI for GUI automation: screenshots, window management, mouse and keyboard control, strict JSON in / JSON out. Zero dependencies — no crates, one small binary, direct OS APIs.

_{Real gridhand output — orient to a labeled grid → zoom into a cell → click the crosshair → verify.}

Built for AI desktop agents (Claude Code, Codex, Gemini CLI, and friends), but it works fine as a general-purpose GUI automation tool.

Why it's different: agents are bad at guessing pixel coordinates from a screenshot, and every other automation tool makes them try. gridhand removes pixels entirely — it overlays a labeled grid with a crosshair at each cell's center, the agent names a cell, and the click lands exactly on that crosshair. Need more precision? Zoom into the cell for a sub-grid and name a sub-cell (C7.F3). It's the whole workflow in one line: orient → zoom → click → verify.

It sees pixels, not an accessibility tree. Most desktop-automation tools for agents find elements by reading the OS accessibility tree — which only helps when that tree exists, is complete, and is correct. gridhand never touches it. It works from the actual rendered screen, so it drives what tree-based tools can't see: games, <canvas> / WebGL apps, Flutter and other custom-drawn UIs, remote desktops and VNC, video, and any app with missing, mislabeled, or wrong accessibility data. If a human can see it, gridhand can click it.

Everything is hand-rolled against raw OS APIs — its own PNG encoder, D-Bus client, JSON output, and DEFLATE — so there are no crates to audit, no build surprises, and it runs natively on GNOME/Wayland where xdotool and pyautogui give up.

_{End-to-end in a real browser — navigate, click a field by cell, type, and submit, no pixel coordinates anywhere.}

What it's for

Concrete things it's been used for:

• Automating apps with no CLI or API — drive GUI-only software (legacy tools, proprietary apps, installers, settings panels) the same way a person would, so an agent can script a workflow that otherwise has no headless path.
• Debugging GUI applications — let an agent open the app, reproduce a bug by clicking through it, screenshot each state, and read what's actually on screen — hands and eyes for the parts a log file doesn't capture.
• Reaching what the accessibility tree can't — pixel-precise grid zoom clicks targets in apps whose accessibility data is missing, incomplete, or wrong: games, <canvas>/WebGL, Flutter and other custom-drawn UIs, remote desktops, and video.

Features

• Grid targeting: Overlay a labeled grid on screenshots with red crosshairs at each cell center. Click by cell label — no pixel coordinates. Supports recursive zoom (B2.C1) and between-cell targeting (D3+E3).
• No accessibility tree required: Targets purely from what's on screen — never AT-SPI, UI Automation, or the AX APIs. Works where the accessibility tree is missing, incomplete, or wrong: games, <canvas>/WebGL, custom-drawn UIs (Flutter, etc.), and remote desktops.
• Contextual zoom: Zoomed views show the target cell with a coarser sub-grid, surrounded by dimmed context from adjacent cells with parent-level labels for spatial orientation.
• No dependencies: Pure std Rust, direct FFI to OS APIs (CoreGraphics, user32.dll, D-Bus). Compiles to a single small binary.
• Wayland support: Works natively on GNOME/Wayland via XDG Desktop Portals and the window-calls extension, where tools like xdotool and pyautogui break.
• JSON output: Every command returns structured JSON, so agents don't have to parse text output.

Grid Targeting