vnc-remote-control

vnc-remote-control

A small command-line tool to drive a VNC/RFB server: type text, press named keys, click, and grab a screenshot. It speaks plain RFB (None or VNC-password security, raw encoding) and sends literal keysyms like a standard VNC client, so it works with any VNC/RFB server; the guest keyboard layout is a server-side setting, not something this client compensates for. A layout-aware server such as openvmm is the ideal target (see the keyboard section), but it is not required.

The CLI is styled with rich-click; the RFB protocol and pixel drawing are pure standard library, and screenshots are written as PNG with Pillow. The tool also needs one external program: tesseract, used for OCR. See the Install section.

When this is useful

The connection is to a VNC/RFB server, so nothing runs on the target machine. That makes the tool a good fit when:

• A machine has no network of its own. A VM that isn't on the network yet (or never will be) can still be driven through its hypervisor's VNC console, the same screen you'd click in the web UI. This is handy for the first-boot setup where you configure networking before the guest can reach anything.
• You want Claude to operate a machine with no footprint on it. Nothing is installed on the guest: no agent, no service, no open port on the target, nothing visible in its process list. The control happens entirely over the host's VNC.
• A box is only reachable over VNC. For remote administration where VNC is the one door you have, this drives it the same way a person at the console would.
• Legacy desktop software has only a GUI. Old line-of-business apps, dated installers, and vendor tools often have no API, CLI, or accessibility tree to automate against, only a window. Reading the screen with OCR and clicking by label drives them when nothing else can, and legacy UIs rarely change layout, so the coordinates stay stable.

What it does

• type a string into the focused guest field (literal keysyms).
• key presses a single named key (enter, tab, esc, arrows, function keys).
• click sends a left-button click at a pixel position.
• screenshot writes the native-resolution framebuffer to a PNG file, with optional crosshair and grid overlays.
• ocr lists the words on screen with their click centers and confidence.
• click-text clicks the first on-screen word matching a pattern.

Install

Where to install it

vnc-remote-control is a pure client: install it only on the machine you drive it from (your control or development box), never on the targets. It connects out to a target's VNC/RFB port over the network, directly or through an SSH tunnel to a console port. Nothing runs on the machine you control: no agent, no service, no open port on the target, nothing in its process list. So one machine has the tool, and any number of targets are driven over VNC while staying untouched.

Prerequisite: tesseract

tesseract is a required system dependency. OCR is core to the tool (it is how an LLM finds what to click via ocr and click-text), so install it first:

# Debian/Ubuntu
sudo apt-get install tesseract-ocr
# macOS
brew install tesseract
# Windows
choco install tesseract

The package

With pip:

pip install vnc-remote-control

With uv:

uv tool install vnc-remote-control

From a checkout:

pip install -e .

For alternative install paths (pip, pipx, uv, uvx, source builds), see INSTALL.md. Every supported method registers the vnc-remote-control command on your PATH.

Usage