HandsOn
Give Claude eyes and hands. A Claude Code plugin that lets Claude see your screen and interact with any application on your desktop.
Alpha software. HandsOn works and is genuinely useful, but Claude will sometimes stumble through tasks — misclicking sidebar links, scrolling the wrong container, needing multiple attempts to find the right element. Complex multi-step workflows (like filling out a Reddit post) may take several retries to get right. It's getting better with each release, but set your expectations accordingly.
What's New in v0.3.0
- Window-cropped visual detection — When a target window is set, visual analysis is cropped to that window's bounds, eliminating noise from the terminal and other background apps
- 30-50% token reduction — Compact output across all tools: shorter class names, removed redundant formatting, tighter coordinate notation
- Unnamed element filtering —
list_elements now skips unnamed Pane/Group/Custom elements that add noise without value (common in Steam, Electron apps, etc.)
- Process-tree terminal refocus —
set_target_window("") walks the process tree to find and refocus the host terminal instead of guessing by title
- Immediate window focus —
set_target_window() now focuses the target window immediately on set, not just before the next input action
- Visual detect module — New CV-based UI region detection for apps with limited accessibility support
- Self-correcting click —
click_text automatically retries at offset positions when the initial center click produces no visual change
Why HandsOn?
Claude Code is powerful, but it's blind — it can't see your screen or interact with anything outside the terminal. HandsOn changes that. It gives Claude eyes to see what's on your desktop and hands to interact with it, turning it into a general-purpose desktop assistant.
The core idea: Claude can now do anything you can do at a keyboard and mouse — navigate apps, fill out forms, click through wizards, manage windows, launch programs, and verify its own work visually.
What this enables:
- Desktop automation — Automate any application on your computer, even legacy apps with no API
- Visual verification — Claude can see the result of its own work, catch visual bugs, and fix them without you describing what's wrong
- App interaction — Navigate settings, install software, fill out forms, click through dialogs — anything you'd normally alt-tab to do yourself
- Accessibility-first targeting — Uses the platform accessibility tree — Windows UI Automation (UIA) or macOS AXUIElement — not just pixel coordinates, so element targeting is DPI-aware and resolution-independent
- Automatic fallbacks — When accessibility doesn't work (custom widgets, canvas apps, games), it falls back to OCR, then to visual analysis.
smart_find picks the right approach automatically.
- Framework-aware — Detects Qt, WPF, Electron, WinForms, etc., and tells Claude exactly what will and won't work
Quick Start
Install
# From GitHub (recommended)
claude plugin marketplace add 3spky5u-oss/HandsOn
claude plugin install handson
# From GitLab
claude plugin marketplace add git@gitlab.com:3spky5u/HandsOn.git
claude plugin install handson
# From Codeberg
claude plugin marketplace add git@codeberg.org:3spky5u/HandsOn.git
claude plugin install handson
Restart Claude Code after installing. Python dependencies are installed automatically on first run.
To update:
claude plugin marketplace update handson
claude plugin update handson
Requirements: Python 3.10+, Windows 10/11 or macOS 12+ (Monterey or later).