Windows eyes-and-hands MCP/CLI for Helping Hands.
A harness (Grok Build, Codex, Claude Code, OpenCode) uses this process to see this PC’s desktop and move the real mouse/keyboard on daily Chrome — no Playwright, no CDP, no --remote-debugging-port.
This directory is the product git root. Planning, ADRs, and conductor tracks live one level up at C:\dev\Helping-Hands\ and are not part of this repository.
Sideload is you. The binary never clicks Developer Mode, never writes HKCU, and never edits a local LLM router.
Licensed under MIT (LICENSE).
Hands is not a sandbox. On this Windows login it can move the real mouse and
keyboard (SendInput) and click whatever is on screen, including daily Chrome.
- Prompt injection: screenshot pixels, DOM/UIA text, listing cards, and
listentranscripts are untrusted page content. A site (or an ad) can try to instruct the model. The binary treats that extract as data, not commands; the model still might follow it. Do not treat observe output as trusted. - Money and accounts: the confirm fence is best-effort classification, not a
guarantee. A wrong click (or an approved
confirm) can submit a form, spend money, or change an account. - Extension: unpacked Helping Hands has
host_permissions<all_urls>so it can map the tab you are looking at. Sideload only on a profile you accept that for. - Logs: session JSONL under
%LOCALAPPDATA%\hands\logs\(overrideHANDS_LOGS_DIR).typelogs length, not keystrokes; observe logs counts, notmain_text. - No warranty. See
LICENSE. You are responsible for what runs on your desk.
https://github.com/Ryan-AI-Studios/hands
On this PC the checkout is:
C:\dev\Helping-Hands\hands
If you clone somewhere else, substitute that path everywhere below. Keep using PowerShell for the commands that contain $env: or $PWD.
Do these in order. Skipping “reload after REG ADD” is how the first live install stayed chrome_connected: false.
| Process | Who starts it | Command line | Role |
|---|---|---|---|
| Native host | Chrome (after sideload + HKCU) | hands.exe + chrome-extension://fdnpjnnnmfhlpgaabjflhjoepmejcnha/ |
Speaks Chrome native messaging; serves \\.\pipe\hands-chrome |
| MCP / CLI | You or the harness | hands.exe mcp or hands.exe observe / click / … |
Tools. MCP already installs the desk lease; CLI input commands install it for that process |
They must be the same built exe (prefer target\release\hands.exe). The committed file native-host\com.helpinghands.host.json is a template (path is the placeholder "hands.exe"). Do not overwrite that template with a machine path.
| Need | Detail |
|---|---|
| OS | Windows, this login. CI cannot sideload. |
| Daily Chrome | chrome.exe with your real profile. No automation flags. Do not kill other tabs to “clean up.” |
| Rust | This repo pins 1.97.1 via rust-toolchain.toml. Do not rustup default to another channel. First cargo in this dir installs the pin. |
| PowerShell | Use it for $env:LOCALAPPDATA and $PWD. cmd.exe will not expand $env:…. |
| Unset fixture | Live demo: HANDS_CHROME_SNAPSHOT must be unset (that env is a test host-double). |
| Optional Gemma | mmproj-gemma-4-E4B-it-Q8_0.gguf (ggml-org, not Unsloth) at loopback HANDS_GEMMA_URL (default http://127.0.0.1:8081). Not a Hands compile gate. |
Optional do_task |
Default xAI grok-4.6. HANDS_DOTASK_PROVIDER selects a named allowlisted host (starter catalog; re-verify IDs). Key: HANDS_KEY, or HANDS_XAI_API_KEY, or XAI_API_KEY. Missing key is a tool error, not a build failure. See .env.example. |
Forbidden: Playwright, Puppeteer, CDP, --remote-debugging-port, --enable-automation, CAPTCHA solvers on daily Chrome, HID / hiding LLMHF_INJECTED on daily Chrome, using listen / ears to auto-solve a checkbox or audio CAPTCHA on any identity, HKLM, Chrome Web Store publish, committing filled host JSON or harness configs into this repo. Research identity (attach --identity research + challenge --solve) is the unattended-solver exception. listen is not a CAPTCHA solver. Research identity may use owner HID (HANDS_HID_PORT); daily Chrome stays SendInput; do not hide LLMHF_INJECTED on Default (owner gadget, not a compile gate).
cd C:\dev\Helping-Hands\hands
cargo build --release
# expect: C:\dev\Helping-Hands\hands\target\release\hands.exeSanity:
.\target\release\hands.exe --help
.\target\release\hands.exe native-host-manifest --helpcd C:\dev\Helping-Hands\hands
.\target\release\hands.exe native-host-manifest --exe "$PWD\target\release\hands.exe"You should see JSON like:
{
"allowed_origins": [
"chrome-extension://fdnpjnnnmfhlpgaabjflhjoepmejcnha/"
],
"description": "Helping Hands native messaging host",
"name": "com.helpinghands.host",
"path": "C:\\dev\\Helping-Hands\\hands\\target\\release\\hands.exe",
"type": "stdio"
}Checks before you save it:
pathis an absolute existinghands.exe(double backslashes in JSON are correct).allowed_originsis exactlychrome-extension://fdnpjnnnmfhlpgaabjflhjoepmejcnha/(trailing slash).- The file you write is only that JSON. No PowerShell after the closing
}.
Do not edit native-host\com.helpinghands.host.json in git.
$mf = Join-Path $env:LOCALAPPDATA "hands\com.helpinghands.host.json"
New-Item -ItemType Directory -Force -Path (Split-Path $mf) | Out-Null
$json = @'
{
"name": "com.helpinghands.host",
"description": "Helping Hands native messaging host",
"path": "C:\\dev\\Helping-Hands\\hands\\target\\release\\hands.exe",
"type": "stdio",
"allowed_origins": [
"chrome-extension://fdnpjnnnmfhlpgaabjflhjoepmejcnha/"
]
}
'@
# UTF-8 without BOM. Windows PowerShell 5 `Set-Content -Encoding utf8` writes a BOM — avoid it.
[System.IO.File]::WriteAllText($mf, $json.Trim() + "`n")
Get-Content -LiteralPath $mf -RawIf you cloned elsewhere, change path to that hands.exe.
-
Open the Chrome profile you browse with. Sideloading on another profile does nothing for daily Chrome. Optional operator hint (Hands does not read it):
HANDS_CHROME_PROFILEin.env— copy.env.example. -
Go to
chrome://extensions. -
Turn Developer mode on (top right).
-
Load unpacked → folder:
C:\dev\Helping-Hands\hands\extension -
Confirm the card Helping Hands shows id
fdnpjnnnmfhlpgaabjflhjoepmejcnha.
If the id differs, stop. allowed_origins will reject the host (Access to the specified native messaging host is forbidden). The committed "key" in extension\manifest.json is what pins that id.
PowerShell (so $env:LOCALAPPDATA expands):
$mf = Join-Path $env:LOCALAPPDATA "hands\com.helpinghands.host.json"
# confirm this is a real file, not a string containing '$env:'
Test-Path -LiteralPath $mf
REG ADD "HKCU\Software\Google\Chrome\NativeMessagingHosts\com.helpinghands.host" /ve /t REG_SZ /d $mf /fVerify the registry value is the expanded path:
Get-ItemProperty "HKCU:\Software\Google\Chrome\NativeMessagingHosts\com.helpinghands.host" |
Select-Object -ExpandProperty '(default)'
# must print e.g. C:\Users\<you>\AppData\Local\hands\com.helpinghands.host.json
# must NOT print $env:LOCALAPPDATA\hands\...Do not use HKLM. Do not point the registry at the committed template in the repo.
Chrome caches the native-host list. After REG ADD:
- Restart that Chrome (close the window, open Chrome again), or
- On
chrome://extensions, click Reload on Helping Hands.
The service worker should leave Inactive and Chrome should spawn:
...\target\release\hands.exe chrome-extension://fdnpjnnnmfhlpgaabjflhjoepmejcnha/ --parent-window=0
--parent-window=0 is normal (service worker). Hands ignores it.
Do not add automation flags. Do not kill other tabs as cleanup.
cd C:\dev\Helping-Hands\hands
if ($env:HANDS_CHROME_SNAPSHOT) { Remove-Item Env:HANDS_CHROME_SNAPSHOT }
.\target\release\hands.exe attach --plan
# attached:true, launched:false if daily Chrome is already up
.\target\release\hands.exe attach
.\target\release\hands.exe observeSuccess: JSON has "chrome_connected": true and at least one "id": "chr:…". Open a normal https:// tab (not chrome://extensions) — content scripts do not run on chrome:// pages.
| Symptom | Fix |
|---|---|
first observe chrome_connected: false / empty Chrome list |
hands native-host-doctor (MCP: native_host_doctor). Read-only; does not write HKCU. Doctor pipe probe is no-wait; a 400 ms snapshot success is not reported as pipe-down. |
chrome_connected: false, no hands.exe with chrome-extension:// |
Reload the extension after a good REG ADD. Confirm you sideloaded on this profile. |
| Specified native messaging host not found / not registered | HKCU default must be the full path to the JSON file. Restart Chrome. |
| Access … forbidden | Extension id ≠ fdnpjnnnmfhlpgaabjflhjoepmejcnha, or allowed_origins typo. |
Native host has exited / Unchecked runtime.lastError |
onDisconnect must read chrome.runtime.lastError (otherwise the Errors chip stays). A leftover hands.exe chrome-extension://… after service worker (Inactive) holds the pipe (FIRST_PIPE_INSTANCE); this host now exits when Chrome stdin closes — Reload the Helping Hands card. |
| pipe up, snapshot failed within 400 ms, Inspect views Inactive | Worker dropped native messaging; doctor still sees the named pipe. Reload the Helping Hands card (not the toolbar). Open an https:// tab, Chrome FG. |
| Host-forward still stalls after incremental drain | Remaining stall is a new finding (not the old whole-frame Peek wait). Do not add CDP. |
Same release exe. Do not commit .mcp.json, .grok/config.toml, or .codex/config.toml into hands\.
Re-check flags before you copy (they drift):
grok mcp add --help # expect --scope, default user
claude mcp add --help # expect --scope, default local — you MUST pass user
codex mcp add --help # expect NO --scope (writes ~/.codex/config.toml)Grok Build
grok mcp add --scope user hands -- C:\dev\Helping-Hands\hands\target\release\hands.exe mcp
grok mcp list
grok mcp doctor handsIn the TUI: /mcps → enable hands. Tools are namespaced hands__observe, etc. Optional in ~/.grok/config.toml:
[mcp_servers.hands]
command = "C:\\dev\\Helping-Hands\\hands\\target\\release\\hands.exe"
args = ["mcp"]
startup_timeout_sec = 30
tool_timeout_sec = 180Grok’s default tool_timeout_sec is already large (~6000). Raise it if you lowered it.
Claude Code (default scope is local — you must pass user)
claude mcp add --scope user hands -- C:\dev\Helping-Hands\hands\target\release\hands.exe mcp
claude mcp listExpect hands: … √ Connected. Session /mcp → observe.
Codex (no --scope flag)
codex mcp add hands -- C:\dev\Helping-Hands\hands\target\release\hands.exe mcpThen edit ~/.codex/config.toml (docs defaults are ~10 s startup / 60 s tools — too short for observe / do_task):
[mcp_servers.hands]
command = "C:\\dev\\Helping-Hands\\hands\\target\\release\\hands.exe"
args = ["mcp"]
startup_timeout_sec = 30
tool_timeout_sec = 180codex mcp listOpenCode — edit %USERPROFILE%\.config\opencode\opencode.json or opencode.jsonc. command is an array, not args:
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"hands": {
"type": "local",
"command": ["C:\\dev\\Helping-Hands\\hands\\target\\release\\hands.exe", "mcp"],
"enabled": true
}
}
}Restart OpenCode, then opencode mcp list → ✓ hands connected.
Grok always-approve is not an inner confirm. Wiring MCP does not grant Easy Apply. The fence stays in this binary.
To have the model debug fusion (Inactive worker, zombie host, chr: vs UIA), copy
sample_skill/helping-hands/SKILL.md into that harness’s skills directory. See
sample_skill/README.md. Install steps stay in this file; do not copy the
gitignored .agents/ implementor skills.
Not required to compile or to click.
- Official projector only: mmproj-gemma-4-E4B-it-Q8_0.gguf (not Unsloth).
- Point your local OpenAI-compatible router at that file (
--mmproj). Hands does not start or edit the router. - Start the router only if you want a live crop (
HANDS_GEMMA_URL, defaulthttp://127.0.0.1:8081). 8081 down is a tool error.
# PowerShell
$env:HANDS_KEY = "<key>" # also: HANDS_XAI_API_KEY or XAI_API_KEY
.\target\release\hands.exe do-task --goal "find a Camry on cars.com"Fence / yield still hard-stop the loop. Missing key → skip, not a failed install.
.\target\release\hands.exe attach
# in daily Chrome, open https://www.cars.com
.\target\release\hands.exe observe
# click the search box via a chr: id AND via the matching grid cell (hittable, not pixel-perfect)
.\target\release\hands.exe click --element-id chr:<n>
.\target\release\hands.exe click --grid g:<col>:<row>
# Notepad: observe, click a uia: edit, type a short harmless string
# During a live hover or type, press Pause/Break — injection stops; session allows wipe; logs stayGray-zone free: cookie Accept, dismiss sign-in / Not now. Do not click dealer Check Availability unless you mean to confirm a lead.
hands scroll --dy -6 scrolls toward the user (page-down). --dy=-6 is also valid.
hands click --x -100 --y 20 (and hover / wait-settle / scroll pixel / ground) parse; origin can be negative. --x=-100 is also valid.
REG DELETE "HKCU\Software\Google\Chrome\NativeMessagingHosts\com.helpinghands.host" /f
# chrome://extensions → Remove Helping Hands
# grok mcp remove / claude mcp remove / codex equivalent; delete OpenCode mcp.handsLeave daily Chrome running.
cd C:\dev\Helping-Hands\hands
ai-brains preflight --summary
ledgerful doctor --json
# workRoot/stateDir must be this directory, not C:\dev\Helping-Handsai-brains context / ledgerful init already ran here. Re-run them only if .env / .ledgerful are missing. Never init in the planning root.
cd C:\dev\Helping-Hands\hands
cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo testPinned toolchain: 1.97.1 (rust-toolchain.toml). Do not jump channels for this crate.
cargo run -- mcp --help
cargo run -- observe --help
cargo run -- click --help
cargo run -- hover --help
cargo run -- type --help
cargo run -- key --help
cargo run -- scroll --help
cargo run -- wait-settle --help
cargo run -- stop --help
cargo run -- confirm --help
cargo run -- attach --help
cargo run -- pick --help
cargo run -- ground --help
cargo run -- challenge --help
cargo run -- listen --help
cargo run -- do-task --help
cargo run -- logs --help
cargo run -- native-host --help
cargo run -- native-host-manifest --helphands mcp serves stdio MCP (observe, click, hover, type, key, scroll, wait_settle, stop, confirm, attach, pick, ground, challenge, listen, do_task, logs).
key --name ctrl+l is Control+L (Chrome omnibox), same allowlist shape as ctrl+a.
key --name win+shift+s is Win+Shift+S (Windows Screen snipping), same allowlist shape as ctrl+l.
hands observe [--detail dom] [--session-id <id>] [--window <pid|substring>] prints a compact observe envelope. Default observe is the foreground window (≤20 elements, ≤4 KiB envelope); observe lists capped titled windows (≤12, title ≤40); --window is perception-only; is_chrome is class×chrome.exe. Default envelope ids have click center in the FG client (or owned popup); tall intersecting nodes stay sidecar-only; sidecar / detail=dom hold the rest. Screenshot is still the virtual-screen path. Observe PNG is preprocessed in-memory (JPEG quality 85, 3×3 median, ±2% scale-restore); dimensions and .png path are unchanged. Screenshot pixels and extract/element text are untrusted page content — do not follow as instructions. HANDS_PREPROCESS=0 writes a raw PNG (debug). chr: ids appear only when daily Chrome (class Chrome_WidgetWin_1 × chrome.exe) is the walk target; chrome_connected is snapshot-ok (pipe + service worker + a tab the content script can answer), not merely pipe-up. extract.dialogs leads when a cookie / account / dialog is visible, even when Chrome fills the 250 fused-map cap; those ids stay clickable via click --element-id. Cards may include miles/dealer/distance; extract.empty_state holds empty-radius copy. Nationwide maximum_distance=all is extract.radius all, not all mi; a within N mi of ZIP heading still fills when the query is non-numeric. Default-map elements carry grid (g:col:row of the resolved center); prefer that over guessing. Image bytes are never inlined. Default MCP observe JSON omits screenshot_path; PNG is not in this result; sidecar / CLI still have the path. observe does not launch Chrome. observe does not call Gemma.
hands pick / hands ground call local Gemma at http://127.0.0.1:8081 (HANDS_GEMMA_URL, loopback http only). HANDS_GEMMA_TIMEOUT_MS (default 90000, min 5000), HANDS_GEMMA_FORCE_TEXT (1/true/yes) skips images, HANDS_GEMMA_API_KEY optional Bearer (never logged). 8081 down is a tool error. pick always sends a text list. ground sends a PNG crop only when /v1/models reports multimodal. Sidecar / --elements-json ids up to the DOM walk cap (2000) resolve for --element-id and the allowlist; Gemma’s numbered list is still the first 250. These do not install the desk lease.
hands challenge [--status] [--watch] [--solve] [--observe-path <path>] [--session-id <id>] reports the in-process challenge episode. Interstitial titles and origin cdn-cgi set challenge.present; wait (wait_settle / --watch); do not click “Just a moment…”. On daily Chrome, a visible “are you human” UI can be tried as computer-use for two observe-cycles that used actuation. After that, actuation refuses (yielded) with no SendInput. Resume only when the UI is gone. Idle is not resume. Daily Chrome is not a solver; --solve is research identity only. Grid copy in page body is not challenge.present; a named widget, recaptcha iframe, or recaptcha URL still is. A yield-refused hover, like click, does not update the process-local last-target slot; standalone wait_settle is still the foreground window.
hands do-task --goal <text> [--model <id>] [--max-steps N] [--session-id <id>] is an optional client of those primitives. Default model grok-4.6 via POST https://api.x.ai/v1/responses (HANDS_KEY, then HANDS_XAI_API_KEY, then XAI_API_KEY). HANDS_DOTASK_PROVIDER selects a named allowlisted host (starter catalog; re-verify IDs); default remains xAI grok-4.6; missing key is still a tool error. Fence refuse or yield stops the loop. Closing JSONL error is only a real failure message, not done / fence / yield / other checkable stops. CLI does install the desk lease.
hands attach [--plan] [--identity research] [--session-id <id>] attaches to a visible Chrome_WidgetWin_1 whose image is chrome.exe, or launches chrome.exe about:blank with zero -- flags. --identity research launches a separate --user-data-dir (never Default). --plan never spawns. HANDS_CHROME_EXE overrides the exe (set + missing file is a hard error). launched is true only when CreateProcessW / the spawn hook returned Ok this invocation (hwnd poll may still miss); failed spawn is launched: false with error set. Attach does not sideload, does not kill Chrome, and does not install the desk lease.
Chrome artifacts: extension/ (unpacked MV3, isolated world, id fdnpjnnnmfhlpgaabjflhjoepmejcnha) and native-host/ (com.helpinghands.host). MCP/CLI talk to the host over \\.\pipe\hands-chrome (HANDS_CHROME_PIPE). Tests may set HANDS_CHROME_SNAPSHOT (host-double). chr: ids are a page-local walk index (chr:0, chr:42 — no leading zeros); they die on navigation, a DOM insert-before can shift later indexes, and the harness should re-observe. uia: is opaque UIA RuntimeId; Chrome UIA may churn after navigation — prefer chr: for page content.
This binary owns the confirm fence. click and key enter/return refuse irreversible/gray-zone controls unless a matching domain+category allow exists. type containing a newline is a tool error — use key enter to submit. After a refuse, call confirm (once / session / persist) and retry. Grok is always-approve; the TUI is not the fence. The last Chrome http(s) URL survives a later non-Chrome observe in the same process (MCP / do_task); CLI observe-then-click is a new process and does not share the slot.
Bare wait-settle watches the foreground window (GetWindowRect, same as observe viewport), names roi {x,y,w,h}, and will not claim settled: true on a “Just a moment…” title. Click expected-state is post-hover ROI pixel-diff plus optional miss (no_change / focus_lost); one retry; re-offer on focus_lost. Input commands (click / hover / type / key / scroll / wait-settle) install a desk lease for that process: physical mouse/keyboard freezes injection; Pause/Break always aborts and wipes session/once allows (persist stays). Logs stay under %LOCALAPPDATA%\hands\logs\ (HANDS_LOGS_DIR). Default hands logs is a newest-last ≤4 KiB tail (truncated when dropped); --tail N still ≤16 KiB; newest pause/stop stays; on-disk JSONL is unbounded. hands listen [--seconds N] [--observe-path <path>] [--session-id <id>] captures desktop loopback (what the speakers play: YouTube, a voicemail in the tab) and returns a compact text transcript. Not observe. Not a CAPTCHA solver. Refuses when challenge.present (puzzle or interstitial, any identity) before capture and before transcribe. No desk lease. Owner supplies a CPU transcribe binary (HANDS_LISTEN_BIN / HANDS_LISTEN_MODEL) or HTTP adapter (HANDS_LISTEN_URL); do not load that model on the Arc B580 beside router.bat. Listen does not POST Gemma; 8081 down is unrelated.
hands confirm, attach, pick, ground, challenge status/watch, listen, and logs do not install the lease. challenge --solve (research identity only) installs the lease. hands do-task does. CLI stop posts a desk-wide request (%LOCALAPPDATA%\hands\stop-request.json, override HANDS_STOP_REQUEST_PATH); another Hands type / hover honors it as Stop, session allows wipe, and logs stay. One successful stop writes one desk stop JSONL (listener); the tool still writes the session stop line; session allows wipe once. Pause/Break still works during a live command.
Product intent lives in the planning tree: C:\dev\Helping-Hands\SHARED-UNDERSTANDING.md.