FastCUA

A local, accessibility-first Windows control plane for AI agents.

Website · 中文 · Current architecture · Technical paper · Next design

Warning

FastCUA is an experimental project under active development. Use it for testing, not important or unattended work.

FastCUA gives an agent a fast, inspectable interface to Windows applications. It prefers Windows UI Automation text, switches to screenshots and a numbered square grid when semantics are weak, and executes related native actions through one resident local runtime.

FastCUA offers three features:

1. Computer use — desktop automation (UIA text first, visual square grid fallback);
2. Record skill — record a demo → compile evidence → write a Skill → dry-run → approved install;
3. Computer use history — a local on-disk audit timeline of every action, app, outcome, and screenshot.

FastCUA runs headless as a resident background service. The only configuration surface is the local config.json file (default Full access); pause and approval are exposed through named-pipe control methods for host integrations.

FastCUA is agent-neutral, but a complete installation always has two parts in the same agent host:

1. the full skills/computer-use/ operating policy;
2. the sky-computer-use stdio MCP server.

Model requirement

Use one full-capability primary model with text/image understanding, reliable reasoning, Skills, MCP, and enough context for the whole task. Native audio understanding is useful for recorded narration; otherwise use typed notes. Do not configure writer, transcription, fallback, or text-only models.

Why FastCUA

	Vision-first computer use	Browser automation	FastCUA
Main observation	Screenshots	DOM/CDP	UIA text, then vision when needed
Scope	Any visible surface	Web content	Windows apps, browser chrome, cross-app flows
Execution	Often one action per loop	Browser commands	Several native actions per model turn
Runtime state	Often rebuilt per call	Browser session	One warm daemon and native host
Human takeover	Integration-dependent	Browser-limited	Global pause, approve, exit

FastCUA complements in-page browser automation; it does not replace it.

Architecture

flowchart TB
  A["Agent host + computer-use Skill"] -->|"stdio MCP"| B["server.mjs"]
  B -->|"path-scoped named pipe"| C["Resident daemon"]
  C --> D["Rust native host"]
  D --> E["UI Automation / HWND"]
  D --> F["Capture / square grid"]
  D --> G["Keyboard / mouse input"]
  C --> H["Approval / pause"]

Loading

All clients share one daemon, policy state, and physical pointer. A persistent js cell can execute related sky.* actions in one model turn; stale targets, changed focus/cursor, out-of-bounds points, timeouts, and human control signals stop execution.

Observation model

FastCUA optimizes how the agent observes Windows, not only how it acts.

Semantics first. Start with get_window_state({include_text:true}) and read state.uia. When UIA quality is good and the target is a named, bounded element, click its current element_index directly — no screenshot is taken at all. Many standard Windows interactions never enter the visual path.

Distrust UIA gracefully. UIA is graded good / weak / broken, and prefer_vision:true is surfaced when the tree is unreliable (Electron, canvas, custom-rendered UI, hung providers). On weak/broken/[no-hit] or one stale-index failure, stop semantic clicking and switch to vision immediately. A timed-out UIA provider is remembered so later requests fail fast instead of repeatedly blocking.

Coarse grounding first, precise grounding later. When vision is needed, the model never predicts one precise point on a full 4K screenshot. It works through numbered square cells:

1. grid_view({window}) returns one image with numbered square cells. Selecting a number is only a decision — it sends no input.
2. grid_refine({window,grid,cell}) captures only that cell and draws a new 3×3 grid; refine again if needed. Each refinement shrinks the search area by 9× (1/9 → 1/81 → 1/729).
3. Commit exactly once: click_cell (cell center), click_in_cell (cell-local offset), or click_view (exact point in the current view).

Refinement uses region-only native capture (capture_region_rgb via BitBlt), so later steps operate on progressively smaller captures instead of re-shooting the full window.

Custom regions. sky.grid({width, height, left, top, right, bottom}) builds a grid over any window sub-region — useful when the target area is known in advance (e.g. a toolbar). The grid itself is computed locally without a screenshot; refining into one of its cells triggers a region-only capture so the model sees that area zoomed in with a fresh 3×3 grid.

Grid annotation is non-blocking. Cell borders are semi-transparent (38% cyan) with no fill; numbers are small outlined digits at each cell center. If the annotation still covers the target, the model can bypass cell selection entirely and click any point in the view with click_view.

Local coordinates stay local. The model only ever sees the current view's pixels (origin top-left of that view). The runtime maps them back deterministically — x_window = cropLeft + x_view × scale — reversing capture scaling and rejecting out-of-view points rather than clamping them. DPI and multi-monitor geometry are normalized by the host (Per-Monitor-V2), never learned by the model.

Revalidate before side effects. Between grounding and input, the runtime re-checks that the foreground window, cursor position, and HWND identity still match the intent; if anything changed, the action aborts instead of clicking blindly.

Install

Use the PowerShell installer. It installs Node.js through WinGet when needed, downloads the GitHub Release runtime, and verifies its checksum:

irm https://raw.githubusercontent.com/Guojiz/FastCUA/main/install.ps1 | iex

The verified installer writes FastCUA Agent Setup.txt to the desktop. Give it to the agent that will actually use FastCUA. That agent must:

1. install the complete skills\computer-use folder into its own Skill system;
2. configure Node.js + the installed server.mjs as sky-computer-use MCP;
3. reload and verify that the Skill is discoverable;
4. call list_windows successfully.

If either the Skill or MCP is missing, installation is incomplete.

Agent self-setup (recommended)

scripts/agent-setup.ps1 registers the sky-computer-use MCP server and the computer-use Skill into detected agent hosts (Qoder, Claude Code, Claude Desktop, Codex CLI, VS Code, opencode, Kimi Work), backs up every config it touches, and runs a live stdio MCP smoke test:

& "$env:LOCALAPPDATA\FastCUA\app\scripts\agent-setup.ps1" -Action List
& "$env:LOCALAPPDATA\FastCUA\app\scripts\agent-setup.ps1" -Action Install
& "$env:LOCALAPPDATA\FastCUA\app\scripts\agent-setup.ps1" -Action Verify

See docs/AGENT_SETUP.md for per-agent config paths, the component lifecycle, and verification rules.

Verify and update

& "$env:LOCALAPPDATA\FastCUA\app\install.ps1" -Action Doctor
& "$env:LOCALAPPDATA\FastCUA\app\install.ps1" -Action Check
& "$env:LOCALAPPDATA\FastCUA\app\install.ps1" -Action Update

Inside MCP, call runtime_info to confirm the exact server, daemon, native host, version, commit, pipe, and data directory in use.

Configuration and control

FastCUA defaults to Full access: unknown apps run without prompting. All settings live in the local config.json file, which users edit directly; restart the daemon to apply.

To restore per-app approval, set approvalPolicy to "safe" in config.json. In safe mode, only whitelisted apps run without prompting; everything else triggers a resolve_approval decision (allow once / allow and whitelist / deny / switch to full access). The whitelist matches exact executable names and paths — never fuzzy substrings.

The control plane (pause / approve / exit) is exposed through daemon named-pipe methods (pause / resume / resolve_approval / shutdown, etc.) for host integrations. The DeepSeek Harness plugin handles configuration and history browsing. Approval uses exact application identity, not fuzzy name matching.

Visual click example

Given a window returned by list_windows:

let view = await sky.grid_view({ window });       // inspect; choose cell 4
view = await sky.grid_refine({
  window,
  grid: view.grid,
  cell: "4",
});                                               // inspect; choose cell 5
await sky.click_cell({ window, grid: view.grid, cell: "5" });
await sky.close();

Record a Skill (preview)

The optional recorder turns a demonstration into an auditable evidence package before any Skill is written:

record → compile evidence → current primary agent writes → provenance lint
       → dry-run with new values → human-reviewed promotion

Password fields and secure-desktop moments are redacted. The current primary agent writes the Skill from evidence; lint, dry-run, application scope, and explicit promotion approval remain hard gates. See skills/skill-recorder/ and the technical paper.

Note

Using the Skill Recorder may send recorded screen content, interaction evidence, and narration to the configured cloud model provider.

Computer Use History

Every desktop action and control event is appended as JSONL to history/history.jsonl in the local data directory. Entries are metadata-only by default — they record the app, action, summary, outcome, and duration; type_text / set_value store only length, never verbatim content. Screenshots are opt-in: set historyCaptureScreenshots: true in config.json to persist captures under history/shots/, and pass include_screenshots: true to list_history / get_history to embed them. Retention is governed by historyEnabled, historyCaptureScreenshots, historyMaxEntries, historyRetentionDays, and historyMaxShotsPerAction in config.json.

Agents read history with the list_history / get_history MCP tools; the DeepSeek Harness plugin provides visual browsing. All history stays on the local machine.

Develop from source

git clone https://github.com/Guojiz/FastCUA.git
cd FastCUA
.\native-host\build.ps1

Then copy the complete skills\computer-use directory into the active agent's Skill directory and configure the absolute path to server.mjs as a stdio MCP server. Use runtime_info to verify the checkout. Reproduction commands and the test matrix are in the technical paper.

Boundaries

FastCUA currently targets Windows 11 x64. UAC, Secure Desktop, authentication dialogs, password managers, Windows Security, higher-integrity processes, protected surfaces, and applications with unusual capture/accessibility behavior are outside the normal path. Synthetic input is not hardware input, and the current key-chord implementation still uses the superseded keybd_event API. Remaining input, provider, capture, IPC, and evaluation work is tracked in Next design.

Uninstall

& "$env:LOCALAPPDATA\FastCUA\app\uninstall.ps1"

License

MIT. See LICENSE.