FlaUI-MCP

🟢 Actively maintained hard fork of shanselman/FlaUI-MCP. Adds Claude Code plugin install, win-arm64 builds, auto-wait/retry, HWND-based window detection (UWP + localized titles), GDI leak fixes, and MCP spec compliance.

An MCP (Model Context Protocol) server that enables AI agents to automate Windows desktop applications using accessibility APIs - the same way Playwright automates browsers.

Why This Exists

When Playwright's MCP server automates browsers, it provides:

browser_snapshot → Structured accessibility tree with element refs
browser_click ref="..." → Click by ref, not coordinates

FlaUI-MCP brings the same pattern to Windows desktop apps:

windows_snapshot → Accessibility tree with refs like w1e5
windows_click ref="w1e5" → Click element by ref

No screenshot parsing. No coordinate guessing. Just semantic element references.

Quick Demo

Agent: Calculate 3 × 3

1. windows_launch { "app": "calc.exe" }
   → Window handle: w1

2. windows_snapshot { "handle": "w1" }
   → - window "Calculator" [ref=w1]
       - button "Three" [ref=w1e43]
       - button "Multiply by" [ref=w1e35]
       - button "Equals" [ref=w1e38]
       - text "Display is 0" [ref=w1e15]

3. windows_batch { "actions": [
     {"action": "click", "ref": "w1e43"},
     {"action": "click", "ref": "w1e35"},
     {"action": "click", "ref": "w1e43"},
     {"action": "click", "ref": "w1e38"},
     {"action": "snapshot", "handle": "w1"}
   ]}
   → 1. click: Invoked Three
     2. click: Invoked Multiply by
     3. click: Invoked Three
     4. click: Invoked Equals
     5. snapshot: ... "Display is 9" ...

Installation

Prerequisites

Windows 10/11
.NET 8.0 Runtime (for the release binaries) or .NET 8.0 SDK (for the Claude Code plugin install and building from source)

Install as a Claude Code plugin (recommended for Claude Code users)

If you use Claude Code, install FlaUI-MCP as a plugin with two commands — no manual MCP config file editing required:

/plugin marketplace add starpia-forge/FlaUI-MCP
/plugin install flaui-mcp@flaui-mcp-marketplace

Claude Code spawns the MCP server automatically. The 32 windows_* tools become available in your session immediately.

Prerequisite: .NET 8.0 SDK on PATH (the plugin invokes dotnet run; the SDK builds the server from source on first launch). If you only have the .NET runtime, use the release-binary install below instead.

First launch takes ~10–30 s for the initial restore + build. Subsequent launches reuse the cached build (~1–3 s).

Update with /plugin update flaui-mcp@flaui-mcp-marketplace; uninstall with /plugin uninstall flaui-mcp@flaui-mcp-marketplace.

Download Release

Grab the latest build from Releases. Four artifacts are published per release:

Artifact	When to choose
`FlaUI-MCP-win-x64-<v>-self-contained.zip`	Intel/AMD 64-bit Windows, no .NET install required
`FlaUI-MCP-win-x64-<v>-framework-dependent.zip`	Intel/AMD 64-bit Windows, .NET 8 runtime already installed
`FlaUI-MCP-win-arm64-<v>-self-contained.zip`	ARM64 Windows (Surface Pro X, Copilot+ PCs), no .NET install required
`FlaUI-MCP-win-arm64-<v>-framework-dependent.zip`	ARM64 Windows, .NET 8 runtime already installed

Extract the ZIP to any folder; the executable is FlaUI.Mcp.exe.

Configure MCP Client

Add to your MCP configuration (e.g., ~/.copilot/mcp-config.json):

{
  "mcpServers": {
    "windows": {
      "type": "local",
      "command": "C:\\path\\to\\FlaUI-MCP.exe",
      "tools": ["*"]
    }
  }
}

Or using dotnet run:

{
  "mcpServers": {
    "windows": {
      "type": "local",
      "command": "dotnet",
      "args": ["run", "--project", "C:\\path\\to\\src\\FlaUI.Mcp"]
    }
  }
}

Available Tools

Session

Tool	Description
`windows_launch`	Launch a Windows application
`windows_attach`	Attach to a running process by PID or executable name; returns handles for every UIA-visible window (including hidden ones typical of tray-resident apps)

Window

Tool	Description
`windows_list_windows`	List all open windows; pass `includeHidden=true` to surface windows with empty titles (tray-resident apps)
`windows_focus`	Bring a window to foreground
`windows_close`	Close a window
`windows_window_state`	Maximize, minimize, or restore a window via WindowPattern; move or resize via TransformPattern

Inspect

Tool	Description
`windows_snapshot`	Get accessibility tree with element refs (also accepts popup handles `m1`, `m2`, … from `windows_context_menu` or `windows_tray_invoke`). Pass `verbose:true` to include AutomationId and BoundingRect per element.
`windows_inspect`	Dump all UIA properties (AutomationId, ClassName, BoundingRect, …) and supported patterns with current state for one element ref
`windows_focused_element`	Return the element holding keyboard focus as a new ref. Auto-registers the owning window. Use after Tab/Shift+Tab navigation.
`windows_screenshot_diff`	Detect visual changes between two points in time. Store baseline (`store:true`), perform an action, then diff. Returns a bounding rectangle of changed pixels plus percentage. Keyed by handle/ref/fullScreen.
`windows_get_text`	Get text content of an element
`windows_screenshot`	Capture window/element as PNG
`windows_grid_cell`	Access a Grid/Table cell by (row, col) via GridPattern.GetItem; registers the cell as a new ref usable with windows_click, windows_get_value, etc. Works on virtualized data grids.

Values

Tool	Description
`windows_get_value`	Read an element's current value via UIA patterns (Value → RangeValue → Toggle → SelectionItem); use instead of `windows_get_text` for sliders, checkboxes, and combo boxes
`windows_set_value`	Set an element's value — string → Value/SelectionItem pattern, number → RangeValue (slider), boolean → Toggle (checkbox)

Clipboard

Tool	Description
`windows_get_clipboard`	Read system clipboard text content (CF_UNICODETEXT); returns an explicit message when the clipboard is empty or contains non-text data
`windows_set_clipboard`	Write text to the system clipboard; empty string clears the clipboard

Mouse

Tool	Description
`windows_click`	Click an element by ref
`windows_hover`	Move mouse over an element to trigger hover-only UI
`windows_scroll`	Scroll within an element (UIA ScrollPattern or mouse wheel)
`windows_drag`	Drag from one element to another or to absolute coordinates

Keyboard

Tool	Description
`windows_type`	Type text into an element
`windows_fill`	Clear and fill a text field
`windows_keys`	Send keyboard shortcuts or sequences (`Ctrl+S`, `Alt+F4`, `Tab`)

Tray & Menu

Tool	Description
`windows_tray_list`	Enumerate Windows notification-area (system tray) icons; returns refs usable with `windows_tray_invoke`
`windows_tray_invoke`	Click a tray icon by ref (left/right/middle, single or double); right-click auto-registers the context menu as a popup handle
`windows_context_menu`	Right-click an element (or send Shift+F10 / VK_APPS) and register the resulting context menu as a popup handle for `windows_snapshot`
`windows_dismiss_menu`	Send Escape to close an open context menu and remove its popup handle from the registry
`windows_dialog`	Drive Win32 common dialogs (#32770: File Open/Save, message boxes). One call polls for the dialog, optionally fills the path and picks a filter, then invokes Open/Save/OK/Cancel/named button. Auto-registers the dialog as a popup handle.

Flow

Tool	Description
`windows_batch`	Execute multiple actions in one call
`windows_wait_for`	Poll until a condition holds (visible, enabled, textContains, valueEquals, expanded, focused, selectionContains, …)
`windows_assert`	One-shot structured PASS/FAIL condition check

How It Works

The Accessibility Snapshot

When you call windows_snapshot, you get a structured text tree:

- window "Calculator" [ref=w1e1]
  - group "Number pad" [ref=w1e39]
    - button "Seven" [ref=w1e47]
    - button "Eight" [ref=w1e48]
    - button "Nine" [ref=w1e49]
  - text "Display is 0" [ref=w1e15]

This comes from Windows UI Automation - the same API screen readers use. Each element has:

Role (button, text, group, textbox)
Name ("Seven", "Display is 0")
Ref (w1e47) - a handle for interaction
State ([disabled], [readonly], [checked])

Why Not Screenshots?

Approach	Pros	Cons
Accessibility Tree	Semantic, precise, fast, works at any resolution	Requires UI Automation support
Screenshot + Vision	Works with any app	Slow, expensive, imprecise, resolution-dependent

FlaUI-MCP uses accessibility because it's what screen readers use - it's designed for programmatic UI interaction.

Tray-Resident Apps (Discord, Slack, etc.)

Apps that "minimize to tray" have no titled top-level window, so windows_list_windows and windows_focus can't find them. Use windows_tray_list to enumerate the notification-area icons, then windows_tray_invoke to click the owner — its window gets auto-registered and returned as a handle. Alternatively, attach by process name with windows_attach { "processName": "Discord" } and pass includeHidden=true to windows_list_windows to see hidden windows.

Limitation: requires the classic Explorer taskbar (Win10, or Win11 with classic taskbar). The Win11 native taskbar (22H2+) hides Shell_TrayWnd from UIA and is not yet supported.

Context Menus

Right-click context menus (Win32 class #32768) are transient — they dismiss on focus loss. The workflow is:

1. windows_context_menu { "ref": "w1e5" }
   → Popup registered: m1

2. windows_snapshot { "handle": "m1" }
   → - menu [ref=m1e1]
       - menuitem "Cut" [ref=m1e2]
       - menuitem "Copy" [ref=m1e3]
       - menuitem "Paste" [ref=m1e4]

3. windows_click { "ref": "m1e3" }
   → Invoked Copy

4. windows_dismiss_menu { "handle": "m1" }   ← optional; menu auto-closes after click

For tray-icon right-click menus, use windows_tray_invoke with button: "right" — it performs the same discovery and returns a popup handle directly. Do not call windows_list_windows or windows_focus between steps 1 and 3; any window-enumeration call can dismiss the menu before the click lands.

Building from Source

# Clone
git clone https://github.com/starpia-forge/FlaUI-MCP.git
cd FlaUI-MCP

# Build
dotnet build FlaUI.Mcp.sln

# Run
dotnet run --project src/FlaUI.Mcp

Testing

dotnet test FlaUI.Mcp.sln

Unit tests live under tests/FlaUI.Mcp.Tests/ (xUnit) and cover ElementRegistry, SnapshotBuilder, ConditionEvaluator, and KeyMap helpers. The CI workflow runs them automatically for the win-x64 matrix leg.

Some tests are marked [Trait("Category", "Integration")] because they perform real OS input (keyboard/mouse) to exercise production code paths. These are safe to run but will briefly interact with the desktop. Exclude them during headless or focus-sensitive sessions:

dotnet test FlaUI.Mcp.sln --filter "Category!=Integration"

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  AI Agent (GitHub Copilot, Claude, etc.)                        │
│  - Calls MCP tools: windows_snapshot, windows_click, etc.       │
└─────────────────────────────────────────────────────────────────┘
                              │ MCP Protocol (JSON-RPC over stdio)
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  FlaUI-MCP Server (.NET 8)                                      │
│  - Implements MCP tool handlers                                 │
│  - Builds agent-friendly accessibility snapshots                │
│  - Maps element refs ↔ AutomationElements                       │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  FlaUI Library (github.com/FlaUI/FlaUI)                         │
│  - UIA3Automation for modern apps (WPF, UWP, Win32)            │
│  - Control patterns: Invoke, Value, Toggle, Selection           │
│  - Tree walking and element discovery                           │
└─────────────────────────────────────────────────────────────────┘

Supported Applications

Works with any Windows application that supports UI Automation:

✅ Win32 apps (Notepad, Explorer, etc.)
✅ WPF applications
✅ WinForms applications
✅ UWP/Store apps (Calculator, Settings, etc.)
⚠️ Electron apps (partial - depends on accessibility implementation)
❌ Games (typically no UI Automation support)

Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Acknowledgments

shanselman/FlaUI-MCP — The upstream project this fork is built on
FlaUI - The excellent .NET UI Automation library this project is built on
Playwright - Inspiration for the snapshot/ref interaction model
Model Context Protocol - The protocol that makes this possible

Name		Name	Last commit message	Last commit date
.claude-plugin		.claude-plugin
.github		.github
scripts		scripts
src/FlaUI.Mcp		src/FlaUI.Mcp
tests/FlaUI.Mcp.Tests		tests/FlaUI.Mcp.Tests
.gitignore		.gitignore
.releaserc.json		.releaserc.json
CONTRIBUTING.md		CONTRIBUTING.md
FlaUI.Mcp.sln		FlaUI.Mcp.sln
GitVersion.yml		GitVersion.yml
LICENSE		LICENSE
README.md		README.md

Navigation Menu

Repository files navigation

FlaUI-MCP

Why This Exists

Quick Demo

Installation

Prerequisites

Install as a Claude Code plugin (recommended for Claude Code users)

Download Release

Configure MCP Client

Available Tools

Session

Window

Inspect

Values

Clipboard

Mouse

Keyboard

Tray & Menu

Flow

How It Works

The Accessibility Snapshot

Why Not Screenshots?

Tray-Resident Apps (Discord, Slack, etc.)

Context Menus

Building from Source

Testing

Architecture

Supported Applications

Contributing

License

Acknowledgments

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

FlaUI-MCP

Why This Exists

Quick Demo

Installation

Prerequisites

Install as a Claude Code plugin (recommended for Claude Code users)

Download Release

Configure MCP Client

Available Tools

Session

Window

Inspect

Values

Clipboard

Mouse

Keyboard

Tray & Menu

Flow

How It Works

The Accessibility Snapshot

Why Not Screenshots?

Tray-Resident Apps (Discord, Slack, etc.)

Context Menus

Building from Source

Testing

Architecture

Supported Applications

Contributing

License

Acknowledgments

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages