Robot Framework PlatynUI

Cross-platform native UI automation for Robot Framework.

Warning

Preview quality. Packages, keywords, CLI output, and platform behavior may change before the first stable release.

What is PlatynUI?

PlatynUI is intended to become the Robot Framework automation layer for native desktop applications. It should let test suites inspect real application windows, find controls by stable queries, read UI state, perform user-like actions, and capture diagnostic evidence without binding tests to one operating system or accessibility technology.

Instead of exposing Windows UIA, Linux AT-SPI2, macOS AX, and test doubles as separate worlds, PlatynUI normalizes them into one desktop UI model. Tests and tools query that model with XPath-like selectors and then act through platform capabilities such as focus, window control, pointer input, keyboard input, highlighting, and screenshots.

The project is organized around:

• A Robot Framework-first library surface, with the current low-level PlatynUI.BareMetal library and a higher-level PlatynUI library still in migration.
• A shared Rust UI model with control, item, app, and native namespaces.
• An XPath 2.0-inspired query engine tailored to native desktop UI trees.
• Native providers for Windows UIA and Linux AT-SPI2, a Java provider that routes to toolkit backends (experimental Java Access Bridge for Windows Swing/AWT today), planned macOS AX support, and mock providers for deterministic tests.
• Platform devices for user-like interaction, screenshots, highlights, desktop metadata, and window management.
• A CLI (platynui-cli) and GUI inspector (platynui-inspector) for diagnostics, exploration, and query development.
• Python bindings (platynui-native) that connect the Rust runtime to Robot Framework and Python tests.

Current status

• Rust core: active implementation for the UI model, XPath engine, runtime, CLI, platform devices, and providers.
• CLI and Inspector: preview tools are available as binary Python packages and from local source builds.
• Robot Framework library: the high-level Library PlatynUI entry point is still a placeholder while the migration continues. Use Library PlatynUI.BareMetal for the current low-level keyword surface.
• Python requirement: Python 3.12 or newer. The native extension uses PyO3 abi3-py312.
• Rust requirement: Rust 1.95 or newer, Rust 2024 edition.

Install preview tools

Install pre-release tool packages explicitly. The examples below assume uv 0.11.7 or newer. For user-level command-line tools, uv tool is the most convenient path:

uv tool install --prerelease allow platynui-cli
uv tool install --prerelease allow platynui-inspector

Inside an existing virtual environment, install the packages directly:

uv pip install --pre platynui-cli platynui-inspector
# or
pip install --pre platynui-cli platynui-inspector

The Robot Framework library package (robotframework-PlatynUI) is not yet published as the stable end-user package. For local development, see CONTRIBUTING.md.

Try the tools:

platynui-cli list-providers
platynui-cli info --format json
platynui-cli query "//control:Window"
platynui-cli keyboard list | head -n 20
platynui-cli keyboard type "Hello <Ctrl+A>\\u00A7"
platynui-cli snapshot "//control:Window" --pretty
platynui-cli snapshot "//control:Window" --format xml --output windows.xml
platynui-cli highlight "//control:Window"
platynui-cli screenshot desktop.png
platynui-cli element-at-point 100 200
platynui-inspector

The CLI also offers watch (stream provider events), focus, window (activate, minimize, move, resize, ...), and pointer (move, click, scroll, drag) subcommands; see packages/cli/README.md for the full command overview.

Useful Inspector options:

platynui-inspector --theme dark
platynui-inspector --search-result-limit unlimited
PLATYNUI_INSPECTOR_SEARCH_RESULT_LIMIT=5000 platynui-inspector

The Inspector includes a live mouse picker ("Pick Element" in the toolbar): while armed, holding the activation combination (default Ctrl+Alt+Shift, configurable under File → Settings) selects and highlights the element under the mouse cursor, including in other applications. See the platform-support table below for where live picking is available.

By default the Inspector follows the system light/dark theme; --theme <system|light|dark> or PLATYNUI_INSPECTOR_THEME forces one for a single run. It shows the first 5000 XPath search results; --search-result-limit <COUNT|unlimited> or PLATYNUI_INSPECTOR_SEARCH_RESULT_LIMIT overrides that guard. Command-line options take precedence over environment variables. Settings changed in the UI (picker combination, toolbar style, theme) persist in <config dir>/platynui/inspector.ron; PLATYNUI_INSPECTOR_SETTINGS_PATH overrides the settings file location. See packages/inspector/README.md for rendering and troubleshooting options.

Platform support

Component	Windows	Linux X11	Linux Wayland	PlatynUI compositor	macOS	Mock
UI tree provider	✅ UIA	✅ AT-SPI2	✅ AT-SPI2, window-relative coordinates	✅ AT-SPI2 + compositor window geometry	❌ AX stub	✅
Java provider backends	⚠️ experimental in-JVM agent + JAB (Java Access Bridge) fallback, Swing/AWT only	❌ planned in-JVM agent	❌ planned in-JVM agent	❌ planned in-JVM agent	— (JDK implements AX natively)	—
Pointer	✅ SendInput	✅ XTest	⚠️ portal (Mutter/KWin) / virtual-input (wlroots) / EIS	✅ EIS	❌ stub	✅
Keyboard	✅ SendInput	✅ XTest	⚠️ portal (Mutter/KWin) / virtual-input (wlroots) / EIS	✅ EIS	❌ stub	✅
Desktop info	✅ Win32	✅ XRandR/root geometry	⚠️ wl_output + Mutter/KWin D-Bus enrichment	✅ wl_output	❌ stub	✅
Screenshot	✅ GDI	✅ XGetImage	❌ planned (portal / screencopy)	✅ compositor capture	❌ stub	✅
Highlight	✅ layered window	✅ override-redirect windows	❌	✅ compositor overlay	❌ stub	✅
Window management	✅ Win32	⚠️ partial EWMH	❌	✅	❌ stub	✅
Inspector	✅	✅	⚠️ tree and queries only	✅	❌	✅ with mock feature
Inspector live mouse picker	✅ UIA	✅ AT-SPI	❌ no live cursor position	✅	❌ AX stub	—

Linux uses platynui-platform-linux as a runtime session mediator. It detects X11 vs Wayland from the environment and delegates to the matching backend. X11 remains the most complete Linux path today. On generic Wayland compositors, capabilities that need compositor cooperation (global coordinates, screenshots, highlighting, window management, live picking) are not available yet; portal- and protocol-based backends are planned. See the working notes under dev-docs/ and apps/wayland-compositor/docs/ for current protocol work.

The PlatynUI compositor column is apps/wayland-compositor, the project's own Wayland test compositor: it exposes input injection (EIS), window geometry and management, screenshots, highlighting, and modifier state over a control socket, optionally hosts X11 applications through XWayland, and backs the Linux acceptance test lanes (windowed or headless). Under it, the full capability set works on Wayland today.

For Java UI toolkits on Linux, java-atk-wrapper is deliberately not part of the strategy (it is fragile and requires modifying the target application's launch): Swing/AWT and JavaFX are planned to be reached through an in-JVM agent provider, while SWT is already covered by the native providers on every platform. See dev-docs/java-toolkits.md for the full detection and coverage picture.

Package docs

• packages/cli/README.md - CLI package and command overview.
• packages/inspector/README.md - GUI inspector package and usage notes.
• packages/native/README.md - native Python bindings and mock-provider setup.
• crates/xpath/README.md - XPath engine notes.
• apps/wayland-compositor/README.md - test compositor overview.
• apps/wayland-compositor-ctl/README.md - compositor control CLI.

Documentation

Developer, design, and planning documentation lives under dev-docs/ — architecture, platform internals, input, testing strategy, Python bindings, and planning notes (see dev-docs/README.md for an index). The docs/ directory is reserved for user-facing documentation, which is still being built out.

Additional working docs live next to some components:

• apps/wayland-compositor/docs/ - Wayland test compositor usage, configuration, and control protocol notes.
• crates/xpath/docs/ - XPath engine coverage notes.

Contributing

Contributions are welcome. Start with these guides:

• CONTRIBUTING.md - setup, contribution expectations, just task runner workflow, coding standards, testing guidance, PR checklist, and packaging notes.

The short version is: keep changes focused, use Conventional Commits, run the relevant just checks, and update docs when behavior changes.

License

Apache-2.0. See LICENSE in this repository.