browser

Support an App in your assistant that acts as a browser.

A Vellum plugin that puts a browser in the workspace panel. Address bar, back/forward/reload, and the live page: you scroll, click, and type on it the way you would a real window. It drives Playwright in-process, with its own Chromium and its own profile, so signing in somewhere stays signed in the next time you open it.

What it ships

One surface does the work and two support it:

Surface	Path	What it is
App	apps/browser/	The browser UI, rendered in the workspace panel. This is the plugin.
HTTP routes	routes/	The app's backend, served under /x/plugins/browser/.
Lifecycle hooks	hooks/	init installs Chromium at boot; shutdown closes a running window.

Everything under src/ is internal: the browser lifecycle, the live frame stream, address-bar resolution, and shared HTTP helpers.

Install

Not in the marketplace catalog yet, so install straight from this repo:

assistant plugins install https://github.com/vellum-ai/browser

The CLI prints a warning naming the source, because an unreviewed plugin's code runs inside the assistant. Then open it from the workspace panel — the app is addressed as plugins~browser~browser.

How it works

Playwright is a direct dependency, and the page is a live object held in the daemon process:

app (sandboxed frame)
  └─ window.vellum.fetch("/x/plugins/browser/…")
       └─ route  ──▶  Playwright  ──▶  Chromium (own profile, in data/)

Holding the page rather than addressing it a command at a time is what buys the two things that make this feel like a browser:

History is the page's own. Back and forward are page.goBack() and page.goForward(), so they move through the real session history, including entries a single-page app pushed itself, which no remembered list of URLs could reproduce.

The page is live. Most sites refuse to be iframed (X-Frame-Options), and the app cannot embed Chromium, so the panel shows a CDP screencast: JPEG frames painted onto a canvas, with wheel, pointer, and keyboard events forwarded to the real page. Scrolling happens in Chromium. The next frame shows the result. The Playwright viewport is resized to the panel, so the picture fills it instead of showing a cropped still with empty space around it.

Getting a Chromium

A plugin's dependencies install with --ignore-scripts, so Playwright's own postinstall never runs and a fresh install has no browser binary. Resolution walks four options, best first:

1. A Google Chrome already on the machine.
2. Playwright's own Chrome for Testing, if the pinned revision is present.
3. A standalone Chromium the image ships (/opt/pw-browsers/chromium, /usr/bin/chromium, …). Playwright resolves its browser by exact revision, so a perfectly good Chromium at a plain path is invisible to it — and driving one beats downloading a second copy of the same browser.
4. Otherwise Chrome for Testing, downloaded on demand — a few minutes, once.

That download runs this plugin's Playwright CLI by absolute path, not bunx playwright. bunx resolves against the working directory, which belongs to the daemon and not to the plugin, so it misses the copy in node_modules/ and fetches whatever the registry serves — downloading a browser at that version's revision while executablePath() still points at the one this package pins. The install appears to succeed and the browser is still missing.

init kicks the download off at boot so the wait is paid in the background rather than by whoever opens the app. It does not open a window and it does not block: the hook returns immediately, and a start that arrives mid-install joins the install already running.

The app opens the window when it loads (POST /start). The Start button stays as a retry if that launch fails, or if the user later closes the browser. When a launch fails, the reason is kept and surfaced with the remediation the route reported, so a machine that gains a Chromium (or a download that fails once) does not need an assistant restart to recover.

The app

• Address bar: a URL, a bare host (example.com:8080/health works), or a search phrase, which goes to DuckDuckGo. Only http and https are opened; javascript:, data:, and file: are refused.
• Browser state: the window opens when the app loads. Before a page is open, a line says whether the browser is ready, starting, or down. Down shows the reason and a Start / Retry button.
• The page: a live view of Chromium. Scroll, click, and type on it. The panel does not scroll a screenshot; wheel and pointer events go to the page.
• Tabs and windows: add and close both. There is always at least one window and at least one tab. A target="_blank" popup becomes a tab in the window that opened it.
• Settings: the gear opens Browser settings. Engines picks Chromium Debugging (the default, always installed) or Lightpanda (optional install). Lightpanda is headless and has no live page view.
• Back / forward / reload: the page's real history.

Configuration

The default engine is Chromium Debugging. Browser settings can switch it to Lightpanda, which writes { "engine": "lightpanda" } to the plugin's config.json. Lightpanda is an optional AGPL binary downloaded into data/ on Install; it is not a package dependency. The Chromium profile still lives in data/profile. Search is DuckDuckGo.

Routes

Served under /x/plugins/browser/. The app reaches them through window.vellum.fetch; a bare fetch from the sandboxed frame carries no gateway URL and no auth and fails.

Route	Method	Purpose
/status	GET	Whether the browser is up, which Chromium backs it, and why a launch failed. Never launches.
/start	POST	Open the window, or retry after a failed launch. Called on app load. Answers like /status.
/navigate	POST	{ input }: raw address-bar value. Returns the page identity (URL, title).
/frame	GET	Latest live picture of the page. ?since= is the last seq the canvas painted; unchanged polls omit the JPEG.
/input	POST	{ events, since? } (or a single event): wheel, move, down, up, click, key, or resize. Answers immediately; a newer JPEG is piggybacked when since is behind.
/act	POST	{ action }: back, forward, or reload.
/session	GET	Windows and tabs. Empty when the browser is not running.
/session	POST	{ action }: new-tab, close-tab, select-tab, new-window, close-window, select-window. At least one window and one tab remain.
/engines	GET	Installed engines and the default.
/engines	POST	{ action, engine }: install, uninstall, or set-default. Chromium Debugging cannot be uninstalled.
/snapshot	GET	Interactive elements on the page, for the assistant.
/element	POST	{ action, eid, text? }: click or type a snapshotted element.
/extract	GET	Page text, for the assistant. ?includeLinks=1 appends its links.
/close	POST	Shut the browser down. The profile is kept.

Every failure answers with { error, hint? }, and the hint is the actionable half: a missing Chromium comes back with the command that installs it.

input and act validate their type/action against an allowlist rather than passing it through, so a request cannot reach behavior the app does not offer.

The app runs in a sandboxed frame

The host renders the app in <iframe sandbox="allow-scripts allow-popups allow-popups-to-escape-sandbox">. No allow-forms, no allow-same-origin, and no top-level navigation — which rules out three things a UI reaches for by reflex:

• No <form> and no type="submit". Submission is blocked, and the submit event never fires. There is no exception to catch: the button and the Enter key simply do nothing. Submitting is a click handler plus an explicit Enter key handler.
• No <a href>. Navigation is blocked, so the link is a dead control.
• No bare fetch. The frame's origin is opaque and carries no gateway URL or auth; window.vellum.fetch is the only way to reach the routes.

The first two fail silently, which is what makes them worth a rule rather than a code review. src/__tests__/app-sandbox.test.ts fails the build if any of them come back.

Page content is untrusted

The page title, URL, and extracted body text are authored by whoever controls the page. The plugin renders them as text; nothing in the routes or the app treats them as instructions.

Development

bun install        # Playwright, plus the devDependencies for typecheck and tests
bun test           # address-bar unit tests, plus collector tests against a real Chromium
bun run typecheck  # tsc over src/, routes/, hooks/, and the app

The collector tests drive an actual browser, which is the only way to exercise code that runs inside the page. They skip when no Chromium is available rather than failing and looking like a broken collector.

An installed plugin has no devDependencies — the installer runs with --omit=dev — and resolves @vellumai/plugin-api from the workspace shim. The app's bundle is compiled by the assistant with its own esbuild and preact.

To iterate without reinstalling, copy the directory into your workspace:

cp -R . "$VELLUM_WORKSPACE_DIR/plugins/browser"

The plugin source watcher picks up changes: routes are re-read on the next request, and the app is rebuilt from apps/browser/src into apps/browser/dist and served on the next open. dist/ is generated — never commit it.

Not in this version

• A model-visible tool. The assistant drives this browser through the plugin skill (skills/browser/) and its HTTP routes, not through a catalog tool.
• A marketplace listing. Install from the repo URL until an entry lands in plugins/marketplace.json upstream.

License

MIT. See LICENSE.