bench: reproducible export-time benchmark against the competition (macOS + Windows)

benchmark/.gitignore

@@ -0,0 +1,4 @@
+.cache/
+results/*/
+results/install.json
+results/preflight.json

benchmark/REMOTE.md

@@ -0,0 +1,108 @@
+# Driving the benchmark remotely
+
+The run takes one to three hours and needs nobody watching it. This is how to start it, check on
+it, and pick it up again from a Claude Code session on your phone, in the browser, or dispatched
+from another machine.
+
+The design constraint behind all of it: **every prompt that needs a human is provoked up front**,
+and everything after that writes its state to disk so a session that disconnects loses nothing.
+
+---
+
+## The one interactive gate
+
+Do this while you are at the keyboard. It is the only part that cannot be remote, because macOS
+security prompts must be answered on the machine itself.
+
+```bash
+node benchmark/bench.mjs preflight --launch
+```
+
+It will:
+
+1. Print the machine's fitness to measure — chip, cores, RAM, OS build, free disk, power source,
+   thermal state — and refuse quietly-wrong conditions rather than producing a quietly-wrong number.
+2. List every download with its size and licence terms, and wait for you to approve the set.
+3. Provoke each app's **"… wants access to control …"** Apple Events prompt one at a time, so you
+   answer them all in one sitting instead of being ambushed six times during the run. Nothing here
+   clicks *Allow* for you — these are security settings.
+4. Open each GUI app once so its first-launch dialogs (onboarding surveys, update nags, usage-data
+   consent) can be dismissed while you are there.
+
+When it prints `preflight complete`, the machine is ready and you can leave.
+
+## Starting a run remotely
+
+```bash
+node benchmark/bench.mjs install                     # skips anything already present
+node benchmark/bench.mjs calibrate                   # once per machine; ~5 min
+node benchmark/bench.mjs run --reps 3 --id nightly
+```
+
+`run` is safe to launch in the background and disconnect from:
+
+```bash
+nohup node benchmark/bench.mjs run --reps 3 --id nightly > /tmp/bench-nightly.log 2>&1 &
+```
+
+## Checking on it
+
+```bash
+node benchmark/bench.mjs status --json
+```
+
+```json
+{
+  "runId": "nightly",
+  "phase": "running",
+  "current": { "app": "camtasia", "index": 3, "of": 6 },
+  "completed": ["ffmpeg-baseline", "openscreen-cli"],
+  "pending": ["kap", "cap"]
+}
+```
+
+`status.json` is rewritten atomically, so polling it can never read a half-written document.
+`results/<runId>/events.ndjson` is append-only and carries one line per app started, finished or
+skipped — `tail -f` it for a live view without touching the run.
+
+Partial results are written after **every** app, not at the end. A run that dies at app four
+still leaves four apps' worth of `results.json`, and `bench.mjs report --run <id>` will render
+what exists.
+
+## Picking up where it stopped
+
+```bash
+node benchmark/bench.mjs run --apps camtasia,kap --id nightly   # same id: same output folder
+node benchmark/bench.mjs report --run nightly
+```
+
+There is no magic resume flag, deliberately: naming the apps you still need is clearer than a
+flag that guesses, and re-running one app is cheap.
+
+## Notes for an agent driving this
+
+- **Do not run two benchmarks at once, and do not do anything else heavy on the machine while one
+  is running.** The measurement is wall-clock on a shared 8-core SoC; a concurrent build makes
+  every number wrong without making any of them look wrong. `preconditionCheck()` catches
+  throttling and battery, not a competing process.
+- **`run` is long.** Expect ~2 minutes per repetition per app plus 45 s of cooldown between them —
+  roughly 25 minutes for six apps at three reps. Poll `status --json` on a slow cadence; do not
+  busy-wait.
+- **A GUI app can leave a window open** if a run is killed mid-export. `bench.mjs doctor` reports
+  what is running; quitting the app by hand is always safe between runs.
+- **Never interpret a missing app as a slow app.** Skipped rows carry a `reason`; report it
+  verbatim rather than omitting the row.
+- **The report is the deliverable, not the terminal output.** `results/<runId>/report.html` is
+  self-contained and can be published as an artifact directly.
+
+## What still needs a human, and when
+
+| Moment | Why | Frequency |
+|---|---|---|
+| Apple Events prompts | macOS security; only the user can grant them | once per app, ever |
+| First-launch dialogs | vendor onboarding, consent, update nags | once per app, ever |
+| Screen Studio activation | export is licence-gated | once, if you own a licence |
+| Nothing else | — | — |
+
+If a prompt does appear mid-run, `lib/permissions.mjs → pendingPermissionDialog()` reads its text,
+so a session can report *what* is being asked rather than just noticing that everything stalled.

benchmark/apps.mjs

@@ -0,0 +1,160 @@
+/**
+ * The app registry: what is in the benchmark, where it comes from, and what it costs to get.
+ *
+ * Separate from the drivers on purpose — `preflight` has to be able to show the user the whole
+ * download list, with sizes and licence terms, and get one approval for all of it *before*
+ * anything is fetched. Everything after that approval runs unattended.
+ */
+
+/**
+ * Download URLs are pinned to a version wherever the vendor exposes one, because "latest"
+ * makes a benchmark unreproducible: two machines run a month apart would measure two products.
+ * `bench.mjs refresh-urls` re-resolves them and prints the diff.
+ */
+export const APPS = {
+	"openscreen-cli": {
+		driver: "./drivers/openscreen-cli.mjs",
+		default: true,
+		install: {
+			method: "github-release",
+			repo: "getopenscreen/openscreen",
+			assetPattern: /macOS-Apple-Silicon.*\.dmg$/i,
+			appName: "Openscreen.app",
+			approxMB: 250,
+			licence: "MIT — free, no account, no watermark",
+		},
+	},
+	"openscreen-gui": {
+		driver: "./drivers/openscreen-gui.mjs",
+		default: true,
+		sharesInstallWith: "openscreen-cli",
+	},
+	"screen-studio": {
+		// Off by default: export is licence-gated, so an unactivated machine would only ever
+		// record a failure. Enable it explicitly once a licence is activated.
+		// macOS only, and export is licence-gated even there.
+		driver: { darwin: "./drivers/screen-studio.mjs" },
+		default: false,
+		install: {
+			method: "dmg",
+			url: "https://screenstudioassets.com/releases/3.7.5-4595/Screen%20Studio%203.7.5-4595%20Apple%20Silicon.dmg",
+			version: "3.7.5-4595",
+			appName: "Screen Studio.app",
+			approxMB: 349,
+			licence: "commercial — trial exports carry a watermark (which does not change render time)",
+			notes: [
+				"No CLI and no scripting dictionary; only screen-studio://record-* deeplinks exist, none for export.",
+			],
+		},
+	},
+	cap: {
+		driver: "./drivers/cap.mjs",
+		default: true,
+		install: {
+			method: "dmg",
+			url: "https://cap.so/download/apple-silicon",
+			appName: "Cap.app",
+			approxMB: 123,
+			licence: "AGPL-3.0 — free; signing in is optional and not needed for a local export",
+			notes: [
+				"Ships a real CLI at Cap.app/Contents/MacOS/cap-cli — `cap export` renders a .cap project.",
+			],
+		},
+	},
+	camtasia: {
+		driver: { darwin: "./drivers/camtasia.mjs", win32: "./drivers/camtasia-win.mjs" },
+		default: true,
+		install: {
+			method: "dmg",
+			url: "https://download.techsmith.com/camtasiamac/releases/Camtasia.dmg",
+			appName: "Camtasia.app",
+			approxMB: 412,
+			licence: "commercial — 30-day trial, watermarked output",
+			notes: ["No CLI on macOS. Driven through the File → Export menu."],
+		},
+	},
+	focusee: {
+		driver: { darwin: "./drivers/focusee.mjs", win32: "./drivers/focusee-win.mjs" },
+		// On macOS the import is broken in 2.4.1 (see drivers/focusee.mjs); on Windows the
+		// vendor ships the real application rather than a downloader stub, so it is in the
+		// default set there.
+		default: process.platform === "win32",
+		install: {
+			method: "manual",
+			url: "https://focusee.imobie.com/go/download.php?product=fs",
+			appName: "FocuSee.app",
+			approxMB: 5,
+			licence: "commercial — trial exports are watermarked",
+			notes: ["The vendor ships a GUI installer stub; run it once during preflight."],
+		},
+	},
+	kap: {
+		// macOS only — Wulkano ships no Windows build.
+		driver: { darwin: "./drivers/kap.mjs" },
+		default: true,
+		install: {
+			method: "dmg",
+			url: "https://github.com/wulkano/Kap/releases/download/v3.6.0/Kap-3.6.0-arm64.dmg",
+			version: "3.6.0",
+			appName: "Kap.app",
+			approxMB: 119,
+			licence: "MIT — free",
+			notes: [
+				"Has no background, zoom, corner-radius or shadow features at all, so it cannot express the",
+				"full-demo scenario. It is kept as a reduced-fidelity reference: a real app doing a real",
+				"export, with none of the compositing. Its row is marked partial in the report.",
+			],
+		},
+	},
+	"ffmpeg-baseline": {
+		driver: "./drivers/ffmpeg-baseline.mjs",
+		default: true,
+		install: null,
+	},
+};
+
+export async function loadDriver(id) {
+	const entry = APPS[id];
+	if (!entry) throw new Error(`Unknown app "${id}". Known: ${Object.keys(APPS).join(", ")}`);
+	const mod = await import(driverPath(entry, id));
+	return mod.default;
+}
+
+/** Apps that can run at all on this platform — the default set is filtered through this. */
+export function availableOn(platform = process.platform) {
+	return Object.entries(APPS)
+		.filter(([, a]) => typeof a.driver === "string" || !!a.driver?.[platform])
+		.map(([id]) => id);
+}
+
+/** Every distinct thing that has to be downloaded for the given app ids. */
+export function installPlan(appIds) {
+	const seen = new Set();
+	const plan = [];
+	for (const id of appIds) {
+		const entry = APPS[id];
+		if (!entry) continue;
+		const target = entry.sharesInstallWith ?? id;
+		if (seen.has(target)) continue;
+		seen.add(target);
+		const spec = (APPS[target] ?? entry).install;
+		if (spec) plan.push({ id: target, ...spec });
+	}
+	return plan;
+}
+
+export const DEFAULT_APPS = Object.entries(APPS)
+	.filter(([id, a]) => a.default && availableOn().includes(id))
+	.map(([id]) => id);
+
+/** Which driver file implements this app on this platform. */
+function driverPath(entry, id) {
+	if (typeof entry.driver === "string") return entry.driver;
+	const p = entry.driver?.[process.platform];
+	if (!p) {
+		throw new Error(
+			`"${id}" has no driver for ${process.platform}. Supported: ${Object.keys(entry.driver ?? {}).join(", ") || "none"}.`,
+		);
+	}
+	return p;
+}