[RFC] Self-hosted visual regression testing for Storybook (replace Chromatic)

[soroushm]

Affected area: web

What's the Motivation? 🤔

We use Chromatic (the paid cloud service from the Storybook maintainers) for visual regression testing — screenshot every story, diff against a baseline, review the difference. The visual-diff capability is genuinely valuable and we want to keep it. The dependency is not:

• External paid SaaS + a CHROMATIC_PROJECT_TOKEN secret we must maintain.
• CI cost/time — the Publish to Chromatic step (ci.yml) runs a second full build:storybook, uploads it, and snapshots, adding minutes to the already ~30-minute web job (which is currently hitting its timeout-minutes: 30).
• Redundant hosting — we now self-host Storybook at storybook.soroush.tech ([Task] Deploy Storybook to Cloudflare at storybook.soroush.tech from cd-web #260), so Chromatic's Storybook-hosting half (whose URL only feeds SB_URL for nicer failing-test links) is duplicated.

Goal: own the visual-regression capability end-to-end on our own infra (Playwright + Cloudflare/R2), drop the paid dependency, and reduce CI time — without losing the "here's what changed, approve it" workflow.

What are the requirements? ❓

• Capture a screenshot of every Storybook story (with a changed-only fast path).
• Diff each against a stored baseline and produce a human-reviewable expected / actual / diff artifact.
• Deterministic rendering — no false diffs from font/antialiasing/OS differences.
• A baseline management + approval flow (how a legitimate visual change gets blessed).
• Runs in CI, change-gated; decide blocking vs non-blocking (Chromatic is currently non-blocking via exitZeroOnChanges: true).
• No new paid SaaS; permissively-licensed OSS only (all candidates are MIT/Apache-2.0).
• Reuse existing infra where possible: Playwright, storybook-static, Cloudflare R2.
• Net reduce CI time versus the current Chromatic step.

What are our options? 💡

#	Option	Diff review UX	Baselines	Notes
A	Playwright toHaveScreenshot() over stories	Playwright HTML report (expected/actual/diff)	committed PNGs in git	Leanest; reuses our Playwright setup; approval = regenerate + commit
B	storycap + reg-suit	Hosted HTML report + PR comment	R2 (off-git)	Closest to Chromatic's review UX; more moving parts
C	Keep Chromatic (status quo)	Chromatic web app	Chromatic cloud	Paid; adds CI time; external dependency
D	Other SaaS (Percy / Applitools)	their UI	their cloud	Still paid — rejected, same downside as C

Legality (see #— discussion): re-implementing the functionality with OSS is fine; only copying Chromatic's code or brand would not be. All candidate tools are MIT/Apache-2.0.

Proposed solution 🟢

Phase 1 — Option A, behind determinism guarantees.

1. Build storybook-static (already scripted: build:storybook).
2. A Playwright spec reads Storybook's index.json, visits each story's iframe.html?id=<id>, waits for render, and runs await expect(page).toHaveScreenshot().
3. Run screenshotting only inside the pinned mcr.microsoft.com/playwright Docker image — same version locally and in CI — so baselines are reproducible.
4. Baselines committed alongside stories; updating them is --update-snapshots in a dedicated PR (the "approval").
5. In CI, replace the Publish to Chromatic step with this job (change-gated, reusing the Playwright browser cache). Point SB_URL at storybook.soroush.tech instead of Chromatic's output.

Phase 2 (optional) — layer Option B if we want a hosted review report: reg-suit stores baselines in R2, publishes an HTML diff report, and posts a PR comment with pass/fail + link — recovering Chromatic's review ergonomics on our own infra.

Rollout is incremental: run the new job alongside Chromatic until parity is proven, then delete the Chromatic step and CHROMATIC_PROJECT_TOKEN.

Risks & mitigations 🚨

• Rendering non-determinism (false diffs) — the Configure Renovate #1 risk. Mitigation: containerize screenshotting in the pinned Playwright Docker image; set per-story maxDiffPixelRatio/threshold; disable animations/caret.
• Baseline bloat in git (Option A) — many PNGs. Mitigation: keep only chromium + a small viewport set initially; if it grows, move baselines to R2 (Option B).
• Approval friction — no hosted "Approve" button. Mitigation: the update-baselines PR is the approval; Phase 2 (reg-suit + PR comment) restores a review report.
• Maintenance burden vs SaaS — we now own the tooling. Mitigation: start minimal (Option A); only adopt B if the review UX is missed.
• Loss of Chromatic TurboSnap/cloud parallelism — snapshots run on our runners. Mitigation: change-gate stories (only-changed), reuse the browser cache; net CI time should still drop since we remove the extra Storybook build + upload.
• Reversibility — low risk: keep the Chromatic step until the self-hosted job reaches parity; rollback is deleting the new job.

Resources and benchmarks 🔗

• Storybook deploy that makes hosting redundant: [Task] Deploy Storybook to Cloudflare at storybook.soroush.tech from cd-web #260 (part of Epic [Epic] Build Design System #40).
• Current CI usage: Publish to Chromatic step in .github/workflows/ci.yml; SB_URL consumed in apps/web/vitest.config.ts.
• Tools: Playwright test screenshots (Apache-2.0), reg-suit (MIT), storycap (MIT), Loki (MIT), BackstopJS (MIT).
• Live self-hosted Storybook: https://storybook.soroush.tech

[soroushm]

Name Suggestions

DiffChroma: A direct combination of "Visual Diff" (comparing changes) and "Chroma" (UI elements/color). It is highly memorable and easily understood by tech

Achromatic: It implies your software strips away the noise to show the raw visual differences. (meaning clean, baseline, or grayscale)