Run the component-tester Playwright suites nightly (BL-16799) - #8292
Conversation
Nothing ran src/BloomBrowserUI/react_components/component-tester, which is how its harness sat broken for weeks on a React 17 pin plus a config bug. The nightly workflow now runs it as a fifth suite, with its own check run and its own junit report, next to the front-end, C#, BloomE2E and visual-regression suites. It is the component config only. The sibling playwright.bloom-exe.config.ts attaches over CDP to a Bloom the developer already has running, so it needs the src/BloomE2E launch fixture before it can run unattended, and that config's testIgnore keeps its specs out of this one. The job needs neither build: the harness serves the components from its own Vite dev server, which playwright.config.ts starts. It does need the src/BloomBrowserUI install, for the axios and jquery that lib/localizationManager imports, and the "pnpm install" step above it provides that. So the group depends on nothing else in the workflow and runs even when the builds failed. --timeout=120000 raises the per-test 30 seconds in playwright.config.ts, which assumes a dev server that has already transformed the module graph. Every CI run starts cold and the first request for a component pays for that transform. A passing test still returns as soon as it passes, so this costs a green run nothing. Two tests that failed about once per full-suite run and passed on their own get a longer wait for the state they are waiting for: the LinkTargetChooser error message (1000ms was not enough on a loaded machine) and the registration opt-out button (a 2000ms buffer on top of the component's own 2-second delay). Both are waits, not sleeps, so a green run pays nothing for the larger number. Retires the AUTOMATION-DEBT.md entry "The component-tester Playwright suites are not in CI". Verified: the suite passes locally through the same command the job runs, 144 passed and 25 skipped, both cold and warm. The job itself is unverified until CI runs it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c43474d to
2a95d13
Compare
Master added a Zulip failure report to .github/workflows/nightly.yml while this branch was open, and both changes edit the manual-run inputs and the end of the job. The merge keeps both: the component-tester suite gets its toggle, its two steps and its own check run, and the Zulip report stays last. The report names every suite it can tell you about, so the component-tester suite is now one of them. A suite missing from that list fails without anybody being told, which is the thing the report exists to prevent. In src/BloomE2E/AUTOMATION-DEBT.md, master reworked the table of this stack to give each pull request its number. This branch deletes its own row and the entry it pays off, so the merge applies both deletions to master's text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BaWw4EHWrbFF2fNJvMJVC8
| # nothing above it, and runs even when the builds failed. | ||
| - name: Set up component-tester tests | ||
| id: setup_component_tests | ||
| if: ${{ !cancelled() && env.RUN_COMPONENT_TESTS == 'true' }} |
There was a problem hiding this comment.
[Devin] Bug: Known hangs skip component coverage
When visual regression consumes the job timeout, setup_component_tests never starts because it runs afterward. The nightly then provides no component coverage.
(.github/workflows/nightly.yml:483)
There was a problem hiding this comment.
[Devin] Investigate: Notification understates suite count
The Zulip link still advertises four suite reports after this change adds a fifth. Failure notifications will present a stale count.
(.github/workflows/nightly.yml:773)
There was a problem hiding this comment.
[Devin] Investigate: Ordering guidance retains stale count
The notification ordering comment still names four publishing steps. Five now precede it, so the maintenance guidance is outdated.
(.github/workflows/nightly.yml:616)
|
[Claude Opus 5 from Hatton's machine during devin-review] Consulted Devin on 2026-09-03 22:45 UTC up to commit |
Nothing ran src/BloomBrowserUI/react_components/component-tester, which is how
its harness sat broken for weeks on a React 17 pin plus a config bug. The
nightly workflow now runs it as a fifth suite, with its own check run and its
own junit report, next to the front-end, C#, BloomE2E and visual-regression
suites.
It is the component config only. The sibling playwright.bloom-exe.config.ts
attaches over CDP to a Bloom the developer already has running, so it needs the
src/BloomE2E launch fixture before it can run unattended, and that config's
testIgnore keeps its specs out of this one.
The job needs neither build: the harness serves the components from its own
Vite dev server, which playwright.config.ts starts. It does need the
src/BloomBrowserUI install, for the axios and jquery that
lib/localizationManager imports, and the "pnpm install" step above it provides
that. So the group depends on nothing else in the workflow and runs even when
the builds failed.
--timeout=120000 raises the per-test 30 seconds in playwright.config.ts, which
assumes a dev server that has already transformed the module graph. Every CI
run starts cold and the first request for a component pays for that transform.
A passing test still returns as soon as it passes, so this costs a green run
nothing.
Two tests that failed about once per full-suite run and passed on their own get
a longer wait for the state they are waiting for: the LinkTargetChooser error
message (1000ms was not enough on a loaded machine) and the registration
opt-out button (a 2000ms buffer on top of the component's own 2-second delay).
Both are waits, not sleeps, so a green run pays nothing for the larger number.
Retires the AUTOMATION-DEBT.md entry "The component-tester Playwright suites
are not in CI".
Verified: the suite passes locally through the same command the job runs, 144
passed and 25 skipped, both cold and warm. The job itself is unverified until
CI runs it.
This is one of eleven stacked pull requests (BL-16799)
Each one pays down one entry of
src/BloomE2E/AUTOMATION-DEBT.md, and each branches off the one before it. Base:BL-16799-vr-collect-failures. Review only this pull request's own commit; the ones below it are reviewed in their own pull requests. The first six change test and tooling code only; the last five also change product code.BL-16799-automation-scripts— Make the bloom-automation scripts safe to ask for helpBL-16799-vr-collect-failures— Report every failed image comparison in a visual-regression case, not the firstBL-16799-component-tests-in-ci— Run the component-tester Playwright suites nightlyBL-16799-vite-port— Let an e2e run test the working tree's front endBL-16799-type-in-one-call— Type into a text box in one call, not one key press per characterBL-16799-page-screenshot— Capture a whole book page from a testBL-16799-toolbox-registration— Register the toolbox tools from one list both callers shareBL-16799-shell-document— Stop a test attaching to a shell document Bloom does not driveBL-16799-tab-test-ids— Click a workspace tab by a test id, not by its localized labelBL-16799-page-change— Refuse a page change the Edit tab cannot do, and wait before askingBL-16799-collection-languages— Set a collection's languages through an e2e hook, not by writing XMLReplaces #8276, which did all of this in one pull request.
Verification of the whole stack, at its tip: the C# suite passes (3338 passed, 13 skipped), the front-end vitest suite passes (781 passed, 5 skipped), and the
src/BloomE2Esuite passes against a Vite dev server on the working tree (36 passed, 0 skipped, 8.2 minutes). Each pull request also has its own type check and lint.🤖 Generated with Claude Code
Devin review
This change is