Skip to content

Run the component-tester Playwright suites nightly (BL-16799) - #8292

Merged
hatton merged 2 commits into
masterfrom
BL-16799-component-tests-in-ci
Sep 3, 2026
Merged

Run the component-tester Playwright suites nightly (BL-16799)#8292
hatton merged 2 commits into
masterfrom
BL-16799-component-tests-in-ci

Conversation

@hatton

@hatton hatton commented Sep 3, 2026

Copy link
Copy Markdown
Member

Nothing ran src/BloomBrowserUI/react_components/component-tester, which is how
its harness sat broken for weeks on a React 17 pin plus a config bug. The
nightly workflow now runs it as a fifth suite, with its own check run and its
own junit report, next to the front-end, C#, BloomE2E and visual-regression
suites.

It is the component config only. The sibling playwright.bloom-exe.config.ts
attaches over CDP to a Bloom the developer already has running, so it needs the
src/BloomE2E launch fixture before it can run unattended, and that config's
testIgnore keeps its specs out of this one.

The job needs neither build: the harness serves the components from its own
Vite dev server, which playwright.config.ts starts. It does need the
src/BloomBrowserUI install, for the axios and jquery that
lib/localizationManager imports, and the "pnpm install" step above it provides
that. So the group depends on nothing else in the workflow and runs even when
the builds failed.

--timeout=120000 raises the per-test 30 seconds in playwright.config.ts, which
assumes a dev server that has already transformed the module graph. Every CI
run starts cold and the first request for a component pays for that transform.
A passing test still returns as soon as it passes, so this costs a green run
nothing.

Two tests that failed about once per full-suite run and passed on their own get
a longer wait for the state they are waiting for: the LinkTargetChooser error
message (1000ms was not enough on a loaded machine) and the registration
opt-out button (a 2000ms buffer on top of the component's own 2-second delay).
Both are waits, not sleeps, so a green run pays nothing for the larger number.

Retires the AUTOMATION-DEBT.md entry "The component-tester Playwright suites
are not in CI".

Verified: the suite passes locally through the same command the job runs, 144
passed and 25 skipped, both cold and warm. The job itself is unverified until
CI runs it.

This is one of eleven stacked pull requests (BL-16799)

Each one pays down one entry of src/BloomE2E/AUTOMATION-DEBT.md, and each branches off the one before it. Base: BL-16799-vr-collect-failures. Review only this pull request's own commit; the ones below it are reviewed in their own pull requests. The first six change test and tooling code only; the last five also change product code.

  1. BL-16799-automation-scripts — Make the bloom-automation scripts safe to ask for help
  2. BL-16799-vr-collect-failures — Report every failed image comparison in a visual-regression case, not the first
  3. BL-16799-component-tests-in-ci — Run the component-tester Playwright suites nightly
  4. BL-16799-vite-port — Let an e2e run test the working tree's front end
  5. BL-16799-type-in-one-call — Type into a text box in one call, not one key press per character
  6. BL-16799-page-screenshot — Capture a whole book page from a test
  7. BL-16799-toolbox-registration — Register the toolbox tools from one list both callers share
  8. BL-16799-shell-document — Stop a test attaching to a shell document Bloom does not drive
  9. BL-16799-tab-test-ids — Click a workspace tab by a test id, not by its localized label
  10. BL-16799-page-change — Refuse a page change the Edit tab cannot do, and wait before asking
  11. BL-16799-collection-languages — Set a collection's languages through an e2e hook, not by writing XML

Replaces #8276, which did all of this in one pull request.

Verification of the whole stack, at its tip: the C# suite passes (3338 passed, 13 skipped), the front-end vitest suite passes (781 passed, 5 skipped), and the src/BloomE2E suite passes against a Vite dev server on the working tree (36 passed, 0 skipped, 8.2 minutes). Each pull request also has its own type check and lint.

🤖 Generated with Claude Code

Devin review


This change is Reviewable

Nothing ran src/BloomBrowserUI/react_components/component-tester, which is how
its harness sat broken for weeks on a React 17 pin plus a config bug. The
nightly workflow now runs it as a fifth suite, with its own check run and its
own junit report, next to the front-end, C#, BloomE2E and visual-regression
suites.

It is the component config only. The sibling playwright.bloom-exe.config.ts
attaches over CDP to a Bloom the developer already has running, so it needs the
src/BloomE2E launch fixture before it can run unattended, and that config's
testIgnore keeps its specs out of this one.

The job needs neither build: the harness serves the components from its own
Vite dev server, which playwright.config.ts starts. It does need the
src/BloomBrowserUI install, for the axios and jquery that
lib/localizationManager imports, and the "pnpm install" step above it provides
that. So the group depends on nothing else in the workflow and runs even when
the builds failed.

--timeout=120000 raises the per-test 30 seconds in playwright.config.ts, which
assumes a dev server that has already transformed the module graph. Every CI
run starts cold and the first request for a component pays for that transform.
A passing test still returns as soon as it passes, so this costs a green run
nothing.

Two tests that failed about once per full-suite run and passed on their own get
a longer wait for the state they are waiting for: the LinkTargetChooser error
message (1000ms was not enough on a loaded machine) and the registration
opt-out button (a 2000ms buffer on top of the component's own 2-second delay).
Both are waits, not sleeps, so a green run pays nothing for the larger number.

Retires the AUTOMATION-DEBT.md entry "The component-tester Playwright suites
are not in CI".

Verified: the suite passes locally through the same command the job runs, 144
passed and 25 skipped, both cold and warm. The job itself is unverified until
CI runs it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hatton
hatton force-pushed the BL-16799-component-tests-in-ci branch from c43474d to 2a95d13 Compare September 3, 2026 19:40
Master added a Zulip failure report to .github/workflows/nightly.yml while this
branch was open, and both changes edit the manual-run inputs and the end of the
job. The merge keeps both: the component-tester suite gets its toggle, its two
steps and its own check run, and the Zulip report stays last.

The report names every suite it can tell you about, so the component-tester
suite is now one of them. A suite missing from that list fails without anybody
being told, which is the thing the report exists to prevent.

In src/BloomE2E/AUTOMATION-DEBT.md, master reworked the table of this stack to
give each pull request its number. This branch deletes its own row and the
entry it pays off, so the merge applies both deletions to master's text.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BaWw4EHWrbFF2fNJvMJVC8
Base automatically changed from BL-16799-vr-collect-failures to master September 3, 2026 22:27
@hatton
hatton merged commit c2b91a4 into master Sep 3, 2026
1 check passed
# nothing above it, and runs even when the builds failed.
- name: Set up component-tester tests
id: setup_component_tests
if: ${{ !cancelled() && env.RUN_COMPONENT_TESTS == 'true' }}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Devin] Bug: Known hangs skip component coverage

When visual regression consumes the job timeout, setup_component_tests never starts because it runs afterward. The nightly then provides no component coverage.

(.github/workflows/nightly.yml:483)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Devin] Investigate: Notification understates suite count

The Zulip link still advertises four suite reports after this change adds a fifth. Failure notifications will present a stale count.

(.github/workflows/nightly.yml:773)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Devin] Investigate: Ordering guidance retains stale count

The notification ordering comment still names four publishing steps. Five now precede it, so the maintenance guidance is outdated.

(.github/workflows/nightly.yml:616)

@hatton

hatton commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

[Claude Opus 5 from Hatton's machine during devin-review]

Consulted Devin on 2026-09-03 22:45 UTC up to commit afdab04fdcbfa1974a00959c85d072ed999056a3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant