Skip to content

ci: retry a hung visual shard on the PR run too - #3104

Closed
mfal wants to merge 2 commits into
mainfrom
ci/retry-visual-shards-on-pr
Closed

ci: retry a hung visual shard on the PR run too#3104
mfal wants to merge 2 commits into
mainfrom
ci/retry-visual-shards-on-pr

Conversation

@mfal

@mfal mfal commented Sep 2, 2026

Copy link
Copy Markdown
Member

The retry from #3090 landed on the label and scheduled workflows only. test.yml's visual job called vitest itself, so the shard invocation and its retry lived in two places. This points the job at run-visual-shard, the action the other two paths already use.

What changes

  • browser input on the action. Empty stays both browsers — what the scheduled and label runs need — and test.yml passes webkit, keeping the existing trade of the dark theme for half the wall clock. The input also keys the Playwright cache, so a WebKit-only save can no longer leave a full run without Firefox.
  • install: "false" on prepare-browser-tests. The visual job uses it as the nx affected gate only; the action installs the browser it renders in. Every other caller is untouched.
  • Timeout 20 → 35 minutes. Three attempts of a shard have to fit inside the job. A shard is 3-6 minutes green and ~8 when every screenshot times out, plus ~3 for setup and the build.
  • CONTRIBUTE.md documents the retry, and both it and the comments name what the retry does not cover — see below.

What the retry does not fix

The first push of this PR asserted that the retry would have saved run 33626981366. Its own CI falsified that, so the claim is out of the comments.

Shard 1 here and shard 5 of the label run failed all three attempts identically — 85 of 86 tests, and 118 of 120, reporting Could not capture a stable screenshot within 5000ms from the first test on, in both browsers, while every other shard passed in four minutes. The attempts share the runner, so a machine that produces this fails all of them. The retry covers a hung browser process, which is the Firefox failure mode; it cannot cover this one.

Nor is it this PR's regression: the same whole-shard red hits roughly 4 of every 10 runs on main, a different shard each time (33631694282, 33626556173, 33608121174, 33607745081, 33599165776), and run 33626981366 was one of them. It wants a fix at the root — worth its own issue.

That leaves a question for review: on the WebKit-only PR path attempts: "3" now triples the time to a red signal on a genuine diff (26 minutes instead of 8), and WebKit has not been seen to hang. Keeping it aligns the three paths; setting it to 1 here keeps the fast red. I lean towards keeping it aligned, but it is a real trade.

Verification

Both paths run on this PR: CONTRIBUTE.md makes remote-react-components affected, so test.yml's visual job exercises the new WebKit path, and the run-visual-tests label runs the changed action in its both-browsers default. The retry loop itself is confirmed working — three attempts, a warning between them, the shard's own exit code.

One-off cost: the action's cache key changes from playwright-<os>-<hash> to playwright-all-<os>-<hash>, so the label and scheduled runs install both browsers cold once.

related #3090

🤖 Generated with Claude Code

The retry landed on the label and scheduled workflows only. `test.yml`'s
`visual` job called vitest itself, so the path every PR crosses was the one
without it — run 33626981366 failed 85 of 86 tests with "Could not capture a
stable screenshot within 5000ms", across all 21 files, on a shard whose
baselines the PR did not touch. A starved runner, reported as a diff, on a
branch that was already up to date with the retry commit.

Point the job at `run-visual-shard` instead of duplicating the loop:

- `browser` input, so the action can render in one browser. Empty stays both,
  which is what the scheduled and label runs need; `test.yml` passes `webkit`
  and keeps trading the dark theme for half the wall clock. It also keys the
  browser cache, so a WebKit-only save no longer leaves a full run without
  Firefox.
- `install: "false"` on `prepare-browser-tests`, which the job now uses as the
  `nx affected` gate only — the action installs what it renders in.
- Timeout 20 -> 35 minutes, because three attempts of a shard have to fit
  inside the job. A shard is 3-6 minutes green and ~8 starved, plus ~3 for
  setup and the build.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mfal
mfal requested a review from a team September 2, 2026 13:02
@mfal mfal self-assigned this Sep 2, 2026
@mfal mfal added the run-visual-tests Runs the full visual regression suite against the existing baselines and fails the check on mismatch label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for ./packages/components/

Status Category Percentage Covered / Total
🔵 Lines 78.69% 746 / 948
🔵 Statements 78.57% 763 / 971
🔵 Functions 80.09% 165 / 206
🔵 Branches 70.33% 377 / 536
File CoverageNo changed files found.
Generated in workflow #6593 for commit 5ca6225 by the Vitest Coverage Report Action

@github-actions github-actions Bot removed the run-visual-tests Runs the full visual regression suite against the existing baselines and fails the check on mismatch label Sep 2, 2026
@mfal
mfal enabled auto-merge (squash) September 2, 2026 13:10
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

Preview environments are ready:

Type URL
docs pr-3104.docs.review.flow-components.de
storybook pr-3104.storybook.review.flow-components.de

Images:

  • docs: ghcr.io/mittwald/flow/docs:pr-3104
  • storybook: ghcr.io/mittwald/flow/storybook:pr-3104

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

❌ Visual Regression Tests Failed

At least one shard did not pass. If snapshots differ from the committed baselines, download the visual-diffs-* artifacts from this run to inspect the actual/diff images (a shard that failed before comparison, e.g. during install or browser setup, leaves none).

If the differences are intentional, update the baselines by adding the update-screenshots label to the PR.

Run details

This PR's own run falsified the justification the comments carried. Shard 1 of
`test.yml` and shard 5 of the label run failed all three attempts identically —
85 of 86 tests, and 118 of 120, reporting "Could not capture a stable screenshot
within 5000ms" from the first test on, in both browsers, while every other shard
passed in four minutes. The attempts share the runner, so a machine that
produces this fails all of them; a retry cannot recover it.

It is not this PR's regression either: the same whole-shard red hits roughly 4 of
every 10 runs on `main`, a different shard each time, and it is what run
33626981366 was.

So the comments and CONTRIBUTE.md now claim only what the retry does — a hung
browser process, which is Firefox — and name the failure it does not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mfal

mfal commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

Closing this. The retry it wires into test.yml is being reverted on the label path in #3107, for the reason this PR's own CI produced: three attempts of a shard fail identically, because they share the runner.

The failure is filed as #3106. The parts of this PR that stand on their own — pointing test.yml's visual job at run-visual-shard instead of a duplicated vitest invocation, and the browser input that lets the action render in one browser — are worth having, but they are deduplication rather than a fix, and they read better once #3106 has settled what the shard invocation should look like. The branch stays on ci/retry-visual-shards-on-pr.

🤖 Generated with Claude Code

@mfal mfal closed this Sep 2, 2026
auto-merge was automatically disabled September 2, 2026 13:55

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant