ci: retry a hung visual shard on the PR run too - #3104
Conversation
The retry landed on the label and scheduled workflows only. `test.yml`'s `visual` job called vitest itself, so the path every PR crosses was the one without it — run 33626981366 failed 85 of 86 tests with "Could not capture a stable screenshot within 5000ms", across all 21 files, on a shard whose baselines the PR did not touch. A starved runner, reported as a diff, on a branch that was already up to date with the retry commit. Point the job at `run-visual-shard` instead of duplicating the loop: - `browser` input, so the action can render in one browser. Empty stays both, which is what the scheduled and label runs need; `test.yml` passes `webkit` and keeps trading the dark theme for half the wall clock. It also keys the browser cache, so a WebKit-only save no longer leaves a full run without Firefox. - `install: "false"` on `prepare-browser-tests`, which the job now uses as the `nx affected` gate only — the action installs what it renders in. - Timeout 20 -> 35 minutes, because three attempts of a shard have to fit inside the job. A shard is 3-6 minutes green and ~8 starved, plus ~3 for setup and the build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coverage Report for ./packages/components/
File CoverageNo changed files found. |
🚀 Preview DeploymentPreview environments are ready:
Images:
|
❌ Visual Regression Tests FailedAt least one shard did not pass. If snapshots differ from the committed baselines, download the visual-diffs-* artifacts from this run to inspect the actual/diff images (a shard that failed before comparison, e.g. during install or browser setup, leaves none). If the differences are intentional, update the baselines by adding the |
This PR's own run falsified the justification the comments carried. Shard 1 of `test.yml` and shard 5 of the label run failed all three attempts identically — 85 of 86 tests, and 118 of 120, reporting "Could not capture a stable screenshot within 5000ms" from the first test on, in both browsers, while every other shard passed in four minutes. The attempts share the runner, so a machine that produces this fails all of them; a retry cannot recover it. It is not this PR's regression either: the same whole-shard red hits roughly 4 of every 10 runs on `main`, a different shard each time, and it is what run 33626981366 was. So the comments and CONTRIBUTE.md now claim only what the retry does — a hung browser process, which is Firefox — and name the failure it does not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Closing this. The retry it wires into The failure is filed as #3106. The parts of this PR that stand on their own — pointing 🤖 Generated with Claude Code |
Pull request was closed
The retry from #3090 landed on the label and scheduled workflows only.
test.yml'svisualjob called vitest itself, so the shard invocation and its retry lived in two places. This points the job atrun-visual-shard, the action the other two paths already use.What changes
browserinput on the action. Empty stays both browsers — what the scheduled and label runs need — andtest.ymlpasseswebkit, keeping the existing trade of the dark theme for half the wall clock. The input also keys the Playwright cache, so a WebKit-only save can no longer leave a full run without Firefox.install: "false"onprepare-browser-tests. The visual job uses it as thenx affectedgate only; the action installs the browser it renders in. Every other caller is untouched.CONTRIBUTE.mddocuments the retry, and both it and the comments name what the retry does not cover — see below.What the retry does not fix
The first push of this PR asserted that the retry would have saved run 33626981366. Its own CI falsified that, so the claim is out of the comments.
Shard 1 here and shard 5 of the label run failed all three attempts identically — 85 of 86 tests, and 118 of 120, reporting
Could not capture a stable screenshot within 5000msfrom the first test on, in both browsers, while every other shard passed in four minutes. The attempts share the runner, so a machine that produces this fails all of them. The retry covers a hung browser process, which is the Firefox failure mode; it cannot cover this one.Nor is it this PR's regression: the same whole-shard red hits roughly 4 of every 10 runs on
main, a different shard each time (33631694282, 33626556173, 33608121174, 33607745081, 33599165776), and run 33626981366 was one of them. It wants a fix at the root — worth its own issue.That leaves a question for review: on the WebKit-only PR path
attempts: "3"now triples the time to a red signal on a genuine diff (26 minutes instead of 8), and WebKit has not been seen to hang. Keeping it aligns the three paths; setting it to1here keeps the fast red. I lean towards keeping it aligned, but it is a real trade.Verification
Both paths run on this PR:
CONTRIBUTE.mdmakesremote-react-componentsaffected, sotest.yml'svisualjob exercises the new WebKit path, and therun-visual-testslabel runs the changed action in its both-browsers default. The retry loop itself is confirmed working — three attempts, a warning between them, the shard's own exit code.One-off cost: the action's cache key changes from
playwright-<os>-<hash>toplaywright-all-<os>-<hash>, so the label and scheduled runs install both browsers cold once.related #3090
🤖 Generated with Claude Code