ci: retry a hung visual shard on the label workflow - #3090
Merged
Conversation
`run-visual-shard` has an `attempts` input for the flake it documents — firefox intermittently hangs on a single file in headless CI, which vitest's per-test `retry` cannot recover, only a fresh browser process can. It defaults to 1. `test-visual-scheduled.yml` passes "3"; `test-visual-label.yml` passed nothing. So the one path contributors are told to use — the `run-visual-tests` label, which the guidance says to add proactively on any styling or layout PR — was the only consumer of the action without the retry. One contended runner turns into a red check on an unrelated PR, and the cheapest-looking next move is the `update-screenshots` label, which is how a wrong baseline gets committed (#2985). The retry cannot mask a regression. Every attempt runs the same `test:visual --shard=N/M` over the whole shard, and that script carries no `--update` (only `test:visual:update` does, which the action never calls). A genuine diff is re-rendered and re-compared against the unchanged committed baseline on every attempt, and the loop exits 0 only when the whole shard passes — as the input's own docs put it, a real diff fails every attempt. The timeout goes to 45 minutes to match the scheduled run, so three attempts fit inside the job instead of being cut off as a cancellation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage Report for ./packages/components/
File CoverageNo changed files found. |
Contributor
🚀 Preview DeploymentPreview environments are ready:
Images:
|
ins0
approved these changes
Sep 2, 2026
This was referenced Sep 2, 2026
mfal
added a commit
that referenced
this pull request
Sep 2, 2026
Same evidence, and this is where the retry started. The three attempts predate the sharding of #3006 — they were a step-level loop over the whole unsharded suite, on the same reasoning #3090 later carried to the label path: a hung firefox process needs a fresh browser, which vitest's per-test `retry` cannot give it. The failure that actually shows up does not need a fresh browser, it needs a different runner. Run 33633352648 ran all three attempts of a shard to 118 of 120 failed tests, in both browsers, at 1110s; the `test.yml` shard beside it did 85 of 86 over 474s, 472s and 472s. Nightly runner time is cheaper than a PR's, but three identical failures buy nothing at any price, and the red they report is the same either way. Timeout 45 -> 30, the label path's number for this same shape at one attempt. 45 was sized for the unsharded job, where one attempt alone took 24-34 minutes. `attempts` stays on the action with no caller. It documents a real mechanism for a hang that shows up on its own, and deleting it would reach past this revert into #3006. Its description now says the failure it does not cover, and points at #3106. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.github/actions/run-visual-shardhas anattemptsinput for the flake itdocuments, and defaults it to
1. Only one of its two callers passes it:attemptstest-visual-scheduled.yml"3"test-visual-label.ymlrun-visual-testslabel — before nowSo the path contributors are told to use was the one without the retry.
AGENTS.md says to add the
run-visual-testslabel proactively on anystyling/layout PR, which makes this the most-travelled visual path in the repo.
What it costs
Observed on #3076, a change that cannot move rendered output (an
Optionkeyfix).
run-visual-tests (2)went red with 85 failed tests across 22scenarios, every one carrying the same message from
testScreenshot(
packages/remote-react-components/src/tests/lib/environments.tsx:221):Nothing was ever compared:
pixels (ratio …) differin the job loggh run download 33610043161→ "no valid artifacts found". The "Collectthe visual diffs" step found no
*-diff-*.png, becausetoMatchScreenshotnever cleared its stability gate and so never wrote reference, actual or diff.
The workflow's own summary already anticipates this shape: "a shard that
failed before comparison … leaves none".
And it was the whole shard, not a subset:
vitest list --project=visual --shard=2/6returns 14 files, and all 14 failed —DonutChart,Kbd,CodeEditor,TimeFieldand the rest. 63 of the 64 failure blocks werefirefox-linux. The five sibling shards passed, and the same commit was greenin
test.yml's own visual shards.A reviewer reaching that check sees a red gate with no diff images. The
cheapest-looking next move is the
update-screenshotslabel — which runs--updateover the whole suite andgit add -A, and is exactly how #2945committed a tooltip-less frame as a baseline and kept the scheduled run red
(#2985).
Why the retry cannot hide a regression
The property that makes this safe, from
run-visual-shard's loop:test:visual --shard=N/M, over thewhole shard — not just the previous attempt's failures.
test:visualisvitest run --project=visual --browser.headless. No--update. Onlytest:visual:updatecarries it, and the action never callsit. An attempt therefore cannot rewrite a baseline; it can only re-render and
re-compare against the committed one.
exit 0s only when the entire shard passes; otherwise it fallsthrough to
exit 1.A genuine diff is deterministic — same code, same baseline, same
browser/platform — so it reproduces on every attempt. The input's own
documentation states the invariant: "A real diff fails every attempt." What a
retry absorbs is only the non-deterministic case: a hung firefox process or a
runner too contended to produce two identical frames.
test.yml's visual job — checked, deliberately not includedIt does not use the action. It calls vitest directly,
--browser.name=webkitonly, with no retry of any kind. So it is not a second caller defaulting to
1;it has no retry mechanism to switch on.
It is also materially less exposed rather than merely lucky: the documented flake
is a firefox hang, and that job never starts firefox — its own header says
"Using Webkit for now, because Firefox has issues with parallel visual tests."
The evidence matches, 63 firefox blocks to 1 webkit.
Not immune, though — that one webkit block was real. Left out of this PR anyway,
for two reasons: giving it a retry means either restructuring it onto the shard
action or duplicating a retry loop, and it is the
visualjob that the requiredmaincheck aggregates, so changing its failure semantics deserves its ownreview rather than riding along in a one-line fix. Worth a follow-up.
Also
timeout-minutes30 → 45, matching the scheduled workflow, so three attemptsfit inside the job. Otherwise a third attempt can be cut off and reported as a
cancellation instead of a clean retry. Passing shards on the observed run took
3m40s–4m46s, so the normal case is unaffected.
Not fixed here, noted for later: the 5000ms stability budget is vitest's default
and nothing in the repo raises it, while
waitForPaintedContentright beside itdeliberately carries 20s with a comment that CI hardware is slower than the
machine its numbers came from. The same argument applies to the screenshot's
stability window.
Verification
prettier --checkclean. Behaviour is CI-only, so it verifies by running: thelabel path exercises this workflow on the next PR that uses it, and the
scheduled run has been on
attempts: "3"all along.🤖 Generated with Claude Code