Skip to content

ci: retry a hung visual shard on the label workflow - #3090

Merged
ins0 merged 1 commit into
mainfrom
ci/visual-label-shard-attempts
Sep 2, 2026
Merged

ci: retry a hung visual shard on the label workflow#3090
ins0 merged 1 commit into
mainfrom
ci/visual-label-shard-attempts

Conversation

@mfal

@mfal mfal commented Sep 2, 2026

Copy link
Copy Markdown
Member

.github/actions/run-visual-shard has an attempts input for the flake it
documents, and defaults it to 1. Only one of its two callers passes it:

Workflow attempts Trigger
test-visual-scheduled.yml "3" cron
test-visual-label.yml unset → 1 the run-visual-tests label — before now

So the path contributors are told to use was the one without the retry.
AGENTS.md says to add the run-visual-tests label proactively on any
styling/layout PR, which makes this the most-travelled visual path in the repo.

What it costs

Observed on #3076, a change that cannot move rendered output (an Option key
fix). run-visual-tests (2) went red with 85 failed tests across 22
scenarios
, every one carrying the same message from testScreenshot
(packages/remote-react-components/src/tests/lib/environments.tsx:221):

Could not capture a stable screenshot within 5000ms.

Nothing was ever compared:

  • 0 occurrences of pixels (ratio …) differ in the job log
  • gh run download 33610043161"no valid artifacts found". The "Collect
    the visual diffs" step found no *-diff-*.png, because toMatchScreenshot
    never cleared its stability gate and so never wrote reference, actual or diff.
    The workflow's own summary already anticipates this shape: "a shard that
    failed before comparison … leaves none"
    .

And it was the whole shard, not a subset: vitest list --project=visual --shard=2/6 returns 14 files, and all 14 failed — DonutChart, Kbd,
CodeEditor, TimeField and the rest. 63 of the 64 failure blocks were
firefox-linux. The five sibling shards passed, and the same commit was green
in test.yml's own visual shards.

A reviewer reaching that check sees a red gate with no diff images. The
cheapest-looking next move is the update-screenshots label — which runs
--update over the whole suite and git add -A, and is exactly how #2945
committed a tooltip-less frame as a baseline and kept the scheduled run red
(#2985).

Why the retry cannot hide a regression

The property that makes this safe, from run-visual-shard's loop:

  • Each attempt runs the identical command, test:visual --shard=N/M, over the
    whole shard — not just the previous attempt's failures.
  • test:visual is vitest run --project=visual --browser.headless. No
    --update.
    Only test:visual:update carries it, and the action never calls
    it. An attempt therefore cannot rewrite a baseline; it can only re-render and
    re-compare against the committed one.
  • The loop exit 0s only when the entire shard passes; otherwise it falls
    through to exit 1.

A genuine diff is deterministic — same code, same baseline, same
browser/platform — so it reproduces on every attempt. The input's own
documentation states the invariant: "A real diff fails every attempt." What a
retry absorbs is only the non-deterministic case: a hung firefox process or a
runner too contended to produce two identical frames.

test.yml's visual job — checked, deliberately not included

It does not use the action. It calls vitest directly, --browser.name=webkit
only, with no retry of any kind. So it is not a second caller defaulting to 1;
it has no retry mechanism to switch on.

It is also materially less exposed rather than merely lucky: the documented flake
is a firefox hang, and that job never starts firefox — its own header says
"Using Webkit for now, because Firefox has issues with parallel visual tests."
The evidence matches, 63 firefox blocks to 1 webkit.

Not immune, though — that one webkit block was real. Left out of this PR anyway,
for two reasons: giving it a retry means either restructuring it onto the shard
action or duplicating a retry loop, and it is the visual job that the required
main check aggregates, so changing its failure semantics deserves its own
review rather than riding along in a one-line fix. Worth a follow-up.

Also

timeout-minutes 30 → 45, matching the scheduled workflow, so three attempts
fit inside the job. Otherwise a third attempt can be cut off and reported as a
cancellation instead of a clean retry. Passing shards on the observed run took
3m40s–4m46s, so the normal case is unaffected.

Not fixed here, noted for later: the 5000ms stability budget is vitest's default
and nothing in the repo raises it, while waitForPaintedContent right beside it
deliberately carries 20s with a comment that CI hardware is slower than the
machine its numbers came from. The same argument applies to the screenshot's
stability window.

Verification

prettier --check clean. Behaviour is CI-only, so it verifies by running: the
label path exercises this workflow on the next PR that uses it, and the
scheduled run has been on attempts: "3" all along.

🤖 Generated with Claude Code

`run-visual-shard` has an `attempts` input for the flake it documents — firefox
intermittently hangs on a single file in headless CI, which vitest's per-test
`retry` cannot recover, only a fresh browser process can. It defaults to 1.
`test-visual-scheduled.yml` passes "3"; `test-visual-label.yml` passed nothing.

So the one path contributors are told to use — the `run-visual-tests` label,
which the guidance says to add proactively on any styling or layout PR — was the
only consumer of the action without the retry. One contended runner turns into a
red check on an unrelated PR, and the cheapest-looking next move is the
`update-screenshots` label, which is how a wrong baseline gets committed (#2985).

The retry cannot mask a regression. Every attempt runs the same
`test:visual --shard=N/M` over the whole shard, and that script carries no
`--update` (only `test:visual:update` does, which the action never calls). A
genuine diff is re-rendered and re-compared against the unchanged committed
baseline on every attempt, and the loop exits 0 only when the whole shard
passes — as the input's own docs put it, a real diff fails every attempt.

The timeout goes to 45 minutes to match the scheduled run, so three attempts fit
inside the job instead of being cut off as a cancellation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mfal mfal self-assigned this Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for ./packages/components/

Status Category Percentage Covered / Total
🔵 Lines 78.69% 746 / 948
🔵 Statements 78.57% 763 / 971
🔵 Functions 80.09% 165 / 206
🔵 Branches 70.33% 377 / 536
File CoverageNo changed files found.
Generated in workflow #6545 for commit 52d393c by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

Preview environments are ready:

Type URL
docs pr-3090.docs.review.flow-components.de
storybook pr-3090.storybook.review.flow-components.de

Images:

  • docs: ghcr.io/mittwald/flow/docs:pr-3090
  • storybook: ghcr.io/mittwald/flow/storybook:pr-3090

@mfal
mfal marked this pull request as ready for review September 2, 2026 11:47
@mfal
mfal requested a review from a team September 2, 2026 11:47
@ins0
ins0 merged commit 2aed064 into main Sep 2, 2026
22 checks passed
@ins0
ins0 deleted the ci/visual-label-shard-attempts branch September 2, 2026 11:48
mfal added a commit that referenced this pull request Sep 2, 2026
Same evidence, and this is where the retry started. The three attempts predate
the sharding of #3006 — they were a step-level loop over the whole unsharded
suite, on the same reasoning #3090 later carried to the label path: a hung
firefox process needs a fresh browser, which vitest's per-test `retry` cannot
give it.

The failure that actually shows up does not need a fresh browser, it needs a
different runner. Run 33633352648 ran all three attempts of a shard to 118 of
120 failed tests, in both browsers, at 1110s; the `test.yml` shard beside it did
85 of 86 over 474s, 472s and 472s. Nightly runner time is cheaper than a PR's,
but three identical failures buy nothing at any price, and the red they report
is the same either way.

Timeout 45 -> 30, the label path's number for this same shape at one attempt.
45 was sized for the unsharded job, where one attempt alone took 24-34 minutes.

`attempts` stays on the action with no caller. It documents a real mechanism for
a hang that shows up on its own, and deleting it would reach past this revert
into #3006. Its description now says the failure it does not cover, and points
at #3106.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants