ci: commit only baselines that reproduce, and only baselines - #2995
Closed
mfal wants to merge 3 commits into
Closed
Conversation
`update-screenshots` ran `pnpm test:visual --update` over the whole suite and then `git add -A`. Neither half was scoped, and both cost us. `--update` rewrites whatever renders now. It cannot tell a baseline the PR meant to move from one belonging to a scenario that merely happens to be failing or flaky — so #2945, a CodeBlock change, picked up a tooltip-less frame of the racy `Tooltip` scenario and committed it as a baseline. It contradicted the three beside it and kept the scheduled visual run red in both environments for days, in a file nobody reviewing a CodeBlock PR would think to look at. `git add -A` then widened the blast radius past screenshots entirely: a lockfile, a `.vitest-attachments` leftover, a regenerated artifact — anything the job left behind would ride along in a commit titled "update visual regression screenshots". So: - Stage only `packages/remote-react-components/src/tests/visual/__screenshots__`, the one directory `test:visual` can write. - Derive the owning test files from the changed baseline paths and re-run just those scenarios against the frames that were written. A baseline the suite cannot reproduce is a race, not a new truth; nothing is committed, and the summary says which scenarios to fix. Scoped, so this costs seconds. - Report the failure case in the summary (`if: always()`), so a blocked update explains itself instead of leaving a red step. Also drop the `git status | awk '{print $2}'` parse, which mis-reads a rename ("R old -> new") and any path containing a space, for `git diff --cached --name-only`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage Report for ./packages/components/
File CoverageNo changed files found. |
`visual-diffs` has never existed. `.vitest-attachments` is a dot directory, and `actions/upload-artifact` skips hidden paths unless told otherwise, so every run logged "No files were found with the provided path" and uploaded nothing — including runs whose failure was a plain pixel mismatch with reference, actual and diff images sitting right there on disk. The failure comment points reviewers at that artifact anyway, and vitest truncates the step log before its own failure summary, so a red visual check offered no reference image, no diff, and no list of which scenarios failed. Diagnosing #2985 meant reading a snapshot's byte size and its git history to work out what had changed. `include-hidden-files: true` is the whole fix. Note the scheduled visual workflow uploads nothing at all — its failures stay just as opaque. Left alone here; it needs the upload step added, not a flag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
🚀 Preview DeploymentPreview environments are ready:
Images:
|
`--update` writes every baseline it can and *then* the suite still fails on anything a rewritten snapshot cannot fix — a scenario that never captures a stable frame, most often. The job stopped at that step, so one unstable scenario anywhere discarded the baselines already written for every other one, and the label looked like it had done nothing at all. That is not hypothetical: it is what the sweep on #2985 did. It ran the full suite, wrote the two `-linux` tooltip baselines the PR needs, then exited 1 and committed nothing, leaving a required check red with no way to fix it from the PR. So the update run is allowed to be red, and the verification added in the previous commit becomes the actual gate: a baseline is committed if re-running its scenario reproduces it, whatever else the suite did. A scenario that fails for reasons a snapshot cannot fix no longer blocks the ones that do. The summary distinguishes the cases, because "red and wrote nothing" must not read as "everything was already correct". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5 tasks
Member
Author
|
This is only needed, if updating screenshots is flaky, which was improved lately |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
update-screenshotsranpnpm test:visual --updateover the whole suite andthen
git add -A. Neither half was scoped, and both cost us.--updaterewrites whatever renders now. It cannot tell a baseline the PRmeant to move from one belonging to a scenario that merely happens to be failing
or flaky. 6bf57c7 (
fix(CodeBlock): …, #2945) hit exactly that: it picked up atooltip-less frame of the racy
Tooltipscenario and committed it as thefirefox-linuxbaseline forTooltip - visible, contradicting the threebaselines beside it. The scheduled visual run stayed red in both environments for
days, over a file nobody reviewing a CodeBlock PR would think to open. (Fixed in
#2985; this is the mechanism that let it happen.)
git add -Awidened the blast radius past screenshots entirely. A lockfile, a.vitest-attachmentsleftover, a regenerated artifact — anything the job leftbehind would ride along in a commit titled "update visual regression
screenshots".
What changes
Stage only baselines.
git add -- <baseline dir>instead ofgit add -A,scoped to
packages/remote-react-components/src/tests/visual/__screenshots__—the one directory
test:visualcan write, since that package is the onlyproject with the target.
Verify that the new baselines reproduce. The changed baseline paths name
their own test files (
…/__screenshots__/<test file>/<snapshot>.png), so thosescenarios are re-run against the frames just written. A baseline the suite
cannot reproduce is a race, not a new truth: nothing is committed, the job goes
red, and the summary names the scenarios to fix. Scoped to the affected files,
so it costs seconds — not a second full suite.
This is the part that would have stopped fix(CodeBlock): balance the padding and unify the editor line colors #2945: the tooltip-less frame does not
reproduce, because the tooltip is normally still painted.
Report the failure case (
if: always()on the summary), so a blockedupdate explains itself instead of leaving a red step with no context.
Drop the
git status --porcelain | awk '{print $2}'parse, which mis-readsa rename (
R old -> new) and any path containing a space, forgit diff --cached --name-only.Third commit: the update run no longer discards its own work
--updatewrites every baseline it can and then vitest can still exit 1 —not because a test failed, but because the run had unhandled errors. Vitest
treats those as a failed run regardless of test results. Reproduced locally:
The job stopped at that step, so
Commit changesnever ran and nothing wascommitted — while the finished baselines sat on the runner's disk. That is why
the
update-screenshotslabel looked like a no-op on #2985: its sweep log showsno
FAILand no failing file at all, only repeated[Error: ResizeObserver loop completed with undelivered notifications.].So the update run is allowed to be red, and the verification from the first
commit becomes the actual gate: a baseline is committed if re-running its
scenario reproduces it, whatever else the run reported. The summary distinguishes
"red and wrote nothing" from "everything was already correct".
This treats the symptom. The unhandled errors are real bugs of their own —
TypeError: listener is not a functioncomes from the remote-dom fork, whereusePropsForRemoteElementwraps everyeventListenersentry without theif (listener)guard itseventPropsbranch has, so a removed listener (keyretained, value
undefined) becomes a callable that throws when invoked. Fixingthat belongs in
mfal/remote-domorpatches/, not here.Second commit: the diff artifact never existed
Separate one-line bug, found while diagnosing #2985 and included because it is
what made that diagnosis so expensive.
visual-diffshas never been uploaded, on any run..vitest-attachmentsisa dot directory and
actions/upload-artifactskips hidden paths by default, soevery run logged
No files were found with the provided path— including runswhose failure was a plain pixel mismatch with reference, actual and diff images
sitting right there on disk. The failure comment points reviewers at that
artifact regardless, and vitest truncates the step log before its own failure
summary, so a red visual check offered no image and no list of failing
scenarios.
include-hidden-files: trueis the whole fix.Worth knowing: the scheduled visual workflow uploads nothing at all, so its
failures stay just as opaque. Left alone here — it needs an upload step added,
not a flag. Say the word and I'll do it.
Notes for the reviewer
The verification is deliberately strict. A legitimate update to a scenario
that is genuinely flaky will now be blocked rather than committed. That is
the intended direction — the alternative is what this PR is cleaning up — and
the summary says what to do. No retry loop, because a retry is how flakiness
gets hidden.
The file list is built as an array, not left to word splitting. An empty
element would reach vitest as an empty filter, which matches every file and
would quietly turn the scoped check back into a full-suite run.
Merge order: landing this before test(remote-react-components): settle the tree before parking the pointer #2985 gets the label is the nicer
sequence, since
pull_request-triggered workflows run the definition from thebase branch — test(remote-react-components): settle the tree before parking the pointer #2985's sweep would then already be covered by the verification.
Either order works, though; test(remote-react-components): settle the tree before parking the pointer #2985 makes the suite green on its own.
Verification
Run in
bash(as the runner does), not zsh — where the array construction matters:yaml.safe_loadon the workflowprettier --check🤖 Generated with Claude Code