Skip to content

ci: commit only baselines that reproduce, and only baselines - #2995

Closed
mfal wants to merge 3 commits into
mainfrom
ci/scope-update-screenshots-commit
Closed

ci: commit only baselines that reproduce, and only baselines#2995
mfal wants to merge 3 commits into
mainfrom
ci/scope-update-screenshots-commit

Conversation

@mfal

@mfal mfal commented Aug 28, 2026

Copy link
Copy Markdown
Member

What & why

update-screenshots ran pnpm test:visual --update over the whole suite and
then git add -A. Neither half was scoped, and both cost us.

--update rewrites whatever renders now. It cannot tell a baseline the PR
meant to move from one belonging to a scenario that merely happens to be failing
or flaky. 6bf57c7 (fix(CodeBlock): …, #2945) hit exactly that: it picked up a
tooltip-less frame of the racy Tooltip scenario and committed it as the
firefox-linux baseline for Tooltip - visible, contradicting the three
baselines beside it. The scheduled visual run stayed red in both environments for
days, over a file nobody reviewing a CodeBlock PR would think to open. (Fixed in
#2985; this is the mechanism that let it happen.)

git add -A widened the blast radius past screenshots entirely. A lockfile, a
.vitest-attachments leftover, a regenerated artifact — anything the job left
behind would ride along in a commit titled "update visual regression
screenshots".

What changes

  • Stage only baselines. git add -- <baseline dir> instead of git add -A,
    scoped to packages/remote-react-components/src/tests/visual/__screenshots__
    the one directory test:visual can write, since that package is the only
    project with the target.

  • Verify that the new baselines reproduce. The changed baseline paths name
    their own test files (…/__screenshots__/<test file>/<snapshot>.png), so those
    scenarios are re-run against the frames just written. A baseline the suite
    cannot reproduce is a race, not a new truth: nothing is committed, the job goes
    red, and the summary names the scenarios to fix. Scoped to the affected files,
    so it costs seconds — not a second full suite.

    This is the part that would have stopped fix(CodeBlock): balance the padding and unify the editor line colors #2945: the tooltip-less frame does not
    reproduce, because the tooltip is normally still painted.

  • Report the failure case (if: always() on the summary), so a blocked
    update explains itself instead of leaving a red step with no context.

  • Drop the git status --porcelain | awk '{print $2}' parse, which mis-reads
    a rename (R old -> new) and any path containing a space, for
    git diff --cached --name-only.

Third commit: the update run no longer discards its own work

--update writes every baseline it can and then vitest can still exit 1 —
not because a test failed, but because the run had unhandled errors. Vitest
treats those as a failed run regardless of test results. Reproduced locally:

Test Files  1 passed (1)
      Tests  18 passed (18)
EXIT=1

⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯
TypeError: listener is not a function

The job stopped at that step, so Commit changes never ran and nothing was
committed
— while the finished baselines sat on the runner's disk. That is why
the update-screenshots label looked like a no-op on #2985: its sweep log shows
no FAIL and no failing file at all, only repeated
[Error: ResizeObserver loop completed with undelivered notifications.].

So the update run is allowed to be red, and the verification from the first
commit becomes the actual gate: a baseline is committed if re-running its
scenario reproduces it, whatever else the run reported. The summary distinguishes
"red and wrote nothing" from "everything was already correct".

This treats the symptom. The unhandled errors are real bugs of their own —
TypeError: listener is not a function comes from the remote-dom fork, where
usePropsForRemoteElement wraps every eventListeners entry without the
if (listener) guard its eventProps branch has, so a removed listener (key
retained, value undefined) becomes a callable that throws when invoked. Fixing
that belongs in mfal/remote-dom or patches/, not here.

Second commit: the diff artifact never existed

Separate one-line bug, found while diagnosing #2985 and included because it is
what made that diagnosis so expensive.

visual-diffs has never been uploaded, on any run. .vitest-attachments is
a dot directory and actions/upload-artifact skips hidden paths by default, so
every run logged No files were found with the provided path — including runs
whose failure was a plain pixel mismatch with reference, actual and diff images
sitting right there on disk. The failure comment points reviewers at that
artifact regardless, and vitest truncates the step log before its own failure
summary, so a red visual check offered no image and no list of failing
scenarios.

include-hidden-files: true is the whole fix.

Worth knowing: the scheduled visual workflow uploads nothing at all, so its
failures stay just as opaque. Left alone here — it needs an upload step added,
not a flag. Say the word and I'll do it.

Notes for the reviewer

Verification

Run in bash (as the runner does), not zsh — where the array construction matters:

Check Result
Baseline path → test file derivation 4 baselines across 3 files → 3 correct test paths
Verification command, 2 derived files 4 test files (2 × 2 browser projects), 12 tests, passes
yaml.safe_load on the workflow parses; 14 steps in the expected order
prettier --check clean

🤖 Generated with Claude Code

`update-screenshots` ran `pnpm test:visual --update` over the whole suite and
then `git add -A`. Neither half was scoped, and both cost us.

`--update` rewrites whatever renders now. It cannot tell a baseline the PR meant
to move from one belonging to a scenario that merely happens to be failing or
flaky — so #2945, a CodeBlock change, picked up a tooltip-less frame of the racy
`Tooltip` scenario and committed it as a baseline. It contradicted the three
beside it and kept the scheduled visual run red in both environments for days,
in a file nobody reviewing a CodeBlock PR would think to look at.

`git add -A` then widened the blast radius past screenshots entirely: a lockfile,
a `.vitest-attachments` leftover, a regenerated artifact — anything the job left
behind would ride along in a commit titled "update visual regression
screenshots".

So:

- Stage only `packages/remote-react-components/src/tests/visual/__screenshots__`,
  the one directory `test:visual` can write.
- Derive the owning test files from the changed baseline paths and re-run just
  those scenarios against the frames that were written. A baseline the suite
  cannot reproduce is a race, not a new truth; nothing is committed, and the
  summary says which scenarios to fix. Scoped, so this costs seconds.
- Report the failure case in the summary (`if: always()`), so a blocked update
  explains itself instead of leaving a red step.

Also drop the `git status | awk '{print $2}'` parse, which mis-reads a rename
("R old -> new") and any path containing a space, for `git diff --cached
--name-only`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mfal
mfal requested a review from a team August 28, 2026 09:59
@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for ./packages/components/

Status Category Percentage Covered / Total
🔵 Lines 76% 627 / 825
🔵 Statements 75.91% 643 / 847
🔵 Functions 77.95% 145 / 186
🔵 Branches 66.66% 298 / 447
File CoverageNo changed files found.
Generated in workflow #6395 for commit d99ee81 by the Vitest Coverage Report Action

`visual-diffs` has never existed. `.vitest-attachments` is a dot directory, and
`actions/upload-artifact` skips hidden paths unless told otherwise, so every run
logged "No files were found with the provided path" and uploaded nothing —
including runs whose failure was a plain pixel mismatch with reference, actual
and diff images sitting right there on disk.

The failure comment points reviewers at that artifact anyway, and vitest
truncates the step log before its own failure summary, so a red visual check
offered no reference image, no diff, and no list of which scenarios failed.
Diagnosing #2985 meant reading a snapshot's byte size and its git history to work
out what had changed.

`include-hidden-files: true` is the whole fix.

Note the scheduled visual workflow uploads nothing at all — its failures stay
just as opaque. Left alone here; it needs the upload step added, not a flag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

Preview environments are ready:

Type URL
docs pr-2995.docs.review.flow-components.de
storybook pr-2995.storybook.review.flow-components.de

Images:

  • docs: ghcr.io/mittwald/flow/docs:pr-2995
  • storybook: ghcr.io/mittwald/flow/storybook:pr-2995

`--update` writes every baseline it can and *then* the suite still fails on
anything a rewritten snapshot cannot fix — a scenario that never captures a
stable frame, most often. The job stopped at that step, so one unstable scenario
anywhere discarded the baselines already written for every other one, and the
label looked like it had done nothing at all.

That is not hypothetical: it is what the sweep on #2985 did. It ran the full
suite, wrote the two `-linux` tooltip baselines the PR needs, then exited 1 and
committed nothing, leaving a required check red with no way to fix it from the
PR.

So the update run is allowed to be red, and the verification added in the
previous commit becomes the actual gate: a baseline is committed if re-running
its scenario reproduces it, whatever else the suite did. A scenario that fails
for reasons a snapshot cannot fix no longer blocks the ones that do.

The summary distinguishes the cases, because "red and wrote nothing" must not
read as "everything was already correct".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mfal

mfal commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

This is only needed, if updating screenshots is flaky, which was improved lately

@mfal mfal closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant