Make snapshot failures readable and link the images - #172
Conversation
A wrong simulator used to be reported as raw model identifiers
("Switch to using iPhone17,1 ... You are using iPhone14,3"), and a
snapshot mismatch left you to find the images yourself.
- Add `PrefireDeviceIdentifier`, mapping model identifiers to marketing
names, and report both the expected and the running device by name.
The running one prefers `SIMULATOR_DEVICE_NAME` from the simulator
environment and falls back to the table. Closes #160.
- Print the OS mismatch as a version instead of dumping
`OperatingSystemVersion`, and name the right OS on tvOS.
- Append `file://` links to the reference and recorded images, plus a
ready-to-run diff command, to every snapshot failure. The directories
are derived the same way SnapshotTesting derives them, so
`use_grouped_snapshots` and `split_snapshot_directories` are honoured.
Closes #161.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2ca2fe15f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| private func snapshotFile(withPrefix prefix: String, in directory: URL) -> URL? { | ||
| let names = (try? FileManager.default.contentsOfDirectory(atPath: directory.path)) ?? [] | ||
| guard let name = names.filter({ $0.hasPrefix(prefix) }).sorted().first else { return nil } |
There was a problem hiding this comment.
When a PreviewProvider yields multiple unnamed previews or duplicate display names, SnapshotTesting assigns numeric counters such as .1.png and .2.png. On a failure after the first assertion, sorting all matching files and selecting .first still returns the .1.png reference and artifact, so the displayed links and ksdiff command compare the wrong images. Preserve the current counter or obtain the exact paths from the verification failure instead of selecting the first prefix match.
Useful? React with 👍 / 👎.
Snapshots sharing a name are told apart by a counter SnapshotTesting appends (`AuthView.1.png`, `AuthView.2.png`), which a `PreviewProvider` with several unnamed previews produces. Picking the first file matching the name prefix therefore linked the first preview's images for every failure after it, and the `ksdiff` command compared the wrong pair. SnapshotTesting puts the absolute paths into the message it returns, but renders them differently per diff tool (quoted `file://` URLs by default, bare quoted paths for ksdiff, unquoted for a string-literal tool, a percent-encoded `open` line when no reference existed yet). Rather than parse those shapes, or keep a second copy of the counter and hope it stays in step, `PrefireSnapshotFailure.fileLinks(for:snapshotDirectory:)` matches the files on disk against the failure text and links the one it mentions. Without a match it reports the folder instead of guessing.
1 participant
Closes #160
Closes #161
Two changes to what a developer sees when a test fails.
Wrong simulator or OS (#160)
Before:
After:
PrefireDeviceIdentifiermaps identifiers to marketing names. For the running device the name fromSIMULATOR_DEVICE_NAMEwins, which also covers devices newer than the table; the table is the fallback and an unknown identifier is printed as-is. The OS message saystvOSon tvOS instead of alwaysiOS.fatalErroris kept rather than replaced withXCTFail+XCTSkip: a skip would turn a misconfigured CI run green, and failing fromsetUp()would let the suite keep running against the wrong device and bury the real message under hundreds of bogus diffs. The readability problem was the text, not the mechanism.Clickable paths on failure (#161)
A failure message now ends with:
Directories are derived the way swift-snapshot-testing derives them, from the same
argument.filethe template already passes toverifySnapshot, souse_grouped_snapshotsandsplit_snapshot_directoriesare honored automatically; without a configured path the template falls back to#filePath, matchingverifySnapshot's own default. Files are located by listing the directory and matching the sanitized name prefix, so the numeric suffix is never guessed — if nothing is found, the directory link is printed instead.SNAPSHOT_ARTIFACTSis honored.Tests
Four generation tests and seven tests on the identifier mapping.
make test60/0,make test-cli24/0. The emitted Swift was additionally checked withswiftc -parse.Custom templates need to copy the new helpers to keep the links — noted in
Documentation/Templates.md.