test(xtask): gate --append-verify phase-2 redo in the validate matrix

[oferchen]

Why

cargo xtask validate never forced a phase-2 verification redo. checks/append_inplace.rs does run --append-verify, but it seeds a matching prefix, so the re-checksum succeeds and the redo path is never entered. Three real divergences live behind that gap and the matrix could not see any of them.

This PR adds the gate. It does not fix the divergences - it makes them visible and keeps them fixed once they are.

What

New xtask/src/commands/validate/checks/verify_redo.rs, registered next to append-inplace in the "Transfer decisions and deletion" group. Category Validation only, so a bare cargo xtask validate runs it.

Fixture - source payload.bin is 200 KiB of deterministic index-hashed bytes; the destination is pre-seeded with 100 KiB of zeros, so it is both shorter than the source and wrong. --append-verify appends the tail, the re-checksum fails, the receiver retains the update and asks the generator to redo it (receiver.c:1070, send_msg_int(MSG_REDO, ndx)). Flags: -a --append-verify --ignore-times --numeric-ids --stats.

Cells - one per (transport, direction, verbosity). Push is skipped on local (a local push is the same local-copy code path as the pull, so it would be a duplicate cell, not coverage); ssh cells skip when no sshd answers on localhost:22.

Assertions, all four per cell:

1. oc's exit code equals upstream's (via the shared comparison::exit_code_diff facet, so a non-zero oc exit is reported as a divergence rather than swallowed as an unrunnable cell).
2. the destination is byte-identical to the source and to upstream's destination.
3. the failed verification lines are exactly upstream's, at each verbosity, per stream - stdout and stderr are checked separately so a line emitted on the wrong stream is a failure, not a vacuous match.
4. the --stats literal/matched split matches upstream's.

Non-vacuous guards - the seeded destination must really be shorter than and differ from the source before the transfer, and upstream must report Number of regular files transferred: 2 (the one file counted twice is the redo). Either guard failing is a FAIL, not a silent pass.

Correction to the assumed oracle

The premise this check was scoped from said the warning prints at default verbosity. It does not. Upstream prints the line only when

if (msgtype == FERROR_XFER || INFO_GTE(NAME, 1) || stdout_format_has_i)   /* receiver.c:1072 */

The first failure is a FWARNING and the redo then succeeds, so nothing ever reaches the ungated FERROR_XFER form. Measured on rsync 3.4.4 across local, daemon pull and daemon push: stderr is empty without -v, and under -v it is exactly

WARNING: payload.bin failed verification -- update retained (will try again).

So the check encodes the measured oracle at both verbosities: upstream's exact silence by default, and upstream's exact bare-relative-name line under -v. That is strictly stronger than asserting the line at one verbosity - it catches a warning that is missing and one that is emitted where upstream is silent.

Currently failing cells (expected, not papered over)

Measured with upstream rsync 3.4.4 as ground truth, --transport local --transport daemon (ssh cells skipped - no sshd on the measuring host). Each cell reports its first failing assertion.

cell	assertion	oc	upstream
local pull default	4	literal 204,800 / matched 0, 1 transfer	literal 205,300 / matched 101,900, 2 transfers
local pull verbose	3	stderr empty (and stdout empty - not emitted anywhere)	the exact WARNING: line on stderr
daemon pull default	3	the line on stdout, with an absolute path, where upstream is silent on both streams	nothing on either stream
daemon pull verbose	3	the line on stdout with an absolute path; stderr empty	the exact WARNING: line on stderr, stdout empty
daemon push default	4	literal 205,700 / matched 101,500	literal 205,300 / matched 101,900
daemon push verbose	4	literal 205,700 / matched 101,500	literal 205,300 / matched 101,900

Three distinct open divergences, all already tracked separately:

• local never enters the redo at all. oc transfers the file once as whole-file literal (204,800 / 0, one transfer) instead of appending, failing verification, retaining, and redoing. Its silence on both streams is a symptom of that, not a separate reporting bug.
• daemon pull emits the warning on stdout, with an absolute path, at every verbosity - wrong stream, wrong name form, and present where upstream is silent. Behind that it also redoes by re-sending everything as literal (307,200 = 102,400 tail + 204,800 redo, matched 0) instead of re-deltaing against the retained basis.
• daemon push reports correctly (silent by default, exact bare-relative-name line on stderr under -v) but its redo delta is off by 400 bytes in each direction (205,700 / 101,500 against 205,300 / 101,900) - a block-boundary difference, not a whole-file resend.

Note on reading the table: each cell reports only its first failing assertion, so daemon pull's literal/matched divergence is now masked behind the earlier assertion-3 failure and will resurface once the emit site is fixed. That ordering is deliberate - reporting the misrouted warning first points at the nearer cause.

None of the assertions were relaxed to get a green run. A red cell here is the correct result until the underlying behaviour is fixed.

Correction: the receiver-side warning is on stdout, not missing

An earlier revision of this description said daemon pull "never prints the warning, even under -v". That is wrong, and the assertion as first written could not tell the difference. Re-measured on the same fixture against /opt/homebrew/bin/rsync 3.4.4, counting the failed verification line per stream:

run	stdout	stderr
upstream daemon pull, default	0	0
upstream daemon pull, -v	0	1, bare payload.bin
oc daemon pull, default	1, absolute <dst>/payload.bin	0
oc daemon pull, -v	1, absolute <dst>/payload.bin	0

So oc emits the line on the wrong stream, with the wrong name form, at a verbosity where upstream is silent - three divergences at once. Because assertion 3 originally filtered stderr only, the default-verbosity cells compared [] == [] and passed that assertion vacuously; the -v cells failed with a message that read as "oc printed nothing".

Assertion 3 is therefore stream-aware, in the second commit on this branch: the oracle is an ExpectedWarnings { stdout, stderr } pair, stdout is empty at every verbosity (upstream rwrite() maps FWARNING to f = stderr, log.c:314 - verified in the C source, not assumed), and the failure message names the stream and calls out a misrouted line explicitly. Detection still uses the bare contains("failed verification") substring on both streams, so the absolute-path form is caught rather than filtered out as absent; the name shape is then judged by the exact-string comparison against upstream's captured line. Fixed on this branch rather than in a stacked follow-up, since the PR is still open and a vacuous assertion should not be merged and then patched.

Deviations from the sibling check's pattern

• The fixture is not backdated. append-inplace stamps a fixed mtime with touch -d @epoch because its reuse decision depends on mtime equality. Here the transfer is forced by the differing size plus --ignore-times, and every assertion (bytes, exit code, warning lines, delta split) is mtime-independent - so the stamp buys nothing, while depending on it makes the whole check skip on any host whose touch is the BSD one. Dropping it is what let the check actually run and produce the table above.
• The daemon is started per run, not per cell. A push must export the client's own destination, so the module differs between the upstream run and the oc run and cannot be shared. The handle is a local in the run, so it stays alive across output() and is killed on return.
• Category is Validation only, not Wire. The redo is a protocol round-trip, but every assertion here is client-observable behaviour (exit code, bytes, stderr, stats) rather than captured wire frames, which is what capability-string - the sole Wire member - actually inspects. Keeping it in Validation also means a bare cargo xtask validate gates it, which is the point.

Verification

• cargo fmt --all -- --check clean
• cargo clippy -p xtask --all-targets --all-features --no-deps -- -D warnings clean (no allow waivers)
• cargo clippy --locked --workspace --all-targets --all-features --no-deps -- -D warnings clean (the full CI invocation, re-run on the head commit)
• cargo nextest run -p xtask --all-features -E 'test(verify_redo)' - 10 passed. The full -p xtask run is 433/434; the one failure is the pre-existing macOS-only validate::support::tests::backdated_tree_populates_and_backdates_the_root, which needs a GNU touch -d @epoch and is untouched by this change
• cargo xtask validate --list shows verify-redo [validation]; --help unchanged
• the matrix was run end to end over the local and daemon transports, on the head commit, to produce the table above