Skip to content

fix(xtask): pin the validate fidelity oracle to upstream rsync 3.4.4 - #7245

Merged
oferchen merged 6 commits into
masterfrom
fix/xtask-validate-pinned-oracle
Aug 13, 2026
Merged

fix(xtask): pin the validate fidelity oracle to upstream rsync 3.4.4#7245
oferchen merged 6 commits into
masterfrom
fix/xtask-validate-pinned-oracle

Conversation

@oferchen

@oferchen oferchen commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Problem

cargo xtask validate is the drop-in fidelity matrix: every check in it asserts
"oc-rsync matches upstream rsync". It resolved the "upstream" side of that
comparison from PATH:

fn pick_upstream(workspace: &Path) -> TaskResult<std::path::PathBuf> {
    if let Some(system) = system_rsync() {
        return Ok(system);
    }
    ...
}

fn system_rsync() -> Option<std::path::PathBuf> {
    let path_var = std::env::var_os("PATH")?;
    for dir in std::env::split_paths(&path_var) {
        let candidate = dir.join("rsync");
        if candidate.is_file() && /* --version exits 0 */ { return Some(candidate); }
    }
    None
}

The first rsync on PATH won, at any version, with no version check at all.
So a PASS meant "oc matches whatever rsync this host happens to have" - not
the contract - and a FAIL could be an oracle-version difference rather than an
oc defect. Measured: on the aarch64 Linux box the matrix has been run on,
/usr/bin/rsync is 3.4.1, so the whole matrix was scored against 3.4.1.

The fallback path could not have saved it either. It filters
UPSTREAM_VERSIONS, which still reads ["3.0.9", "3.1.3", "3.4.1"], while
tools/ci/run_interop.sh has moved to versions=(3.0.9 3.1.3 3.4.4). The
starts_with("3.4") preference therefore selects 3.4.1, and 3.4.4 was
unreachable by either branch.

Two more holes made the resolution unsafe rather than merely loose: nothing
verified the binary's version, and nothing verified it was rsync at all
(/usr/bin/rsync on macOS is openrsync).

Fix

The oracle becomes an explicit, verified input to the harness instead of
ambient state it sniffs. New xtask/src/commands/validate/oracle.rs:

  • Resolves the pinned build at
    target/interop/upstream-install/3.4.4/bin/rsync - the tree
    tools/ci/run_interop.sh already populates. No second discovery mechanism is
    introduced; the path construction and the install root are hoisted into
    interop::shared::upstream and shared with detect_upstream_binaries.
  • Verifies the release from the --version banner, not the path. This mirrors
    upstream_release_version() in tools/ci/run_interop.sh, which parses the
    banner for exactly the stated reason: "The install path is not authoritative:
    a directory named 3.4.4 may hold any binary, and /usr/bin/rsync on macOS is
    openrsync."
  • Refuses to run when the oracle is absent, unrunnable, not rsync, or the
    wrong release. A fidelity gate that cannot find its oracle must not guess.
  • Prints the resolved path and the banner once, before any result, so every
    PASS and FAIL is attributable to a named upstream release.
  • OC_RSYNC_VALIDATE_UPSTREAM=<path> overrides for deliberate cross-version
    comparison. The override is still probed and still reported - only the
    version requirement is relaxed. The default is strict.

resolve_with() takes the install root, the override, and the version probe as
parameters so the policy is exercisable without a real binary; production passes
the real probe at the single call site. That is the same dependency-inversion
shape as #7240, and for the same reason - the input that decides the outcome was
a global read several frames below the function under test. A plain parameter,
not a trait: one production value, one fixture value.

xtask is oc-specific build tooling. Upstream rsync uses autotools and has no
equivalent command, so there is no upstream behaviour to mirror here and no
upstream citation is claimed for this change. The run_interop.sh reference
above is a reuse citation for the banner-parsing rule, not an upstream-fidelity
claim.

Verification

Local only - GitHub Actions was in a confirmed outage while this was prepared.

cargo fmt --all -- --check                                   clean
cargo clippy --locked --workspace --all-targets
  --all-features --no-deps -- -D warnings                    clean
cargo nextest run --workspace --all-features -E 'test(oracle)'
                                                             7 tests run: 7 passed

Both directions. Relaxing only the two production strictness gates (making them
if false && ...) while keeping the tests turns the two that matter red:

7 tests run: 5 passed, 2 failed
  FAIL absent_pinned_binary_fails_with_a_named_diagnostic
  FAIL wrong_version_fails_and_names_both_versions

so they exercise the resolver rather than passing vacuously. End-to-end, with
real binaries:

oracle absent   -> upstream oracle missing: no rsync 3.4.4 at .../3.4.4/bin/rsync
                   Build it with `bash tools/ci/run_interop.sh`, or point
                   OC_RSYNC_VALIDATE_UPSTREAM at a binary ...
3.4.1 in slot   -> upstream oracle version mismatch: .../3.4.4/bin/rsync reports
                   3.4.1, the fidelity matrix is defined against rsync 3.4.4
correct 3.4.4   -> [validate] upstream oracle: rsync  version 3.4.4  protocol version 32
override=3.4.1  -> [validate] upstream oracle: rsync  version 3.4.1  protocol version 32

Matrix re-run: which failures were oracle artifacts

Same oc-rsync binary, same checks, oracle version as the only variable, both
upstream releases built from source on the same host. Two runs each, identical
results:

cell vs 3.4.1 vs 3.4.4
verbosity: local -v FAIL FAIL
verbosity: local -vv FAIL FAIL
verbosity: local -vvv FAIL FAIL
crtimes: daemon PASS FAIL
chmod: local D2755,F640 FAIL FAIL
chmod: daemon D2755,F640 FAIL FAIL
chown: daemon chown-self FAIL FAIL
106 passed, 6 failed 105 passed, 7 failed
  • The three verbosity failures survive verbatim - line 1: oc "a.txt" vs upstream "./" - so they are genuine oc divergences, not oracle artifacts.
  • crtimes: daemon passes against 3.4.1 and fails against 3.4.4
    (crtime differs at alpha: oc=1614830767 upstream=0). The wrong oracle was
    masking a real divergence. Not fixed here - out of scope for this change -
    but it is the concrete demonstration of the defect this PR closes.
  • The chmod/chown failures are host artifacts of this run, not oc defects:
    upstream itself exits 23 (failed to set permissions on ".": Operation not permitted) and the group mismatch is the macOS default-group rule. They are
    oracle-independent.

Caveats on this measurement, stated rather than hidden: it was run on macOS, so
the 10 ssh-transport cells skipped (no sshd on localhost:22), and it needed GNU
touch on PATH. That is a second, separate defect worth its own issue -
support::build_backdated_tree shells out to touch -h -d @<epoch>, which BSD
touch rejects, so on a stock macOS host 127 of 239 cells skip and the matrix
silently validates almost nothing. Not touched here.

@github-actions github-actions Bot added the bug Something isn't working label Aug 6, 2026
@oferchen
oferchen force-pushed the fix/xtask-validate-pinned-oracle branch from dd4904e to 558bf25 Compare August 7, 2026 01:35
@oferchen
oferchen force-pushed the fix/xtask-validate-pinned-oracle branch from 558bf25 to fc62caf Compare August 7, 2026 23:03
version_banner and parse_release_version are consumed only by
commands::validate::oracle, and validate declares `mod oracle` behind
#[cfg(unix)] because the fidelity matrix is unix-only. Their enclosing
module, interop::shared::upstream, is not gated, so on Windows both
functions had no caller and the deny(dead_code) build failed on the
Windows ACL/xattr, Windows GNU cross-check and Windows IOCP cells.

Gate them to match their sole consumer.

Verified with cargo check -p xtask --all-targets --target
x86_64-pc-windows-msvc (clean) and natively via fmt + clippy -D warnings.
Both sides added a module declaration at the same point in the alphabetical
list: this branch adds `mod oracle;`, master gained `mod skips;` from the
skip-ledger change. Both modules are required, so keep both and preserve the
ordering (oracle < skips < support).
@oferchen
oferchen marked this pull request as ready for review August 12, 2026 16:04
Every sibling module in the validate command carries #[cfg(unix)], and
skips.rs opens with `use super::unix_impl::{CheckOutcome, Status}`, a
module that is itself #[cfg(unix)]. Merging master into this branch
re-ordered the module list to place `mod oracle;` alphabetically and
dropped the attribute from `mod skips;` in the process, so on Windows
the module compiled while the type it imports was configured out.

That broke every Windows job that builds the workspace - not just the
cross-check - because all of them build xtask:

  error[E0432]: unresolved import `super::unix_impl`
    --> xtask/src/commands/validate/skips.rs:30:12

Verified with cargo check -p xtask for x86_64-pc-windows-msvc and for
the native unix target.
@oferchen
oferchen merged commit 6b20f26 into master Aug 13, 2026
55 checks passed
@oferchen
oferchen deleted the fix/xtask-validate-pinned-oracle branch August 13, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant