rustcall-extract without --crate-root: which scan position wins for an implicitly reached file

[terasakisatoshi]

Split out of the review of PR #352 (#343).

The mode

rustcall-extract manifest|wrap --mode crate without --crate-root treats every FILE argument as its own module root. Nothing in RustCall.jl reaches it: scan_crate (src/crate_bindings.jl) and both wrap_crate call sites (src/pyo3.jl) pass crate_root = lib_root. It is a CLI affordance for callers that enumerate .rs files themselves.

#343 gave that mode something it did not have before: files reached implicitly, through an include! and through the mod declarations inside a fragment. A listed file's own mod is still the caller's to list — without a root there is no tree to place it in — but a fragment is not something a caller can name, so it and what it declares are followed. That is the right behaviour, and it opens a class of questions the mode has never had to answer.

The class

1. Which scan wins when a file is both listed and reachable implicitly. Follow out-of-line mod declarations inside include!d files (#343) #352 makes the caller's list win, which fixes the duplicate-symbol failure it was written for (rustcall_deep claimed twice) but keeps the unpositioned scan. For PyO3 the position is the answer: with pub mod api { include!("frag.rs"); } and a #[pyfunction] in frag.rs, the listed-root scan records an empty module_path, so function_wrapper generates crate_name::f instead of crate_name::api::f and the wrapper does not compile. The positioned scan is the one to keep, and the explicit root scan the one to suppress.

2. What identifies a visited file. The key is (canonical path, module_path). FilePosition now also carries reachable, enclosing_cfg, symbol_path and marked. Under a lenient scan, #[cfg(feature = "x")] pub mod api { include!("frag.rs"); } beside #[cfg(not(feature = "x"))] mod api { include!("frag.rs"); } queues one fragment at one module path with two different positions; only the LIFO winner is scanned, so a #[pyfunction] can inherit the private branch's skip reason and crate_needs_pyo3_wrapper then declines to generate a wrapper for a build that does enable x.

Both are visible failures — a wrapper that does not compile, or one that is not generated — not silently wrong bindings, and neither is reachable from Julia.

Acceptance

• a file that is both listed and reachable through an include! is scanned once, at the position the include! gives it, and the manifest records that module_path;
• two cfg-exclusive modules including one fragment yield both positions in a lenient scan;
• the duplicate-symbol case Follow out-of-line mod declarations inside include!d files (#343) #352 fixed stays fixed (extract_manifest([lib, frag]; mode = "crate") yields each item once);
• the crate-root walk is unaffected, and pulled_in stays the one step both walks share.

[terasakisatoshi]

MWE (verified)

Both cases reproduced on main at 47a3473 with the extractor built from that tree.

1. A listed file loses the position its include! gave it

D=$(mktemp -d); mkdir -p $D/src
cat > $D/src/frag.rs <<'EOF'
#[pyfunction]
pub fn shout(s: String) -> String { s }
EOF
cat > $D/src/lib.rs <<'EOF'
pub mod api {
    include!("frag.rs");
}
EOF

EX=deps/rustcall_extract/target/release/rustcall-extract

# (1) the crate-root walk — what RustCall uses
$EX manifest --mode crate --skip-unparsable --crate-root $D/src/lib.rs | grep -E '^name|^module_path'
# (2) rootless, only the including file listed
$EX manifest --mode crate --skip-unparsable $D/src/lib.rs           | grep -E '^name|^module_path'
# (3) rootless, BOTH files listed
$EX manifest --mode crate --skip-unparsable $D/src/lib.rs $D/src/frag.rs | grep -E '^name|^module_path'

(1)  name = "shout"   module_path = ["api"]     ← correct
(2)  name = "shout"   module_path = ["api"]     ← correct
(3)  name = "shout"   module_path = []          ← the listed root scan wins

In case (3) a wrapper crate generated from that manifest calls crate_name::shout instead of crate_name::api::shout, and does not compile. This is the rootless mode only; RustCall always passes --crate-root (src/crate_bindings.jl:158, src/pyo3.jl:1266, :1766).

2. Two cfg-exclusive modules including one fragment — this one is not rootless-only

This case reaches the crate-root walk, so it is a defect in the path RustCall runs, and it has been split out as its own bug. It is left here only as the reason the visited key came under review; see that issue for the MWE and the fix.

Corrected scope

When I posted the scope decision on #352 I described both findings as belonging to the rootless mode. That is right for case 1 and wrong for case 2, which I only established by running it. Case 2 is now tracked separately as a bug; this issue keeps case 1, whose reasoning stands: the mode is a CLI affordance RustCall never uses, and its failure is a wrapper that does not compile rather than a wrong binding.

Acceptance (case 1)

• a file that is both listed and reachable through an include! is scanned once, at the position the include! gives it, and the manifest records that module_path;
• the duplicate-symbol case Follow out-of-line mod declarations inside include!d files (#343) #352 fixed stays fixed (extract_manifest([lib, frag]; mode = "crate") yields each item once);
• the crate-root walk is unaffected, and pulled_in stays the one step both walks share.