Precompile invalidation cannot represent the non-file inputs of a crate's artifact identity

[terasakisatoshi]

Split out of the review of PR #351 (#339), where the class became visible.

The class

A crate's artifact identity (compute_crate_hash, _pyo3_wrapper_build_env) is computed from more than files. RUSTFLAGS, PYO3_PYTHON, RUSTCALL_PYTHON_LIBDIR, a PYO3_CONFIG_FILE that now points at a different file, and the rest of the artifact_build_env allowlist all change what is built.

Julia invalidates a precompile image from files: Base.include_dependency tracks a file's content (or a directory's entry list), and that is the only lever a generated module has. So after #351 the image is invalidated by an edit to the crate, to a path dependency, to .cargo/config.toml, to PYO3_CONFIG_FILE itself, and by a file appearing or disappearing in any of those directories — but not by the environment changing. Every tracked file is then still byte-for-byte what it was, the image stays valid, and the package loads a library built for the other environment.

#351 makes the module record the environment it was generated under and @warn from __init__ when it no longer matches, naming the variables. That converts silence into a message; it does not invalidate anything.

What a real fix would need

Something the image can carry that changes when a non-file input does. Options, none of them free:

• a stamp file per (crate, environment) under RustCall's cache whose content is the artifact key, declared with include_dependency — the difficulty is that only a re-precompile rewrites it, which is the step being skipped, so the stamp has to be written by something that runs on every load, or keyed so that a new environment names a path that does not exist yet;
• Base.CacheFlags-style participation, if Julia gains a hook for package-defined invalidation predicates;
• refusing to load instead of warning, which is honest but breaks a workflow where the difference is deliberate.

Acceptance

• a package precompiled with one PYO3_PYTHON and loaded with another either rebuilds or refuses, rather than warning and continuing;
• the same for RUSTFLAGS;
• no spurious invalidation when nothing relevant changed — the @rust_crate precompile testsets already pin that an unchanged crate is not re-precompiled;
• docs/src/crate_bindings.md describes whatever rule replaces the warning.

[terasakisatoshi]

MWE

Reproduced on fix/rust-crate-precompile at b19703e (macOS, Julia 1.12.7). The package is built under one RUSTFLAGS and loaded under another; nothing else changes.

using RustCall
root = mktempdir(); crate = joinpath(root, "envcrate"); mkpath(joinpath(crate, "src"))
macros = joinpath(dirname(pathof(RustCall)), "..", "deps", "juliacall_macros")
write(joinpath(crate, "Cargo.toml"), """
[package]
name = "envcrate"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ["cdylib"]
[dependencies]
juliacall_macros = { path = "$(replace(abspath(macros), "\\" => "/"))" }
""")
write(joinpath(crate, "src", "lib.rs"), """
use juliacall_macros::julia;
#[julia]
pub fn one() -> i32 { 1 }
""")

name = "EnvProbe355"; uuid = "7c1d9e40-8b23-4a19-9f52-6d3e4c8a1b70"
pkg = joinpath(root, name); mkpath(joinpath(pkg, "src"))
write(joinpath(pkg, "Project.toml"),
      "name = \"$name\"\nuuid = \"$uuid\"\nversion = \"0.1.0\"\n\n[deps]\nRustCall = \"$(Base.PkgId(RustCall).uuid)\"\n")
write(joinpath(pkg, "src", "$name.jl"), """
module $name
using RustCall
@rust_crate $(repr(abspath(crate))) submodule="Bindings"
end
""")

cache = joinpath(root, "cache"); mkpath(cache)
probe = joinpath(root, "probe.jl")
write(probe, """
id = Base.identify_package("$name")
stale = !Base.isprecompiled(id)
using $name
print(stale, " ", basename($name.Bindings._LIB_PATH))
""")
run_it(flags) = withenv("JULIA_LOAD_PATH" => join((pkgdir(RustCall), root, "@stdlib"), ":"),
                        "RUSTCALL_CACHE_DIR" => cache, "RUSTFLAGS" => flags) do
    readchomp(pipeline(`$(Base.julia_cmd()) --startup-file=no $probe`; stderr = devnull))
end

println("A (opt-level=2): ", run_it("-C opt-level=2"))
println("B (opt-level=1): ", run_it("-C opt-level=1"))
println("A again:         ", run_it("-C opt-level=2"))

What it prints

A (opt-level=2): true 28c1a9f79fa87098c35ddd57c8ae5b393cd917d666ea15f3a29a3788aead8d04.dylib
B (opt-level=1): false 28c1a9f79fa87098c35ddd57c8ae5b393cd917d666ea15f3a29a3788aead8d04.dylib
A again:         false 28c1a9f79fa87098c35ddd57c8ae5b393cd917d666ea15f3a29a3788aead8d04.dylib

The first column is "was the image stale?", the second the library the module carries.

• A: no image yet, so it is built — stale = true, and the artifact is the one -C opt-level=2 produces.
• B: stale = false. Every file the image tracks is byte-for-byte what it was, so Julia keeps it, @rust_crate never runs, and the module loads the same 28c1a9f7… library — the one built under A. The -C opt-level=1 artifact is never built. A @warn naming RUSTFLAGS fires (that is what PR @rust_crate at a package's top level can be precompiled (#339) #351 added), and that is all that happens.
• A again: still valid, still the same library — correct this time, by luck.

Note the cache key does separate the two since #351 (compute_crate_hash(...; build_env = artifact_build_env())), and so does the registry name. That is exactly the point: the artifact identity knows the two builds differ, and the invalidation scheme cannot say so, because it has only files to speak with.

What a fix has to beat

The obvious one — a stamp file under RustCall's cache whose content is the artifact key, declared with include_dependency — is circular: only @rust_crate writes it, and @rust_crate is the step being skipped. Whatever replaces the warning has to be evaluated on the load side, or has to make the recorded dependency itself a function of the environment.