@rust_crate at a package's top level can be precompiled (#339)

CHANGELOG.md

@@ -134,6 +134,97 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   is not a list of items (`include!("table.rs")` holding `[1, 2, 3]`), is noted
   on stderr and skipped as a missing `mod` target already was — never a failed
   scan.
+- **A package that uses `@rust_crate` at top level can be precompiled**
+  ([#339](https://github.com/AtelierArith/RustCall.jl/issues/339)). The macro
+  evaluated the generated module into an anonymous `Module` under `Main`, so
+  `Pkg.precompile()` of such a package failed with ``Evaluation into the closed
+  module `##RustCallCrateRuntime#N` breaks incremental compilation``. The
+  module is now defined **inside the module that expands the macro**
+  (`load_crate_bindings(...; target_module = __module__)`), in a hidden,
+  per-call child namespace (`Caller.var"##RustCallCrateRuntime#N"`), so nothing
+  the caller did not name appears in its namespace and repeated calls never
+  collide (the #222 contract). The return value is unchanged, a
+  `RustCall.CrateBindings`; `load_crate_bindings` called without a
+  `target_module` keeps the anonymous module. Two consequences for the
+  generated module: `_LIB_PATH` of an
+  in-memory `@rust_crate` module is now the **durable** library — RustCall's
+  cache copy, or Cargo's output — instead of the per-process generation copy,
+  which is made in `__init__` (as the written file already did since format 6),
+  because after precompilation `__init__` runs in a later session than the one
+  that generated the module (visible only through `Bindings.module_ref._LIB_PATH`);
+  and the module declares that library **and the crate's own input files** —
+  the set its artifact identity is computed from: the crate directory, every
+  local `path` dependency, a workspace root's manifest and lockfile, an
+  out-of-directory `[lib] path` — with `Base.include_dependency`. So editing
+  `src/lib.rs`, or `RustCall.clear_cache()`, makes the package's precompile
+  cache stale and the next `using` re-precompiles it and builds the crate
+  again, instead of `__init__` opening a path that is gone or the package
+  going on calling a build that no longer matches its source. Tracking the
+  library alone would not do the second of those: the library is
+  content-addressed, so a new build lands at a *different* path and leaves the
+  old file untouched (found in review of
+  [#351](https://github.com/AtelierArith/RustCall.jl/pull/351)). The
+  crate is built when the package is precompiled, nothing is written into the
+  package, and the library is not opened during precompilation (`__init__` is
+  deferred to load time), so the bindings are callable after the package's
+  `__init__`, not from its own top level. The generated module imports `Libdl`
+  through RustCall (`import RustCall.Libdl`), so the package does not need
+  `Libdl` among its dependencies.
+
+### Added
+- **`@rust_crate ... submodule="Bindings"`**
+  ([#339](https://github.com/AtelierArith/RustCall.jl/issues/339)) defines the
+  generated module in the calling module under that name, so a package can
+  `using .Bindings: f, T` from it — the idiom that pairs with the precompile
+  fix above, and the same shape as `include("generated/Bindings.jl")`. `name=`
+  is unchanged: it names the generated module and defines nothing, which is
+  what keeps the documented `const MyBindings = @rust_crate path name="MyBindings"`
+  working. The two are separate options on purpose: an earlier cut of this
+  change made `name=` define the module, and a package written that way
+  precompiled and then **segfaulted** on load, because the constant was bound
+  over the module binding the macro had just created (found in review of
+  [#351](https://github.com/AtelierArith/RustCall.jl/pull/351)).
+- **A changed build environment is reported rather than ignored**
+  ([#339](https://github.com/AtelierArith/RustCall.jl/issues/339)). `RUSTFLAGS`,
+  `PYO3_PYTHON` and a `PYO3_CONFIG_FILE` pointing at another file decide the
+  artifact but are not files, so Julia — which invalidates a precompile image
+  from files — keeps the image and the package loads a library built under the
+  previous values. The generated module records the environment it was built
+  under (`artifact_build_env`) and `__init__` warns when it no longer matches,
+  naming the variables that changed and how to force a rebuild.
+  [#355](https://github.com/AtelierArith/RustCall.jl/issues/355) tracks
+  representing such inputs in the invalidation scheme itself.
+- **A plain crate's cache key covers the build environment**
+  ([#339](https://github.com/AtelierArith/RustCall.jl/issues/339)).
+  `compute_crate_hash` was called without `build_env` on the non-PyO3 path, so
+  two `cargo build`s under different `RUSTFLAGS` — or a different `CC` a build
+  script reads, or anything else in the #282 allowlist — shared one cache entry
+  and the second was handed the first one's library. The PyO3 wrapper path
+  already folded `artifact_build_env()` in; the plain path does now too, and
+  like the wrapper it hashes the *contents* of `PYO3_CONFIG_FILE` on top —
+  the allowlist records the path, and a plain build of a crate that depends on
+  pyo3 reads the file, so an in-place edit of the configuration is a different
+  binary under the same key (`_plain_crate_build_env`). One
+  consequence is that the load-time warning above can be acted on: forcing the
+  package to be precompiled again really does rebuild the artifact, instead of
+  finding the stale one under the same key.
+- **`@rust_crate <crate> cache=false` on a crate that RustCall has to wrap**
+  ([#339](https://github.com/AtelierArith/RustCall.jl/issues/339)). A crate
+  whose `[lib]` is not a `cdylib` is bound through a generated wrapper project
+  in a temporary directory, which is deleted as soon as the build returns.
+  With caching on, the library had already been copied into the cache; with
+  `cache = false` nothing copied it, so the generated module named a file that
+  no longer existed and loading it failed with `could not load library
+  ".../rustcall_wrapper_XXXXXX/target/release/..."`. The library is now taken
+  out of the wrapper project before the cleanup — into the cache, or into a
+  directory of its own under the Cargo cache that outlives the process (a
+  package precompiled with `cache = false` is loaded by a *later* process, and
+  a `mktempdir()` cleaned at exit would have taken the recorded `_LIB_PATH`
+  with it; the PyO3 wrapper path had the same `mktempdir()` and uses the same
+  home now). `cache = false` is still not the shape to use inside a
+  package: `docs/src/crate_bindings.md` says which path the module then carries
+  and what makes its precompile cache stale.
+
 
 ## [0.3.0] - 2026-09-08
 

docs/src/crate_bindings.md

@@ -503,7 +503,92 @@ MyMath.fibonacci(UInt32(20))  # => 6765
 
 ## Precompilation Support
 
-For package development, you can generate bindings to a file that will be precompiled with your package, improving startup time.
+A package can carry a crate's bindings in two ways, and both precompile: use
+`@rust_crate` at the package's top level, or write the bindings to a file with
+`write_bindings_to_file` and `include` it.
+
+### Using `@rust_crate` inside a package
+
+```julia
+module MyPackage
+using RustCall
+
+# Build the crate and generate its bindings; define them as `MyPackage.Bindings`.
+@rust_crate joinpath(@__DIR__, "..", "deps", "my_rust_crate") submodule="Bindings"
+using .Bindings: add, multiply, MyStruct
+
+export add, multiply, MyStruct
+end
+```
+
+The generated module is evaluated **inside the module that expands the macro**
+(#339), so it is part of `MyPackage` and is compiled into `MyPackage`'s
+precompile cache like any other submodule. What happens, and when:
+
+- **When the package is precompiled** (the first `using`, or
+  `Pkg.precompile()`): `@rust_crate` scans the crate, builds it — the library
+  goes into RustCall's cache under the depot's scratch space — and generates the
+  module. Nothing is written into the package. The library is *not* opened at
+  this point: Julia defers the generated module's `__init__` to load time, so
+  the bindings are callable after `MyPackage.__init__`, not from the package's
+  own top level.
+- **When the package is loaded** from its cache: the generated module's
+  `__init__` opens the cached library (through a private per-process copy, so
+  Cargo's output and the cache copy stay free to be rebuilt, #309). No
+  scanning, no Cargo.
+- **After an edit to the crate, or `RustCall.clear_cache()`**: the module
+  declared the library *and the crate's own input files* — the set the artifact
+  identity is computed from, so a `path` dependency and a workspace root count
+  too — with `Base.include_dependency`, so the
+  package's precompile cache is stale and the next `using` re-precompiles the
+  package, building the crate again. Deterministic, and never a failed
+  `dlopen` of a path that is gone.
+
+One thing Julia's invalidation cannot see: **the environment**. `RUSTFLAGS`,
+`PYO3_PYTHON`, a `PYO3_CONFIG_FILE` pointing at a *different* file — all of them
+decide the artifact, and none of them is a file the image can track. Change one
+and every tracked file is still what it was, so the image stays valid and the
+package loads the library built under the previous values. The generated module
+records the values it was built under and `@warn`s at load time when they no
+longer match, naming the variables; the fix is to precompile the package again
+(`Pkg.precompile(; force = true)`, or touch a source file of the crate).
+
+`cache=false` is not the shape to use in a package. The library is then not
+the cache copy but whatever the build produced: Cargo's own output under the
+crate's `target/` for a crate that is already a `cdylib`, and a copy in a
+directory of its own under RustCall's Cargo cache — one the cache lookup never
+returns, and that only `RustCall.clear_cache()` removes — for a crate RustCall
+has to wrap. The copy outlives the process that made it on purpose: a package
+precompiled with `cache=false` is loaded by another process, which must still
+find the file. In the first case the next `cargo build` of the crate
+invalidates the package's cache; in the second every precompilation leaves a
+copy behind until the cache is cleared.
+
+The naming rule: **`submodule="Bindings"` is what defines** the module as
+`MyPackage.Bindings`, which is what `using .Bindings: ...` needs. Without it
+the module gets a hidden, per-call name (`MyPackage.var"##RustCallCrateRuntime#N"...`)
+and is reached only through the value the macro returns — nothing the caller
+did not name appears in its namespace. `name="Bindings"` chooses that hidden
+module's name and still defines nothing, which is why
+`const MyBindings = @rust_crate path name="MyBindings"` — the form the macro's
+docstring has always shown — keeps working: the constant is the only binding
+the caller gets.
+
+A second `@rust_crate ... submodule="Bindings"` in the same module replaces the
+module (Julia warns `replacing module Bindings`); values obtained earlier keep
+the module they hold. Do not write `const Bindings = @rust_crate path submodule="Bindings"`:
+that binds the returned value over the module the macro just defined. The
+return value is the same `CrateBindings` in every position — REPL, function
+body, package — so a package may also keep it: `const B = @rust_crate path`
+gives `B.add(...)`, world-age-safe, without any visible module. The generated
+module needs only `RustCall` among the package's dependencies (it reaches
+`Libdl` through RustCall).
+
+Prefer this shape when the machine that loads the package has a Rust toolchain
+and building the crate on first use is acceptable; the bindings can never be
+out of date with respect to the crate. Prefer the written file below when the
+package must load without Rust installed, or when the bindings and the library
+must be inspected, committed or shipped as files.
 
 ### Generating Bindings to a File
 
@@ -650,6 +735,13 @@ If you encounter precompilation issues:
 - Ensure the library path is correct (use `relative_lib_path` for portable packages)
 - Check that the library was copied to the correct location
 - Verify the generated code compiles without errors
+- With `@rust_crate` at a package's top level: the package must depend on
+  `RustCall`, the crate must build on the machine that precompiles the
+  package, and calls into the bindings belong after the package's `__init__`
+  (the library is not open while the package's top level runs during
+  precompilation). `Base.isprecompiled(Base.identify_package("MyPackage"))`
+  says whether the cache is currently valid; it turns `false` after
+  `RustCall.clear_cache()` until the next `using` rebuilds.
 
 ## Object lifetime
 

docs/src/precompilation.md

@@ -17,16 +17,24 @@ When you use `@rust_crate` in a Julia package, the bindings can be precompiled a
 
 ### Runtime vs Precompile Time
 
-**Without precompilation** (using `@rust_crate` directly):
-1. Julia loads your package
-2. `@rust_crate` scans the Rust crate
-3. Rust code is compiled (if not cached)
-4. Bindings are generated and evaluated
-
-**With precompilation** (using `write_bindings_to_file`):
+Two ways to put a crate's bindings in a package, both precompiled:
+
+**`@rust_crate` at the package's top level** (`@rust_crate path submodule="Bindings"`
+followed by `using .Bindings: ...`; see "Using `@rust_crate` inside a package"
+in [Crate Bindings](crate_bindings.md)):
+1. During precompilation: `@rust_crate` scans the crate, builds it (into
+   RustCall's cache) and generates the module, which is compiled into the
+   package's cache; nothing is written into the package
+2. At runtime: the module's `__init__` opens the cached library
+3. After `RustCall.clear_cache()` or a rebuild of the crate: the package's
+   precompile cache is stale and the next `using` re-precompiles it
+
+**`write_bindings_to_file`** (a `deps/build.jl` writes the bindings and copies
+the library into the package):
 1. During development: Generate bindings file once
 2. During precompilation: Julia compiles the bindings module
-3. At runtime: Precompiled bindings load instantly
+3. At runtime: Precompiled bindings load instantly, from the library the
+   package carries — no Rust toolchain needed on the machine that loads it
 
 ### The Generation Process
 

examples/README.md

@@ -305,13 +305,15 @@ clear_cache()
 
 ### Module name confusion
 
-When using `@rust_crate`, the returned bindings object wraps a generated runtime module whose default name is the crate name converted to PascalCase.
+When using `@rust_crate`, the returned bindings object wraps a generated module whose default name is the crate name converted to PascalCase.
 
 Example: `sample_crate` → `SampleCrate`
 
-You can override that internal runtime module name with `name=`, while still using the value returned by `@rust_crate`:
+By default that module is hidden inside the calling module and is reached only through the returned value; `name=` chooses its name, and `submodule=` is what defines it in the calling module so a package can `using` from it (#339):
 ```julia
-const bindings = @rust_crate "/path/to/crate" name="MyCustomName"
+const bindings = @rust_crate "/path/to/crate" name="MyCustomName"        # nothing new is defined here
+@rust_crate "/path/to/crate" submodule="MyCustomName"                   # defines MyCustomName here
+using .MyCustomName: add
 ```
 
 ## Additional Resources

src/artifact_id.jl

@@ -904,6 +904,38 @@ function crate_input_files(dir::AbstractString)
     return "walk", files
 end
 
+"""
+    crate_input_dirs(dir::AbstractString) -> Vector{String}
+
+Every directory `crate_input_files` walks, `dir` itself included, as
+paths relative to `dir` (`"."` for the root). Same walk, same exclusions —
+Cargo's `target/` at the package root, VCS metadata at any depth — so the two
+cannot drift.
+
+Separate from the file list because a *directory* is what tells a consumer that
+a file appeared. `crate_content_digest` hashes the list of files, so creating
+the first file in a directory that was empty changes the artifact; the parent's
+own entry list does not move (the directory was already there), and neither
+does any file. A caller that tracks directories therefore has to know about the
+empty ones too (#339 review).
+"""
+function crate_input_dirs(dir::AbstractString)
+    dir = String(dir)
+    dirs = String[]
+    for (root, subdirs, _) in walkdir(dir)
+        at_root = _canonical_dir(root) == _canonical_dir(dir)
+        filter!(subdirs) do d
+            d in CRATE_INPUT_VCS_DIRS_ANY_LEVEL && return false
+            at_root && d == "target" && return false
+            return true
+        end
+        push!(dirs, replace(relpath(root, dir), '\\' => '/'))
+    end
+    unique!(dirs)
+    sort!(dirs)
+    return dirs
+end
+
 """
     local_path_dependency_dirs(root::AbstractString) -> (strategy::String, dirs::Vector{String})
 
@@ -972,6 +1004,19 @@ function _local_path_dependency_dirs_uncached(root::String)
         end
         if !isempty(found)
             append!(dirs, found)
+            # `cargo tree` resolves the *default* build's graph, and this
+            # function is called with no feature set: an optional `path`
+            # dependency that only `features = [...]` activates is not in
+            # `found`. Every local crate any manifest in the graph declares is
+            # added, transitively, optional or not — a crate the build *can*
+            # pull in is an input of the artifact, and an edit to it must
+            # change the key whether the current feature set pulls it in or
+            # not. Over-approximating costs a rebuild; under-approximating
+            # handed a stale library back under an unchanged key (#339 review).
+            seen = Set{String}()
+            for dir in copy(dirs)
+                _collect_manifest_path_deps!(dirs, dir, seen)
+            end
             unique!(dirs)
             return "cargo-tree", dirs
         end
@@ -1088,9 +1133,40 @@ function _declared_path_dependencies(manifest::AbstractString)::Vector{String}
             end
         end
     end
+    # `[patch.<source>] name = { path = "../local" }` replaces a registry or
+    # git dependency with a local crate — the dependency table itself still
+    # says `version = "..."`, so harvesting it alone never sees the directory.
+    # When the patched dependency is optional and a feature activates it, the
+    # default `cargo tree` graph omits it too, and an edit to the local crate
+    # changed neither the key nor the declared inputs (#339 review). Cargo
+    # honours `[patch]` in the workspace root's manifest (or a crate's own
+    # when it is its own root), and the paths are relative to the manifest
+    # that declares them.
+    _harvest_patch_paths!(out, parsed, dir)
+    root = _workspace_root_dir(dir)
+    if root !== nothing && _canonical_dir(root) != _canonical_dir(dir)
+        root_manifest = _parse_manifest_or_nothing(joinpath(root, "Cargo.toml"))
+        root_manifest isa AbstractDict && _harvest_patch_paths!(out, root_manifest, root)
+    end
     return sort!(unique!(out))
 end
 
+# Every `path` a `[patch.<source>]` table of `parsed` names, made absolute
+# against `dir`, the directory of the manifest that declares it.
+function _harvest_patch_paths!(out::Vector{String}, parsed::AbstractDict, dir::AbstractString)
+    patches = get(parsed, "patch", nothing)
+    patches isa AbstractDict || return nothing
+    for (_, per_source) in patches
+        per_source isa AbstractDict || continue
+        for (_, spec) in per_source
+            spec isa AbstractDict || continue
+            p = get(spec, "path", nothing)
+            p isa AbstractString && push!(out, abspath(joinpath(String(dir), String(p))))
+        end
+    end
+    return nothing
+end
+
 # `[workspace.dependencies]` of this manifest, of the workspace its
 # `[package] workspace = "..."` names, else of the nearest ancestor manifest
 # that declares a workspace. Returns the table together with the directory of