Skip to content

feature(SourceId): use stable hash from rustc-stable-hash - #14917

Merged
epage merged 2 commits into
rust-lang:masterfrom
weihanglo:stable-hash
Dec 11, 2024
Merged

feature(SourceId): use stable hash from rustc-stable-hash#14917
epage merged 2 commits into
rust-lang:masterfrom
weihanglo:stable-hash

Conversation

@weihanglo

@weihanglo weihanglo commented Dec 10, 2024 •
edited by epage
Loading

Copy link
Copy Markdown
Member

What does this PR try to resolve?

This helps -Ztrim-paths build a stable cross-platform path for the
registry and git sources. Sources files then can be found from the same
path when debugging.

It also helps cache registry index all at once for all platforms,
for example the use case in #14795
(despite they should use cargo vendor instead IMO).

Some caveats:

  • Newer cargo will need to re-download files for global caches
    (index files, git/registry sources).
    The old cache is still kept and used when running with older cargoes.
  • Absolute paths on windows iarenot really covered by the "cross-platform" hash,
    because path prefix components like C: are always there.
    That means hashes of some sources kind,
    like local registry and local path,
    are not going to be real cross-platform stable.

Security concern

There might be hash collisions if you have two registries under the same
domain. This won't happen to crates.io, as the infra would have to
intentionally put another registry on index.crates.io to collide.
We don't consider this is an actual threat model, so we are not going to
use any cryptographically secure hash algorithm like BLAKE3.

At least, the current unstable SipHash isn't in a better situation.
We might switch to a cryptographic secure one when needed.

See also #13171 (comment)

How should we test and review this PR?

We have an FCP in #14795 (comment).

This PR implements the proposal,
The path-length concern in #14795 (comment) is automatically addressed
because we don't need cryptographically secure hash for now.

Additional information

See more information and benchmark results in #14116.

@rustbot

rustbot commented Dec 10, 2024

Copy link
Copy Markdown
Collaborator

r? @ehuss

rustbot has assigned @ehuss.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot rustbot added A-cache-messages Area: caching of compiler messages A-layout Area: target output directory layout, naming, and organization A-rebuild-detection Area: rebuild detection and fingerprinting S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 10, 2024
@weihanglo weihanglo changed the title refactor: make room for adapting rustc-stable-hasher feature(SourceId): use stable hash from rustc-stable-hash Dec 10, 2024
@weihanglo weihanglo mentioned this pull request Dec 10, 2024
Closed
epage
epage reviewed Dec 10, 2024
View reviewed changes
Comment thread src/cargo/core/compiler/build_runner/compilation_files.rs
epage
epage reviewed Dec 10, 2024
View reviewed changes
Comment thread src/cargo/util/hasher.rs
epage
epage reviewed Dec 10, 2024
View reviewed changes
Comment thread src/cargo/core/source_id.rs
@weihanglo
feat(SourceId): use stable hash from rustc-stable-hash
5eb7480 
This helps `-Ztrim-paths` build a stable cross-platform path for the
registry and git sources. Sources files then can be found from the same
path when debugging.

It also helps cache registry index all at once for all platforms,
for example the use case in rust-lang#14795
(despite they should use `cargo vendor` instead IMO).

Some caveats:

* Newer cargo will need to re-download files for global caches
  (index files, git/registry sources).
  The old cache is still kept and used when running with older cargoes.
* Windows is not really covered by the "cross-platform" hash,
  because path prefix components like `C:` are always there.
  That means hashes of some sources kind,
  like local registry and local path,
  are not going to be real cross-platform stable.

There might be hash collisions if you have two registries under the same
domain. This won't happen to crates.io, as the infra would have to
intentionally put another registry on index.crates.io to collide.
We don't consider this is an actual threat model, so we are not going to
use any cryptographically secure hash algorithm like BLAKE3.

See also <rust-lang#13171 (comment)>
weihanglo
weihanglo commented Dec 10, 2024
View reviewed changes
Comment thread tests/testsuite/global_cache_tracker.rs
epage
epage approved these changes Dec 11, 2024
View reviewed changes

@epage epage left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FCP in #14795. We likely don't need to wait the 10 day waiting period. If something comes up, we have time to revert before the next release.

@epage
epage added this pull request to the merge queue Dec 11, 2024
@weihanglo weihanglo added the A-caching Area: caching of dependencies, repositories, and build artifacts label Dec 11, 2024
@weihanglo weihanglo mentioned this pull request Dec 11, 2024
Closed
14 of 19 tasks
Merged via the queue into rust-lang:master with commit 94d274d Dec 11, 2024
@weihanglo
weihanglo deleted the stable-hash branch December 11, 2024 20:07
@weihanglo weihanglo mentioned this pull request Dec 13, 2024
Merged
@uncomfyhalomacro uncomfyhalomacro mentioned this pull request Dec 14, 2024
Closed
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 14, 2024
@bors
Auto merge of rust-lang#134278 - weihanglo:update-cargo, r=weihanglo
af4e36e 
Update cargo

18 commits in 20a443231846b81c7b909691ec3f15eb173f2b18..7847c03965260b5dcc8d93218d6af295a717abb6
2024-12-06 21:56:56 +0000 to 2024-12-13 18:06:39 +0000
- fix(base): Support bases in patches in virtual manifests  (rust-lang/cargo#14931)
- fix(resolver): Report invalid index entries  (rust-lang/cargo#14927)
- feat: Implement `--depth workspace` for `cargo tree` command (rust-lang/cargo#14928)
- fix(resolver): In errors, show rejected versions over alt versions (rust-lang/cargo#14923)
- fix: emit_serialized_unit_graph uses the configured shell (rust-lang/cargo#14926)
- fix(script): Don't override the release profile (rust-lang/cargo#14925)
- feature(SourceId): use stable hash from rustc-stable-hash (rust-lang/cargo#14917)
- fix(resolver): Don't report all versions as rejected  (rust-lang/cargo#14921)
- fix(resolver): Report unmatched versions, rather than saying no package (rust-lang/cargo#14897)
- fix(build-rs): Implicitly report rerun-if-env-changed for input (rust-lang/cargo#14911)
- a faster hash for ActivationsKey (rust-lang/cargo#14915)
- feat(build-script): Pass CARGO_CFG_FEATURE  (rust-lang/cargo#14902)
- fix(build-rs): Correctly refer to the item in assert (rust-lang/cargo#14913)
- chore: update auto-label to include build-rs crate (rust-lang/cargo#14912)
- refactor: use Path::push to construct remap-path-prefix (rust-lang/cargo#14908)
- feat(build-rs): Add the 'error' directive (rust-lang/cargo#14910)
- fix(build-std): determine root crates by target spec `std:bool` (rust-lang/cargo#14899)
- SemVer: Add section on RPIT capturing (rust-lang/cargo#14849)
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 14, 2024
@bors
Auto merge of rust-lang#134278 - weihanglo:update-cargo, r=weihanglo
85641f7 
Update cargo

19 commits in 20a443231846b81c7b909691ec3f15eb173f2b18..769f622e12db0001431d8ae36d1093fb8727c5d9
2024-12-06 21:56:56 +0000 to 2024-12-14 04:27:35 +0000
- test(build-std): dont require rustup (rust-lang/cargo#14933)
- fix(base): Support bases in patches in virtual manifests  (rust-lang/cargo#14931)
- fix(resolver): Report invalid index entries  (rust-lang/cargo#14927)
- feat: Implement `--depth workspace` for `cargo tree` command (rust-lang/cargo#14928)
- fix(resolver): In errors, show rejected versions over alt versions (rust-lang/cargo#14923)
- fix: emit_serialized_unit_graph uses the configured shell (rust-lang/cargo#14926)
- fix(script): Don't override the release profile (rust-lang/cargo#14925)
- feature(SourceId): use stable hash from rustc-stable-hash (rust-lang/cargo#14917)
- fix(resolver): Don't report all versions as rejected  (rust-lang/cargo#14921)
- fix(resolver): Report unmatched versions, rather than saying no package (rust-lang/cargo#14897)
- fix(build-rs): Implicitly report rerun-if-env-changed for input (rust-lang/cargo#14911)
- a faster hash for ActivationsKey (rust-lang/cargo#14915)
- feat(build-script): Pass CARGO_CFG_FEATURE  (rust-lang/cargo#14902)
- fix(build-rs): Correctly refer to the item in assert (rust-lang/cargo#14913)
- chore: update auto-label to include build-rs crate (rust-lang/cargo#14912)
- refactor: use Path::push to construct remap-path-prefix (rust-lang/cargo#14908)
- feat(build-rs): Add the 'error' directive (rust-lang/cargo#14910)
- fix(build-std): determine root crates by target spec `std:bool` (rust-lang/cargo#14899)
- SemVer: Add section on RPIT capturing (rust-lang/cargo#14849)
@rustbot rustbot added this to the 1.85.0 milestone Dec 14, 2024
github-actions Bot pushed a commit to rust-lang/miri that referenced this pull request Dec 15, 2024
@bors
Auto merge of #134278 - weihanglo:update-cargo, r=weihanglo
cbe0700 
Update cargo

19 commits in 20a443231846b81c7b909691ec3f15eb173f2b18..769f622e12db0001431d8ae36d1093fb8727c5d9
2024-12-06 21:56:56 +0000 to 2024-12-14 04:27:35 +0000
- test(build-std): dont require rustup (rust-lang/cargo#14933)
- fix(base): Support bases in patches in virtual manifests  (rust-lang/cargo#14931)
- fix(resolver): Report invalid index entries  (rust-lang/cargo#14927)
- feat: Implement `--depth workspace` for `cargo tree` command (rust-lang/cargo#14928)
- fix(resolver): In errors, show rejected versions over alt versions (rust-lang/cargo#14923)
- fix: emit_serialized_unit_graph uses the configured shell (rust-lang/cargo#14926)
- fix(script): Don't override the release profile (rust-lang/cargo#14925)
- feature(SourceId): use stable hash from rustc-stable-hash (rust-lang/cargo#14917)
- fix(resolver): Don't report all versions as rejected  (rust-lang/cargo#14921)
- fix(resolver): Report unmatched versions, rather than saying no package (rust-lang/cargo#14897)
- fix(build-rs): Implicitly report rerun-if-env-changed for input (rust-lang/cargo#14911)
- a faster hash for ActivationsKey (rust-lang/cargo#14915)
- feat(build-script): Pass CARGO_CFG_FEATURE  (rust-lang/cargo#14902)
- fix(build-rs): Correctly refer to the item in assert (rust-lang/cargo#14913)
- chore: update auto-label to include build-rs crate (rust-lang/cargo#14912)
- refactor: use Path::push to construct remap-path-prefix (rust-lang/cargo#14908)
- feat(build-rs): Add the 'error' directive (rust-lang/cargo#14910)
- fix(build-std): determine root crates by target spec `std:bool` (rust-lang/cargo#14899)
- SemVer: Add section on RPIT capturing (rust-lang/cargo#14849)
@UebelAndre UebelAndre mentioned this pull request Jan 10, 2025
Closed
@mizar mizar mentioned this pull request Feb 19, 2025
Closed
@ehuss ehuss mentioned this pull request Feb 21, 2025
Open
smoelius added a commit to trailofbits/dylint that referenced this pull request Feb 21, 2025
@smoelius
Account for Cargo's new hashing algorithm
eb06565 
See: rust-lang/cargo#14917
smoelius added a commit to trailofbits/dylint that referenced this pull request Feb 21, 2025
@smoelius
Account for Cargo's new hashing algorithm
72ada8b 
See: rust-lang/cargo#14917
github-merge-queue Bot pushed a commit to trailofbits/dylint that referenced this pull request Feb 21, 2025
@smoelius
Account for Cargo's new hashing algorithm
a87c4e0 
See: rust-lang/cargo#14917
augustin-v pushed a commit to augustin-v/dylint that referenced this pull request Feb 24, 2025
@smoelius @augustin-v
Account for Cargo's new hashing algorithm
c6358de 
See: rust-lang/cargo#14917
@cbgbt cbgbt mentioned this pull request Feb 26, 2025
Closed
@ncatelli ncatelli mentioned this pull request Aug 15, 2025
Closed
@weihanglo weihanglo mentioned this pull request Sep 18, 2026
Merged
3 of 6 tasks
github-actions Bot pushed a commit to codeandsolder/cargo-ephemeral that referenced this pull request Oct 6, 2026
@epage
feat(trim-paths): stabilize profile.trim-paths (rust-lang#17488)
784be26 
*[View all
comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/cargo/pull/17488)*

# Stabilization report: `profile.trim-paths`

Resolves rust-lang#12137
Resolves rust-lang/rust#111540
RFC: <https://rust-lang.github.io/rfcs/3127-trim-paths.html>

## What is stabilized

The rustc side `--remap-path-scope` was already stabilized in Rust 1.95
via rust-lang/rust#147611

This stabilizes the Cargo side:

* `profile.<name>.trim-paths = "none" | "object" | "all"`
  in both manifest and config
* The remap rules of how Cargo passes `--remap-path-{prefix,scope}`.
  The exact remap prefixes still stay unspecified.
* The unremap file `<artifact>.trim-paths.json` (schema v1),
  which is emitted beside final artifacts when debuginfo is on.
* `CARGO_TRIM_PATHS_SCOPE` and `CARGO_TRIM_PATHS_REMAP` for build
scripts.

When this is merged and sync in rust-lang/rust,
we'll also stabilize

* The `rust-gdb` and `rust-lldb` unremap loaders.
`RUST_GDB_TRIM_PATHS=unstable` and `RUST_LLDB_TRIM_PATHS=unstable` are
not needed anymore.

See doc for details:

https://github.com/rust-lang/cargo/blob/8814ead110e36ed8fdcf1fdd4009baf82bd78523/doc/book/src/reference/unstable.md?plain=1#L1441-L1645

## What is not stabilized / included

* This doesn't guarantee full sanitization. It is a best-effort feature.
* Other `--remap-path-scope` values in rustc (`macro`, `diagnostics`,
`debuginfo`, `coverage`),
  boolean values, and comma-separated list options.
  These are removed in rust-lang#17432.
  They can come back later when needed.
* A default trim-path value for built-in profiles.
  RFC originall proposed to set `release` to `"object"`.
  This is left for future when this is more adopted and battle-tested.
We have loose stability guarantee for changing profile settings anyway.
* `__CARGO_RUSTC_BOOTSTRAP_WS_REMAP`.
  This stays as an internal thing between rustc bootstrap and cargo
  (see rust-lang#17349, rust-lang#17366)
* Doctest remapping and `documentation` scope.
  This will be integrated in the future incrementally
  when those scopes and features are stable.
* The exact remap prefixes are unspecified as documented.
  However, in practice,
  rustc bootstrap and debugger depend on the stabilized shape,
  so any change needs careful coordination with them.
* Unremap files for artifact deps: deferred, non-blocking.
* `build-rs` API for the two build script variables: deferred,
non-blocking.
* If there are new kinds of artifacts, we can decide whether to remap
freely.

### Doors closed

* The trim-paths profile key name, its shape, and its options.
* The unremap file name suffix `.trim-paths.json` and the v1 schema.
* The environment variable `CARGO_TRIM_PATHS_SCOPE` and
`CARGO_TRIM_PATHS_REMAP`.

## Post-RFC changes

* rustc removed `split-debuginfo` scopes,
  and Cargo followed and stopped caring split debuginfo.
* The RFC remapped the current package to relative paths and every
dependency to `<name>-<version>`.
  In rust-lang#17302 we chose workspace members relative remap,
  so debuggers resolve workspace sources with zero configuration.
The RFC worried that relative paths only work when running from the
right directory,
and symbolication tools need a second process for joining workspace
relative paths.
THe unremap file has `workspace_root`, so the join is fairly mechanical.
  Remap prefixes are unspecified anyway,
  so we can still change if it turns out not ideal.
* The unremap file is new (introduced in rust-lang#17303),
  for helping debugging find sources,
  as well as our keeping remap rules unspecified.
  The RFC had no answer about this.
* `CARGO_TRIM_PATHS_REMAP` build script env is new.
It lets build scripts forward the same rules to C/C++ compilers to flags
like `-fmacro-prefix-map`.
  `cc-rs` has integrated that since rust-lang/cc-rs#1794.

## Feedback

* rustc bootstrap builds the compiler and standard library with
`trim-paths`
  via rust-lang/rust#161049 since 2026-09-02.
This exercises the sysroot remap and the workspace prefix override in
rust-lang/rust CI.
* `cc-rs` forwards the remap rules to C/C++ compiler since 1.3.0:
<https://github.com/rust-lang/cc-rs/releases/tag/cc-v1.3.0>
* Zulip thread: [#t-cargo > stabilization plan for
&rust-lang#96;-Ztrim-paths&rust-lang#96;](https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/stabilization.20plan.20for.20.60-Ztrim-paths.60/with/622955580)
* Call for testing posted on 2026-09-09
*
rust-lang#12137 (comment)
* Included in TWiR issue 669
<https://this-week-in-rust.org/blog/2026/09/16/this-week-in-rust-669/>

## Known limitations

This sanitization is best-effort. See

* rustc:
<https://doc.rust-lang.org/rustc/remap-source-paths.html#caveats-and-limitations>
* Cargo:
<https://github.com/rust-lang/cargo/blob/8814ead110e36ed8fdcf1fdd4009baf82bd78523/doc/book/src/reference/unstable.md?plain=1#L1603-L1626>

## Implementation

### History

| PR | Merged | Title |

|--------|------------|-------------------------------------------------|
| rust-lang#12625 | 2023-10-31 | implement RFC 3127 `-Ztrim-paths` |
| rust-lang#12900 | 2023-10-31 | set env `CARGO_TRIM_PATHS` for build scripts |
| rust-lang#12908 | 2023-11-02 | merge `trim-paths` from different profiles |
| rust-lang#13118 | 2023-12-06 | assert `OSO` and `SO` cannot be trimmed |
| rust-lang#14389 | 2024-08-12 | rustdoc supports trim-paths for diagnostics |
| rust-lang#14908 | 2024-12-09 | use Path::push to construct remap-path-prefix |
| rust-lang#14917 | 2024-12-11 | use stable hash from rustc-stable-hash |
| rust-lang#15614 | 2025-06-02 | remap all paths to `build.build-dir` |
| rust-lang#15621 | 2025-06-02 | enable more tests for windows-msvc |
| rust-lang#16536 | 2026-01-21 | `--remap-path-scope` stabilized in 1.95-nightly
|
| rust-lang#17104 | 2026-06-15 | emit `CARGO_TRIM_PATHS_REMAP` for build.rs |
| rust-lang#17221 | 2026-07-15 | exercise GDB on windows-gnu |
| rust-lang#17302 | 2026-08-03 | unambiguous and reversible remap rules |
| rust-lang#17303 | 2026-08-04 | emit unremap files for final artifacts |
| rust-lang#17326 | 2026-08-07 | exercise unremap files with debuggers |
| rust-lang#17338 | 2026-08-08 | `/cargo/deps` fallback sources |
| rust-lang#17337 | 2026-08-10 | workspace remap under -Zroot-dir |
| rust-lang#17349 | 2026-08-11 | honor workspace prefix override from env |
| rust-lang#17366 | 2026-08-26 | custom workspace-relative member paths remap |
| rust-lang#17424 | 2026-09-02 | remove default scope from release profile |
| rust-lang#17425 | 2026-09-02 | docs: add limitations and polish |
| rust-lang#17432 | 2026-09-04 | limit options to `none\|object\|all` |
| rust-lang#17476 | 2026-09-15 | unremap file in one JSON doc |
| rust-lang#17491 | 2026-09-21 | `build-rs` support |

### Test coverage

* Remap for each dependency kind
* `"object"` with every `split-debuginfo` mode
* `"all"` diagnostics remapping for rustc and rustdoc
* The new build-script environment variables.
* Real world debugger exercises with GDB, LLDB, and CDB
* unremap files with rebuilds, `cargo clean`, JSON messages
* rustc bootstrap workspace prefix override
* `-Zbuild-std` backtraces show `/rustc/<hash>` paths

## Follow-ups after stabilization

* [ ] Revisit reproducibility issues, such as
  * rust-lang#13586
  * rust-lang#15122
  * rust-lang#7645
  * rust-lang#10915
Absolute paths of workspace and `CARGO_HOME` still get into
`-Cmetadata`/`-Cextra-filename`/fingerprints.
With `trim-paths = "object"` we might be able to also trim paths in
those places.
* [ ] Revisit a new default for built-in profiles e.g., `release`
* [ ] In rust-lang/rust stabilize loader logic in
`src/etc/gdb_trim_paths.py` and `src/etc/lldb_trim_paths.py`
* [x] A new issue for `build-rs` adding `CARGO_TRIM_PATHS_SCOPE` and
`CARGO_TRIM_PATHS_REMAP` support
* [ ] A new issue for supporting doctest remapping and documentation
scope.
* [ ] Track unremap files support in artifact dependencies tracking
issue

---

🤖 LLM disclosure: impl history was generated. heading was generated.
meats are human-written.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@epage epage epage approved these changes

Assignees

@ehuss ehuss

Labels

A-cache-messages Area: caching of compiler messages A-caching Area: caching of dependencies, repositories, and build artifacts A-layout Area: target output directory layout, naming, and organization A-rebuild-detection Area: rebuild detection and fingerprinting S-waiting-on-review Status: Awaiting review from the assignee but also interested parties.

Projects

None yet

Milestone

1.85.0

Development

Successfully merging this pull request may close these issues.

4 participants

@weihanglo @rustbot @epage @ehuss