feat: stream `.rmeta` artifact notifications for rustc by weihanglo · Pull Request #2875 · mozilla/sccache

weihanglo · 2026-09-27T18:04:47Z

What is this

This basically adds a side channel to stream artifact notifications for rmeta, so that caller (cargo) can see .rmeta notification as early as possible.

This is only added to local daemon and client-side mode. Distributed mode is not touched.

This should help quite a bit when there are some cache misses, especially for large Cargo workspaces.

How to review

Reviewing this commit by commit is highly recommended.

Tests are added as minimum reproducer for client and daemon modes.
The feat commits diff shows the behavior changes through flipping the assertions.
This follows C-TEST commit structure

I personally don't like the post-filter of rmeta notifcation in CompilerFinished, though it seems unavoidable.

Cargo timing HTML

Here we build rust-lang/cargo@07b8049 on master and this PR.

Before, you see almost no codegen (purple) section:

After, pipelining works and overlaps shows:

🤖 LLM disclosure: This was initially generated by LLM after a series of design discussions, and handcrafted afterwards. I've reviewed every single line of this and is responsible for the code change.

This is a preparation for propagating rmeta artifact notification in the middle of a rustc invocation.

`run_input_output` reads the child's stderr to EOF, so nothing can react to a line before the process exits. This add `run_input_output_observing`, which takes an optional stderr observer, which can support streaming and propogations.

tests are running in parallel so need a separate daemon

THis is needed so that tests can observe rmeta artifact notifications

codecov-commenter · 2026-09-27T18:08:28Z

Codecov Report

❌ Patch coverage is 93.55469% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.49%. Comparing base (8396f02) to head (33304cf).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
src/commands.rs	81.53%	12 Missing ⚠️
src/server.rs	93.95%	11 Missing ⚠️
src/test/tests.rs	94.65%	7 Missing ⚠️
tests/sccache_rustc.rs	94.82%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2875      +/-   ##
==========================================
+ Coverage   76.35%   77.49%   +1.14%     
==========================================
  Files          72       72              
  Lines       40219    41265    +1046     
==========================================
+ Hits        30709    31980    +1271     
+ Misses       9510     9285     -225

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sylvestre · 2026-09-28T11:23:44Z

+        // Daemon already emitted `.rmeta` notification and we forwarded it,
+        // so the copy from this local compile must not reach the caller again.
+        //
+        // XXX: This local compile rewrites artifacts the caller may already be


does it really need a 6 lines comment? please make it shorter :)

No. I just wanted to call out that non-deterministic rustc output may hurt this assumption (but anyway sccache currently doesn't yet prepared for the new parallel frontend/--job-frontend flag)

sylvestre · 2026-09-28T11:23:52Z

+    /// The client that has forwarded the notification must drop that copy.
+    ///
+    /// Currently only `.rmeta` notifications are sent (for Cargo pipelining).
+    ArtifactNotification(Vec<u8>),


could you add it at the end of the enum?
inserting here shifts the bincode tags of all the Storage* variants

Nice catch. Hmm… I should have listened to LLM. I ignored it because I felt like it was being too paranoid.

sylvestre · 2026-09-28T11:24:00Z

-                            .and_then(|res| async { Ok(Response::CompileFinished(res)) })
-                            .boxed();
+                        let body = if c.kind() == CompilerKind::Rust {
+                            // Only rustc emits artifact notifications,


start_compile_task already spawns internally, do we need the extra spawn_on here?

I tried without it and it hung. I guess that was because start_compile_task is itself an async fn, so its internal spawn only runs if itself future is polled. However, the later chain won't poll it until rx closes, which needs the tx end be dropped first.

It is kinda a cycle: tx drop -> rx closes -> chain -> start_compile_task -> tx drop

Hence an extra spawn breaking the cycle.

There might be a way we can use select! or something to avoid this extra task, though for readability I feel like the current one is good enough.

sylvestre · 2026-09-28T11:24:06Z

+    let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+    let addr = crate::net::SocketAddr::Net(listener.local_addr().unwrap());
+    let server = thread::spawn(move || {
+        let (mut sock, _) = listener.accept().unwrap();


the fake server setup is almost the same as in the previous test, could be dedup, no?

Dedup'd. (I don't particularly like how it works half the way though)

sylvestre · 2026-09-28T11:24:12Z

+        "second run was not a hit"
+    );
+
+    for (name, stderr, rmeta_lines) in [("miss", &miss.stderr, 1), ("hit", &hit.stderr, 1)] {


rmeta_lines is always 1, do we still need it?

Nice catch. It was vestige. Removed.

This is currently unused though.

Only local rustc compilations that miss the cache stream: * cache hits have no running compiler * distributed compiles never stream because `.rmeta` is not on local disk yet * other compilers are unchanged

`SCCACHE_CLIENT_SIDE` mode already produces the streamed body, so `compile_direct` now takes the CLI's stderr and writes each `CompileStderr` chunk as well.

weihanglo

Thanks for the review!

weihanglo · 2026-09-28T13:53:50Z

+        // Daemon already emitted `.rmeta` notification and we forwarded it,
+        // so the copy from this local compile must not reach the caller again.
+        //
+        // XXX: This local compile rewrites artifacts the caller may already be


No. I just wanted to call out that non-deterministic rustc output may hurt this assumption (but anyway sccache currently doesn't yet prepared for the new parallel frontend/--job-frontend flag)

weihanglo · 2026-09-28T14:26:58Z

+        "second run was not a hit"
+    );
+
+    for (name, stderr, rmeta_lines) in [("miss", &miss.stderr, 1), ("hit", &hit.stderr, 1)] {


Nice catch. It was vestige. Removed.

weihanglo · 2026-09-28T14:27:32Z

+    let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+    let addr = crate::net::SocketAddr::Net(listener.local_addr().unwrap());
+    let server = thread::spawn(move || {
+        let (mut sock, _) = listener.accept().unwrap();


Dedup'd. (I don't particularly like how it works half the way though)

weihanglo · 2026-09-28T14:37:17Z

-                            .and_then(|res| async { Ok(Response::CompileFinished(res)) })
-                            .boxed();
+                        let body = if c.kind() == CompilerKind::Rust {
+                            // Only rustc emits artifact notifications,


I tried without it and it hung. I guess that was because start_compile_task is itself an async fn, so its internal spawn only runs if itself future is polled. However, the later chain won't poll it until rx closes, which needs the tx end be dropped first.

It is kinda a cycle: tx drop -> rx closes -> chain -> start_compile_task -> tx drop

Hence an extra spawn breaking the cycle.

There might be a way we can use select! or something to avoid this extra task, though for readability I feel like the current one is good enough.

weihanglo · 2026-09-28T14:38:30Z

+    /// The client that has forwarded the notification must drop that copy.
+    ///
+    /// Currently only `.rmeta` notifications are sent (for Cargo pipelining).
+    ArtifactNotification(Vec<u8>),


Nice catch. Hmm… I should have listened to LLM. I ignored it because I felt like it was being too paranoid.

weihanglo and others added 4 commits September 26, 2026 20:15

refactor: represent compile body as a stream of responses

5559209

This is a preparation for propagating rmeta artifact notification in the middle of a rustc invocation.

test: give each sccache_rustc test its own daemon port

08f6193

tests are running in parallel so need a separate daemon

test: expose wrapper's stderr

a77751f

THis is needed so that tests can observe rmeta artifact notifications

weihanglo mentioned this pull request Sep 27, 2026

sccache isn't compatible with Cargo pipelining on cache miss #2873

Open

sylvestre reviewed Sep 28, 2026

View reviewed changes

weihanglo added 7 commits September 28, 2026 10:25

feat: add Response::ArtifactNotification

3b106f8

This is currently unused though.

test: show how client mode deals with artifact notification

b278e41

feat: forward artifact notification to stderr before compile finish

b2a1f01

test: show rustc miss deliver.rmeta notification

6ccaded

feat: stream .rmeta artifact notifications on local cache miss

4609fc5

Only local rustc compilations that miss the cache stream: * cache hits have no running compiler * distributed compiles never stream because `.rmeta` is not on local disk yet * other compilers are unchanged

test: show client-side mode compile delivers rmeta notification

1ddd04f

feat: stream .rmeta artifact notifications in client-side mode

33304cf

`SCCACHE_CLIENT_SIDE` mode already produces the streamed body, so `compile_direct` now takes the CLI's stderr and writes each `CompileStderr` chunk as well.

weihanglo force-pushed the stream-rmeta-notification branch from ea00651 to 33304cf Compare September 28, 2026 14:25

weihanglo commented Sep 28, 2026

View reviewed changes

Conversation

weihanglo commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this

How to review

Cargo timing HTML

Uh oh!

codecov-commenter commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weihanglo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

weihanglo commented Sep 27, 2026 •

edited

Loading

codecov-commenter commented Sep 27, 2026 •

edited

Loading