Tracking Issue for integration with `--jobs` options in rustc

[petrochenkov]

Summary

Compiler MCP: rust-lang/compiler-team#1005
Documentation: https://doc.rust-lang.org/nightly/rustc/command-line-arguments.html#-j--jobs---jobs-frontend---jobs-backend---jobs-linker-limit-parallelism

rustc options

rustc now has a number of options to control parallelism - --jobs/-j for the total limit, and more specific options like --jobs-frontend for specific parts of the compiler.
The options are currently unstable, but the plan is to stabilize them as a part of compiler MCP 1005.
Documentation for the options can be found in here.

Even if cargo limits parallelism in rustc dynamically by passing a jobserver to it, it's still benefitial to limit it statically as well by passing the --jobs option, then rustc will know the maximum amount of jobs it will be able to use and account for it when e.g. creating thread pools. If the limit is 1 rustc will be able to perform even more optimizations.

What we expect cargo to do

When the options are stable

If cargo determines that it should use N jobs (from command line, or config file, or env vars, or the thread::available_parallelism default), it passes --jobs N to the compiler.

While the options are not yet stable

If cargo determines that it should use N jobs and detects a nightly compiler, it passes --jobs N -Zunstable-options to the compiler.
Something similar is already done for options like -Zembed-metadata.

When user wants to set frontend parallelism specifically

There are reasons to set --jobs-frontend to 1 even if the parallelism is set to a larger number in general, e.g. to get reproducible diagnostics, or to get reproducible binaries until the binary reproducibility bugs are fixed.

Ideally, cargo should support setting jobs-frontend = M in cargo configs, so that users don't have to pass it through RUSTFLAGS, and other tools like rust-analyzer could also use it consistently from the same source.
Same applies to jobs-backend and jobs-linker, but the motivation may be not so strong.

Unlike --jobs, which should always be passed, the fine grained options should be passed only if they are explicitly set in cargo config.

---

@Bryanskiy will provide an implementation for this if the direction is generally approved by the cargo team.

Unresolved Issues

No response

Future Extensions

If cargo uses N jobs, it can potentially pass some smaller number M to rustc, if it determines that the CPU is already busy enough with compiling other crates and will continue being busy for a while.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

[petrochenkov]

How this interacts with

• Use 2 parallel frontend threads by default on the nightly and dev channel rust#162848

If cargo implements what is described above, then jobs = 1 in cargo config will correctly disable parallel frontend in rustc even if rustc enables it by default (which will eliminate the parallel frontend overhead in particular).

[weihanglo]

Thanks for working toward the stabilization!

Not yet tracked all the progress and reports, but do we have any heuristic numbers of scheduling in mind? For example, how many frontend/backend/link jobs is appropriate for a 16-core machine and that won't throttle the system, including both CPU time and memory usage.

Prior discussions regarding limiting linking and general parallelism: #9157 and #t-cargo > cargo#9157: Restrict parallel linker executions

[petrochenkov]

how many frontend/backend/link jobs is appropriate for a 16-core machine and that won't throttle the system, including both CPU time and memory usage.

"jobs-backend = 16" and "jobs-link = 16" (*) is what would effectively be used now in this situation, are there reports about it resulting in throttling?
"jobs-frontend" will probably be capped by ~8 in rustc anyway, unless --jobs-frontend is passed explicitly, because larger numbers don't give much benefit in practice atm.
Otherwise I don't have any numbers in mind.

(*) with linkers like LLD that support threads.

[epage]

If cargo determines that it should use N jobs (from command line, or config file, or env vars, or the thread::available_parallelism default), it passes --jobs N to the compiler.

To double check, this is talking only about the static cap and we need to handle the balance between many different rustc's separately. We need to make sure this intent is clear whenever we talk about it (and comment it in the code). We also need to figure out how to balance the differe rustc invocations.

While the options are not yet stable
If cargo determines that it should use N jobs and detects a nightly compiler, it passes --jobs N -Zunstable-options to the compiler.
Something similar is already done for options like -Zembed-metadata.

Conditionalizing unstable features off the toolchain is generally discouraged. It was done for -Zembed-metadata only after regular time on nightly and only make dependent on the toolchain when it was determined that more broad testing was needed. When this was done, we then needed a way for nightly users to turn it off.

There are reasons to set --jobs-frontend to 1 even if the parallelism is set to a larger number in general, e.g. to get reproducible diagnostics, or to get reproducible binaries until the binary reproducibility bugs are fixed.

I wonder if --jobs-frontend 1 should be the default until the binary reproducibility issues are resolved. That seems bad to regress on that.

[petrochenkov]

To double check, this is talking only about the static cap and we need to handle the balance between many different rustc's separately. We need to make sure this intent is clear whenever we talk about it (and comment it in the code). We also need to figure out how to balance the differe rustc invocations.

Yes, it's a static cap, the dynamic limit is still controlled by jobserver tokens at any given moment.

I wonder if --jobs-frontend 1 should be the default until the binary reproducibility issues are resolved. That seems bad to regress on that.

That's the current consensus for stable, but on nightly we want to enable it sooner (rust-lang/rust#162848), so some people may need to report issues and then opt out.

[Kobzol]

Just noting that codegen-units is kind of a precedent here, because in addition to specifying the number of CGUs used, it also used to serve essentially as a --jobs-backend parameter. Now that logic has been modified, so that we don't spawn a new thread for every CGU, AFAIK, but it still affects backend parallelism.