cargo test: option to run all tests under the same binary about cargo HOT 5 OPEN

Asked T-compiler for input on --mod at https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Unified.20test.20binaries/near/422637295

from cargo.

Note that with #5609, we plan to run test binaries in parallel. We are actively working towards this with the T-testing-devex team. The first step is json output for libtest for cargo to read and process.

Reducing link times would be great though.

Ways to opt-in

• Edition
• New autotests value
• New test target value (like the proposal for opting custom test harnesses into the json output for parallel test running)

My main question is how to setup cargo/rustc to enable this, both with and without harness = false.

from cargo.

The link you shared is for a multicrate testing though, instead of single crate? Or are you planning on running binaries in the same crate in parallel too?
In any case,

I see that Issue being about running all selected test targets in parallel, whether they belong to the same package or not. There is another issue about running the available test binaries in parallel to compilation.

My plan is to not run a process per test unlike nextest because that runs into shared state issues.

This would alleviate (2), but not the other issues I think?

Correct, this does not help with reducing link time.

And you would manage the number of running threads across all binaries?

My assumption is that we'll use jobserver to limit the number of threads across all processes

from cargo.

@epage

Hi :)

The link you shared is for a multicrate testing though, instead of single crate? Or are you planning on running binaries in the same crate in parallel too?
In any case, This would alleviate (2), but not the other issues I think? And you would manage the number of running threads across all binaries?

Anyway, I think it would be nice if running everything in the same binary per thread would be facilitated, as single process multithreading (for tests in a crate) solves all 4 listed issues.

from cargo.

Had some more thoughts on reducing the link time.

A very rough sketch:

package.autotests and package.autobins gain two new values

• `split
• unified

(names to be bikesheded)

split is the current behavior and true will map to split for the current Edition.

unified will

• enumerate test files / directories
• strip those out that have an explicit target
• create an implicit build target called tests (so cargo test --test tests shows this is redundant with cargo test --tests)
  • This will error if the user has an explicit tests target but I don't see a way to avoid this without a confusing fallback scheme
• pass these with a new rustc --mod <mod>=<path> command-line flag with an empty string being piped in for the crate root.

This still leaves out custom test harnesses. Our working approach to testing is that libtest is effectively frozen and we need to make custom test harnesses a first class experience to allow evolution outside of libtest.

One approach is if we move forward with a target field to specify main from a dep. In this case, the above scheme works.

If we do not have a way to delegate main, we'd likely want to infer tests/main.rs is the main and pass --mods to that (instead of using stdin with an empty string). The problem with this approach is that this will be inconsistent with further nested modules and with other target types. To make this consistent would be re-hashing the mod system discussions from the 2018 Edition.

We likely will need to defer the custom test harness discussion to have a better idea of what other pieces may be available (e.g. delegated main)

Benefits

• Decrease test suite compile time by 3x for cargo itself (source)
• Decrease on-disk artifacts by 5x for cargo itself (source)
• Faster test times as more tests run in parallel, much like cargo nextest but without the orchestration complexity

In future editions we could migrate package.auto* = true to unified (details to be worked out)

Questions for what the long term true should be for autotestss / autobenches:

• Are there sufficient end-user benefits outside of reduced link times / target/ size?
• Would a reduction in link times (e.g. changing the default linker, whether lld, mold, or wild) change the answer to what the default should be? If so, where is the line?

Open questions

• Is --mod <mod>=<path> the right approach?

Prior discussions and related links

• https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html
• https://internals.rust-lang.org/t/running-test-crates-in-parallel/15639/12?u=epage
• https://www.reddit.com/r/rust/comments/lto0qa/blog_post_delete_cargo_integration_tests/gp3wito/
• https://crates.io/crates/automod/

from cargo.