lukaskalbertodt / libtest-mimic Goto Github PK

Hello,

To properly mimic the behavior of libtest, libtest-mimic should default to capturing the output and then displaying it only upon test failure.

TBH, I'm not yet really sure how to implement it, so before jumping into it, I wanted to check: would you be interested by such a feature? (Disclaimer: I don't know yet when I'll actually start doing it, so please read this as “in an ideal would, would you agree that libtest-mimic should have this feature?”)

Anyway, thank you for libtest-mimic! :)

I am using a custom test harness for OSdev, but wondering if this could do no_std as well?

Thanks for libtest-mimic! I have a feature request of sorts. I'd be happy to put in the work to implement it, but since it's not completely trivial and raises a question about MSRV, I wanted to ask how you feel about it before I start on it.

clap switched from using termcolor to anstream for colored text output over a year ago (clap-rs/clap#4765) while libtest-mimic still uses termcolor. This means that any workspace using libtest-mimic and a reasonably recent clap version is pulling in both of those crates and their respective dependencies. This works, but it's a bit silly in principle and the two crates don't 100% agree on which environment variables affect colored output. It's also a bit wasteful to always build two implementation of the same functionality, especially on Windows, where both pull in windows-sys with different sets of features (and potentially two different versions, when windows-sys makes a new semver-breaking release).

So I'd like to switch libtest-mimic to use anstream v0.6, to share dependencies with modern versions of clap. However, anstream and its dependencies currently have a MSRV of 1.64 or 1.65 (not 100% sure, Cargo.toml and the commit that modified it give different versions). libtest-mimic declares 1.60 MSRV in Cargo.toml (though 08e76d3 suggest reality may be more complicated). Perhaps enough time has passed that bumping MSRV to 1.65 is alright?

It would be useful if a test could timeout. As a start, a global timeout value would suffice.

It seems pretty straightforward to implement. However, it may be considered ideal to have the test itself implement the timeout, which allows each test to specify their own timeout.

Thoughts?

I think this library could use some improvements in various areas. Unfortunately, I don't actively use it currently. And since my time is short and I have other projects that I'm more interested in, I will not implement all those things now, but just post this issue with a random list of improvements. Note that the library is still totally usable, it just doesn't have the perfect API (IMO) and does not support all of the newest libtest CLI flags/features.

Get good integration tests first!

• --include-ignored
• print total time rust-lang/rust@470c059
• get rid of rayon (I think it's overkill and makes the dependency chain longer to compile)
• catch panics and report as failures
• --logfile PATH Write logs to the specified file
• when single thread, run on main thread
• Ignore message rust-lang/rust@bb3b557
• --shuffle ordering rust-lang/rust@fa23d4f
  • --shuffle-seed
  • Alphabetical order otherwise?
  • RUST_TEST_SHUFFLE
• RUST_TEST_THREADS
• Run tests in new process
• multiple filters

Unstable

• --format json/junit
• --ensure-time
• --exclude-should-panic Excludes tests marked as should_panic (UNSTABLE)
• --report-time
• --force-run-in-process Forces tests to run in-process when panic=abort

I have tests which I need to run in a custom way, and I'd like to be able to benefit from libtest-mimic's work to precisely emulate the standard test harness interface, but the most I can use, it seems, is using the public Arguments struct to parse arguments — there's no way to use the filtering implementation but not the runner.

Could you add a function which takes an Arguments and a set of test descriptions (like run()), and returns an iterator of which tests should be run, instead of running them? (Or in the case of --list, it would do the expected printing and return a value indicating to just exit.)

(It is of course already possible to insert placeholders as the individual test functions, but that would require each placeholder function to access global state for the real test harness, and to block until the test actually runs in order to report success/failure, which seems inelegant and potentially problematic.)

libtest-mimic seems to be vulnerable to CVE-2022-23639. Please update the crate crossbeam-channel to the latest version to fix the issue.

Currently, the argument --test-threads is ignored. This shouldn't be the case.

In nextest-rs/datatest-stable#20 I discovered that coverage data was not being generated for some of my tests. I came to the conclusion (at nextest-rs/datatest-stable#20 (comment)) that the problem was the use of std::process::exit from within libtest_mimic.

It seems that on Windows (but not linux), coverage data isn't written to disk when the process is terminated with std::process::exit. The documentation on which says

Note that because this function never returns, and that it terminates the process, no destructors on the current stack or any other thread's stack will be run.

I presume that the profiling data is written to disk in the destructor for some type, and that isn't run when you terminate abruptly with std::process::terminate. I have no idea why this only affects Windows.

To work around this problem, in datatest-stable the maintainer used libtest_mimic::Conclusion::has_failed to return a std::process::ExitCode from the generated main function (see nextest-rs/datatest-stable@33d4725).
When ExitCode is returned from main, threads are cleaned up and coverage data is generated properly.

As discussed in nextest-rs/datatest-stable#20 (comment) libtest-mimic may want to deprecate libtest_mimic::Conclusion::exit in favour of a method that returns an ExitCode.

Hi Lukas,

Thanks for addressing the ExitCode issue so quickly! Noticed that the logic looks inverted though, would be good to get another fix out soon.

I'd like to use libtest-mimic to support MPI, which will involve running a function in the current executable with no test harness semantics. If I were to add such a thing to simple.rs, the new check_mpi() would involve running a Command that amounts to

mpiexec -n 2 target/debug/examples/simple-60c49fe0be82d931 --just-run-function-with-no-harness main_for_check_mpi

This creates multiple processes that communicate between themselves. The check_mpi() would check that they complete successfully and produce desired output. I want main_for_check_mpi() to be able to live in the same source file and executable as check_mpi(), but haven't found a way to do this with libtest-mimic.

Is this a use case that might be in scope for libtest-mimic? Either way, do you have implementation suggestions?

https://docs.rs/libtest-mimic/0.5.2/libtest_mimic/index.html << Renders fine

https://docs.rs/libtest-mimic/0.6.0/libtest_mimic/index.html << Broken

Confirmed its not my local browser caching a docs.rs hiccup

I use this crate to generate roughly 20k testcases. When I run my testsuite I can add > /dev/null to shave 500ms off of my 3.2s test runtime.

I want to be able to stop printing one line per test output, and rather either:

1. print one dot per passing testcase (and batch/buffer the printing of dots in large batches) -- this is what rustc-test (fork of std test) appears to do in case there are many testcases: https://github.com/servo/rustc-test
2. customize on the Test struct, or via Arguments, what kind of output I want for this testfile/item.

hope that makes sense

In 0.5, Outcome was made private and callers lost the ability to mark a test as ignored/skipped at runtime. This is very useful for test runners built on top that support runtime skipping to communicate what happened to the user.

When using IntelliJ Rust, cargo test is specialcased to add extra arguments to allow the IDE to parse the results. Unfortunately, these arguments aren't supported and so you can't run tests from inside the IDE if using libtest-mimic.

When running cargo test it actually adds: --format=json -Z unstable-options --show-output

• --format=json is tracked by #1
• -Z unstable-options Is used to enable --format=json
• --show-output perhaps we should also just do nothing, since that's the current behaviour I believe?

Libtest --help output says:

    --show-output   Show captured stdout of successful tests             
-Z unstable-options Enable nightly-only flags:                           
                    unstable-options = Allow use of experimental features

It relies on the duration always being 0s and apparently my PC is being pessimistic and rounds it up.

thread 'terse_output' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
 running 16 tests
 .F.Fiiii.F.Fiiii
 failures:
 
 ---- dog ----
 was not a good boy
 
 ---- bunny ----
 jumped too high
 
 ---- blue ----
 sky fell down
 
 ---- green ----
 was poisoned
 
 
 failures:
     dog
     bunny
     blue
     green
 
<test result: FAILED. 4 passed; 4 failed; 8 ignored; 0 measured; 0 filtered out; finished in 0.01s
>test result: FAILED. 4 passed; 4 failed; 8 ignored; 0 measured; 0 filtered out; finished in 0.00s

', tests\mixed_bag.rs:502:5
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library\std\src\panicking.rs:575
   1: core::panicking::panic_fmt
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library\core\src\panicking.rs:65
   2: mixed_bag::terse_output
             at .\tests\mixed_bag.rs:502
   3: mixed_bag::terse_output::closure$0
             at .\tests\mixed_bag.rs:493
   4: core::ops::function::FnOnce::call_once<mixed_bag::terse_output::closure_env$0,tuple$<> >
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6\library\core\src\ops\function.rs:251
   5: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library\core\src\ops\function.rs:251
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Hi, I'm wondering if mimic would help with this: when running cargo test:

1. A setup function is executed
2. All the tests are run just like cargo test would without mimic
3. A teardown function is executed

If I understand correctly, mimic helps with this ... eliminating the boilerplate when writing a custom harness?

Currently, --format=json is not supported. This should change.