randomize tests about busted HOT 11 CLOSED

It isn't built into busted yet, but I can tag this as a feature request.

from busted.

question with #48 is then whether it is limited to randomizing per testfile, or it would automatically disable threaded testing and randomize all available tests.

from busted.

Good question - I assume randomizing per testfile. I don't have a strong opinion - if we want a truly global random, then we could probably either:

1. run each test in its own thread
2. run with one single thread

from busted.

It seems like globally random would be ideal, but per file would certainly be sufficient (particularly if there is any kind of limitation running tests across files). Are there any constraints related to setup/teardown only running once? I THINK most test frameworks that randomize do so for each level of tests - this would imply file-level randomization. I am open to working on this feature if noone else is available/willing to pick it up, although I would certainly need some guidance.

from busted.

I think a couple of items should be combined;

• threaded testing #48
• implementing snapshots (see lunarmodules/luassert#38) on each nested level
• randomizing tests

This requires some rework on the engine. Currently the bootstrapper loads and executes the testfiles (does not run the tests itself) and then hands the set of tests to the busted core and there the tests are executed.

To make this work properly, executing the testfiles should also move into busted core, because each test file needs to be wrapped in a luassert snapshot.
Or stated otherwise; now there are nested levels of describe() and it() blocks, and the testfile itself is not a block but should be added as one.

from busted.

Just posted #93

Which triggered me that the randomization can not go beyond the scope of a describe() block, because the setup() and teardown() only run once per describe() block.

from busted.

when doing randomization, on what level should this be set?

• global through a commandline option
• per describe() block through a function eg randomize(true/false) which should then make all tests in that describe() (and nested ones) randomized (or explicitly not randomized)

from busted.

@ToadJamb I think the current master should make it possible to add the randomization. It has to be done on the level of a describe() block, so the run_context() function in core.lua seems to be the right spot.

from busted.

A global --randomize could work with tagged tests. It'd also be easy to add in randomize() as a function if it's in the bottom of a describe block - describe would hit this and scramble the indexes in the context tree. This wouldn't be recursive by default, although we could add in a flag like randomize({ recursive = true }) to have it recurse through sub-describes. This would also allow you to say "randomize these tests that are above me", if you only want to randomize a subset.

Would that flexibility be a feature, or just a leftover of taking the lazy way to implement it?

from busted.

for simplicity rather randomize_all() then randomize({ recursive = true }). And then the option --randomize should behave as a randomize_all() on the top-level describe of testfiles.

and the functions should overrule the command line option.

from busted.

Yes, that makes sense.

from busted.