[FEATURE REQUEST] Different testing levels about rev HOT 4 CLOSED

What tests will be a part of what levels?

from rev.

That's for the solver of this issue to decide

from rev.

#238 (comment)

from rev.

You can look at the time each test takes by running it today. From there, you can do a Huffman-like compression, by sorting the times, and taking all of the fastest tests which take 1/3 of the total time. That would be the "Low" category. Then among the remaining tests, take the fastest which add up to 1/3 of the total time. They and the Low tests would be in the Med category. Finally, the remaining tests and Med would be in the High category. That's one way to do it in thirds.

But another way would be to put all of the fastest tests in the Low category up to where there is a huge climb in the timing of the tests. Sort the tests by runtime, and plot the runtimes. At the first large jump, put everything below that jump in the Low category. Then do the same with the remaining tests, finding a large jump and putting all tests under that jump in the Med category. Put any remaining tests in the High category.

Numbers like 1, 2, 3 can be used instead of category names, so that in CMake you can add a tag like testlevel=1 to each test with a number on it, and then only run a test if its number is less than or equal to the testing level.

from rev.