Improve performance when running a large number of JUnit tests about rules_java HOT 1 OPEN

Thanks for the great writeup!

I also feel the single test class restriction is kind of clunky and we should definitely fix bazelbuild/bazel#2539. But perhaps that would be best done once the rules are out of bazel and in this repo.

re: flaky tests, I think the general guidance is always to fix flakiness :). Internally, what I see is a mix of re-running + ignoring (based on flakiness stats).

Generating a @Suite (either at compile time or dynamically via something like AllTests) clashes with --flaky_test_attempts as, if any test case fails, the entire suite is detected as having failed and so all tests are run again. This can be somewhat mitigated by sharding but there's a cap of 50 on the number of shards.

A partial mitigation could be to extract all your known flaky tests into separate, individual targets, and use the AllTests mechanism for all the other well-behaved targets. It's manual toil but I'm uncertain if there's anything one could do - flaky tests are just inherently problematic.

from rules_java.