Add support for parallel execution of tests about jqwik HOT 16 CLOSED

So parallelism is in my backlog. I cannot promise anything about when I'll tackle it, though.

As for a master/worker architecture: Without having thought about it too deeply, my guess is that it must be solved on the JUnit platform level since that is how build tools - and thereby CI tools - talk to jqwik and other engines. So you might want to open an issue on https://github.com/junit-team/junit5 for that.

from jqwik.

any news on that issue ?

from jqwik.

FYI, lightweight threads and co-routines aren't really any benefit if the type of work is CPU bound (what I suspect of most jqwik work).

For CPU bound work, the best thing to use is the common forkjoin pool ForkJoinPool.commonPool() It was specially built for CPU bound work. It was also shipped with java 8.

In that case, the usage is the same as it would be with a lightweight thread or coroutine. Spawn n CompletableFutures (CompletableFuture.supplyAsync(task)) doing the various scenarios then join them up and merge the results.

from jqwik.

Since the repo for jqwik2 is not in place yet there is no right place for this question ;-)

To answer your question: yes, the current plan is to have a framework-independent way to execute properties. In addition to a JUnit test engine.

from jqwik.

So parallelism is in my backlog. I cannot promise anything about when I'll tackle it, though.

As for a master/worker architecture: Without having thought about it too deeply, my guess is that it must be solved on the JUnit platform level since that is how build tools - and thereby CI tools - talk to jqwik and other engines. So you might want to open an issue on https://github.com/junit-team/junit5 for that.

Just as sidenote, this can already be done using "manual" exclusion of tests and running multiple disjunctive sets of tests. Have a look at jenkins plugin for this approach for example: https://plugins.jenkins.io/parallel-test-executor

from jqwik.

This feature makes sense considering that Junit 5 Jupiter already has it: https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution

There are, however, at least two axes that can be used to parallelize:

1. Run different properties/tests in parallel
2. Run different parameter sets of same property in parallel

Option 1 is easier because the potential for concurrency problems (e.g. race conditions) is smaller since different properties use different instances of the container class. It would also enlarge the overall potential for speed up.

Option 2 - which is what you suggest I believe - comes with more difficulties implementation-wise so I wonder if there is a certain reason that you prefer this one?

from jqwik.

Hi @jlink,

Yes, I was looking for the 2nd option, but I think parallelizing on both axes would be even better.

As a first step, we can limit this issue to the easier option. Then later we can implement parallelization in the other axis too.

Another nice to have option, especially when tests are run in CI/CD environments with dozens of agents, is to have a master/worker architecture that distributes the tests among multiple machines for even more parallelism. Then each agent will in turn parallelize the work assigned to it in the first two dimensions.

from jqwik.

We are also trying to use parallel test execution on the class-level.
We achieve this with the mechanisms gradle provides, using parallel test JVMs, so maximum isolation here.
The main thing (executing the tests) works well, but we have issues with jqwik reading/updating its database: We regularly get java.io.UTFDataFormatExceptions in net.jqwik.engine.recording.TestRunDatabase#readAllTestRuns.
For now it's just annoying having to know that these exceptions (typically) are not relevant but getting rid of them would be greatly appreciated.

from jqwik.

@tmohme-hypoport My guess is that the test run database does not make sense when you run tests parallelly from the outside. Would an option to disable test run recording altogether fit your needs? E.g. setting database = in jqwik.properties?

from jqwik.

Imho the test run database would still make sense even when running the tests in parallel since the underlying problem it is meant to solve still exists: How to reproduce a failing test.
But I also see that it would probably be a big effort to make the database safe when running multiple tests completely isolated as we do (at least I have no idea for an easy solution).

Right now (in the scenarios where we run test in parallel) we don't have the benefit that the database is meant to provide anyway, so an option to disable the recording altogether would improve the situation for us as we would get rid of the mentioned exceptions in the log.

Mostly the "established" tests help us to catch regressions and these are most of the time not rare edge-cases so they are typically easy to reproduce.
We see the main benefit of the database (the support for repeatability) when we develop new features until they have stabilised. But in this phase we typically focus on running just the tests for "the one thing in development" and not the whole suite so there's no reason to run them in parallel.

Long story short: Yes, an option to disable test run recording altogether would be greatly appreciated :)

from jqwik.

@tmohme-hypoport In version 1.2.2-SNAPSHOT you can now configure

database=

and disable test run recording altogether.

from jqwik.

@tmohme-hypoport BTW, reproduction of a failing property is still possible because the random seed will be logged and can be used in a manual test run through

@Property(seed="....")
void myProperty() {...}

from jqwik.

With the modification in 1.2.2-SNAPSHOT ("database=") our build with parallel test execution in multiple VMs runs smoothly without irritating error messages - thx :)
Regarding the reproduction of failing tests: The logged seed came to my mind shortly after my post, but nevertheless thx for the hint.

from jqwik.

@fduminy Do you have a special need that cannot be solved by the approach described further up in the discussion:

We are also trying to use parallel test execution on the class-level.
We achieve this with the mechanisms gradle provides, using parallel test JVMs, so maximum isolation here.

The thing is that all investment in thread-based parallelism (requiring isolation and Synchronisation in many places) may become futile with lightweight threads or co-routine-based approaches. One of these days/months/years jqwik will switch to a higher minimum Java version, thereby getting better tools.

from jqwik.

jqwik 2 will support parallel execution. So I close this feature request which refers to jqwik 1.

from jqwik.

@jlink sorry if it's a wrong place to ask, cant find a milestone page: will jqwik 2 support manual running of properties? API like JQwik.forAll(arbitrary, property) to call manually inside a bare JUnit @Test? Thanks

from jqwik.