Support property testing with Spring and SpringBoot about jqwik HOT 17 CLOSED

As far as junit-team/junit5#1722 is sketched out it won't help much in this case. jqwik already has its own (internal) extension and lifecycle model. The exact Spring extension couldn't be used anyway due to differences in how the lifecycle works. I'll come back to Spring integration as one of the first things after 1.0 has been released.

from jqwik.

I suggest to support Spring (and SpringBoot) integration testing in as similar a way as it work with Jupiter. This is described here: https://docs.spring.io/spring-framework/docs/current/spring-framework-reference/testing.html

Most annotation should be supported. @SpringJUnitConfig must be replaced by something equivalen for jqwik, e.g. @SpringJqwikConfig.

Since jqwik's lifecycle is somewhat different, not everything will work exactly as with Jupiter, for example, there must be a way to specifiy if Spring's clean up will happen either:

• Per property: Spring beans will be kept around for all tries in a property. This has better performance, but only makes sense if an individual try does not change the beans' states in a way that will influence the next try.
• Per try: Spring beans will be freshly initialized for each try.

For Spring Boot, there must be a replacement for @SpringBootTest since this is made for Jupiter.

from jqwik.

@jlink Unfortunately I don't have any clue of how Spring and Jupiter work in tests (coming from using Hypothesis in python with pytest and QuickCheck in haskell it seems over-complicated to me), so I don't have any clue about implementation details. What I do know is that there's lot of Spring code out there and it would benefit from property based testing.

I would do something similar to Hypothesis. Specify what generators to use via annotations (don't forget data.draw to get something from generator right in test). Then run test trying to falsify property. Detect junit test failure (because of falsifying example) and do shrinking in the end reporing minimal failure.

Scenario is very simple. I'm using MockMvc and I need to autowire repositories to setup database etc.

from jqwik.

@luvarqpp Writing a source for parameterized tests that uses jqwik under the hood should be doable. Give it a shot! As for PBT in integrated testing: It requires a bit more thinking and obviously more time to run the tests. Nevertheless, when you watch some of John Hughes' presentations you will see that this seems to be his main application of PBT. Since he's more or less the inventor of PBT there's probably something to learn there.

from jqwik.

Many thanks for the comment. To be frank I've been waiting for such a suggestion :-) I'd love to see a concrete example of how you want to use properties in a Spring context. Under the hood jqwik already has something similar to Jupiter's extension model, so it wouldn't be too hard to write a Spring extension.

What I won't do is implementing jqwik as a Jupiter extension so that it can work with other Jupiter extensions. The reason is that jqwik's lifecycle is too different from Jupiter's lifecycle. Having both together would require massive changes in both - and my impression is that it wouldn't really make sense.

So I'm happy to look at your scenario which might give me motivation to enhance and publish jqwik's life cycle extension model.

from jqwik.

OK. I'll change the title of the issue to "Support property testing with Spring and SpringBoot" and take it as a driving use case for enhancing jqwik's extension model. Might take a while, though, until it gets to the top of the feature list.

from jqwik.

I believe this issue will be easier to implement if or when junit-team/junit5#1722 is implemented.

from jqwik.

Yeah this would be awesome

from jqwik.

It would be really useful

from jqwik.

Depends on at least partial implementation of #61

from jqwik.

Just to add another usecase:
I have some integration tests, which does start full application and in-memory database (spring things with embedded tomcat and H2 in-memory database). I am also not able to directly use PBT in its whole glory. My "workaround" is at the end of this comment.

My testing class is something like this:

@ExtendWith({RestDocumentationExtension.class, SpringExtension.class})
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@AutoConfigureRestDocs(outputDir = "target/snippets", uriScheme = "https", uriHost = "asdf.qpp.sk", uriPort = 443)
@Slf4j
public class AbstractHttpHelper {
    protected static final String DEFAULT_HOST_FOR_EXAMPLES = "asdf.qpp.sk";
    protected static final int DEFAULT_PORT_FOR_EXAMPLES = 433;
    protected static final String DEFAULT_PROTOCOL_FOR_EXAMPLES = "https";
    protected static final String DEFAULT_PREFIX_FOR_EXAMPLES = "/afds_backend-jib";
    protected static final RequestParametersSnippet pathParametersSnippetForSortPageSize = requestParameters(
            parameterWithName("sort").description("Optional sort style. You specify column name and optionaly direction of sort. Format is more specifically defined as '($propertyname,)+[asc|desc]?'. For more info, have a look at sample usage in <<Get test cases sorted>>, or go to https://docs.spring.io/spring-data/rest/docs/3.1.8.RELEASE/reference/html/#paging-and-sorting.sorting[spring reference documentation].").optional(),
            parameterWithName("page").description("Optional page (default 0, i.e. first one) number to be returned.").optional(),
            parameterWithName("size").description("Optional size of page (default 20), i.e. number of resources (entities) that will be returned at once.").optional()
    );
    protected final boolean initializeSomeBearer;
    /**
     * @see #webClientBase
     */
    @Autowired
    protected TokenProvider tokenProvider;

    /**
     * Modified (mutated) {@link #webClientBase} instance, which does have pre-set these things:
     * <ul>
     *     <li>X-Forwarded-Host -> simulating that we are going to our testing (publicly available) site. Documentation is than generated with correct sample commands with correct url, etc</li>
     *     <li>X-Forwarded-Port -> simulating test site</li>
     *     <li>X-Forwarded-prefix -> simulating test site</li>
     *     <li>X-Forwarded-proto -> simulating test site</li>
     *     <li>Authorization -> adding "Bearer" JWT authorization token for master user to make http request authorized</li>
     * </ul>
     *
     * @see #setUp(RestDocumentationContextProvider) initialization of mutated webClient is done here
     */
    protected WebTestClient webClient;

    @LocalServerPort
    protected int port;
    /**
     * Using {@link Autowired} annotation in flavour of using constructor with these parameters, because in that way,
     * all extenders of this abstract class have to call given constructor and additionally they all have to use
     * annotation like: <pre>@TestConstructor(autowireMode = TestConstructor.AutowireMode.ALL)</pre>.
     * <p>
     * Non-modified web client wired here. It will be mutated in {@link #setUp(RestDocumentationContextProvider)}
     * method and stored in {@link #webClient} field. Given field should be used for actual testing of project.
     */
    @Autowired
    private WebTestClient webClientBase;

    @BeforeEach
    public void setUp(RestDocumentationContextProvider restDocumentation) {
        final URIUpdaterPreprocessorWithPrefix preprocessor = new URIUpdaterPreprocessorWithPrefix(
                DEFAULT_HOST_FOR_EXAMPLES,
                DEFAULT_PORT_FOR_EXAMPLES,
                DEFAULT_PROTOCOL_FOR_EXAMPLES,
                DEFAULT_PREFIX_FOR_EXAMPLES
        );

        final WebTestClient.Builder wcBuilder = this.webClientBase.mutate()
                //.baseUrl("https://api.example.com")
                .filter(documentationConfiguration(restDocumentation)
                        .operationPreprocessors()
                        .withRequestDefaults(preprocessor)
                        .withResponseDefaults(prettyPrint()))
                .defaultHeader("X-Forwarded-Host", DEFAULT_HOST_FOR_EXAMPLES)
                .defaultHeader("X-Forwarded-Port", "," + DEFAULT_PORT_FOR_EXAMPLES)
                .defaultHeader("X-Forwarded-prefix", DEFAULT_PREFIX_FOR_EXAMPLES)
                .defaultHeader("X-Forwarded-proto", DEFAULT_PROTOCOL_FOR_EXAMPLES);
        if (this.initializeSomeBearer) {
            final String AUTHORIZATION_TOKEN_FOR_USER_1 = this.tokenProvider.createToken(1L);
            wcBuilder.defaultHeader("Authorization", "Bearer " + AUTHORIZATION_TOKEN_FOR_USER_1);
        }
        this.webClient = wcBuilder.build();
        log.info(
                "webClient for test has been set-up. SUT (service under test) is accessible at {}.",
                this.webClient.get().uri("api").exchange().returnResult(String.class).getUrl()
        );
        log.info("Testing server should listen on port {}.", port);
    }
}

Note, that I am using:

• some autowiring of beans from SUT (system under the test, i.e. springboot application with embedded tomcat and H2),
• @beforeeach annotation to set-up webclient before each test,
• some junit extensions (SpringExtension and RestDocumentationExtension). They provide me for example possibility to document my testing calls from http client (webClient) and make my documentation always aligned with tests (i.e. actual code).

To use randomly generated data (from unit tests and other things which does not require spring to be alive during test), I am using jqwick to provide me some sample data while my tests are being executed by standard (JUnit Jupiter) executor. I am using my prepared Arbitrary instances to get sample instance using code like this:

import lombok.Builder;
import lombok.Data;
import net.jqwik.api.Arbitraries;
import net.jqwik.api.Arbitrary;
import net.jqwik.api.Combinators;

import java.util.Random;

@Data
@Builder
public class UserForTests {
    private static final Random rnd = new Random();

    public String name;
    public String email;
    public String password;

    public static UserForTestsBuilder aUser() {
        Arbitrary<UserForTestsBuilder> r = Combinators.combine(
                Arbitraries.strings().all().ofMinLength(1).ofMaxLength(42), // name
                Arbitraries.strings().alpha().numeric().ofMinLength(1).ofMaxLength(42), //first part of email
                Arbitraries.strings().alpha().numeric().ofMinLength(1).ofMaxLength(42), //part of email after '@'
                Arbitraries.strings().alpha().ofMinLength(2).ofMaxLength(5), // domain of email (suffix after '.')
                Arbitraries.strings().all().ofMinLength(1).ofMaxLength(42) // password
        ).as(
                (name, email1, email2, emailDomain, pwd) -> new UserForTestsBuilder()
                        .name(name)
                        .email(email1 + '@' + email2 + '.' + emailDomain)
                        .password(pwd)
        );
        return r.generator(0).next(rnd).value();
    }
}

To use generated sample builder instance, I use this code in normal test (annotated by @Test):

    UserForTests a = UserForTests.aUser()
                .email("invalidEmail!")
                .build();
        SignUpAndLoginHelper.doSignUpRequest(this.webClient, a.name, a.email, a.password)
                .jsonPath("$.success").isBoolean()
                .jsonPath("$.success").isEqualTo(false);

For more info, you can have a look at another issue (question): #76 (comment)

from jqwik.

What will never (well, for the forseeable future) be possible is to combine jqwik properties (@Property) with Jupiter extensions because Jupiter has a completely differenty lifecycle that doesn't combine with jqwik's own lifecycle. But, of course, using jqwik generators directly as you do it here will always be an option.

I recently did some research into enabling spring boot testing into jqwik. It's doable and it's jqwik's next big feature - if my planning survives contact with next year's reality.

from jqwik.

I get it that there are way to different lifecycle. I would like just to add one more usecase from "real life".

To cope with existing limitations, one "solution" came to my mind. What about to use jqwik as source of values in @ParameterizedTest? It does not solve this issue, but it can in my opinion bring some basics of generated data in input. Shrinking can be done probably "by hand", where one can invoke your shrinker within jupiter test execution directly, if needed. We have used many times PBT library (scala) just as data generator and it was imho success.

Sidenote: Using PBT in integration tests for anything more than generating a few "realistic" data samples, is probably bad practice. Does it make sense to make this type of usage easier? :)

from jqwik.

Since release 1.2.4 is in the making and focused on implementing the necessary hooks under the hood (#61) work on this issue is scheduled for version 1.2.5.

from jqwik.

Work started in independent project repository: https://github.com/jlink/jqwik-spring

from jqwik.

Basic Spring mechanism and test annotations work: https://github.com/jlink/jqwik-spring/blob/master/src/test/java/net/jqwik/spring/SimpleSpringTests.java

I pushed version "0.1.0-SNAPSHOT" so that you can already try this feature in your projects.

from jqwik.

With Release 1.2.6 the internal Lifecycle Hooks API is stable enough to support Spring and Spring Boot.

All further issues and requests should go to jqwik-spring which will probably have its first release in a few hours (or days max).

from jqwik.