Usability with failed Tests about jqwik HOT 12 CLOSED

I'm not quite sure what you mean with "in the beginning". Do you mean before "tries" et al. is reported? In that case it's the IDE that decides when and where reported values are displayed. I'm just using the JUnit platform reporting mechanism.

What I could do is to redundantly report the assertion error's message text (without stack trace) first thing before anything else is reported. Would that help?

from jqwik.

@3nigm4 Thanks for the carefully thought out suggestion. I will think about ways to realize it and come up with a first attempt in 2 to 3 weeks (due to vacation).

from jqwik.

I think we mean the same thing. Just to be sure.

This is a typical intellij output:
(1)
"C:\Program Files\Java\jdk1.8.0_181\bin\java.exe" ...

(2)
timestamp = 2019-02-20T12:38:40.490
tries = 10
checks = 10
generation-mode = RANDOMIZED
after-failure = PREVIOUS_SEED
seed = -8932427284849257286
original-sample = [-2147483648]
sample = [-2147483648]

(3)
...................... some exceptions here ..............

(4)
timestamp = 2019-02-20T12:38:40.490
tries = 10
checks = 10
generation-mode = RANDOMIZED
after-failure = PREVIOUS_SEED
seed = -8932427284849257286
original-sample = [-2147483648]
sample = [-2147483648]

(5)
java.lang.AssertionError:
Expecting:
<0>
to be greater than:
<0>

(6)
...................... some exceptions here ..............

Most of the time I am looking for (5) and currently I need to scroll and search for it.
Another problem is that there is no grouping of realted info.

I can not easily determine reading through the output, which exceptions (3), (6) and property info blocks (2), (4) belong to which assertion error (5) [[ the one below or above ??? ]]

So as a first step, yes it would make a difference, if you put the assertion error (5) redundantly between (1) and (2). That way I could see it right away. Maybe like so:

java.lang.AssertionError in TESTNAME
Expecting:
<0>
to be greater than:
<0>

from jqwik.

I'm still a bit puzzled why (2) and (3) is duplicated.

When I click a single failing test in IntelliJ I only see (4), (5) and (6) depending on the exception and type of failure.

from jqwik.

You are right, it is that way when you click the failing test. I was reporting the total view of "TestResults" (see picture).

I 'd still prefer if the assertion like (5) came before the timestampt etc. (2) ... and if everything were a bit more grouped and formatted. I can think of a proposal beginning next week ...

from jqwik.

Ok. I'm awaiting your ideas...

from jqwik.

Hi there,

after going through a couple of ideas, I realized that most of the things I want to change are not in control of jqwik and I have to admit I do not understand the extent of the proposed change.
Anyway here is my proposal:

--------------- Assumptions ---------------
a) you can not easily change the format of errors reported by third party assertion librarys such as e.g. assertJ
b) because you can not do that, it is hard to put information next to each other horizontally because you do not know how spacious an e.g. assertion error is going to be
c) ... therefore for an practical first solution, there is only vertical reordering, restructuring and grouping of info thats alrady there

--------------- Guiding Opinion ---------------
Vertical Space is most precious, because we have less of it
Most people use the default position of the rest reporter in their IDE
-> we have a lot of unused horizontal space and need to carefully prioritise the vertical order of test output. Higher up meaning more important for the user.

--------------- Good Failures ---------------
I wanted to group things together with boxes, but did not go for it, because it would require one line on top which carries no information

--------------- Vertical Order ---------------
The example below shows the following order / priorities

(1) Output of Test Result / Assertion
(2) Line Number and Content of where the assertions was triggered (I thought you could get that out of the stacktrace -> but I think it does not work if the test method returns boolean :(
(3) Jqwik Info with some helpers
(4) Stacktrace

--------------- Before and After Pics ---------------
I used pics because I can show context but also because I just couldn't get make github protect whitespace

New Jqwik Report Proposal:

In context: 14 inch Laptop, before:

In context: 14 inch Laptop, after:

from jqwik.

Here is the file used in the pic:

https://github.com/3nigm4/JqwikReportProposal/blob/master/ExampleTestReportHappyPath

As I have also written in the repo, I accept the Jqwik contributors agreement (should this become a contribution).

from jqwik.

@3nigm4 I attempted a first shot in b123184

It's available in "1.1.1-SNAPSHOT". Please give it a try and tell me what you think.

timestamp = 2019-02-27T12:29:30.207, ConstrainingValuesExamples.squareOfRootIsOriginalValue = 

java.lang.AssertionError: 
Expecting:
  <NaN>
to be close to:
  <-1.0>
by less than 10% but difference was NaN%.
(a difference of exactly 10% being considered valid)

                              |---------- Jqwik ----------
tries = 1                     | # of calls to property
checks = 1                    | # of not rejected calls
generation-mode = RANDOMIZED  | parameters are randomly generated
after-failure = SAMPLE_ONLY   | only try the previously failed sample
seed = 8391794339428545543    | random seed to reproduce generated values
sample = [-1.0]
original-sample = [-199.42]

from jqwik.

The latest snapshot also allows to configure that only failures will be reported.

properties.jqwik:
reportOnlyFailures=true

from jqwik.

Will be available in 1.1.1 (deployment ongoing). Feel free to reopen if something should be improved.

from jqwik.

Thank you ! I haven't had the opportunity to test it yet, because of a serious family illness. I will get back to you as soon as I can get back to and catch up with work. Thanks for your openness !

from jqwik.