Implement coverage checking for statistical values about jqwik HOT 17 CLOSED

Current implementation target

@Property(generation = GenerationMode.RANDOMIZED)
void simpleStats(@ForAll RoundingMode mode) {
	Statistics.collect(mode);
	Statistics.coverage(coverage -> {
		coverage.check(RoundingMode.CEILING).percentage(p -> p > 5.0);
		coverage.check(RoundingMode.FLOOR).count(c -> c > 2);
	});
}

Similar for labelled statistics:

@Property(generation = GenerationMode.RANDOMIZED)
void labeledStatistics(@ForAll @IntRange(min = 1, max = 10) Integer anInt) {
	String range = anInt < 3 ? "small" : "large";
	Statistics.label("range").collect(range);
	Statistics.label("value").collect(anInt);

	Statistics.coverageOf("range", coverage -> coverage.check("small").percentage(p -> p > 20.0));
	Statistics.coverageOf("value", coverage -> coverage.check(0).count(c -> c > 0));
}

from jqwik.

1. Sometimes there are more than single parameter. How it will look for a such case? Would be something like this?:

@Property(generation = GenerationMode.RANDOMIZED)
void labeledStatistics(@ForAll @IntRange(min = 1, max = 10) Integer anInt, @ForAll @IntRange(min = 41, max = 43) Integer answerInt) {
	String range = anInt < 3 ? "small" : "large";
        String combinedComplexity = Math.round(Math.log(55.4 * anInt / answerInt));

	Statistics.label("range").collect(range);
	Statistics.label("value").collect(anInt);
        Statistics.label("value2").collect(answerInt);
        Statistics.label("combinedComplexity").collect(combinedComplexity);


	Statistics.coverageOf("range", coverage -> coverage.check("small").percentage(p -> p > 20.0));
        // ensure at least single example with anInt = 0:
        Statistics.coverageOf("value", coverage -> coverage.check(0).count(c -> c > 0));
        // I expect at least two different combinedComplexity values (it would get values 4 and 5)
	Statistics.coverageOf("combinedComplexity", coverage -> coverage.valuesCount(c -> c >= 2));
}

2. Does Statistics.check(...) influence number of examples tested? If I remember correctly, John Hughes has mentioned something like running so much tests, that framework can be "certain", that it is not just bad luck to not have enough percentage of samples to pass check.

3. What about Statistics.printHistogram() with optional parameter for label? Nice and simple "ascii art" histogram is shown here for example. I would like to see after finishing all iterations on given test, what was distribution.

4. Is there any option to not include statistics/label/coverage code in testing code block? I.e. using annotation parameters for this?

Or searching for each test for method named "${originalTestName}Stats" with same parameters (signature), which is not annotated and is private with returning value of type "CoverageRestrictions"? For example something like this:

@Property(generation = GenerationMode.RANDOMIZED)
void simpleStats(@ForAll RoundingMode mode) {
	Statistics.collect(mode);
	Statistics.coverage(coverage -> {
		coverage.check(RoundingMode.CEILING).percentage(p -> p > 5.0);
		coverage.check(RoundingMode.FLOOR).count(c -> c > 2);
	});
}

private CoverageRestrictions simpleStatsStats(RoundingMode mode) {
	Statistics.collect(mode);
        Statistics.printHistogram();
        // perhaps consuming histogram by own code can be nice and we could that way check for deviation of arbitrates values over test runs...
        return Statistics.noCoverageRestrictions();
}

Hmm, what is right balance between "magic naming" and method boilerplate? I address this issue due to small LOC of testing methods, which can be doubled just by adding handful number of Statistics lines.

from jqwik.

@luvarqpp Let me comment your remarks...

ad 1) There is already the possibility to combine values on collection. I'm thinking of doing coverage checking analogous, e.g.

@Property
void combinedStats(@ForAll int anInt) {
	String posOrNeg = anInt > 0 ? "positive" : "negative";
	String evenOrOdd = anInt % 2 == 0 ? "even" : "odd";
	Statistics.collect(posOrNeg, evenOrOdd);

	Statistics.coverage(coverage -> {
		coverage.check("positive", "even").count(c -> c > 0);
		coverage.check("negative", "even").count(c -> c > 0);
		coverage.check("positive", "odd").count(c -> c > 0);
		coverage.check("negative", "odd").count(c -> c > 0);
	});
}

Alternatively something like checkAll would be feasible:

Statistics.coverage(coverage -> {
	coverage.checkAllCombinations(
		asList("positive", "negative"),
		asList("even", "odd")
	).count(c -> c > 0);
});

I also feel that a different kind of statistics API could help here. What if just the raw values were collected and aggregation into classifiers would come later? Something like

enum Sign {POSITIVE, NEGATIVE}
enum Oddity {EVEN, ODD}

@Property
void combinedStats3(@ForAll int anInt) {
	Statistics.collect(anInt);

	Statistics.coverage(coverage -> {
		Function<Integer, Sign> posOrNeg = i -> i > 0 ? Sign.POSITIVE : Sign.NEGATIVE;
		Function<Integer, Oddity>  evenOrOdd = i -> i % 2 == 0 ? Oddity.EVEN : Oddity.ODD;
		Classifier classifier = Classifier.from(posOrNeg, evenOrOdd);

		coverage.classify(posOrNeg, evenOrOdd)
				.checkAll().count(c -> c > 0);
				.check(Sign.NEGATIVE, Oddity.ODD).percentage(p -> p > 5.0);
	});
}

ad 2) I've seen his presentation about it and it is indeed clever. I won't go for that in the beginning but it's a worthwhile step down the road.

ad 3) Looks cool. Different feature I'd say. I opened a new issue: #83

ad 4) I haven't thought about that yet. First reaction: coverage checking is part of a property and should be visible there. I conceed it's not exactly the same as a property invariant. Let it sink in with me. Maybe you want to open an issue with that suggestion.

from jqwik.

Checking of counts is now available in 1.2.3-SNAPSHOT:

interface CoverageChecker {

	/**
	 * Check the number of occurrences returning true (ok) or false (fail)
	 *
	 * @param countChecker a predicate to accept a select value sets number of occurrences
	 */
	void count(Predicate<Integer> countChecker);

	/**
	 * Check the number of occurrences using one or more assertions.
	 *
	 * @param countChecker a consumer to accept a select value sets number of occurrences
	 */
	void count(BiPredicate<Integer, Integer> countChecker);

	/**
	 * Check the number of occurrences returning true (ok) or false (fail).
	 *
	 * @param countChecker a predicate to accept a select value sets number of occurrences
	 *                     and the count of all submitted value sets to compare with
	 *                     or make a calculation
	 */
	void count(Consumer<Integer> countChecker);

	/**
	 * Check the number of occurrences using one or more assertions.
	 *
	 * @param countChecker a predicate to accept a select value sets number of occurrences
	 *                     and the count of all submitted value sets to compare with
	 *                     or make a calculation
	 */
	void count(BiConsumer<Integer, Integer> countChecker);
}

from jqwik.

Percentage checking is now possible:

// Make sure that at least 40% of all generated integers are greater than 0
@Property
void percentageCheckPredicate(@ForAll int anInt) {
	Statistics.collect(anInt > 0);
	Statistics.coverage(coverage -> {
		coverage.check(true).percentage(p -> p > 40.0);
	});
}

from jqwik.

Ad hoc queries allow to do simililar stuff:

// Make sure that about 50% of generated values are equal or smaller than 50
@Property(generation = GenerationMode.EXHAUSTIVE)
void checkPercentageForQuery(@ForAll @IntRange(min = 1, max = 100) int anInt) {
	Statistics.collect(anInt);

	Statistics.coverage(coverage -> {
		Predicate<List<Integer>> query = params -> params.get(0) <= 50;
		coverage.checkQuery(query).percentage(p -> {
			Assertions.assertThat(p).isCloseTo(50.0, Offset.offset(0.1));
		});
	});
}

from jqwik.

BTW, Statistics has moved to package net.jqwik.api.statistics.

net.jqwik.api.Statstics has been deprecated.

Current implementation is available in 1.2.3-SNAPSHOT

from jqwik.

Released in version 1.2.3

Documentation is here: https://jqwik.net/docs/1.2.3/user-guide.html#checking-coverage-of-collected-statistics

from jqwik.

@jlink hi, I have put Statistics to work just now and I have some small things. If they are worth of separate issue, I can make them.

1. using labels and collect was not "intuitive" without having a look at userguide. I think, that javadoc can be improved (on the other hand, verbosity in javadoc can lead to other problems, so be strict in review). PR is on its way and perhaps today will be sent.

2. Some CoverageChecker.count methods does have wrong description of its parameter (and I am not sure about Predicate versus Consumer disproportion).

3. What about fluent usage of label, collect and coverage? Now there are two possibilities:

        Statistics.label("passwdLength").collect(pwdLen);
        Statistics.label("passwdLength").coverage(checker -> {
            checker.check("zero").count(x -> x > 10);
            checker.check("more than 13").count(x -> x > 10);
        });

        Statistics.label("passwdLength").collect(pwdLen);
        Statistics.coverageOf("passwdLength", checker -> {
            checker.check("zero").count(x -> x > 10);
            checker.check("more than 13").count(x -> x > 10);
        });

I think that using more fluent approach can be handy and perhaps enough as the only way of writing it to code:

        Statistics.label("passwdLength")
                .collect(pwdLen)
                .coverageOf(checker -> {
                    checker.check("zero").count(x -> x > 10);
                    checker.check("more than 13").count(x -> x > 10);
        });

What do you think?

4. Just cherry on the top of cake, what about to report statistics like this?:

timestamp = 2020-01-29T15:01:29.600693, [SignUpTests:arbitraryQaronUserIsAlwaysValid] (2000) passwdLength = 
    more than 13 (1360) : 68 %       #checked by "percentage > 3.14"
    9..13        ( 228) : 11 %
    4..8         ( 222) : 11 %
    1..3         ( 124) :  6 %
    zero         (  66) :  3 %                #checked by "count > 10"

I am not sure, how easy it would be to put natural language (or exact source code string) there (like "checked that arbitraries count is higher than 10") there, but at least some mark, that any check is present, would be nice.

PS: Great work on statistics. I like it just from beginning. Current report printout is very readable and also javadoc of things is handy in many cases. Thanks.

from jqwik.

@luvarqpp ad 2: Fixed it

from jqwik.

@luvarqpp ad 4:

The idea of reporting if something was checked is interesting. I've never thought of that before.
I don't see a way to get the source code of a condition. Parsing the byte code one could probably get close (without the parameter name unless specific compiler options are set).
The alternative is to have more specific check methods like checkPercentageAtLeast. I found the more generic approach more generic (ok that's obvious) especially since you can use assertions that report well in case of failures. But that doesn't help for reporting.

The simplest thing I can think of is s.t. like "checked count" or "checked percentage"

Added #87 to reflect that

from jqwik.

@luvarqpp ad 3:

I did consider that and rejected the idea for a reason I cannot remember :-(
I'll reconsider and get back.

from jqwik.

ad 3 again:

Magically

@Property
void passwords(
	@ForAll @AlphaChars @NumericChars @StringLength(min = 0, max = 30) String password) {
	int pwdLen = password.length();

	Statistics.label("passwdLength").collect(pwdLen);
	Statistics.label("passwdLength").coverage(checker -> {
		checker.check(0).count(x -> x > 10);
		Predicate<List<Integer>> moreThan13 = p -> p.get(0) > 13;
		checker.checkQuery(moreThan13).count(x -> x > 10);
	});
}

already works! It's just not documented.

What doesn't work yet is

Statistics
	.label("passwdLength")
	.collect(pwdLen);
	.coverage(checker -> {
		checker.check(0).count(x -> x > 10);
		...
	});

But it would be a piece of cake to implement - indeed more or less a single added return this;.

What I'm not sure about is removing Statistics.coverage(checker -> ...) because it allows to put coverage checking at the end of a property which is otherwise not possible for statistics that do not have a label of their own.

from jqwik.

In #89 I am delivering the goods. Please review it at least for grammar and style.

In #90 I have tried to "return this" for collect method. If you reconsider its value, you can just use it. It is through extra simple change.

from jqwik.

What I'm not sure about is removing Statistics.coverage(checker -> ...) because it allows to put coverage checking at the end of a property which is otherwise not possible for statistics that do not have a label of their own.

I removed Statistics.coverageOf after applying #90.

from jqwik.

Applied #89 and published "1.2.4-SNAPSHOT"

from jqwik.

Here's the snapshot javadoc: https://jqwik.net/docs/snapshot/javadoc/net/jqwik/api/statistics/Statistics.html

from jqwik.