1jajen1 / propcheck Goto Github PK

Function generation has some quirks that need to be generated.
List of topics to cover:

• Implementing Func and Coarbitrary
• Using it in forAlls and the rules that function generation follows (determinism etc)
• Gotchas in regards to shrinking (very weak support for shrinking atm, will be a bit better when #13 is done)

Several things:

• A custom test runner with no dependencies other than kotlin mpp libraries (seperate module)
• Modules for interop with existing testing libraries
• As a step towards mpp: Drop all jvm dependencies

Custom test runner:

Just a basic dsl to write property based tests that then will be detected and executed, reporting failure. This should be mpp if possible.

Interop with testing libraries:

A custom test runner is all great and fine, but interop with other libraries is a good thing as well. As this means simply offering wrappers and dsl's to build tests with propCheck this should be a easy task. Just a module with the correct dependencies and wrappers around their methods/classes.

Mpp:

There are only two jvm dependencies atm:

• arrow
• commons-math

arrow is going to get mpp support soon, no idea about IO tho, so it's best to wait on that for now.
commons-math is used in one place only and that is for statistics analysis. Either replace it with a pure kotlin lib or implement that method here.

This is kept as one issue atm as a reminder. I'll split this soon when I have though about this some more.
The interop libraries would be amazing to have for 1.0, a test runner would not be bad as well but can be put off. Mpp is not a priority until arrow has it's IO or something equivalent mpp compatible.

Currently the counterexample added by eqv just prints both values one after the other. A proper diff can make it a lot clearer what is happening.

I have a working version that uses kotlin-pretty for nice output, a parser for generic toString output (works best with data classes) and a decent diffing algorithm. The result is quite nice and I plan on releasing it together with the internals rewrite soon.

This can be taken a lot further with showing errors and values at correct positions in code etc.

There is a lot still undocumented or in need of examples. The readme file is already too long, so split into several wiki like pages.

I am sort of cheating by using kotlin.Random and generating tuples of A (value generated) and a Long (new seed). That should be fine for running tests, but there might be better ways. A review would be great :)

I am by no means an expert when it comes to gradle and especially the publish part could use a second pair of eyes :).

Currently the rose monad is not stacksafe and thus breaks MonadLaw tests. This is rarely a problem in practice, but definitly a bug that should be fixed.

• More and better examples.
• In general more descriptive text
• Ank checked examples to get results

Benefits:

• A user would not have to write a shrinking function for most if not all data-types
• Shrinking automatically follows invariants by construction. This reduces duplication in logic that checks/maintains those when manually shrinking
• Allows to get rid of Arbitrary which is in itself not a big problem, but jut a hassle to implement, carry around and lookup.
• Since shrinking is coupled to generators and changes to gen also change shrinking, which is a good thing if for example range is changed you'd not want shrunk results to be outside of that range

This comes with a few drawbacks:

• When two generators are combined monadically they are shrunk one after the other and if the first one fails to shrink it will not be shrunk again. This can be overcome with using Applicative combinators wherever possible. Yet another way to overcome this is to simply add back a shrinker. While certainly not great, it gives the same level of control as type based shrinking (Or at least I think it does)
• Generating functions relies on Coarbitrary and Function and some unsafe methods. I do not want to loose generating functions so there must be a way. I'd be fine keeping the typeclasses for the two, also the unsafe methods, but this needs to be done in a way that still allows integrated shrinking to work

My take on this atm is: Convenience over the best possible shrinking (which can still be recovered by extra work if needed). I'll start playing around with integrated shrinking once the internal rewrite is done.

This is a very good resource on the differences between the two and their drawbacks: http://www.well-typed.com/blog/2019/05/integrated-shrinking/#fnref10

Hedgehog has a much nicer api than quickcheck (although I don't think it's too bad), which also comes with tradeoffs:

One such tradeoff is shrinking is limited in monadic context to be an in order shrink, which if the failure is on the second param can lead to unshrunk first parameters being displayed. This is especially bad for shrinking functions because unshrunk functions aren't displayed.
Another tradeoff is, in kotlin an exact port would have higher-kinded return types etc, which is not really feasable, so some sort of compromise needs to be found (that is the reason I ported quickcheck and not hedgehog in the first place)

Things I would like:
First class effectful testing and generating. While easy in haskell this is an api burden in kotlin, needs some tests too see how that will look.
Merging shrinkers with generators in a lawful way. This will get back a lot of convenience (removing the need for Arbitrary altogether). If it can be done without sacrificing too much in terms of shrinking then it'll be amazing!
The AMAZING printout hedgehog gives! This should be possible!

Convenience for kotlin:
Exception handling needs first class support
Suspend function support

This is just a list of ideas so I don't forget, feel free to comment with more, but open seperate issues if you have something more concrete to do.

Regardless of what gets implemented: The current internal implementation is quite rigid, to make it easier to test and write these changes a serious rewrite with more polymorphic functions is needed. That could be possible without any api change at all. Since the library is somewhat feature complete now, that will be the next focus.

Not necessarily against other libraries. The main focus here is to check regressions when dealing with performance related issues. Just setting up something that runs on the ci and reports regressions is enough. Actual benchmarks will follow eventually.

Shrinking always terminates if implemented correctly, but in the meantime due to missing lazyness very large lists will be generated and sometimes even retained in memory. The worst offender here is shrinkList, it would be much better if it operated on the Sequence data-type and only forced loading the entire thing if it's needed (when the list cannot be cut down to smaller sizes anymore, or that cutting had no effect). This is not an easy task because the implicit strictness of everything in kotlin makes for some fun situations (for example m(): Sequence is not lazy in generation, but sequenceOf(Unit).flatMap(m) is) It is quite tricky to find all of those situations, but it needs to be done to get shrinking back to a performant level on large sequences.

First the smaller issue - a typo in the readme:

testImplementation: 'propCheck:propCheck-kt:0.9.3'

the : should be removed.

Adding the maven repository and the gradle dependency (without :) I get the following error:

Execution failed for task ':compileTestKotlin'.
> Could not resolve all files for configuration ':testCompileClasspath'.
   > Could not find propCheck:propCheck-kt:0.9.3.
     Searched in the following locations:
       - https://jcenter.bintray.com/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.pom
       - https://jcenter.bintray.com/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.jar
       - https://repo.maven.apache.org/maven2/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.pom
       - https://repo.maven.apache.org/maven2/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.jar

Right now exception handling in tests is left to the user writing the test and any exception thrown will not be caught. That is fine for most cases (when programming in a functional way and prefering total functions) but it may be a restriction that does not need to be there. Any ideas on how to approach this (if at all) are welcome :)

Statemachine testing is available in both sequential and parallel mode, but is not yet documented.
See the tests for it to understand how it works.

Topics to cover:

• Implementing a state-machine test
• When to use state-machine tests
• Parallel tests and their uses (Catching race conditions,...)
• Some gotchas regarding shrinking parallel tests (non-deterministic failure means shrinking is not deterministic either)

fun main() {
    propCheck(Args(maxShrinks = 10000)) {
        forAll { (f): Fun<Int, Int> ->
            f(0) == f(1)
        }
    }
}

Sometimes shrinks down to [0 -> 1, _ -> 0] (or similar) very fast, but sometimes just seems to loop somewhere. This is kind of hard to debug, but needs a fix!