1jajen1 / propcheck Goto Github PK
View Code? Open in Web Editor NEWProperty based testing for kotlintest ported from quickcheck
License: Other
Property based testing for kotlintest ported from quickcheck
License: Other
Currently the rose monad is not stacksafe and thus breaks MonadLaw tests. This is rarely a problem in practice, but definitly a bug that should be fixed.
Currently the counterexample added by eqv
just prints both values one after the other. A proper diff can make it a lot clearer what is happening.
I have a working version that uses kotlin-pretty
for nice output, a parser for generic toString
output (works best with data classes) and a decent diffing algorithm. The result is quite nice and I plan on releasing it together with the internals rewrite soon.
This can be taken a lot further with showing errors and values at correct positions in code etc.
First the smaller issue - a typo in the readme:
testImplementation: 'propCheck:propCheck-kt:0.9.3'
the :
should be removed.
Adding the maven repository and the gradle dependency (without :
) I get the following error:
Execution failed for task ':compileTestKotlin'.
> Could not resolve all files for configuration ':testCompileClasspath'.
> Could not find propCheck:propCheck-kt:0.9.3.
Searched in the following locations:
- https://jcenter.bintray.com/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.pom
- https://jcenter.bintray.com/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.jar
- https://repo.maven.apache.org/maven2/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.pom
- https://repo.maven.apache.org/maven2/propCheck/propCheck-kt/0.9.3/propCheck-kt-0.9.3.jar
Function generation has some quirks that need to be generated.
List of topics to cover:
Func
and Coarbitrary
I am sort of cheating by using kotlin.Random and generating tuples of A (value generated) and a Long (new seed). That should be fine for running tests, but there might be better ways. A review would be great :)
Not necessarily against other libraries. The main focus here is to check regressions when dealing with performance related issues. Just setting up something that runs on the ci and reports regressions is enough. Actual benchmarks will follow eventually.
There is a lot still undocumented or in need of examples. The readme file is already too long, so split into several wiki like pages.
I am by no means an expert when it comes to gradle and especially the publish part could use a second pair of eyes :).
fun main() {
propCheck(Args(maxShrinks = 10000)) {
forAll { (f): Fun<Int, Int> ->
f(0) == f(1)
}
}
}
Sometimes shrinks down to [0 -> 1, _ -> 0]
(or similar) very fast, but sometimes just seems to loop somewhere. This is kind of hard to debug, but needs a fix!
Shrinking always terminates if implemented correctly, but in the meantime due to missing lazyness very large lists will be generated and sometimes even retained in memory. The worst offender here is shrinkList
, it would be much better if it operated on the Sequence
data-type and only forced loading the entire thing if it's needed (when the list cannot be cut down to smaller sizes anymore, or that cutting had no effect). This is not an easy task because the implicit strictness of everything in kotlin makes for some fun situations (for example m(): Sequence is not lazy in generation, but sequenceOf(Unit).flatMap(m) is) It is quite tricky to find all of those situations, but it needs to be done to get shrinking back to a performant level on large sequences.
Benefits:
Arbitrary
which is in itself not a big problem, but jut a hassle to implement, carry around and lookup.This comes with a few drawbacks:
My take on this atm is: Convenience over the best possible shrinking (which can still be recovered by extra work if needed). I'll start playing around with integrated shrinking once the internal rewrite is done.
This is a very good resource on the differences between the two and their drawbacks: http://www.well-typed.com/blog/2019/05/integrated-shrinking/#fnref10
Right now exception handling in tests is left to the user writing the test and any exception thrown will not be caught. That is fine for most cases (when programming in a functional way and prefering total functions) but it may be a restriction that does not need to be there. Any ideas on how to approach this (if at all) are welcome :)
Several things:
Just a basic dsl to write property based tests that then will be detected and executed, reporting failure. This should be mpp if possible.
A custom test runner is all great and fine, but interop with other libraries is a good thing as well. As this means simply offering wrappers and dsl's to build tests with propCheck this should be a easy task. Just a module with the correct dependencies and wrappers around their methods/classes.
There are only two jvm dependencies atm:
arrow
is going to get mpp support soon, no idea about IO
tho, so it's best to wait on that for now.
commons-math
is used in one place only and that is for statistics analysis. Either replace it with a pure kotlin lib or implement that method here.
This is kept as one issue atm as a reminder. I'll split this soon when I have though about this some more.
The interop libraries would be amazing to have for 1.0, a test runner would not be bad as well but can be put off. Mpp is not a priority until arrow
has it's IO or something equivalent mpp compatible.
Hedgehog has a much nicer api than quickcheck (although I don't think it's too bad), which also comes with tradeoffs:
One such tradeoff is shrinking is limited in monadic context to be an in order shrink, which if the failure is on the second param can lead to unshrunk first parameters being displayed. This is especially bad for shrinking functions because unshrunk functions aren't displayed.
Another tradeoff is, in kotlin an exact port would have higher-kinded return types etc, which is not really feasable, so some sort of compromise needs to be found (that is the reason I ported quickcheck and not hedgehog in the first place)
Things I would like:
First class effectful testing and generating. While easy in haskell this is an api burden in kotlin, needs some tests too see how that will look.
Merging shrinkers with generators in a lawful way. This will get back a lot of convenience (removing the need for Arbitrary altogether). If it can be done without sacrificing too much in terms of shrinking then it'll be amazing!
The AMAZING printout hedgehog gives! This should be possible!
Convenience for kotlin:
Exception handling needs first class support
Suspend function support
This is just a list of ideas so I don't forget, feel free to comment with more, but open seperate issues if you have something more concrete to do.
Regardless of what gets implemented: The current internal implementation is quite rigid, to make it easier to test and write these changes a serious rewrite with more polymorphic functions is needed. That could be possible without any api change at all. Since the library is somewhat feature complete now, that will be the next focus.
Statemachine testing is available in both sequential and parallel mode, but is not yet documented.
See the tests for it to understand how it works.
Topics to cover:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.