leanovate / gopter Goto Github PK

Looking for clarification on how a complex state is supposed to be handled along with creation and destruction of a SUT.

For example, testing a circular buffer you could have a container/list mirror the contents of the circular buffer to verify behaviour in a state structure. However, I do not see how to correlate re-initializing the state when the SUT is destroyed and recreated. Is there something I am missing?

Is there any interest in supporting generics in gopter?

I'm willing to start taking a stab at it, but I want to make sure the work gets reviewed by someone.

Release 0.1.0 was made in April 2016. There have been changes since that would benefit users. With the increasing prevalence of dep, the 0.1.0 release will be chosen by default. We should make a new release for dep to use.

when you have a generator that looks like this:

genStruct({
  a: genSlice(genStruct({
    b: genString().SuchThat(...SomeCondition),
    c: genString().SuchThat(...SomeOtherCondition)
  }))
})

An index out of bounds exception is thrown here:

it happens because for some reason baseSieve only has a single value (SomeCondition) but ups has two values (b and c) in it. So when you loop over ups and grab the corresponding baseSieve item for the up (at i) you get that panic.

This doesn't happen on the latest release but does happen on master.

minimal example:

type TestBook struct {
	Title   string
	Content string
}

func genTestBook() gopter.Gen {
	return gen.Struct(reflect.TypeOf(&TestBook{}), map[string]gopter.Gen{
		"Title": gen.AlphaString().SuchThat(func(s string) bool {
			fmt.Println("NonEmptyString1")
			return s != ""
		}),
		"Content": gen.AlphaString().SuchThat(func(s string) bool {
			fmt.Println("NonEmptyString2")
			return s != ""
		}),
	})
}

type TestLibrary struct {
	Books []TestBook
}

func genTestLibrary() gopter.Gen {
	return gen.Struct(reflect.TypeOf(&TestLibrary{}), map[string]gopter.Gen{
		"Books": gen.SliceOf(genTestBook()),
	})
}

This should probably block #60

Hi @untoldwind,

We've recently started to use gopter (which is a great project thanks) and came across a bug when using multiple workers and a failure occurred it wouldn't return the full test result error, labels or prop args, making hard to understand what generated value caused the test failure. Which I raised a new PR for review: #36

However that exposed another bug unfortunately with math/rand. Essentially rand.New(rand.NewSource(..)) is not thread safe when calling for new rand Int - running across multiple workers and you can validate with this smallest example - it doesn't happen consistently but happens regularly

https://gist.github.com/jnormington/a832472edf9fa0db020ebd028b7849b9
Issues on golang;

• golang/go#318
• golang/go#3611

I've looked at this and there are a few solutions;

1. Use rand.Int... directly from the rand package and set the seed
2. Create a new rand instance with the same seed across multiple workers
3. Copy lockedSource (how golang does it as lockedSource isn't exposed)
   https://github.com/golang/go/blob/master/src/math/rand/rand.go#L371-L410

There are a few problems with 1 & 2 though

1. Means changing the code in a lot of places which I don't feel is great to use rand we lose some control
2. This means multiple workers could generate the same values as they would each have the same seed in a new initialized rand.New(rand.NewSource(..)) call
3. I like three it means very minimal change but gives multiple workers thread safe access to rand source. With the same control with genParams as we have today without a breaking change.

We've updated our vendored version and works great.

So I wanted to get your thoughts on the issue and whether you had any ideas or was happy with this approach or if you had other concerns I haven't thought of. Below is an example of point 3

diff --git a/gen_parameters.go b/gen_parameters.go
index aca147c..a146b10 100644
--- a/gen_parameters.go
+++ b/gen_parameters.go
@@ -51,7 +51,7 @@ func (p *GenParameters) CloneWithSeed(seed int64) *GenParameters {
 		MinSize:        p.MinSize,
 		MaxSize:        p.MaxSize,
 		MaxShrinkCount: p.MaxShrinkCount,
-		Rng:            rand.New(rand.NewSource(seed)),
+		Rng:            rand.New(NewLockedSource(seed)),
 	}
 }
 
@@ -63,6 +63,6 @@ func DefaultGenParameters() *GenParameters {
 		MinSize:        0,
 		MaxSize:        100,
 		MaxShrinkCount: 1000,
-		Rng:            rand.New(rand.NewSource(seed)),
+		Rng:            rand.New(NewLockedSource(seed)),
 	}
 }
diff --git a/rand_source.go b/rand_source.go
new file mode 100644
index 0000000..34f5241
--- /dev/null
+++ b/rand_source.go
@@ -0,0 +1,77 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+// Taken from golang lockedSource implementation https://github.com/golang/go/blob/master/src/math/rand/rand.go#L371-L410
+
+package gopter
+
+import (
+	"math/rand"
+	"sync"
+)
+
+type lockedSource struct {
+	lk  sync.Mutex
+	src rand.Source64
+}
+
+// NewLockedSource takes a seed and returns a new
+// lockedSource for use with rand.New
+func NewLockedSource(seed int64) *lockedSource {
+	return &lockedSource{
+		src: rand.NewSource(seed).(rand.Source64),
+	}
+}
+
+func (r *lockedSource) Int63() (n int64) {
+	r.lk.Lock()
+	n = r.src.Int63()
+	r.lk.Unlock()
+	return
+}
+
+func (r *lockedSource) Uint64() (n uint64) {
+	r.lk.Lock()
+	n = r.src.Uint64()
+	r.lk.Unlock()
+	return
+}
+
+func (r *lockedSource) Seed(seed int64) {
+	r.lk.Lock()
+	r.src.Seed(seed)
+	r.lk.Unlock()
+}
+
+// seedPos implements Seed for a lockedSource without a race condition.
+func (r *lockedSource) seedPos(seed int64, readPos *int8) {
+	r.lk.Lock()
+	r.src.Seed(seed)
+	*readPos = 0
+	r.lk.Unlock()
+}
+
+// read implements Read for a lockedSource without a race condition.
+func (r *lockedSource) read(p []byte, readVal *int64, readPos *int8) (n int, err error) {
+	r.lk.Lock()
+	n, err = read(p, r.src.Int63, readVal, readPos)
+	r.lk.Unlock()
+	return
+}
+
+func read(p []byte, int63 func() int64, readVal *int64, readPos *int8) (n int, err error) {
+	pos := *readPos
+	val := *readVal
+	for n = 0; n < len(p); n++ {
+		if pos == 0 {
+			val = int63()
+			pos = 7
+		}
+		p[n] = byte(val)
+		val >>= 8
+		pos--
+	}
+	*readPos = pos
+	*readVal = val
+	return
+}

I have a use case where I want to generate strings with a length between n and m. Currently I am doing this:

	makeStringGen := func(min, max int) gopter.Gen {
		return gen.Sized(func(size int) gopter.Gen {
			return gen.SliceOfN(size%max+1, gen.Rune()).
				SuchThat(func(rs []rune) bool { return len(rs) >= min }).
				Map(func(v []rune) string { return string(v) })
		})
	}

but I imagine that using % would probably mess with the distribution of sizes and using SuchThat is fine in my use case since the min is generally 0 or 1 but it seems suboptimal as a general solution. What's the best way of doing this sort of thing?

Property based tests are great tools to assert properties of an API. Though I haven't done it in Go, this package looks useful.

I'm more curious about the implementation and the algorithm. Do you know any references or papers I could read?

It seems to me this library has less powerful "generator inference" than the simple testing/quick one, which has no problem with this case or with #27

Given this input program

package main

import (
	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/arbitrary"
)

type MyIDType int32

type Foo struct {
	Name string
	Id   MyIDType
}

func main() {
	parameters := gopter.DefaultTestParameters()
	parameters.Rng.Seed(1234)

	arbitraries := arbitrary.DefaultArbitraries()

	properties := gopter.NewProperties(parameters)

	properties.Property("Foo", arbitraries.ForAll(
		func(foo *Foo) bool {
			return true
		}))

	properties.Run(gopter.ConsoleReporter(false))
}

Output:

! Foo: Error on property evaluation after 0 passed tests: Check paniced:
   reflect.Set: value of type int32 is not assignable to type main.MyIDType
   goroutine 1 [running]:
runtime/debug.Stack(0xc42004b788, 0x4f0980, 0xc42007e4d0)
	/usr/x86_64-pc-linux-gnu/lib/go/src/runtime/debug/stack.go:24 +0xa7
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Save
  Prop.func1.1(0xc42004bd28)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:19 +0x6e
panic(0x4f0980, 0xc42007e4d0)
	/usr/x86_64-pc-linux-gnu/lib/go/src/runtime/panic.go:505 +0x229
reflect.Value.assignTo(0x4f0340, 0xc42008bdf8, 0x85, 0x521cbc, 0xb,
   0x4f0400, 0x0, 0x4f0340, 0x4f0340, 0xc42008bdf8)
	/usr/x86_64-pc-linux-gnu/lib/go/src/reflect/value.go:2235 +0x427
reflect.Value.Set(0x4f0400, 0xc4200bc4d0, 0x185, 0x4f0340, 0xc42008bdf8,
   0x85)
	/usr/x86_64-pc-linux-gnu/lib/go/src/reflect/value.go:1373 +0xa4
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter/gen.
  StructPtr.func1(0xc4200bc480, 0x1)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/gen/struct.go:64 +0x327
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter/prop
  .ForAll.func1(0xc4200bc480, 0x529668)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop/forall.go:31 +0x140
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Save
  Prop.func1(0xc4200bc480, 0x0)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:24 +0x6c
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Prop
  .Check.func1(0x0, 0xc42007e470, 0x5d4f60)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:52 +0x15c
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.(*ru
  nner).runWorkers(0xc420082720, 0x0)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/runner.go:45 +0x2d9
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Prop
  .Check(0xc42007e450, 0xc4200a05c0, 0x520b0c)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:110 +0x1af
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.(*Pr
  operties).Run(0xc4200822a0, 0x538c20, 0xc4200bc440, 0x3)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/properties.go:37 +0xc9
main.main()
	/home/julian/go/src/github.com/hasufell/gopter-examples/gopterexamples.go:
  28 +0x19d

Hi,

I really like gopter so far. Thanks for providing such a useful library. I've just run into something unexpected with stateful testing though.

What I want is after each operation to not only compare the Result of the SystemUnderTest command with the model state (State) as in Postcondition, but also to ensure that the internal state of the SystemUnderTest is logically equivalent to the model state. I tried including a pointer to the SystemUnderTest in State so I could just perform this check in Postcondition, but it appears that by the time Postcondition runs, the pointer is pointing to a different value than what was initialized and returned from NewSystemUnderTest.

The rationale for this feature is to be able to test mutable methods on types that don't return values. So far I think the only workaround is to provide some introspection methods that we add as commands so we can then run and compare the output to the model state.

I was wondering if you could provide this capability, via another callback like invariant/1 in eqc_statem, or if there is a specific reason you think it should not be included. And yes, effort is a fine justification :) Thanks!

It's important that generators produce the same value for the same GenParameters (same seed, etc.) The enumeration of Go maps is non-deterministic, in fact the Go designers have gone out of their way to make it random! This means that the values produced by the Gens in the map passed to gen.Struct are most likely to be different from one call to the next given the same PRNG.

I am looking for a way to generate values which depend on each other. In particular, I want to recreate the following simple example from ScalaCheck's User Guide (https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md#generators), where two integer are generated, the first in the range of 10..20, the second has a lower bound which is twice as large as the first value:

// ScalaCheck Example
val myGen = for {
  n <- Gen.choose(10,20)
  m <- Gen.choose(2*n, 500)
} yield (n,m)

My impression was that the gen.FlatMap() should provide the required functionality (in Scala, <- is a monadic assignment, implemented by FlatMap), but I failed to find a way to succeed.

I defined a simple struct to generate two values which can be fed into the property:

type IntPair struct {
		Fst int
		Snd int
	}
properties.Property("ScalaCheck example for a pair", prop.ForAll(
		func(p IntPair) bool {
			a := p.Fst
			b := p.Snd
			return a*2 <= b
		},
		genIntPairScala(),
	))

The generator is a straight translation of the Scala code, first generating an integer and then generating a second via accessing the generated value of the first. Both generators are finally stored in the struct generator:

genIntPairScala := func() gopter.Gen {
		n := gen.IntRange(10, 20).WithLabel("n (fst)")
		m := n.FlatMap(func(v interface{}) gopter.Gen {
			k := v.(int)
			return gen.IntRange(2*k, 50)
		}, reflect.TypeOf(int(0))).WithLabel("m (snd)")

		var gen_map = map[string]gopter.Gen{"Fst": n, "Snd": m}
		return gen.Struct(
			reflect.TypeOf(IntPair{}),
			gen_map,
		)
	}

However, it does not work:

=== RUN   TestGopterGenerators
! ScalaCheck example for a pair: Falsified after 10 passed tests.
n (fst), m (snd): {Fst:17 Snd:32}
n (fst), m (snd)_ORIGINAL (1 shrinks): {Fst:19 Snd:32}
Elapsed time: 233.121µs
    properties.go:57: failed with initial seed: 1617636578517672000

Remark: I set the upper bound to 50 instead of 500. The property must still hold, but the generator has a smaller pool to pick suitable values: setting the upper bound to 500 often results in a passing property!

Gopter is a great tool for exploring a set of possible codepaths and discovering bugs. However, if I happen to find a bug in my code that comes from an unusual codepath, I would want to capture the sequence of operations that exposed that bug, and quickly turn them into a regression test to live in my suite independent of my normal Gopter tests. In other words, I want to be able to have my Gopter tests run, and then write regression tests in terms of Gopter.

E.g. Let's say my Gopter test for my simple counter fails with the sequence of events

initialState: {0}
sequential=[INCREMENT INCREMENT RESET DECREMENT]

I want some way to write that sequence of commands into a new test that will essentially run those exact commands within the Gopter framework, the same way I would expect them to run when running a more exploratory Gopter test.

Is there any way to leverage the current library to do this?

In the docs I can see that GenParameters.WithSize() can be used to control the maximum length of string arbitraries, but I can not figure out how to use it to limit the length of strings generated by gen.AlphaString(), gen.AnyString, gen.NumString(), etc.

In the code they all call genString(), which in turn calls SliceOf(), which then makes use of genParams.Size (from GenParameters), but genParams gets passed in by some internal function and I do not know how to control this.

On the flip side I see that SliceOfN() will allow an int parameter to specify the length, but my attempts to create a generator using it fails, as using the same map as genString() wants a rune and not []rune.

First of all, I do appreciate all your work on this project. This library is fantastic. I was playing with gopter and found that the property doesn't seem to hold the original state of the sample for reporting, which gives confusing results.

First, I prepared a small target function named biggest, which is expected to return the biggest value in the given slice of int.

func biggest(ns []int) (int, error) {
	if len(ns) == 0 {
		return 0, fmt.Errorf("slice must have at least one element")
	}
	return ns[0], nil
}

As the PBT of this function, I wrote the following test:

func TestBiggest(t *testing.T) {
	properties := gopter.NewProperties(nil)
	properties.Property("Non-zero length small int slice", prop.ForAll(
		func(ns []int) bool {
			result, _ := biggest(ns)
			sort.Slice(ns, func(i, j int) bool {
				return ns[i] > ns[j]
			})
			return result == ns[0]
		},
		gen.SliceOf(gen.IntRange(0, 20),
			reflect.TypeOf(int(0))).
			SuchThat(func(v interface{}) bool {
				return len(v.([]int)) > 0
			}),
	))
	properties.TestingRun(t)
}

The result of this test was:

! Non-zero length small int slice: Falsified after 0 passed tests.
ARG_0: [13 0]
ARG_0_ORIGINAL (1 shrinks): [14 13]
Elapsed time: 164.792µs
--- FAIL: TestBiggest (0.00s)
    /home/ymotongpoo/personal/pbt/sample/properties.go:57: failed with initial seed: 1602330526004760703
FAIL

But these arguments, both the original and the shrunk, should pass the test. Though I haven't dug into the implementation of the default shrinker and reporting, I assumed that the shrinker doesn't hold the original state of the sample. I changed the test code a bit to keep the original state of sample and it returned the expected report.

func TestBiggest(t *testing.T) {
	properties := gopter.NewProperties(nil)
	properties.Property("Non-zero length small int slice", prop.ForAll(
		func(ns []int) bool {
			nns := make([]int, len(ns))
			copy(nns, ns)
			result, _ := biggest(nns)
			sort.Slice(nns, func(i, j int) bool {
				return nns[i] > nns[j]
			})
			return result == nns[0]
		},
		gen.SliceOf(gen.IntRange(0, 20),
			reflect.TypeOf(int(0))).
			SuchThat(func(v interface{}) bool {
				return len(v.([]int)) > 0
			}),
	))
	properties.TestingRun(t)
}

This test made the following report that makes sense to me:

! Non-zero length small int slice: Falsified after 0 passed tests.
ARG_0: [0 1]
ARG_0_ORIGINAL (6 shrinks): [3 19]
Elapsed time: 245.025µs
--- FAIL: TestBiggest (0.00s)
    /home/ymotongpoo/personal/pbt/sample/properties.go:57: failed with initial seed: 1602333441977570032
FAIL

Sometimes we get a failure with the below code

Error on property evaluation after 1 passed
   tests: Check paniced: reflect: Call using *schema.PathMatcherType_Prefix
   as type *schema.PathMatcherType_Path

Could you please suggest if you see any issue ?

Code:

func genPathMatcherType() gopter.Gen {
	return gen.StructPtr(reflect.TypeOf(&pb_schema.PathMatcherType{}), map[string]gopter.Gen{
		"PathMatch": gen.OneGenOf(genPathMatcherType_Prefix(), genPathMatcherType_Path(), genPathMatcherType_Regex()),
	})
}
func genPathMatcherType_Prefix() gopter.Gen {
	return gen.StructPtr(reflect.TypeOf(&pb_schema.PathMatcherType_Prefix{}), map[string]gopter.Gen{
		"Prefix": gen.AlphaString(),
	})
}
func genPathMatcherType_Path() gopter.Gen {
	return gen.StructPtr(reflect.TypeOf(&pb_schema.PathMatcherType_Path{}), map[string]gopter.Gen{
		"Path": gen.AlphaString(),
	})
}
func genPathMatcherType_Regex() gopter.Gen {
	return gen.StructPtr(reflect.TypeOf(&pb_schema.PathMatcherType_Regex{}), map[string]gopter.Gen{
		"Regex": gen.AlphaString(),
	})
}

Hi folks,

I've got a test whose runtime is real slow and I wonder if I'm doing something very wrong. Or, if I'm not doing something wrong if there are any options I've got to improve the test runtime. Here's a representative example of my actual tests:

package main

import (
        "github.com/leanovate/gopter"
        "github.com/leanovate/gopter/arbitrary"
        "github.com/leanovate/gopter/gen"
        "reflect"
        "testing"
)

type TestBook struct {
        Title   string
        Content string
}

func genTestBook() gopter.Gen {
        return gen.Struct(reflect.TypeOf(&TestBook{}), map[string]gopter.Gen{
                "Title":   gen.AlphaString(),
                "Content": gen.AlphaString(),
        })
}

type TestLibrary struct {
        Name       string
        Librarians uint8
        Books      []TestBook
}

func genTestLibrary() gopter.Gen {
        return gen.Struct(reflect.TypeOf(&TestLibrary{}), map[string]gopter.Gen{
                "Name":       gen.AlphaString(),
                "Librarians": gen.UInt8Range(1, 255),
                "Books":      gen.SliceOf(genTestBook()),
        })
}

type CityName = string
type TestCities struct {
        Libraries map[CityName][]TestLibrary
}

func genTestCities() gopter.Gen {
        return gen.StructPtr(reflect.TypeOf(&TestCities{}), map[string]gopter.Gen{
                "Libraries": gen.MapOf(gen.AlphaString(), gen.SliceOf(genTestLibrary())),
        })
}

func TestLibraries(t *testing.T) {
        parameters := gopter.DefaultTestParameters()
        parameters.MinSuccessfulTests = 10
        parameters.MaxSize = 4
        arbitraries := arbitrary.DefaultArbitraries()
        arbitraries.RegisterGen(genTestCities())

        properties := gopter.NewProperties(parameters)

        properties.Property("no unsupervised libraries", arbitraries.ForAll(
                func(tc *TestCities) bool {
                        for _, libraries := range tc.Libraries {
                                for _, library := range libraries {
                                        if library.Librarians == 0 {
                                                return false
                                        }
                                }
                        }
                        return true
                },
        ))

        properties.TestingRun(t)
}

My test builds a tree structure -- TestCities -> TestLibrary -> TestBook -- and then runs a property to assert that no libraries are unsupervised. By construction this property will always pass: genTestLibrary does not allow for TestLibrary.Libarians to be 0. As expected, the property does pass but the runtime is around 30 seconds on my host. An example:

> go test
+ no unsupervised libraries: OK, passed 10 tests.
Elapsed time: 356.71µs
PASS
ok  	_/Users/btroutwine/go/propexample	26.830s

Am I doing anything obviously wrong here?

I've been playing around with gopter for a little while now, trying to understand how to write my own generator for my use case, which is not as straight forward as those in the repo. My use case is the following; I want to test a function (ReadQF) that should return a value and true when enough, i.e. a quorum of replies have been received and passed in to the ReadQF function. It should return false otherwise.

I've hacked together something that seems to work in the following:

https://github.com/relab/byzq/blob/master/authdataspec_property_test.go#L37
https://github.com/relab/byzq/blob/master/authdataspec_property_test.go#L63

However, I suspect it isn't quite in the spirit of property-based testing, and I'm struggling to break it up into multiple generators, since the input parameter n to the NewAuthDataQ constructor that creates a qspec object and computes the parameter q, which is used to decide the minimal/maximal length of the replies array. And furthermore, I need access to the qspec object in the end to decide if a quorum has been received.

I would really appreciate to get some feedback on my two tests linked above, especially, if you can provide some advice on how to decouple things.

(Below is an initial attempt at writing a generator, but I don't know how to get both the quorumSize and qspec parameters out of the generator for consumption in the condition function passed to prop.ForAll().)

func genQuorums(min, max int, hasQuorum bool) gopter.Gen {
	return func(genParams *gopter.GenParameters) *gopter.GenResult {
		rangeSize := uint64(max - min + 1)
		n := int(uint64(min) + (genParams.NextUint64() % rangeSize))
		qspec, err := NewAuthDataQ(n, priv, &priv.PublicKey)
		if err != nil {
			panic(err)
		}
		// initialize as non-quorum
		minQuorum, maxQuorum := math.MinInt32, qspec.q
		if hasQuorum {
			minQuorum, maxQuorum = qspec.q+1, qspec.n
		}
		rangeSize = uint64(maxQuorum - minQuorum + 1)
		quorumSize := int(uint64(minQuorum) + (genParams.NextUint64() % rangeSize))

		genResult := gopter.NewGenResult(qspec, gopter.NoShrinker)
		return genResult
	}
}

A test I wrote using PtrOf sometimes results in a panic like this:

Parent ids not equal
! fields round-trip: Error on property evaluation after 0 passed tests:
   Check paniced: reflect: Call using zero Value argument goroutine 14
   [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x80
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.SaveProp.
  func1.1(0xc8202a9ba8)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop.go:19 +0x53
panic(0x79ffc0, 0xc8203daaf0)
        /usr/lib/go/src/runtime/panic.go:426 +0x4e9
reflect.Value.call(0x8311a0, 0xc8203da2d0, 0x13, 0x8e40c8, 0x4,
   0xc820395f90, 0x3, 0x3, 0x0, 0x0, ...)
        /usr/lib/go/src/reflect/value.go:367 +0x48b
reflect.Value.Call(0x8311a0, 0xc8203da2d0, 0x13, 0xc820395f90, 0x3, 0x3,
   0x0, 0x0, 0x0)
        /usr/lib/go/src/reflect/value.go:303 +0xb1
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter/prop.chec
  kConditionFunc.func2(0xc8203b2900, 0x3, 0x3, 0x3)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop/check_condition_func.go:45 +0x1db
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter/prop.ForA
  ll.func1.1(0x0, 0x0, 0x86a401)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop/forall.go:51 +0x112
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter/prop.firs
  tFailure(0xc8203cea80, 0xc8202a9b40, 0xc8203cea80, 0x0, 0x0)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop/forall.go:104 +0x5f
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter/prop.shri
  nkValue(0x3e8, 0xc820395e00, 0x778480, 0xc8203da7e0, 0xc820395f40,
   0xc8202a9b40, 0xc820395f40, 0x0, 0x0)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop/forall.go:88 +0xb5
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter/prop.ForA
  ll.func1(0xc8203edc00, 0x9b53c8)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop/forall.go:52 +0x53d
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.SaveProp.
  func1(0xc8203edc00, 0x0)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop.go:24 +0x68
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.Prop.Chec
  k.func1(0x0, 0xc8203da360, 0xd44b20)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop.go:51 +0x19e
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.(*runner)
  .runWorkers(0xc8203b25a0, 0x0)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/runner.go:45 +0x102
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.Prop.Chec
  k(0xc8203da340, 0xc8201f4140, 0x938f20)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/prop.go:106 +0x1c2
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.(*Propert
  ies).Run(0xc8203b2480, 0x7ff522cf6968, 0xc8203ed860, 0x45085e)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/properties.go:35 +0xe7
github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gopter.(*Propert
  ies).TestingRun(0xc8203b2480, 0xc8200845a0)
        /golang/src/github.com/ClusterHQ/dataplane/vendor/github.com/leanovate/gop
  ter/properties.go:46 +0x119
github.com/ClusterHQ/dataplane/dataset.TestAddSnapshot(0xc8200845a0)
        /golang/src/github.com/ClusterHQ/dataplane/dataset/dataset_test.go:409
   +0x295
testing.tRunner(0xc8200845a0, 0xd2da98)

I think I minimized the test case that provokes this behavior. You can see it at master...ClusterHQ:call-using-zero-value-argument

I wasn't quite sure where to put this test or even if it's a real problem I've found or just a mis-use of PtrOf. Finishing up for today, will probably look more tomorrow, thought I'd see if you had any thoughts on the issue.

The following code panics sometimes:

package main

import (
	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/gen"
	"github.com/leanovate/gopter/prop"
)

type A struct {
	Value *float64
}

func genA() gopter.Gen {
	return gopter.DeriveGen(
		func(f *float64) A {
			return A{Value: f}
		},
		func(a A) *float64 {
			return a.Value
		},
		gen.PtrOf(gen.Float64()),
	)
}

type B struct {
	A1 A
}

func genB() gopter.Gen {
	return gopter.DeriveGen(
		func(a1 A) B {
			return B{a1}
		}, func(b B) A {
			return b.A1
		},
		genA(),
	)
}

func main() {
	properties := gopter.NewProperties(nil)

	properties.Property("should not panic", prop.ForAll(
		func(b B) bool {
			return false
		},
		genB(),
	))

	properties.Run(gopter.ConsoleReporter(false))
}

with the output being:

! should not panic: Error on property evaluation after 0 passed tests:
   Check paniced: reflect: call of reflect.Value.Interface on zero Value
   goroutine 1 [running]:
runtime/debug.Stack(0xc4200556b0, 0x10f6840, 0xc42000aa80)
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/debug/stack.go:24 +0xa7
github.com/leanovate/gopter.SaveProp.func1.1(0xc420055d30)
	/Users/joel/code/go/src/github.com/leanovate/gopter/prop.go:19 +0x6e
panic(0x10f6840, 0xc42000aa80)
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/panic.go:502 +0x229
reflect.valueInterface(0x0, 0x0, 0x0, 0x1, 0x10e49e0, 0x0)
	/usr/local/Cellar/go/1.10.3/libexec/src/reflect/value.go:953 +0x1a3
reflect.Value.Interface(0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.10.3/libexec/src/reflect/value.go:948 +0x44
github.com/leanovate/gopter/gen.PtrShrinker.func1(0x10e49e0, 0x0, 0x1)
	/Users/joel/code/go/src/github.com/leanovate/gopter/gen/ptr_shrink.go:28
   +0xc1
github.com/leanovate/gopter.CombineShrinker.func1(0x10e8c80, 0xc42000aa60,
   0x10e8c80)
	/Users/joel/code/go/src/github.com/leanovate/gopter/shrink.go:168 +0x139
github.com/leanovate/gopter.(*derivedGen).Shrinker(0xc420094500,
   0x10fc000, 0x0, 0xc42000c0e0)
	/Users/joel/code/go/src/github.com/leanovate/gopter/derived_gen.go:81
   +0xe3
github.com/leanovate/gopter.(*derivedGen).Shrinker-fm(0x10fc000, 0x0, 0x1)
	/Users/joel/code/go/src/github.com/leanovate/gopter/derived_gen.go:31
   +0x3e
github.com/leanovate/gopter.CombineShrinker.func1(0x10e8c80, 0xc42000aa00,
   0x10e8c80)
	/Users/joel/code/go/src/github.com/leanovate/gopter/shrink.go:168 +0x139
github.com/leanovate/gopter.(*derivedGen).Shrinker(0xc420094780,
   0x10fc080, 0x0, 0xc420055b90)
	/Users/joel/code/go/src/github.com/leanovate/gopter/derived_gen.go:81
   +0xe3
github.com/leanovate/gopter.(*derivedGen).Shrinker-fm(0x10fc080, 0x0,
   0x10fc080)
	/Users/joel/code/go/src/github.com/leanovate/gopter/derived_gen.go:31
   +0x3e
github.com/leanovate/gopter/prop.shrinkValue(0x3e8, 0xc420094a00,
   0x10fc080, 0x0, 0xc420094a50, 0xc420055cb8, 0x10c4dbc, 0x10fc980,
   0xc42000e480)
	/Users/joel/code/go/src/github.com/leanovate/gopter/prop/forall.go:99
   +0x4e
github.com/leanovate/gopter/prop.ForAll.func1(0xc42000a620, 0x1122320)
	/Users/joel/code/go/src/github.com/leanovate/gopter/prop/forall.go:47
   +0x532
github.com/leanovate/gopter.SaveProp.func1(0xc42000a620, 0x0)
	/Users/joel/code/go/src/github.com/leanovate/gopter/prop.go:24 +0x6c
github.com/leanovate/gopter.Prop.Check.func1(0x0, 0xc42000e4c0, 0x11c69a0)
	/Users/joel/code/go/src/github.com/leanovate/gopter/prop.go:52 +0x15c
github.com/leanovate/gopter.(*runner).runWorkers(0xc420088570, 0x0)
	/Users/joel/code/go/src/github.com/leanovate/gopter/runner.go:49 +0x2d9
github.com/leanovate/gopter.Prop.Check(0xc42000e4a0, 0xc42001e6c0,
   0x111b6c4)
	/Users/joel/code/go/src/github.com/leanovate/gopter/prop.go:110 +0x1af
github.com/leanovate/gopter.(*Properties).Run(0xc4200882d0, 0x1131560,
   0xc42000a5e0, 0x10)
	/Users/joel/code/go/src/github.com/leanovate/gopter/properties.go:37 +0xc9
main.main()
	/Users/joel/code/gopter_sample.go:50 +0x1b0

It seems to only happen if the shrinker is used. If I change, for example, the return false in the property to return true it does never happen.

Somehow it requires these two nested structs. Using just struct A does not result in a panic... Not sure why or if I was just unlucky when reproducing...

It looks like the problem is in the Ptr shrink implementation where the call to Interface panics on a nil value.

I'm happy to help to fix this but I need a bit of guidance :)

Currently, gen.Struct (and gen.StructPtr) create results by taking elements from field generators, and combining them into a struct, which is emitting by the struct generator. However, this struct is emitted using NoShrinker, so it cannot be shrunk. In order to get shrinkable structs, I have been manually writing Gens using DeriveGen, but it would be much nicer to be able to use the helper functions like gen.Struct, and if the default arbitrary generators gave shrinkable results as well.

I've been skimming through the docs, and coming from ScalaCheck i was looking for generating parallel command sequences. Is it supported?

Really sorry about this, turns out I've messed up on the types of the Shrinker. Shrunk values are still time.Time rather than *time.Time. I'll make some time to fix this up over the next few days.

The adjusted example from the doc:

package main

import (
	"errors"
	"math/cmplx"

	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/arbitrary"
	"github.com/leanovate/gopter/gen"
)

type QudraticEquation struct {
	A, B, C complex128
}

func (q QudraticEquation) Eval(x complex128) complex128 {
	return q.A*x*x + q.B*x + q.C
}

func (q QudraticEquation) Solve() (complex128, complex128, error) {
	if q.A == 0 {
		return 0, 0, errors.New("No solution")
	}
	v := q.B*q.B - 4*q.A*q.C
	v = cmplx.Sqrt(v)
	return (-q.B + v) / 2 / q.A, (-q.B - v) / 2 / q.A, nil
}

func main() {
	parameters := gopter.DefaultTestParameters()
	parameters.Rng.Seed(1234) // Just for this example to generate reproducable results

	arbitraries := arbitrary.DefaultArbitraries()
	arbitraries.RegisterGen(gen.Complex128Box(-1e8-1e8i, 1e8+1e8i)) // Only use complex values within a range

	properties := gopter.NewProperties(parameters)

	properties.Property("Quadratic equations can be solved", arbitraries.ForAll(
		func(quadratic QudraticEquation) bool {
			x1, x2, err := quadratic.Solve()
			if err != nil {
				return true
			}

			return cmplx.Abs(quadratic.Eval(x1)) < 1e-5 && cmplx.Abs(quadratic.Eval(x2)) < 1e-5
		}))

	properties.Property("Quadratic equations can be solved alternative", arbitraries.ForAll(
		func(a, b, c complex128) bool {
			quadratic := &QudraticEquation{
				A: a,
				B: b,
				C: c,
			}
			x1, x2, err := quadratic.Solve()
			if err != nil {
				return true
			}

			return cmplx.Abs(quadratic.Eval(x1)) < 1e-5 && cmplx.Abs(quadratic.Eval(x2)) < 1e-5
		}))

	// When using testing.T you might just use: properties.TestingRun(t)
	properties.Run(gopter.ConsoleReporter(false))
}

Output:

! Quadratic equations can be solved: Error on property evaluation after 0
   passed tests: Check paniced: runtime error: invalid memory address or nil
   pointer dereference goroutine 1 [running]:
runtime/debug.Stack(0xc42004ba90, 0x4ffa40, 0x5d0ca0)
	/usr/x86_64-pc-linux-gnu/lib/go/src/runtime/debug/stack.go:24 +0xa7
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Save
  Prop.func1.1(0xc42004bd20)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:19 +0x6e
panic(0x4ffa40, 0x5d0ca0)
	/usr/x86_64-pc-linux-gnu/lib/go/src/runtime/panic.go:505 +0x229
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter/prop
  .ForAll.func1(0xc420098460, 0x52aaf8)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop/forall.go:31 +0x13b
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Save
  Prop.func1(0xc420098460, 0x0)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:24 +0x6c
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Prop
  .Check.func1(0x0, 0xc42007e3d0, 0x5d7f60)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:52 +0x15c
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.(*ru
  nner).runWorkers(0xc420082510, 0x0)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/runner.go:45 +0x2d9
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.Prop
  .Check(0xc42007e3a0, 0xc4200a05c0, 0x52741e)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/prop.go:110 +0x1af
github.com/hasufell/gopter-examples/vendor/github.com/leanovate/gopter.(*Pr
  operties).Run(0xc420082480, 0x53a700, 0xc420098420, 0x2d)
	/home/julian/go/src/github.com/hasufell/gopter-examples/vendor/github.com/
  leanovate/gopter/properties.go:37 +0xc9
main.main()
	/home/julian/go/src/github.com/hasufell/gopter-examples/gopterexamples.go:
  64 +0x2af


+ Quadratic equations can be solved alternative: OK, passed 100 tests.

When transitioning a model using Command.NextState -> commands.State, is it dangerous to share pointers or reference types between model states?

For example,

type State struct {
  index map[string]string
}

func (writeFooBarCommand) NextState(s commands.State) commands.State {
  current := s.(State)
  next := current
  next.index["foo"] = "bar"
  return next
}

I believe that the existing model state and the new model state will share a reference to the underlying map, meaning manipulations via the old state instance or the new state instance will reflect in both objects. I currently make it a habit to deep copy my model states when transitioning, but my question is whether or not the deep copy is necessary.

Hello! Thanks for this great library and apologies in advance if this is me just missunderstanding it (go newbie).

I've got a stateful system, a bitarray and the state is a []bool. I'm trying to assert that the bitarray is equivalent to the state ([]bool).

I'm using 2 gen.IntRange:

• for the initial state (to explore different array sizes)
• for the command (set value at Index to be true/false, using IntRange to avoid out-of-bounds Index values)

Despite using IntRange to stay within the bounds of he SUT and state, I eventually see illegal values (I suspect during the shrink phase, as they only appear after the post condition fails), which causes the test to panic.

Is this intended behaviour for the gen.IntRange (to shrink outside the range)?

Another interesting detail: if I remove the IntRange from the initial state and only keep it in the command, the Index values for the command stay within range and I never see the panic.

See an example here: https://play.golang.org/p/xplrfKS_Nvm

Thanks!

There are generators for slices, but none for maps. What I have been doing to work around this is generating helper structs containing slices and then computing the maps after generation. This seems to work fine, but is a bit ugly.

Is there a fundamental reason why generating maps is hard, or is it just a matter of work to build it?

passing an array type to GenForType returns nil, and this only panics once the properties are actually ran so it isn't obvious what the issue is, it produces the following stack trace:

! roundtrip: Error on property evaluation after 0 passed tests: Check
   paniced: runtime error: invalid memory address or nil pointer dereference
goroutine 19 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/leanovate/gopter/prop.ForAll.SaveProp.func3.1()
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/prop.go:20 +0x6b
panic({0x6c7ec0?, 0x96c6a0?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/leanovate/gopter/prop.ForAll.func1(0xc0000be2e0)
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/prop/forall.go:33 +0xc7
github.com/leanovate/gopter/prop.ForAll.SaveProp.func3(0x7fc1b4964b98?)
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/prop.go:25 +0x5b
github.com/leanovate/gopter.Prop.Check.func1(0x0, 0xc000090580)
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/prop.go:53 +0x132
github.com/leanovate/gopter.(*runner).runWorkers(0xc000094f90)
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/runner.go:49 +0x1cd
github.com/leanovate/gopter.Prop.Check(0xc000090560, 0xc0000f0b80)
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/prop.go:112 +0x1be
github.com/leanovate/gopter.(*Properties).Run(0xc0000d9f38, {0x78f100, 0xc0000be2a0})
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/properties.go:37 +0xa9
github.com/leanovate/gopter.(*Properties).TestingRun(0xc0000d9f38, 0xc000084ea0, {0x0, 0x0, 0x6b14a0?})
        /home/wessie/go/pkg/mod/github.com/leanovate/[email protected]/properties.go:56 +0xfc

GenForType should probably panic if there is an unsupported type

I've got the following situation:

// regular generator using gen.* types with gopter.CombineGens/gopter.Map
func genX() gopter.Gen { ... } 

// more complicated generator, which needs the output of `genX`, and the ability
// to use the rng from `*gopter.GenParams`.
func genXY() gopter.Gen {}

I can't use any of Map/FlatMap/MapResultin genXY(), as I still need access to the *gopter.GenParams. I ended up doing something hacky like:

func genXY() gopter.Gen {
  return func(params *gopter.GenParams) *gopter.GenResult {
    x := genX()(params)
    // using the result of x, especially the shrinkers to create a new result
    // along with the rng from `params`
  }
}

This works but causes data races in the rng if I'm using multiple workers. Can we expose a new API (e.g. MapExtended(params *gopter.GenParams, result gopter.GenResult)) to allow for such composition as a first class in the API?

NB: in an un-related note, thank you so much for this library! My team & I have caught so many issues using it. Massive massive props.

I have a scenario where I'd like to generate commands for a system under test based on the current state of the system. For example, let's say my system under test is an on-disk set of strings and I've modeled it with an in-memory set. It supports two operations: Add and Has:

Add() string adds a random string to the set and returns this string
Has(s string) bool returns true if the given string is in the set

To test this system I might have a generator for Add and Has commands and my test would apply the commands to each system and check that the results match. When I generate my Has commands it would be useful if the current in-memory state could inform my generator so that I ensure that the string I'm passing to Has actually was added to the set. Would the state parameter of the GenCommandFunc function allow me to do something like this?

Here's the general idea I had for how I might structure the generators:

type MySet struct {
    s map[string]bool
}

func (s *MySet) Add() string {
    r := randomString()
    s.s[r] = true
    return r
}

func (s *MySet) Has(s string) bool {
    _, ok := s.s[s]
    return ok
}

func AddCommand() gopter.Gen {
    return gen.Const(&addCommand{})
}

func HasCommand(state *MySet) gopter.Gen {
    return gen.Int().Map(func(i int) commands.Command {
        if len(state.s) == 0 {
            return hasCommand("")
        }

        i = i % len(state.s)
        
        for _, str := range {
            if i == 0 {
                return hasCommand(str)
            }

            i--
        }
    })
}

In a scenario like this it's difficult because passing any random string to Has() would probably result in it returning false. It's very unlikely that a randomly generated string would match one returned by the system under test. What is the intended use of that parameter in the GenCommandFunc function? Is there a better way to construct generators for commands where the parameters of one command might rely on the result of another?

I do a couple of things to work around the default interfaces exposed by gopter

(1) I like using time.Now() as a seed for my tests, but also having this reported upon failure so I can reproduce the failure. My tests end up looking like:

func TestSomethingOrTheOther(t *testing.T) {
	parameters := gopter.DefaultTestParameters()
	seed := time.Now().UnixNano()
	parameters.Rng = rand.New(rand.NewSource(seed))
	properties := gopter.NewProperties(parameters)
...
	properties.Property("...", prop.ForAll(...)
...
	reporter := gopter.NewFormatedReporter(true, 160, os.Stdout)
	if !properties.Run(reporter) {
		t.Errorf("failed with initial seed: %d", seed)
	}
}

(2) Overriding width for the formatted reporter
The default test reporter truncates the output to 79 columns (iirc); I've always overriden this to a larger value so that stacktraces do not get chopped into different pieces.

Would you be open to changing the API (exposing new ctors) to make these the defaults? I'm happy to contribute the changes if so.

Hi,

I got an error when trying to generating a slice of pointers using the code below:

package main

import (
	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/gen"
	"github.com/leanovate/gopter/prop"
	"reflect"
)

type Empty struct {
	ID uint
}

func main() {
	properties := gopter.NewProperties(nil)

	properties.Property("just test pointer", prop.ForAll(
		func(nodes []*Empty) bool {
			return true
		},
		gen.SliceOf(gen.Struct(reflect.TypeOf(&Empty{}), map[string]gopter.Gen{
			"ID": gen.UInt(),
		})),
	))
	properties.Run(gopter.ConsoleReporter(false))

}

The error was:

! just test pointer: Error on property evaluation after 0 passed tests:
   Check paniced: reflect: Call using []main.Empty as type []*main.Empty
ARG_0: []

I think reflect.TypeOf(&Empty{}) returns *main.Empty instead of main.Empty, but according to the error messages, the generated type is []main.Empty; so I just wonder whether there's something wrong with my code here.

Hello - thanks for a great testing library!

I'm trying to use gopter to test a simple stateful object. The object is initialized with no existing state, and then methods are called. After each method is called, I'd like to execute a check on the internal state of the object. Note this means I don't want to maintain a separate State object, I'd basically like my SystemUnderTest to also act as the State object and I don't want to have to do any kind of updates in NextState. Ideally the Run method will be called, executing one of the object's methods, and then PostCondition will check some property of the object.

I believe I managed to get this working, but a few things that aren't clear:

• Why does InitialStateGen run so many times? I would expect it to run once for each set of commands being executed but it seems to run much more than that.
• Is it wrong to try and use the commands.SystemUnderTest as the commands.State? If I try to do so, it doesn't seem to work, since the commands.State and the commands.SystemUnderTests objects seem to come from different calls to InitialStateGen, so the commands.State doesn't see the changes effected to the commands.SystemUnderTest during calls to Run. I can work around this by returning the SystemUnderState as the result in Run and then checking the result in PostCondition while just ignoring the State. Is that the right thing to do? Related #25

I've included a minimum viable example below.

Thanks!

package keeper

import (
	"fmt"
	"testing"

	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/commands"
	"github.com/leanovate/gopter/gen"
)

type sys struct {
	height  int
	heights map[int]uint8
}

// store i under the latest height value and incremenet the height value
func (s *sys) New(i uint8) {
	h := s.height
	s.height += 1

	// bug when height is 5
	if s.height == 5 {
		return
	}

	s.heights[h] = i
}

type newCommand uint8

func (value newCommand) Run(q commands.SystemUnderTest) commands.Result {
	s := q.(*sys)
	s.New(uint8(value))
	return s
}

// expect there to be an entry in the `sys.heights` map for each height up to the `sys.height`
func (newCommand) PostCondition(state commands.State, result commands.Result) *gopter.PropResult {
	// XXX: if I try to use the state instead of the result here, it doesn't work
	// (state doesn't seem to see the effects from Run)
	s := result.(*sys)
	for i := 0; i < s.height; i++ {
		_, ok := s.heights[i]
		if !ok {
			return &gopter.PropResult{Status: gopter.PropFalse}
		}
	}
	return &gopter.PropResult{Status: gopter.PropTrue}
}

func (value newCommand) NextState(state commands.State) commands.State {
	s := state.(*sys)
	return s
}

func (newCommand) PreCondition(state commands.State) bool {
	return true
}

func (value newCommand) String() string {
	return fmt.Sprintf("New(%d)", value)
}

var genNewCommand = gen.UInt8().Map(func(value uint8) commands.Command {
	return newCommand(value)
})

func cbCommands(t *testing.T) *commands.ProtoCommands {
	return &commands.ProtoCommands{
		NewSystemUnderTestFunc: func(initState commands.State) commands.SystemUnderTest {
			return initState
		},
		InitialStateGen: func(p *gopter.GenParameters) *gopter.GenResult {
			// XXX: this function seems to execute many more times than Run
			result := &sys{
				height:  0,
				heights: make(map[int]uint8),
			}
			return gopter.NewGenResult(result, gopter.NoShrinker)
		},
		GenCommandFunc: func(state commands.State) gopter.Gen {
			return gen.OneGenOf(genNewCommand)
		},
	}
}

func TestSys(t *testing.T) {
	parameters := gopter.DefaultTestParametersWithSeed(1235)
	properties := gopter.NewProperties(parameters)
	properties.Property("circular buffer", commands.Prop(cbCommands(t)))
	properties.TestingRun(t)
}

Do you want to add the badge from goreportcard link to the README.md to get listed in awesome-go list. For more info refer to this PR.

I find that code like the following:

package panics

import (
	"testing"
	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/gen"
	"github.com/leanovate/gopter/prop"
)

func TestPanic(t *testing.T) {
	properties := gopter.NewProperties(nil)
	properties.Property("Will panic", prop.ForAll(
		func(i int) bool {
			if i%2 == 0 {
				panic("hi")
			}
			return true
		},
		gen.Int().WithLabel("number")))
	properties.TestingRun(t)
}

does catch the panic, but while it prints the random seed, it does not print the parameters that resulted in the panic:

$ go test -run TestPanic .
! Will panic: Error on property evaluation after 0 passed tests: Check
   paniced: hi goroutine 20 [running]:
runtime/debug.Stack(0xc0000b36e8, 0x130fa80, 0x13ded80)
        /usr/local/Cellar/go/1.11.5/libexec/src/runtime/debug/stack.go:24 +0xa7
github.com/leanovate/gopter.SaveProp.func1.1(0xc0000b3cf0)
        /Users/asf/go/src/github.com/leanovate/gopter/prop.go:19 +0x6e
panic(0x130fa80, 0x13ded80)
        /usr/local/Cellar/go/1.11.5/libexec/src/runtime/panic.go:513 +0x1b9
github.com/antifuchs/o_test.TestPanic.func1(0x0)
        /Users/asf/go/src/github.com/antifuchs/o/ranges_properties_test.go:131
   +0x39
reflect.Value.call(0x130d6c0, 0x13a01f0, 0x13, 0x138a736, 0x4, 0x1643500,
   0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:447 +0x454
reflect.Value.Call(0x130d6c0, 0x13a01f0, 0x13, 0x1643500, 0x0, 0x0,
   0x13898c0, 0x1, 0x1643500)
        /usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:308 +0xa4
github.com/leanovate/gopter/prop.checkConditionFunc.func2(0x1643500, 0x0,
   0x0, 0x1643500)
        /Users/asf/go/src/github.com/leanovate/gopter/prop/check_condition_func.go
  :37 +0x5e
github.com/leanovate/gopter/prop.ForAll.func1(0xc00009a560, 0x13a0258)
        /Users/asf/go/src/github.com/leanovate/gopter/prop/forall.go:40 +0x266
github.com/leanovate/gopter.SaveProp.func1(0xc00009a560, 0x0)
        /Users/asf/go/src/github.com/leanovate/gopter/prop.go:24 +0x6c
github.com/leanovate/gopter.Prop.Check.func1(0x0, 0xc000084fd0, 0x1625ca0)
        /Users/asf/go/src/github.com/leanovate/gopter/prop.go:52 +0x155
github.com/leanovate/gopter.(*runner).runWorkers(0xc000089170, 0x0)
        /Users/asf/go/src/github.com/leanovate/gopter/runner.go:49 +0x2d1
github.com/leanovate/gopter.Prop.Check(0xc000084fb0, 0xc000090840,
   0x138be08)
        /Users/asf/go/src/github.com/leanovate/gopter/prop.go:110 +0x194
github.com/leanovate/gopter.(*Properties).Run(0xc000089110, 0x13e0d60,
   0xc00009a520, 0xc00008b1e0)
        /Users/asf/go/src/github.com/leanovate/gopter/properties.go:37 +0xbc
github.com/leanovate/gopter.(*Properties).TestingRun(0xc000089110,
   0xc000116100, 0x0, 0x0, 0x0)
        /Users/asf/go/src/github.com/leanovate/gopter/properties.go:56 +0x137
github.com/antifuchs/o_test.TestPanic(0xc000116100)
        /Users/asf/go/src/github.com/antifuchs/o/ranges_properties_test.go:133
   +0x126
testing.tRunner(0xc000116100, 0x13a01f8)
        /usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
        /usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:878 +0x35c
Elapsed time: 275.319µs

--- FAIL: TestPanic (0.00s)

properties.go:57: failed with initial seed: 1552922262110846000

FAIL

FAIL    github.com/antifuchs/o  0.015s

What I would expect to see is to see the "Labels of the failing property" printed.

To work around this, I hard-code the random seed in the test options and return false at the start of my function if I am interested in the values, but this seems like a thing gopter could & should do automatically, especially as it already captures the panic.

Trying to get a handle on how WithLabel could help decoding inputs in the event of test failure. For the simple example shown on https://godoc.org/github.com/leanovate/gopter with two simple inputs, it seems fine. But here is what I have after I have put WithLabel on most everything in my test generators:

Project_Name, Project_Id, Rules: {{ﶄ⺗ມ𑌃ଃ b- {} [] 0} [{5p7 -
   𑊋﷼ⴐ🃭ἲ 1 [0xc0002e1a40 0xc0002e4c30 0xc0002e5e00 0xc000558ff0
   0xc00055c1e0] {} [] 515} {xm x ዓ𖭖🄉࠾᳆ 1 [0xc0005aba90
   0xc0005aec80 0xc0005afe50 0xc0005b7040 0xc0005fe8c0] {} [] 0} {h5 w
   ే𐬹𝕐𐄡Ӂ 0 [0xc000575130 0xc00053a690 0xc00053b900 0xc000554dc0
   0xc000538410] {} [] 0} {-y -y ৗए𑃰𬐐࿑ 0 [0xc000525450
   0xc000520f50 0xc00051ea00 0xc00051c460 0xc00051df40] {} [] 0} {2-9v a
   ූᩥᬥආⵣ 1 [0xc00050f8b0 0xc00050aff0 0xc000506730 0xc000507f40
   0xc000505720] {} [] 0}]}

First, it seems to have shown the labels only for the topmost items.
Second, it put all the labels first, not intermixed with their values.
Here is how those labels match the inputs:

Just to give some feel for how my generators are organized, here it is in pseudo-code:

createProjectAndRuleGen => 
	CombineGens(projectReqGen, gen.SliceOf(ruleReqGen)

projectReqGen => (id, name)
ruleReqGen => CombineGens(gen.Bool(), ruleReqGenEvent, ruleReqGenNode)

ruleReqGenEvent => (id, name, conditions1, and some other properties...)
ruleReqGenNode => (id, name, conditions2, and some other properties...)

conditions1 => gen.SliceOf(conditionsGenEvent)
conditions2 => gen.SliceOf(conditionsGenNode)

conditionsGenEvent => (type, value)
conditionsGenNode => (type, value)

Looking for tips on how to make the input blob be less indecipherable.

Hi,
I'm seeing a performance issue when using generators in somewhat high numbers.
DefaultGenParameters() function is calling rand.New(rand.NewSource(seed)) which internally uses locks among other things in Go. However if one will call DefaultGenParameters() once and use this instance instead of repeatedly calling the function it can improve performance dramatically.
In all the examples shown the usage of DefaultGenParameters() function is only for the side effect and not the result.

https://github.com/leanovate/gopter/blob/master/derived_gen.go#L109
https://github.com/leanovate/gopter/blob/master/gen.go#L51
https://github.com/leanovate/gopter/blob/master/gen.go#L105

I can create a pull request if needed.

I'm just getting started with gopter (also being fairly new to property based testing in general) and I was trying to write a custom generator for a struct type of mine with private fields/initalizers.

A simplified example:

type Struct struct {
    name    string
    version string
}

func NewStruct(n, v string) *Struct {
    return &Struct{n, v}
}

func (s Struct) String() string {
    return s.name //+ ":" + s.version
}

func GenStruct() gopter.Gen {
    return gopter.CombineGens(
        gen.Identifier(),
        gen.NumString().SuchThat(nonZeroLen),
    ).Map(func(values []interface{}) Struct {
        name := values[0].(string)
        ver := values[1].(string)
        return NewStruct(name, ver)
    })
}

This works but as documented Map looses the shrinker from CombineGens. (Okay, slightly surprising but documented.)

My attempt at writing a version that adds a Shrinker ran into some problems - it seemed to loop and not shrink as well (or almost at all) as the CombineGen case. I was trying this:

func GenStructShrink() gopter.Gen {
    combines := gopter.CombineGens(
        gen.Identifier().WithLabel("name"),
        gen.NumString().SuchThat(nonZeroLen).WithLabel("ver"),
    )
    return func(genParams *gopter.GenParameters) *gopter.GenResult {
        input := combines(genParams)
        values, ok := input.Retrieve()

        var s *Struct
        if !ok {
            return gopter.NewEmptyResult(reflect.TypeOf(s))
        }
        mapper := func(strings []interface{}) *Struct {
            name := strings[0].(string)
            ver := strings[1].(string)
            return NewStruct(name, ver)
        }
        s = mapper(values.([]interface{}))

        myShrinker := func(value interface{}) gopter.Shrink {
            fmt.Printf("Creating chrinker for %#v\n", value)
            return input.Shrinker(values).Map(mapper)
        }

        return gopter.NewGenResult(s, myShrinker)
    }
}

I would have expected it to create the shrinker once and call it again, but it seems to create a new version of the shrinker many times which means it resets the shrink (because I'm ignoring the input value to try and get back at the underlying generators.

I guess this is a long way of saying: what is the exact interface of the Shrinkers? What is the Shrinker called with? When is it called? How often?

The following test panics:

package main_test

import (
	"reflect"
	"testing"

	"github.com/leanovate/gopter"
	"github.com/leanovate/gopter/gen"
	"github.com/leanovate/gopter/prop"
)

type Box struct {
	Content interface{}
}

func BoxOf(inner gopter.Gen) gopter.Gen {
	return gen.Struct(reflect.TypeOf(Box{}), map[string]gopter.Gen{
		"Content": inner,
	})
}

func TestPanics(t *testing.T) {
	properties := gopter.NewProperties(nil)

	properties.Property("Should not panic",
		prop.ForAll(
			func(e Box) bool {
				return reflect.DeepEqual(e, Box{0})
			},
			BoxOf(gen.Int()),
		))

	properties.TestingRun(t)
}

What seems to be happening is that I can't use a generator of int to fill a variable of type interface{} in Box. What I expected to happen is for this to be fine. After all, there's nothing wrong with var i interface{} = 3 so why is there a problem with BoxOf(gen.Int())?

I can work around by using an explicit Map() to change the type of the Int() generator to interface{}:

			func(e Box) bool {
				return reflect.DeepEqual(e, Box{0})
			},
-			BoxOf(gen.Int()),
+			BoxOf(
+				gen.Int().Map(func(in *gopter.GenResult) *gopter.GenResult {
+					in.ResultType = reflect.TypeOf((*interface{})(nil)).Elem()
+					return in
+				}),
+			),
		))

	properties.TestingRun(t)

but this feels pretty heavyweight. Is it possible to get structs to accept concrete generators for interface types?

Thanks for this package, it's great!

I'm working on something and running into a bit of trouble with a recursive data type. How might you use this package for recursive data? For instance in the ScalaCheck documentation, there's an example of binary integer trees. How could you emulate that example with this package?

I'd assume you start like:

type tree interface {
	isTree()
}

type node struct {
	left  tree
	right tree
	v     int
}

func (*node) isTree() {}

type leaf struct{}

func (*leaf) isTree() {}

var genLeaf gopter.Gen = gen.Const(&leaf{})

var genNode gopter.Gen = gen.Struct(reflect.TypeOf(&node{}), map[string]gopter.Gen{
	"left":  genTree,
	"right": genTree,
	"v":     gen.Int(),
})

var genTree gopter.Gen = gen.OneGenOf(genLeaf, genNode)

But that ends up with an initialization loop. Converting genNode and genTree to functions allows it to be initialized:

func genNode() gopter.Gen {
	return gen.Struct(reflect.TypeOf(&node{}), map[string]gopter.Gen{
		"left":  genTree(),
		"right": genTree(),
		"v":     gen.Int(),
	})
}

func genTree() gopter.Gen { return gen.OneGenOf(genLeaf, genNode()) }

But, using either one ends up with a stack overflow.

Any thoughts on how to implement that binary int tree example?

Hey!

I am trying to generate unique ids however I am having some troubles with it. I have tried adding a list to which ids have already been generated and then discard any that matches. questionIdGenOptions.alreadyGeneratedQuestions is an array of strings. Utils.Contains is a function which returns true if the given array contains the corresponding string otherwise false.

func QuestionIdGen(questionIdGenOptions *QuestionIdGenOptions) gopter.Gen{
	if questionIdGenOptions != nil && questionIdGenOptions.alreadyGeneratedQuestions == nil {
		var newAlreadyGeneratedQuestions = []string{}
		questionIdGenOptions.alreadyGeneratedQuestions = &newAlreadyGeneratedQuestions
	}
	pgen := gen.PtrOf(gen.AnyString())
	if questionIdGenOptions != nil {
		pgen = pgen.SuchThat(
			func(generatedId *string) bool {
				//Some characters look the same however their bytes are not i.e. it is therefore unique.
				if questionIdGenOptions.GenUniqueId != nil && *questionIdGenOptions.GenUniqueId == true && (generatedId == nil || Utils.Contains(*questionIdGenOptions.alreadyGeneratedQuestions, *generatedId)) {
					return false
				}

				newAlreadyGeneratedQuestions := append(*questionIdGenOptions.alreadyGeneratedQuestions, *generatedId)
				questionIdGenOptions.alreadyGeneratedQuestions = &newAlreadyGeneratedQuestions
				return true
			})
	}
	return pgen
}

However atm. with this if statement, the wrapper generator which uses this function fails to generate anything... And I am unsure why almost like something internal error is happening. If I remove it everything works fine except the ids are not unique. Is there another way of doing it or?

I'm running some tests but for some reason gopter.Run doesn't stop. The tests make calls against a db that isn't very fast and after around 2000 tests I just kill the gopter.Run process because it takes too long. Is there any way to limit the amount of tests that get run before gopter says that the tests have passed?

Also I don't know why but other tests don't need nearly as many attempts.

I'm thinking of writing a gen.StructStrict, which panics if gens contains keys which are not fields on rt, so for instance the "NotThere" key in struct_test.go::TestStruct would cause a failure. Would you be likely to accept this as a PR?

Looks like there have been a bunch of changes since the last release (1 year ago) including gen.Sized which looks quite useful. Could we cut a new release?

I have this massive generator which creates an arbitrary complex struct however each time I run the test I get a different result. So I started implementing tests for the generators to see if I could reproduce results from the same generator using the same seed in in two instances of properties. This is the layout of the test and its support functions:

func SetupReproducibleProperties(seed *int64) (*gopter.Properties, *gopter.Properties) {
	var parameters1 *gopter.TestParameters
	var parameters2 *gopter.TestParameters
	if seed != nil {
		parameters1 = gopter.DefaultTestParametersWithSeed(*seed)
		parameters2 = gopter.DefaultTestParametersWithSeed(*seed)
	}else{
		parameters1 = gopter.DefaultTestParameters()
		parameters2 = gopter.DefaultTestParametersWithSeed(parameters1.Seed)
	}
	properties1 := gopter.NewProperties(parameters1)
	properties2 := gopter.NewProperties(parameters2)
	return properties1, properties2
}

func GetHashStringFromInterface(i interface{}) string {
	newJsonBytes, _ := json.Marshal(i)
	newHash := md5.Sum(newJsonBytes)
	return string(newHash[:])
}

func Test(t *testing.T) {
	var seedToUse *int64
	//seedToUse = Utils.Int64ToPtr(1604056858868077464)
	properties1, properties2 := SetupReproducibleProperties(seedToUse)
	generatorToUse := SomeGenerator
	var oldGens []GeneratedStruct
	properties1.Property("Generate first iteration", prop.ForAll(
		func(value GeneratedStruct) bool {
			oldGens = append(oldGens, value)
			return true
		},
		generatorToUse,
	))
	counter := Utils.IntToPtr(0)
	properties2.Property("Comparing hashes should be a match", prop.ForAll(
		func(value GeneratedStruct) bool {
			currentHash := Utils.GetHashStringFromInterface(value)
			storedValue := oldGens[*counter]
			storedHash := Utils.GetHashStringFromInterface(storedValue)
			counter = Utils.IntToPtr(*counter + 1)
			if storedHash == currentHash {
				return true
			}
			return false
		},
		generatorToUse,
	))
	properties1.TestingRun(t)
	properties2.TestingRun(t)
}

Running this test always succeeds. Same goes if you replace GeneratedStruct with string and SomeGenerator with gen.AnyString(). However when I close the test down and start it up again I start seeing different results. If I use the AnyString example it generates the same value when I run the test and restart it. If I do it with my custom generated struct it does not generate the same result.

type SelectOption struct {
	// gopg specific fields to make database management easier
	tableName struct{} `pg:"selectoptions"`

	// Fields specified in AsyncAPI configuration
	Value           string              `json:"value" `
	Selected        *bool               `json:"selected,omitempty" `
	Name            string              `json:"name" `
	SubsetQuestions *QuestionsKeyObject `json:"subsetQuestions,omitempty" `
}

type Question struct {
	// gopg specific fields to make database management easier
	tableName struct{} `pg:"questions"`

	// Fields specified in AsyncAPI configuration
	Id                       *string             `json:"id,omitempty" `
	Type                     QuestionTypesEnum   `json:"type" `
	Hint                     *string             `json:"hint,omitempty" `
	Placeholder              *string             `json:"placeholder,omitempty" `
	DefaultValue             *string             `json:"defaultValue,omitempty" `
	Options                  []SelectOption      `json:"options,omitempty" `
	CheckedSubsetQuestions   *QuestionsKeyObject `json:"checkedSubsetQuestions,omitempty" `
	UncheckedSubsetQuestions *QuestionsKeyObject `json:"uncheckedSubsetQuestions,omitempty" `
	Questions                *QuestionsKeyObject `json:"questions,omitempty" `
	Mappers                  []Mapper            `json:"mappers,omitempty" `
}

type QuestionTypesEnum int

type questionTypesStruct struct {
	Text      QuestionTypesEnum `json:"text"`
	Select    QuestionTypesEnum `json:"select"`
	Container QuestionTypesEnum `json:"container"`
	Textarea  QuestionTypesEnum `json:"textarea"`
	Checkbox  QuestionTypesEnum `json:"checkbox"`
	_values   [5]QuestionTypesEnum
	_names    [5]string
}
var QuestionTypes = &questionTypesStruct{
	Text:      0,
	Select:    1,
	Container: 2,
	Textarea:  3,
	Checkbox:  4,
	_values: [5]QuestionTypesEnum{
		0,
		1,
		2,
		3,
		4},
	_names: [5]string{
		"text",
		"select",
		"container",
		"textarea",
		"checkbox",
	},
}

func TestQuestionGenShouldReproduceResults(t *testing.T) {
	var seedToUse *int64
	//seedToUse = Utils.Int64ToPtr(1604056858868077464)
	properties1, properties2 := SetupReproducibleProperties(seedToUse)
	generatorToUse := QuestionGenWithOptions()
	var oldGens []schema.Question
	properties1.Property("Generate first iteration", prop.ForAll(
		func(question schema.Question) bool {
			oldGens = append(oldGens, question)
			return true
		},
		generatorToUse,
	))
	counter := Utils.IntToPtr(0)
	properties2.Property("Comparing hashes should be a match", prop.ForAll(
		func(question schema.Question) bool {
			questionHash := Utils.GetHashStringFromInterface(question)
			storedQuestion := oldGens[*counter]
			storedHash := Utils.GetHashStringFromInterface(storedQuestion)
			counter = Utils.IntToPtr(*counter+1)
			if storedHash == questionHash {
				return true
			}
			return false
		},
		generatorToUse,
	))
	properties1.TestingRun(t)
	properties2.TestingRun(t)
}

func QuestionGenWithOptions() gopter.Gen {
	questionGenOptionsGen := gen.PtrOf(QuestionGenOptionsGen()).WithLabel("QuestionGenOptionsGenPtrOf")
	questionGenOptionsGen = questionGenOptionsGen.FlatMap(func(i interface{}) gopter.Gen {
		var questionGenOptions *QuestionGenOptions
		if i != nil {
			questionGenOptions = i.(*QuestionGenOptions)
		}
		return SpecificQuestionGenWithOptions(schema.Question{}, questionGenOptions)
	}, reflect.TypeOf(schema.Question{})).WithLabel("QuestionGenOptionsGenFlatMap")
	return questionGenOptionsGen
}

func QuestionIdGenOptionsGen() gopter.Gen {
	presetGens := map[string]gopter.Gen{
		"DifferentThen": gen.PtrOf(gen.SliceOfN(10, gen.PtrOf(gen.AnyString()))).WithLabel("QuestionIdGenOptionsGenDifferentThen"),
		"NotNil": gen.PtrOf(gen.Bool()).WithLabel("QuestionIdGenOptionsGenNotNil"),
		"UseNil": gen.PtrOf(gen.Bool()).WithLabel("QuestionIdGenOptionsGenUseNil"),
		"GenUniqueId": gen.PtrOf(gen.Bool()).WithLabel("QuestionIdGenOptionsGenGenUniqueId"),
	}
	return gen.Struct(reflect.TypeOf(QuestionIdGenOptions{}), presetGens).WithLabel("QuestionIdGenOptionsGen")
}

func QuestionGenOptionsGen() gopter.Gen {
	presetGens := map[string]gopter.Gen{
		"IdGenOptions": gen.PtrOf(QuestionIdGenOptionsGen()).WithLabel("QuestionGenOptionsGenIdGenOptions"),
		"ExcludeId": gen.PtrOf(gen.Bool()).WithLabel("QuestionGenOptionsGenExcludeId"),
	}
	return gen.Struct(reflect.TypeOf(QuestionGenOptions{}), presetGens).WithLabel("QuestionGenOptionsGen")
}

type QuestionGenOptions struct {
	//Use a custom generator for ids
	IdGen *gopter.Gen
	//Affect the current id generator by providing options
	IdGenOptions *QuestionIdGenOptions
	//Dont generate id, it will then always get the value nil
	ExcludeId *bool
}

type QuestionIdGenOptions struct {
	//Used to not generate the same id
	DifferentThen *[]*string
	//Used to ensure that nil are not generated, the generator returned are no longer a pointer
	NotNil *bool
	UseNil *bool
	//Generate unique ids
	GenUniqueId               *bool
	alreadyGeneratedQuestions *[]string
}

func QuestionIdGen(questionIdGenOptions *QuestionIdGenOptions) gopter.Gen {
	if questionIdGenOptions != nil && questionIdGenOptions.alreadyGeneratedQuestions == nil {
		var newAlreadyGeneratedQuestions = []string{}
		questionIdGenOptions.alreadyGeneratedQuestions = &newAlreadyGeneratedQuestions
	}
	var pgen gopter.Gen
	if questionIdGenOptions != nil && questionIdGenOptions.UseNil != nil && *questionIdGenOptions.UseNil == true {
		var data *string = nil
		pgen = gen.Const(data).WithLabel("QuestionIdGenNil")
	}else{
		pgen = gen.PtrOf(gen.AnyString()).WithLabel("QuestionIdGenPtrOf")
	}
	if questionIdGenOptions != nil {
		pgen = pgen.SuchThat(
			func(generatedId *string) bool {
				//If they want nil dont make it different then
				if (questionIdGenOptions.UseNil == nil || questionIdGenOptions.UseNil != nil && *questionIdGenOptions.UseNil == false) && questionIdGenOptions.DifferentThen != nil && generatedId != nil && Utils.ContainsPointers(*questionIdGenOptions.DifferentThen, generatedId) {
					return false
				}
				if questionIdGenOptions.NotNil != nil && *questionIdGenOptions.NotNil == true && generatedId == nil {
					return false
				}

				//Even if the characters are the same the bytes are not i.e. it is therefore unique.
				//if questionIdGenOptions.GenUniqueId != nil && *questionIdGenOptions.GenUniqueId == true && (generatedId == nil || Utils.Contains(*questionIdGenOptions.alreadyGeneratedQuestions, *generatedId)) {
				//	return false
				//}
				if generatedId != nil {
					newAlreadyGeneratedQuestions := append(*questionIdGenOptions.alreadyGeneratedQuestions, *generatedId)
					questionIdGenOptions.alreadyGeneratedQuestions = &newAlreadyGeneratedQuestions
				}
				return true
			}).WithLabel("QuestionIdGenSuchThat")
	}
	return pgen.WithLabel("QuestionIdGen")
}

func SpecificQuestionGenWithOptions(presetQuestions schema.Question, options *QuestionGenOptions) gopter.Gen {
	presetGens := map[string]gopter.Gen{
		"Type": gen.OneConstOf(
			schema.QuestionTypes.Checkbox,
			schema.QuestionTypes.Select,
			schema.QuestionTypes.Container,
			schema.QuestionTypes.Text,
			schema.QuestionTypes.Textarea).WithLabel("QuestionTypeGen"),
	}

	//Based on the already existing presetQuestions decide if we should keep the values
	if presetQuestions.Id != nil {
		presetGens["Id"] = gen.Const(presetQuestions.Id).WithLabel("QuestionIdGen1")
	} else {
		if options != nil && options.IdGen != nil {
			presetGens["Id"] = (*options.IdGen).WithLabel("QuestionIdGen2")
		} else if options != nil {
			presetGens["Id"] = QuestionIdGen(options.IdGenOptions).WithLabel("QuestionIdGen3")
		}else{
			presetGens["Id"] = QuestionIdGen(nil).WithLabel("QuestionIdGen4")
		}
	}

	if presetQuestions.Hint != nil {
		presetGens["Hint"] = gen.Const(presetQuestions.Hint).WithLabel("QuestionHintGen1")
	} else {
		presetGens["Hint"] = gen.PtrOf(gen.AnyString()).WithLabel("QuestionHintGen2")
	}

	if presetQuestions.Placeholder != nil {
		presetGens["Placeholder"] = gen.Const(presetQuestions.Placeholder).WithLabel("QuestionHintGen1")
	} else {
		presetGens["Placeholder"] = gen.PtrOf(gen.AnyString()).WithLabel("QuestionHintGen1")
	}
	if presetQuestions.DefaultValue != nil {
		presetGens["DefaultValue"] = gen.Const(presetQuestions.DefaultValue).WithLabel("QuestionDefaultValueGen1")
	} else {
		presetGens["DefaultValue"] = gen.PtrOf(gen.AnyString()).WithLabel("QuestionDefaultValueGen1")
	}
	if presetQuestions.Questions != nil {
		presetGens["Questions"] = gen.Const(presetQuestions.Questions).WithLabel("QuestionQuestionsGen")
	}
	if presetQuestions.CheckedSubsetQuestions != nil {
		presetGens["CheckedSubsetQuestions"] = gen.Const(presetQuestions.CheckedSubsetQuestions).WithLabel("QuestionCheckedSubsetQuestionsGen")
	}
	if presetQuestions.Mappers != nil {
		presetGens["Mappers"] = gen.Const(presetQuestions.Mappers).WithLabel("QuestionMappersGen")
	}
	if presetQuestions.UncheckedSubsetQuestions != nil {
		presetGens["UncheckedSubsetQuestions"] = gen.Const(presetQuestions.UncheckedSubsetQuestions).WithLabel("QuestionUncheckedSubsetQuestionsGen")
	}

	standardQuestion := gen.Struct(reflect.TypeOf(schema.Question{}), presetGens).WithLabel("QuestionStandardGen")
	if presetQuestions.Options == nil {
		standardQuestion = standardQuestion.FlatMap(
			func(generatedValue interface{}) gopter.Gen {
				generatedQuestion := generatedValue.(schema.Question)
				presetGens := map[string]gopter.Gen{
					"Type": gen.Const(generatedQuestion.Type),
					"Hint": gen.Const(generatedQuestion.Hint),
					"Id": gen.Const(generatedQuestion.Id),
					"Placeholder": gen.Const(generatedQuestion.Placeholder),
					"DefaultValue": gen.Const(generatedQuestion.DefaultValue),
					"Questions": gen.Const(generatedQuestion.Questions),
					"CheckedSubsetQuestions": gen.Const(generatedQuestion.CheckedSubsetQuestions),
					"Mappers": gen.Const(generatedQuestion.Mappers),
					"Options": gen.Const(generatedQuestion.Options),
					"UncheckedSubsetQuestions": gen.Const(generatedQuestion.UncheckedSubsetQuestions),
				}
				if presetQuestions.Options != nil {
					presetGens["Options"] = gen.Const(presetQuestions.Options).WithLabel("QuestionOptionsGen")
				}else{
					switch generatedQuestion.Type {
					case schema.QuestionTypes.Select:
						presetGens["Options"] = generateOptions()
						break
					}
				}
				return gen.Struct(reflect.TypeOf(schema.Question{}), presetGens)
			},
			reflect.TypeOf(schema.Question{}),
		).WithLabel("QuestionOptionsGenFlatMap")
	}
	return standardQuestion
}
func generateOptions() gopter.Gen {
	optionsGen := func(genParams *gopter.GenParameters) *gopter.GenResult {
		genParams.MinSize = 1
		genParams.MaxSize = 10
		presetGens := map[string]gopter.Gen{
			"Value": gen.AnyString().WithLabel("OptionsSliceValueGen"),
			"Name":  gen.AnyString().WithLabel("OptionsSliceNameGen"),
		}
		return gen.SliceOf(gen.Struct(reflect.TypeOf(schema.SelectOption{}), presetGens).WithLabel("OptionGen")).WithLabel("OptionsSliceGen")(genParams)
	}
	return optionsGen
}

Do you have any idea what might do this? I'll investigate further and keep adding comments, but thought it was time to share the bug or if it is my wrong doing. Keep in mind Go is new for me so if anything looks weird I apologise 😄

For example:

$ go test ./...
--- FAIL: TestRunnerParallelWorkers (3.02s)
    runner_test.go:174: [0] expected result time to be positive number but got 0s
    runner_test.go:174: [1] expected result time to be positive number but got 0s
    runner_test.go:174: [2] expected result time to be positive number but got 0s
! always fail: Falsified after 0 passed tests.
ARG_0: 0
ARG_0_ORIGINAL (1 shrinks): 299854320
Elapsed time: 0s
! always fail: Falsified after 0 passed tests.
ARG_0: 0
ARG_0_ORIGINAL (1 shrinks): 299854320
Elapsed time: 0s
FAIL
FAIL    github.com/leanovate/gopter     3.320s
ok      github.com/leanovate/gopter/arbitrary   0.683s
ok      github.com/leanovate/gopter/commands    0.739s
--- FAIL: TestOneConstOf (0.00s)
    one_of_test.go:51: Not all consts where generated: map[string]bool{"four":true, "two":true}
--- FAIL: TestOneGenOf (0.00s)
    one_of_test.go:51: Not all consts where generated: map[string]bool{"four":true, "three":true}
+ PtrOf: OK, passed 100 tests.
Elapsed time: 2.0028ms
--- FAIL: TestWeighted (0.01s)
    weighted_test.go:33: Result 250 for A falls outside acceptable range 50, 150
    weighted_test.go:33: Result 356 for B falls outside acceptable range 150, 250
    weighted_test.go:33: Result 394 for C falls outside acceptable range 650, 750
FAIL
FAIL    github.com/leanovate/gopter/gen 1.632s
ok      github.com/leanovate/gopter/prop        0.197s
FAIL

Sometimes the only way to know what the next state should be is if you let the system tell you. So it would be nice to be able to have the Result of Run in the NextState callback. Right now I am modifying the state in the post condition.

For example in a CRUD application you can know all of the parameters except for the unique id that gets generated by the system.

Concrete use case:

• command: create user

  • next state: save user with id to model state.
  • post condition: ensure creation response conforms

• command: get user

  • pre condition: at least one user must exist
  • post condition: ensure application response matches model state

Although this would be reasonably involved, it would be really nice if it were possible to generate strings from an instance of the RangeTable struct from unicode.

This seems like it would be reasonably difficult to do given the variable strides and variable width of the ranges in order to generate a uniform distribution. But seems like quite a useful feature for testing properties under certain conditions.

I will take a look to see if I can see an obvious way to implement it.

Test framework panics on false precondition. Example change to expose the panic can be seen in jamesd@d411fc1