norfairking / validity Goto Github PK

View Code? Open in Web Editor NEW

156.0 9.0 30.0 1.45 MB

Validity and validity-based testing

Home Page: https://www.youtube.com/watch?v=eIs9qNh17SM

License: MIT License

Haskell 93.65% Nix 6.35%

invariants explicit-invariants invalid-values sanity-test validity

validity's Introduction

Validity and validity-based testing

Why?

to make writing correct software cheaper.

Free generators
Free shrinking
Cheap properties

Cheap properties

Property Combinators:

specify "inverse functions" $ inverseFunctions (+1) (-1)
specify "equivalent functions" $ equivalent ((+ 1) . (- 1)) id
specify "transitivity" $ transitivity ((>) :: Int -> Int -> Bool)
specify "symmetry" $ symmetry ((==) :: Int -> Int -> Bool)

Test suite combinators:

ordSpec @Int

  Ord Int
    (<=) :: Int -> Int -> Bool
      is reflexive for "even Int"'s
        +++ OK, passed 100 tests.
      is antisymmetric for "even Int"'s
        +++ OK, passed 100 tests.
      is transitive for "even Int"'s
        +++ OK, passed 100 tests.
      is equivalent to (\a b -> compare a b /= GT) for "even Int"'s
        +++ OK, passed 100 tests.
    (>=) :: Int -> Int -> Bool
      is reflexive for "even Int"'s
        +++ OK, passed 100 tests.
      is antisymmetric for "even Int"'s
        +++ OK, passed 100 tests.
      is transitive for "even Int"'s
        +++ OK, passed 100 tests.
      is equivalent to (\a b -> compare a b /= LT) for "even Int"'s
        +++ OK, passed 100 tests.
    (<) :: Int -> Int -> Bool
      is antireflexive for "even Int"'s
        +++ OK, passed 100 tests.
      is transitive for "even Int"'s
        +++ OK, passed 100 tests.
      is equivalent to (\a b -> compare a b == LT) for "even Int"'s
        +++ OK, passed 100 tests.
    (>) :: Int -> Int -> Bool
      is antireflexive for "even Int"'s
        +++ OK, passed 100 tests.
      is transitive for "even Int"'s
        +++ OK, passed 100 tests.
      is equivalent to (\a b -> compare a b == GT) for "even Int"'s
        +++ OK, passed 100 tests.

Free generators and shrinking

data MyType = MyType
  { myRational :: Rational
  , myBool :: Bool
  } deriving (Show, Eq, Generic)

instance Validity MyType -- Implementation is derived via Generic
instance GenValid MyType -- Default implementation via Generic

genValid :: Gen MyType -- Free generator
shrinkValid :: MyType -> [MyType] -- Free _valid_ shrinking function

Cachix cache

There is a cachix cache for this project.

To use it, use cachix use validity or add the appropriate details to your nixos configuration.

validity's People

Contributors

Stargazers

Watchers

validity's Issues

genvalidity-hspec >= 0.7.0.1 requires genvalidity-property >=0.5

While building for Stackage LTS 14:

--  While building package genvalidity-hspec-0.7.0.2 using:
      /var/stackage/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_6
4-linux/Cabal-2.4.0.1 build lib:genvalidity-hspec test:genvalidity-hspec-doctests test:genvalidity-hspec-test --ghc-options ""
    Process exited with code: ExitFailure 1
    Logs have been written to: /var/stackage/work/unpack-dir/.stack-work/logs/genvalidity-hspec-0.7.0.2.log

    Preprocessing library for genvalidity-hspec-0.7.0.2..
    Building library for genvalidity-hspec-0.7.0.2..
    [ 9 of 15] Compiling Test.Validity.GenValidity
    
    /var/stackage/work/unpack-dir/unpacked/genvalidity-hspec-0.7.0.2-a8dbf7e5b1b4f82d68e0d88817729944ea1485be6426b70ecde659f6e11f
99d9/src/Test/Validity/GenValidity.hs:99:26: error:
        • Couldn't match expected type ‘Property’
                      with actual type ‘(a -> [a]) -> Property’
        • Probable cause: ‘genGeneratesValid’ is applied to too few arguments
          In the expression: genGeneratesValid @a genValid
          In an equation for ‘genValidGeneratesValid’:
              genValidGeneratesValid = genGeneratesValid @a genValid
       |
    99 | genValidGeneratesValid = genGeneratesValid @a genValid
       |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    
    /var/stackage/work/unpack-dir/unpacked/genvalidity-hspec-0.7.0.2-a8dbf7e5b1b4f82d68e0d88817729944ea1485be6426b70ecde659f6e11f
99d9/src/Test/Validity/GenValidity.hs:110:30: error:
        • Couldn't match expected type ‘Property’
                      with actual type ‘(a -> [a]) -> Property’
        • Probable cause: ‘genGeneratesInvalid’ is applied to too few arguments
          In the expression: genGeneratesInvalid @a genInvalid
          In an equation for ‘genInvalidGeneratesInvalid’:
              genInvalidGeneratesInvalid = genGeneratesInvalid @a genInvalid
        |
    110 | genInvalidGeneratesInvalid = genGeneratesInvalid @a genInvalid
        |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Same for 0.7.0.1

Not all packages compatible with genvalidity 0.8

At least, that's what I think is happening, apologies if mistaken. The two examples:

Building library for genvalidity-vector-0.2.0.3..
[1 of 2] Compiling Data.GenValidity.Vector ( src/Data/GenValidity/Vector.hs, dist/build/Data/GenValidity/Vector.o )

src/Data/GenValidity/Vector.hs:18:10: error:
    • Could not deduce (GenUnchecked v)
        arising from a use of ‘Data.GenValidity.$dmshrinkValid’
      from the context: GenValid v
        bound by the instance declaration
        at src/Data/GenValidity/Vector.hs:18:10-44
      Possible fix:
        add (GenUnchecked v) to the context of the instance declaration
    • In the expression: Data.GenValidity.$dmshrinkValid @(V.Vector v)
      In an equation for ‘shrinkValid’:
          shrinkValid = Data.GenValidity.$dmshrinkValid @(V.Vector v)
      In the instance declaration for ‘GenValid (V.Vector v)’
   |
18 | instance GenValid v => GenValid (V.Vector v) where
   |

[20 of 22] Compiling Test.Validity.Functions.CanFail ( src/Test/Validity/Functions/CanFail.hs, dist/build/Test/Validity/Functions/CanFail.o )

src/Test/Validity/Functions/CanFail.hs:182:48: error:
    • Could not deduce (GenUnchecked b)
        arising from a use of ‘genUnchecked’
      from the context: (Show a, Show b, Show c, Show (f c),
                         GenInvalid a, GenInvalid b, CanFail f)
        bound by the type signature for:
                   failsOnInvalid2 :: forall a b c (f :: * -> *).
                                      (Show a, Show b, Show c, Show (f c), GenInvalid a,
                                       GenInvalid b, CanFail f) =>
                                      (a -> b -> f c) -> Property
        at src/Test/Validity/Functions/CanFail.hs:(170,1)-(180,15)
      Possible fix:
        add (GenUnchecked b) to the context of
          the type signature for:
            failsOnInvalid2 :: forall a b c (f :: * -> *).
                               (Show a, Show b, Show c, Show (f c), GenInvalid a, GenInvalid b,
                                CanFail f) =>
                               (a -> b -> f c) -> Property
    • In the fourth argument of ‘failsOnGens2’, namely ‘genUnchecked’
      In the first argument of ‘(.&&.)’, namely
        ‘failsOnGens2
           func genInvalid shrinkInvalid genUnchecked shrinkUnchecked’
      In the expression:
        failsOnGens2
          func genInvalid shrinkInvalid genUnchecked shrinkUnchecked
          .&&.
            failsOnGens2
              func genUnchecked shrinkUnchecked genInvalid shrinkInvalid
    |
182 |     failsOnGens2 func genInvalid shrinkInvalid genUnchecked shrinkUnchecked .&&.
    |                                                ^^^^^^^^^^^^

validity-aeson fails to build with aeson-2.0

validity-aeson                 > /tmp/stack-c3c07f9871ea1838/validity-aeson-0.2.0.4/src/Data/Validity/Aeson.hs:15:27: error:
validity-aeson                 >     • No instance for (Validity Object)
validity-aeson                 >         arising from a use of ‘annotate’
validity-aeson                 >     • In the expression: annotate o "Object"
validity-aeson                 >       In an equation for ‘validate’:
validity-aeson                 >           validate (Object o) = annotate o "Object"
validity-aeson                 >       In the instance declaration for ‘Validity Value’
validity-aeson                 >    |
validity-aeson                 > 15 |     validate (Object o) = annotate o "Object"
validity-aeson                 >    |                           ^^^^^^^^^^^^^^^^^^^

Genvalidity support for ghc-8.2 and prior

word to floating point number casting was added in ghc-8.4, so genvalidity fails to compile with:

src/Data/GenValidity/Utils.hs:282:22: error:
    Variable not in scope: castWord32ToFloat :: w0 -> Float
    |
282 | genFloat = genFloatX castWord32ToFloat
    |                      ^^^^^^^^^^^^^^^^^

src/Data/GenValidity/Utils.hs:286:23: error:
    Variable not in scope: castWord64ToDouble :: w1 -> Double
    |
286 | genDouble = genFloatX castWord64ToDouble
    |                       ^^^^^^^^^^^^^^^^^^

So it be nice to update the lower bounds to: base >= 4.11

Consider making `genValidStructurally` the default

Currently the default implementation for GenValid is

genValid = genUnchecked `suchThat` isValid

Pros:

It's always correct, even if isValid explicitly doesn't require validity of a subpart.
For example in

newtype InvalidRational = InvalidRational Rational

instance Validity InvalidRational where
  validate = trivialValidation

Admittedly, this is a pretty rare occurrence.

It does not require a Generic instance (as long as GenUnchecked didn't need one either).

The alternative is to have the default implementation be:

genValid = genValidStructurally

Pros: It's closer to what you want, usually. Eventhough an override is still necessary when you want genValidStructurallyWithoutExtraChecking.
Cons: It's not always correct, see the first pro of the current approach.

GenUnchecked shouldn't be required for GenValid

Right now, a type needs to be an instance of GenUnchecked in order to be made an instance of GenValid.
I think this might be unnecessary. Abstractly speaking, I don't think it needs to be, and it might be more elegant if the code reflected this.

Bounds for base are off

The lower bound on genvalidity for base should be set to to base >= 4.10

genvalidity       > /tmp/stack-3feadd8e35b44dfe/genvalidity-0.10.0.1/src/Data/GenValidity/Utils.hs:274:22: error:
genvalidity       >     Variable not in scope: castWord32ToFloat :: w0 -> Float
genvalidity       >     |                         
genvalidity       > 274 | genFloat = genFloatX castWord32ToFloat
genvalidity       >     |                      ^^^^^^^^^^^^^^^^^
genvalidity       >                               
genvalidity       > /tmp/stack-3feadd8e35b44dfe/genvalidity-0.10.0.1/src/Data/GenValidity/Utils.hs:278:23: error:
genvalidity       >     Variable not in scope: castWord64ToDouble :: w1 -> Double
genvalidity       >     |                         
genvalidity       > 278 | genDouble = genFloatX castWord64ToDouble
genvalidity       >     |                       ^^^^^^^^^^^^^^^^^^
genvalidity       >

README: empty prime factorisation

The README claims that only numbers greater than one have a prime factorisation and that the prime factorisation is always a non-empty list. I'd say that 1 has a perfect prime factorisation, namely the empty list. It holds product [] == 1.

Document RelativeValidity

The explanation is pretty sparse, and there aren't any instances to get a sense of what this class is for. I have no idea what it means.

Deriving useful implementation for genValid

In practise, the Generics-based default genValid instance is often too slow.
This means genValid often needs to be implemented by hand.
However, many composite data types have a validity instance saying
"MyType is valid iff its fields are valid". Because of the ubiquity of this pattern,
I propose to implement a Generics-based implementation (just like was done for
genUnchecked, but without making it the defaults implementation).

Tests fail on 32-bit architectures

After updating validity to 0.11.0.0 on Debian, tests have started failing for genvalidity (both versions 0.10.0.2 and 0.11.0.0 on 32-bit architectures. The error is the following:

Failures:

  test/Data/InstanceSpec.hs:156:11: 
  1) Data.Instance.Word32 genValid of Word32 generates only valid values
       Falsifiable (after 1 test):
         3581183768
       'validate' reported this value to be invalid:
       3581183768
       Violated: The contained integer is smaller than 2^32 = 4294967296
       

  To rerun use: --match "/Data.Instance/Word32/genValid of Word32 generates only valid values/"

  test/Data/InstanceSpec.hs:165:13: 
  2) Data.Instance.Word32 shrinkValid of Word32 shrinks to only valid values
       Falsifiable (after 1 test):
         3581183768
       'validate' reported this value to be invalid:
       0
       Violated: The contained integer is smaller than 2^32 = 4294967296
       

  To rerun use: --match "/Data.Instance/Word32/shrinkValid of Word32 shrinks to only valid values/"

  test/Data/InstanceSpec.hs:156:11: 
  3) Data.Instance.Int32 genValid of Int32 generates only valid values
       Falsifiable (after 1 test):
         1433700120
       'validate' reported this value to be invalid:
       1433700120
       Violated: The contained integer is smaller than 2^31 = 2147483648
       

  To rerun use: --match "/Data.Instance/Int32/genValid of Int32 generates only valid values/"

  test/Data/InstanceSpec.hs:165:13: 
  4) Data.Instance.Int32 shrinkValid of Int32 shrinks to only valid values
       Falsifiable (after 1 test):
         1433700120
       'validate' reported this value to be invalid:
       0
       Violated: The contained integer is smaller than 2^31 = 2147483648
       

  To rerun use: --match "/Data.Instance/Int32/shrinkValid of Int32 shrinks to only valid values/"

The complete build log can be found here. Any idea what is going on and how we can fix it? Thanks.

Are there replacements for `GenInvalid` and `GenUnchecked`?

I'm trying out this library and the few docs/intros that I can find often mention GenUnchecked and GenInvalid. But as far as I can tell they're not part of genvalidity anymore. Are there any replacements?

How would I go, for instance, about writing tests like the ones below:

-- I want to test this
sortPair :: Ord a => (a, a) -> (a, a)
sortPair (x, y) | x > y = (y, x)
sortPair p              = p

newtype Sorted = Sorted (Int, Int) deriving (Generic, Show)
instance Validity Sorted where validate (Sorted (x, y)) = declare "The pair is sorted" $ x <= y
instance GenValid Sorted
instance GenInvalid Sorted
instance GenUnchecked Sorted

spec :: Spec
spec =
  describe "sortPair" do
    it "is equivalent to `id` for sorted pairs" $
      forAllValid \(Sorted p) ->
        sortPair p `shouldBe` id p
    it "is equivalent to `swap` for unsorted pairs" $
      forAllInvalid \(Sorted p) ->
        sortPair p `shouldBe` swap p
    it "is idempotent" $
      forAllUnchecked \(Sorted p) ->
        sortPair (sortPair p) `shouldBe` sortPair p

Validity is wrong for rational numbers

The code that checks whether Rational is valid looks like this:

    isValid (d :% n) = isValid n && isValid d && d > 0

This confuses numerator and denominator, and checks whether the numerator is positive:

import Data.Validity
isValid (0::Rational)
False

Validity of Float special values

λ> isValid $ 0/0
    True

Is that OK?

validity-path documentation renders incorrectly

see https://hackage.haskell.org/package/validity-path-0.3.0.2/docs/Data-Validity-Path.html

genvalidity-text-1.0.0.1 benchmark compilation errors

Maybe something related changed recently?

Building benchmark 'genvalidity-text-bench' for genvalidity-text-1.0.0.1..                                               [22/184958]
[1 of 2] Compiling Main [Data.GenValidity.Text changed]           
                                                                  
/var/stackage/work/unpack-dir/unpacked/genvalidity-text-1.0.0.1-947a4d0faa7c03d509664d69857ba4e65f3d81a3a4c2363d7cddbfc1385a3a2b/ben
ch/Main.hs:20:11: error:
    • Variable not in scope:                                                                                                        
        genBenchSizes :: String -> Gen Strict.Text -> Benchmark   
    • Perhaps you meant ‘genBenchSized’ (imported from Data.GenValidity.Criterion)                                                  
   |                                                                                                                                20 |         [ genBenchSizes "Strict.Text" (genValid @Strict.Text),                                                                 
   |           ^^^^^^^^^^^^^
                                                                  
/var/stackage/work/unpack-dir/unpacked/genvalidity-text-1.0.0.1-947a4d0faa7c03d509664d69857ba4e65f3d81a3a4c2363d7cddbfc1385a3a2b/ben
ch/Main.hs:21:11: error:
    • Variable not in scope:                                                                                                        
        genBenchSizes :: String -> Gen Lazy.Text -> Benchmark     
    • Perhaps you meant ‘genBenchSized’ (imported from Data.GenValidity.Criterion)                                                  
   |                                                                                                                                21 |           genBenchSizes "Lazy.Text" (genValid @Lazy.Text)
   |           ^^^^^^^^^^^^^

/var/stackage/work/unpack-dir/unpacked/genvalidity-text-1.0.0.1-947a4d0faa7c03d509664d69857ba4e65f3d81a3a4c2363d7cddbfc1385a3a2b/ben
ch/Main.hs:25:11: error:
    • Variable not in scope:                                      
        genBenchSizes :: String -> Gen Strict.Text -> Benchmark
    • Perhaps you meant ‘genBenchSized’ (imported from Data.GenValidity.Criterion)
   |
25 |         [ genBenchSizes "via list (old version)" $ Strict.pack <$> genValid,
   |           ^^^^^^^^^^^^^

/var/stackage/work/unpack-dir/unpacked/genvalidity-text-1.0.0.1-947a4d0faa7c03d509664d69857ba4e65f3d81a3a4c2363d7cddbfc1385a3a2b/ben
ch/Main.hs:26:11: error:
    • Variable not in scope:
        genBenchSizes :: String -> Gen Strict.Text -> Benchmark
    • Perhaps you meant ‘genBenchSized’ (imported from Data.GenValidity.Criterion)
   |
26 |           genBenchSizes "genText" genText,
   |           ^^^^^^^^^^^^^

/var/stackage/work/unpack-dir/unpacked/genvalidity-text-1.0.0.1-947a4d0faa7c03d509664d69857ba4e65f3d81a3a4c2363d7cddbfc1385a3a2b/ben
ch/Main.hs:27:11: error:
    • Variable not in scope:
        genBenchSizes :: String -> Gen Strict.Text -> Benchmark
    • Perhaps you meant ‘genBenchSized’ (imported from Data.GenValidity.Criterion)
   |
27 |           genBenchSizes "genTextBy genValid" $ genTextBy genValid,
   |           ^^^^^^^^^^^^^

/var/stackage/work/unpack-dir/unpacked/genvalidity-text-1.0.0.1-947a4d0faa7c03d509664d69857ba4e65f3d81a3a4c2363d7cddbfc1385a3a2b/ben
ch/Main.hs:28:11: error:
    • Variable not in scope:
        genBenchSizes :: String -> Gen Strict.Text -> Benchmark
    • Perhaps you meant ‘genBenchSized’ (imported from Data.GenValidity.Criterion)
   |
28 |           genBenchSizes "genTextBy (choose (minBound, maxBound)) (currently in use)" $
   |           ^^^^^^^^^^^^^

Validity not defined for Ratio

Validity can check Rational numbers, but not the generic Ratio type. (It is Rational = Ratio Integer.)

genvalidity-time-1.0.0.0: No instance for (Validity CalendarDiffDays)

In the Stackage Nightly build:

/var/stackage/work/unpack-dir/unpacked/genvalidity-time-1.0.0.0-52c81aff7fd255e6be2a0ebeaa728a8629f4a6f90c25e97b66cf0fb23c693bf5/sr
c/Data/GenValidity/Time/Calendar.hs:38:10: error:                
    • No instance for (Validity CalendarDiffDays)                                                                                  
        arising from the superclasses of an instance declaration                                                                   
    • In the instance declaration for ‘GenValid CalendarDiffDays’                                                                  
   |                                                                                                                               
38 | instance GenValid CalendarDiffDays where                                                                                      
   |          ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                            
                                                                                                                                   
/var/stackage/work/unpack-dir/unpacked/genvalidity-time-1.0.0.0-52c81aff7fd255e6be2a0ebeaa728a8629f4a6f90c25e97b66cf0fb23c693bf5/sr
c/Data/GenValidity/Time/Calendar.hs:42:10: error:                
    • No instance for (Validity DayOfWeek)                                                                                         
        arising from the superclasses of an instance declaration                                                                   
    • In the instance declaration for ‘GenValid DayOfWeek’                                                                         
   |                                                             
42 | instance GenValid DayOfWeek where                                                                                             
   |          ^^^^^^^^^^^^^^^^^^

doctest failures

I'm seeing a number of test suite failures on the stackage build server

genvalidity-hspec-aeson: BuildFailureException Process exited with ExitFailure 1: dist/build/genvalidity-hspec-aeson-doctests/genvalidity-hspec-aeson-doctests
genvalidity-hspec-binary: BuildFailureException Process exited with ExitFailure 1: dist/build/genvalidity-hspec-binary-doctests/genvalidity-hspec-binary-doctests
genvalidity-hspec-cereal: BuildFailureException Process exited with ExitFailure 1: dist/build/genvalidity-hspec-cereal-doctests/genvalidity-hspec-cereal-doctests
genvalidity-hspec-optics: BuildFailureException Process exited with ExitFailure 1: dist/build/genvalidity-hspec-optics-doctests/genvalidity-hspec-optics-doctests
genvalidity-property: BuildFailureException Process exited with ExitFailure 1: dist/build/genvalidity-property-doctests/genvalidity-property-doctests

The errors all seem to be the same issue:

    Variable not in scope:
      mkName
        :: [Char]
           -> template-haskell-2.14.0.0:Language.Haskell.TH.Syntax.Name

I haven't taken a close look, but something like the following (near the top of each affected module) might fix it:

-- $setup
-- >>> import Language.Haskell.TH (mkName)

See: https://github.com/sol/doctest#setup-code

Commutative properties type

Is there a reason for commutative property to take only functions that have the same result type as its arguments?

That is,
commutativeOnGens :: (Show a, Eq a) => (a -> a -> a) -> Gen (a, a) -> ((a, a) -> [(a, a)]) -> Property
instead of
commutativeOnGens :: (Show a, Eq b, Show b) => (a -> a -> b) -> Gen (a, a) -> ((a, a) -> [(a, a)]) -> Property.

Aeson tests should support golden round trip tests

I'd like something similar to hspec-golden-aeson allowing GenValidity to create a sufficient consistent set of values that can be serialized and tested against in a golden test.

Right now, one must think of all of the ways to build an object (including transitive values) and write tests for all the combinations. It'd be great to do this automatically.

Aeson tests don't validate toJSON and toEncoding agree

It's recommended to implement toEncoding when instantiating ToJSON, but there's no validation ensuring it agrees with toJSON (which is a requirement since it doesn't go the other way).

A simple workaround might be to encode toJSON v and v both while testing round tripping. They don't need to end with identical representations, but the decoder needs to agree that they're equivalent.

genvalidity-sydtest specs for Enum/Bounded

I've been working on a set of specs for testing manually-written Enum instances (toEnum and fromEnum round-trip, if the type is Bounded too then pred/succ fail only on minBound/maxBound, etc). I was going to write up a pull request when done.

Now I have rethought what I was doing a little so that I no longer have a use-case for an Enum instance that can't be auto-derived. So I don't really need specs for Enum instances any more. I'm happy to finish and submit the pull request, if you think it would be useful to have in genvalidity-sydtest. But given I don't have an immediate use case, I thought I'd check first whether it's something you would actually want to provide.

Generate validity

Upon defining a type to be an instance of validity by defining an isValid function,
the validate function is automatically generated. However, this function does not always
actually compile. I would propose to try and find a good default implementation, as you
are right now, but if this implementation doesn't work, to use validate = validateByCheckingName [name of the type].

Compile failure w/ genvalidity-path-0.1.0.1 & base-4.7

The cabal file states compatibility w/ base >= 4.7, however GHC 7.8.4 disagrees:

Configuring component lib from genvalidity-path-0.1.0.1...
Preprocessing library genvalidity-path-0.1.0.1...
[1 of 1] Compiling Data.GenValidity.Path ( src/Data/GenValidity/Path.hs, /tmp/matrix-worker/1489783673/dist-newstyle/build/x86_64-linux/ghc-7.8.4/genvalidity-path-0.1.0.1/build/Data/GenValidity/Path.o )

src/Data/GenValidity/Path.hs:13:25:
    Not in scope: ‘<$>’
    Perhaps you meant ‘</>’ (imported from Path)

GenUnchecked instance for strict ByteString is utterly bad

As the documentation states:

-- | WARNING: Unchecked ByteStrings are *seriously* broken.
-- The pointer may still point to something which is fine, but
-- the offset and length will most likely be complete nonsense.
-- This will most-likely lead to segfaults.

It does indeed lead to consistent segfaults, as such should either be removed or fixed. Even something like that would be better than what there is now: genUnchecked = SB.pack <$> genUnchecked

GenUnchecked is unnecessary for GenValid

Right now, a type needs to be an instance of GenUnchecked in order to be made an instance of GenValid.
This is logically unnecessary.

Show/Read test suite combinator

When both Show and Read is instantiated for a given type, I'd like to be able to test, with a test suite combinator, that read (show a) == a holds for any qualifier value.

genvalidity-hspec doesn't build with hspec 2.10

[ 2 of 13] Compiling Test.Validity.Utils ( src/Test/Validity/Utils.hs, dist/build/Test/Validity/Utils.dyn_o )

src/Test/Validity/Utils.hs:71:57: error:
    • Couldn't match type: [SpecTree b]
                     with: (base-4.15.1.0:Data.Semigroup.Internal.Endo Config,
                            [SpecTree b])
      Expected: (r,
                 (base-4.15.1.0:Data.Semigroup.Internal.Endo Config, [SpecTree a]))
                -> (r,
                    (base-4.15.1.0:Data.Semigroup.Internal.Endo Config, [SpecTree b]))
        Actual: (r, [SpecTree a]) -> (r, [SpecTree b])
    • In the first argument of ‘fmap’, namely ‘(second (map f))’
      In the first argument of ‘mapWriterT’, namely
        ‘(fmap (second (map f)))’
      In the first argument of ‘SpecM’, namely
        ‘(mapWriterT (fmap (second (map f))) specs)’
    • Relevant bindings include
        f :: SpecTree a -> SpecTree b
          (bound at src/Test/Validity/Utils.hs:71:14)
        mapSpecTree' :: (SpecTree a -> SpecTree b)
                        -> SpecM a r -> SpecM b r
          (bound at src/Test/Validity/Utils.hs:71:1)
   |
71 | mapSpecTree' f (SpecM specs) = SpecM (mapWriterT (fmap (second (map f))) specs)
   |                                                         ^^^^^^^^^^^^^^

genvalidity-hspec-0.6.2.1 test suite failure (from hackage tarball)

It looks like some sort of -Wall failure, rather than an actual test failure. Here's what I saw on the stackage build server:

[14 of 14] Compiling Main             ( test/Spec.hs, dist/build/genvalidity-h
spec-test/genvalidity-hspec-test-tmp/Main.o )
Linking dist/build/genvalidity-hspec-test/genvalidity-hspec-test ...
> /tmp/stackage-build13/genvalidity-hspec-0.6.2.1$ dist/build/genvalidity-hspe
c-doctests/genvalidity-hspec-doctests

src/Test/Validity/Utils.hs:80:13: warning: [-Wmissing-fields]
    • Fields of ‘Item’ not initialised: itemIsFocused
    • In the first argument of ‘Leaf’, namely
        ‘Item
           {itemRequirement = s, itemLocation = Nothing,
            itemIsParallelizable = Nothing,
            itemExample = \ _ _ _
                            -> do let ...
                                  ....}’
      In the expression:
        Leaf
          Item
            {itemRequirement = s, itemLocation = Nothing,
             itemIsParallelizable = Nothing,
             itemExample = \ _ _ _
                             -> do let ...
                                   ....}
      In an equation for ‘go’:
          go sp
            = Leaf
                Item
                  {itemRequirement = s, itemLocation = Nothing,
                   itemIsParallelizable = Nothing, itemExample = \ _ _ _ -> do ...}
   |
80 |             Item
   |             ^^^^...

src/Test/Validity/Utils.hs:80:13: warning: [-Wmissing-fields]
    • Fields of ‘Item’ not initialised: itemIsFocused
    • In the first argument of ‘Leaf’, namely
        ‘Item
           {itemRequirement = s, itemLocation = Nothing,
            itemIsParallelizable = Nothing,
            itemExample = \ _ _ _
                            -> do let ...
                                  ....}’
      In the expression:
        Leaf
          Item
            {itemRequirement = s, itemLocation = Nothing,
             itemIsParallelizable = Nothing,
             itemExample = \ _ _ _
                             -> do let ...
                                   ....}
      In an equation for ‘go’:
          go sp
            = Leaf
                Item
                  {itemRequirement = s, itemLocation = Nothing,
                   itemIsParallelizable = Nothing, itemExample = \ _ _ _ -> do ...}
Examples: 19  Tried: 19  Errors: 0  Failures: 0
> /tmp/stackage-build13/genvalidity-hspec-0.6.2.1$ dist/build/genvalidity-hspec-test/genvalidity-hspec-test
genvalidity-hspec-test: src/Test/Validity/Utils.hs:(80,13)-(96,17): Missing field in record construction itemIsFocused

genvalidity-text-1.0.0.0 testsuite failed in Stackage Nightly build

    Failures:
    
      test/Data/GenValidity/TextSpec.hs:89:28: 
      1) Data.GenValidity.Text, Strict Text, textWithoutAnyOf, never contains any of the given chars
           Falsifiable (after 62 tests):
             "Sf\177445j\ETB\1042812\STX\RS\ESC\54796K,\145365%\165630\r\ETB\990043r\1090611V\b{\1075011\135158P\1020487 \1080484\53064\65533o[\169085<\SOH\186133Yj}g4fOp"
             "\65533\1090810\170309\265568\1108101\65533\595966\1010967\986930\916105\2241\860537\660185\658381\1001861\65533\30519\485547\717184\65533\419308\1022746\347921\65533\446086\628980\974780\461121\1113039\796610\65533\251864\716438\65533\492899"
           predicate succeeded on: "\65533\1090810\170309\265568\1108101\65533\595966\1010967\986930\916105\2241\860537\660185\658381\1001861\65533\30519\485547\717184\65533\419308\1022746\347921\65533\446086\628980\974780\461121\1113039\796610\65533\251864\716438\65533\492899"
    
      To rerun use: --match "/Data.GenValidity.Text/Strict Text/textWithoutAnyOf/never contains any of the given chars/"
    
    Randomized with seed 18824094
    
    Finished in 0.0146 seconds
    13 examples, 1 failure

genvalidity-0.11.0.2: Variable not in scope: isLineSeparator

Building library for genvalidity-0.11.0.2..                                                                                         
[2 of 4] Compiling Data.GenValidity                               
                                                                                                                                    
/var/stackage/work/unpack-dir/unpacked/genvalidity-0.11.0.2-3d6a0489512630870494f29bc8efefa8f2cf6dfa55f45eb6373e17849e5d5598/src/Data/GenValidity.hs:652:50: error:                                   
    • Variable not in scope: isLineSeparator :: Char -> Bool                                                                        
    • Perhaps you meant ‘genLineSeparator’ (line 649)                                                                               
    |                                                                                                                               
652 | genNonLineSeparator = genValid `suchThat` (not . isLineSeparator)                                                             
    |                                                  ^^^^^^^^^^^^^^^

It's possible to create cyclic instances

This is possible, and loops.

data Example = …

instance Validity Example where
  validate example = delve "Loops" example

It would be nice to catch this, somehow.

Char should not be trivially valid

Currently in validity Char is trivially valid. But if you check the chr function it is expected to be:

| isTrue# (int2Word# i# `leWord#` 0x10FFFF##) = C# (chr# i#)

While when read from memory only 31 bits are expected: readWideCharOffAddr#

In other words I can construct an invalid Char# that can mess things up:

λ> C# (chr# 2147483647#)
'\2147483647'
λ> chr 2147483647
*** Exception: Prelude.chr: bad argument: 2147483647

In fact it is possible to construct even larger 64bit Chars that are ven negative 🙂

λ> C# (chr# 9223372036854775807#)
'\9223372036854775807'
λ> C# (chr# -9223372036854775808#)
'\-9223372036854775808'

genvalidity-hspec-0.3.0.0 uses TypeApplications which was introduced only in GHC-8.0

Please make a revision that adds a lower bound on base.

If possible I'd propose removing the use of that extension so the package can be used with older versions of base.

genvalidity-hspec-optics is named confusingly

It's about lens not optics!

Instances for all vectors

Validity
GenUnchecked
GenValid
GenInvalid

All different vectors in http://hackage.haskell.org/package/vector

Missing function isValidValidation

This library is working great so far me apart for a minor catch.
I can't find a way to go from a Validation to a Bool.
I know that most of the time this is not needed as we can use isValid.
However in my case I have other functions producing a Validation than validate.
Example:

composable :: T -> T -> Validation

Slow tests

genvalidity:genvalidity-test timed out on test machine for Stackage builds (the current test suite timeout is 10 minutes), the last lines were

  (Float,Float,Float,Float,Float)
    genUnchecked of (Float,Float,Float,Float,Float) does not crash while validating
      +++ OK, passed 100 tests.
    shrinkUnchecked of (Float,Float,Float,Float,Float) only produces values that do not crash while validating
      +++ OK, passed 20 tests.
    shrinkUnchecked of (Float,Float,Float,Float,Float) does not shrink to itself
      +++ OK, passed 20 tests.
    genValid of (Float,Float,Float,Float,Float) generates only valid values
      +++ OK, passed 100 tests.
    shrinkValid of (Float,Float,Float,Float,Float) shrinks to only valid values
      +++ OK, passed 20 tests.

And on my machine 3rd run took

genvalidity-0.7.0.0: Test suite genvalidity-test passed

real	16m19,237s
user	25m46,936s
sys	17m34,115s

and it looked like the most problematic tests were for Float and Double tuples.

I will add this package into expected-test-failures section in build constraints of Stackage.

Validity for ShortByteString

There is currently no instance for ShortByteString in validity-bytestring.

genvalidity-hspec-0.6.1.0 build failure with hspec-core >= 2.5.3

It seems that mapSpecTree was removed from Test.Hspec.Core.Spec in hspec-core-2.5.3. (Which is odd, because this is a breaking change which should have warranted a major version bump.)

[ 2 of 14] Compiling Test.Validity.Utils ( src/Test/Validity/Utils.hs, dist/build/Test/Validity/Utils.o )

src/Test/Validity/Utils.hs:66:18: error:
    • Variable not in scope:
        mapSpecTree
          :: (SpecTree () -> SpecTree ()) -> SpecWith () -> SpecWith ()
    • Perhaps you meant ‘mapSpecItem’ (imported from Test.Hspec.Core.Spec)
   |
66 | failsBecause s = mapSpecTree go
   |                  ^^^^^^^^^^^

genvalidity-containers-0.9.0.0 test compilation failure

Found in stackage nightly, will disable the tests there for now - please file a PR if they get fixed and you'd like us to run them again.

    [3 of 6] Compiling Data.GenValidity.Containers.SetSpec

    /var/stackage/work/unpack-dir/unpacked/genvalidity-containers-0.9.0.0-fea514851cc3b29e4ff63455e90861dc68b973346e0e03613ba173c0f9f159f8/test/Data/GenValidity/Containers/SetSpec.hs:33:52: error:
        Variable not in scope: distinctOrd :: [Int] -> prop0
       |
    33 |         forAll (genSeperate genUnchecked) $ \ls -> distinctOrd (ls :: [Int])
       |                                                    ^^^^^^^^^^^

How to build

I tried building validity with these steps

Going into a nix shell with nix-shell
cd validity
stack build

I then get this https://gist.github.com/piq9117/ca163f79352bcb171ce6ae537208cf44

I apologize if there is already docs on how to do this but I've looked in docs/ and couldn't find anything. I can update the docs if I learn how to build this project.

Check validity for Functor instances?

If I define my own instance of Functor, is there a convenient way to have validity check that it satisfies the functor laws? I could cobble something together, but my question is:

Are the functor laws already described in validity? I couldn't find them.
If now, how would I best generically define the laws?
How would I apply this then to my own Functor instance?

genvalidity-time test compilation failure

I haven't tracked down exactly what changed yet, but I might update this later

Building test suite 'genvalidity-time-test' for genvalidity-time-0.2.1.0..
[1 of 3] Compiling Data.GenValidity.TimeSpec ( test/Data/GenValidity/TimeSpec.hs, dist/build/genvalidity-time-test/genvalidity-time-test-tmp/Data/GenValidity/TimeSpec.o )

test/Data/GenValidity/TimeSpec.hs:32:5: error:
    • No instance for (Eq ZonedTime)
        arising from a use of ‘shrinkValiditySpec’
    • In a stmt of a 'do' block: shrinkValiditySpec @ZonedTime
      In the expression:
        do genValidSpec @Day
           shrinkValidSpec @Day
           genValidSpec @UniversalTime
           shrinkValidSpec @UniversalTime
           ....
      In an equation for ‘spec’:
          spec
            = do genValidSpec @Day
                 shrinkValidSpec @Day
                 genValidSpec @UniversalTime
                 ....
   |
32 |     shrinkValiditySpec @ZonedTime
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

GenUnchecked and abstract data types

I really like the GenUncheck vs GenValid distinction. However I'm running into an issue using it in practice. Pretty much all of the meaningful types in my programs are composed of at least one abstract data type-- usually Text, but sometimes others. This causes a problem because I lose the usefulness of the GenUnchecked vs. GenValid distinction.

Eg: imagine I'm implementing a programming language. I'd like GenUnchecked to generate all possible ADTs so that I can assert they can all roundtrip through serialization-parsing, etc. Then I'd like GenValid to generate only ones that typecheck, so that I can assert things like they give the same results when run through different interpreters. However, since I usually have Text in there somewhere I can't do this, because GenUnchecked is forbidden for Text.

Does that make sense? If it does I think the problem comes from different users having different perspectives on things like Text. An implementer of the text package would like a GenUnchecked instance for Text that truly generates all possible instances of the internal data type.

However, a user of the text package likely wants to be able to trust the abstraction and that the smart constructors like pack are implemented correctly, and have GenUnchecked for Text generate only what could be generated with pack.

What are your thoughts on this? I think a GenUnchecked Text instance that generates valid Text is what almost all users are going to want from genvalidity-text, but I'm curious to hear the reasoning behind the current setup.

Missing documentation at http://hackage.haskell.org/package/validity

I can't tell what this package is about from looking at the above Hackage page (nor from looking at the README in this repo).

It would be great if you could add some info to the description field in the .cabal file(s) and make sure that the haddocks is generated or upload it yourself. I believe the neil tool can do the latter for you.

You could also consider adding validity to stackage, where the haddocks are generated more reliably.

Generate validate

If I define "isValid", the automatically generated "validate" function should always compile.
If you have enough tools to build a type-specific validate function, great.
If not, define validate = validateByCheckingName [name of the type].

compile failure with genvalidity-0.3.2.0

I noticed this compile failure here with GHC 8.0:

Configuring component lib from genvalidity-0.3.2.0...
Preprocessing library genvalidity-0.3.2.0...
[1 of 2] Compiling Data.GenValidity ( src/Data/GenValidity.hs, /tmp/matrix-worker/1489769093/dist-newstyle/build/x86_64-linux/ghc-8.0.2/genvalidity-0.3.2.0/build/Data/GenValidity.o )

src/Data/GenValidity.hs:260:10: error:
    • No instance for (Validity Word8)
        arising from the superclasses of an instance declaration
    • In the instance declaration for ‘GenValid Word8’

src/Data/GenValidity.hs:265:10: error:
    • No instance for (Validity Word16)
        arising from the superclasses of an instance declaration
    • In the instance declaration for ‘GenValid Word16’

and this one with GHC 7.8:

Configuring component lib from genvalidity-0.3.2.0...
Preprocessing library genvalidity-0.3.2.0...
[1 of 2] Compiling Data.GenValidity ( src/Data/GenValidity.hs, /tmp/matrix-worker/1489768554/dist-newstyle/build/x86_64-linux/ghc-7.8.4/genvalidity-0.3.2.0/build/Data/GenValidity.o )

src/Data/GenValidity.hs:143:10:
    Could not deduce (Validity (Either a b))
      arising from the superclasses of an instance declaration
    from the context (GenValid a, GenValid b)
      bound by the instance declaration
      at src/Data/GenValidity.hs:(143,10)-(144,30)
    In the instance declaration for ‘GenValid (Either a b)’

src/Data/GenValidity.hs:148:10:
    Could not deduce (Validity (Either a b))
      arising from the superclasses of an instance declaration
    from the context (GenInvalid a, GenInvalid b)
      bound by the instance declaration
      at src/Data/GenValidity.hs:(148,10)-(149,32)
    In the instance declaration for ‘GenInvalid (Either a b)’

src/Data/GenValidity.hs:230:10:
    No instance for (Validity ())
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid ()’

src/Data/GenValidity.hs:235:10:
    No instance for (Validity Bool)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Bool’

src/Data/GenValidity.hs:240:10:
    No instance for (Validity Ordering)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Ordering’

src/Data/GenValidity.hs:245:10:
    No instance for (Validity Char)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Char’

src/Data/GenValidity.hs:250:10:
    No instance for (Validity Int)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Int’

src/Data/GenValidity.hs:255:10:
    No instance for (Validity Word)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Word’

src/Data/GenValidity.hs:260:10:
    No instance for (Validity Word8)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Word8’

src/Data/GenValidity.hs:265:10:
    No instance for (Validity Word16)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Word16’

src/Data/GenValidity.hs:274:10:
    No instance for (Validity Float)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenInvalid Float’

src/Data/GenValidity.hs:283:10:
    No instance for (Validity Double)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenInvalid Double’

src/Data/GenValidity.hs:289:10:
    No instance for (Validity Integer)
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid Integer’

src/Data/GenValidity.hs:297:10:
    No instance for (Validity (Ratio Integer))
      arising from the superclasses of an instance declaration
    In the instance declaration for ‘GenValid (Ratio Integer)’

src/Data/GenValidity.hs:303:10:
    Could not deduce (Validity (Fixed a))
      arising from the superclasses of an instance declaration
    from the context (HasResolution a)
      bound by the instance declaration
      at src/Data/GenValidity.hs:(303,10)-(304,27)
    In the instance declaration for ‘GenValid (Fixed a)’

Use `Data.Map.Lazy.valid`

There could be invalid structurally Map k v: http://hackage.haskell.org/package/containers-0.5.7.1/docs/Data-Map-Lazy.html#v:valid

related: parent packages should note that there are companion instance packages.

norfairking / validity Goto Github PK

validity's Introduction

Validity and validity-based testing

Why?

Cheap properties

Free generators and shrinking

Cachix cache

validity's People

Contributors

Stargazers

Watchers

Forkers

validity's Issues

Recommend Projects

Recommend Topics

Recommend Org