emc2 / hunit-plus Goto Github PK

There should be a function, similar to testName for creating a TestSuite.

There is presently no way to log sysout and syserr output from tests.

Design and implement an intelligent, extensible timeout mechanism for tests. This needs to be able to cleanly support long-running tests and tests with unpredictable running times. A crude mechanism like "kill the test after N seconds" is undesirable.

A better mechanism would allow tests to periodically issue a "heartbeat", which would reset the timeout counter. Ideally, the timeout period could be changed on the fly as well.

This occurs when the name contains underscores. Likely due to a parsing error in filters.

When reporting errors, failures, and output from a test such as Outer.Inner.testname, text reporting modes will only show the path (example: "Error in Outer.Inner: ...")

The terminal output mode needs to output a \n at the end of execution.

XML reports seem to be giving the total number of assertions made in the test run in the "assertions" attribute for a test case.

The unit tests that should have caught this also need to be updated.

There should be a variant of assertThrows that catches SomeException, allowing for assertions about multiple exception types.

The terminal output mode can produce extremely large log files when used with 'cabal test'. The default output mode should be switched to text.

I am using HUnit-Plus to write tests to be run with stack test, with type: detailed-0.9, and ran into undefined, even on a test as simple as:

tests :: IO [Test]
tests = return ["testName" ~: True ~?= True]

And I traced it to L133 of Test.HUnitPlus.Base:

testinfo = unsafePerformIO $! newIORef undefined

Indeed, testinfo is used by the ~?= operator, but is reset only by executeTest, which I don't call since cabal or stack should take care of running the tests.

Initializing it to a defined value seems to fix it.

In some cases, (as evidenced by the unit tests), text reporting incorrectly outputs text to stderr sometimes as opposed to accumulating it as string output.

HUnit depended on deepseq, but the code no longer uses it. The dependency should be removed.

Citing from http://hydra.cryp.to/build/912674/nixlog/1/raw:

Building HUnit-Plus-1.0.1...
Preprocessing library HUnit-Plus-1.0.1...
[ 1 of 10] Compiling Test.HUnitPlus.Terminal ( src/Test/HUnitPlus/Terminal.hs, dist/build/Test/HUnitPlus/Terminal.o )
[ 2 of 10] Compiling Test.HUnitPlus.Filter ( src/Test/HUnitPlus/Filter.hs, dist/build/Test/HUnitPlus/Filter.o )
[ 3 of 10] Compiling Test.HUnitPlus.Reporting ( src/Test/HUnitPlus/Reporting.hs, dist/build/Test/HUnitPlus/Reporting.o )

src/Test/HUnitPlus/Reporting.hs:27:1: Warning:
    The import of ‘Data.Word’ is redundant
      except perhaps to import instances from ‘Data.Word’
    To import instances alone, use: import Data.Word()

<no location info>: 
Failing due to -Werror.

Text mode output is a bit lacking. It should report total counts. Verbose mode output should also report starting and ending suites, and execution time.

Replace all uses of String with Text for better performance.

1.1.0 is unbuildable but git HEAD is ok

It would be more in line with the library if assertThrows' first argument was a function of type "e -> Assertion" instead of "e -> Bool".

Currently, when a test is given a synthetic name, the result is that all such tests have the name "synthetic". The name of every synthetically-named test should be different.

The top-level test executor incorrectly ignores uncaught exceptions. They should be treated as errors.

The following error occurs when attempting to run a test program:

UnitTest: System.Console.CmdArgs.Implicit, unexpected mode: FlagType "FILE"

There should be an assertion that a given computation throws an exception

The code:

import Test.HUnitPlus
suite name = testSuite name [name ~: assertSuccess]
main = createMain [suite "Foo", suite "Bar"]

Console output:

Cases: 1  Tried: 1  Errors: 0  Failures: 0  Assertions: 1  Skipped: 0

Text report:

Test suite Foo starting
Test case Foo starting
Test Foo completed in 0.000000 sec
Test suiteFoo completed in 0.000000 sec
Test suite Bar starting
Test case Bar starting
Test Bar completed in 0.000000 sec
Test suiteBar completed in 0.000000 sec
Tests completed in 0.000000 sec
Cases: 2  Tried: 2  Errors: 0  Failures: 0  Assertions: 2  Skipped: 0

Same difference is observed between -c text and -c terminal.

Switch containers library to unordered-containers for better performance.

Assertions are causing tests to terminate early in some cases. Likely residual behavior from HUnit.

Options should be fully supported. This includes the following:

• Test execution should properly maintain options state during execution
• Tests need a better way of accessing options than what Distribution.TestSuite provides (possible additions to TestData)
• It should be possible to specify options in filter expressions
• Execution may need to execute a test or a test suite multiple times, with different options
• Some options should be automatically set by the test environment (OS, architecture, time of day, hostname, haskell version, etc)

Output is very vague, but since this has been happening a lot, I expect that this is because the test suite is checking something returned from unordered-containers in a way that is ordering-sensitive which is not guaranteed to be stable between releases.

Running 1 test suites...
Test suite RunTests: RUNNING...




Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Tests completed in 0.000000 sec
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Tests completed in 0.000001 sec
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Tests completed in 0.000001 sec
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Tests completed in 0.000001 sec
Cases: 0  Tried: 0  Errors: 0  Failures: 0  Assertions: 0  Skipped: 0
Tried: 14030  Errors: 0  Failures: 76
Test suite RunTests: FAIL
Test suite logged to: dist/test/HUnit-Plus-2.0.0-RunTests.log
0 of 1 test suites (0 of 1 test cases) passed.

Should be reproducible with cabal v2-test --constraint='unordered-containers == 0.2.16.0'.