openprov / interop-test-harness Goto Github PK

Investigate a way to invoke the rerunning of a test job automatically. This can happen, say, when a ProvPy build on Travis is successful. Then, the provpy-interop-job will automatically be run.

The ProvPy repo page, for example, can include another icon for the status of its interop tests and a developer can be aware of the existence of any interop issue without having to invoke the interop tests manually.

The ConverterTestCase class has a method:

def initialise_test_harness()

which initialises the test harness and provide the test cases as a generator. This is called back from:

@parameterized.expand(initialise_test_harness(), testcase_func_name=test_case_name)
def test_case(self, index, ext_in, file_ext_in, ext_out, file_ext_out)

and is used to provide the test cases so that nose-parameterized can auto-generate one test method per test case.If running Sphinx to create API documentation then the test harness initialisation is not done and, instead, a generator that contains zero test cases is returned. This is a hack to workaround Sphinx's execution of the Python it parses. Sphinx is detected if sys.argv[0] (the current command) contains the text sphinx-build.

If this is not done, then Sphinx documentation generation fails for this class and sub-classes as it tries to initialise the test harness.

Perhaps a less hacky solution can be adopted. I tried setting the environment variable, PROV_HARNESS_CONFIGURATION to point to a simple harness.yml file which points to a directory with no test cases, but getting the environment variable through Make and Sphinx proved problematic and would have needed the introduction of another shell script, which seemed like overengineering.

ProvToolbox takes a long time to install as the "mvn clean install" command not only builds the software but runs all its tests. This makes it very time consuming when using within the interoperability test harness (possibly the most time consuming part of the whole process). If there is a way to build ProvToolbox, without running its tests, when that should be tried in the Travis CI and Jenkins build configurations to see if it reduces the overall execution time for test jobs using ProvToolbox as a converter or comparator.

Improve the design of the harness framework so that:

• the framework will install/download its default comparator, not the specific job. The job should not have to change the comparator configurations (e.g. path to comparator script) unless it really needs to.
• a specific job will be able to define its converter class(es) and configurations by extending the base classes provided by the framework.

On the second point, the framework currently provides the converter classes for ProvPy, ProvToolbox, ProvStore, and ProvTranslator. I think those should be moved to the corresponding job repos. By so doing, we will also provide examples for converters developed not by Southampton team how to take advantage of the interop test framework without the need to modify the test framework's code.

There are over 7,500 tests currently generated and running jobs over those is time consuming, especially with those having to access remote services. Since all tests are independent to one another, parallelising their running will potentially speed up the test jobs significantly.

nose-testconfig plugin supports test configuration files including YAML and JSON. It also supports providing values for configuration parameters at the command-line, which could be used for (for example) specifying service URLs.

Consider replacing the YAML file handling in prov_interop/interop_tests with the use of this plugin.

When running a set of tests, a Travis CI test run can terminate with the following warning, if the log grows beyond 4MB:

The log length has exceeded the limit of 4 Megabytes (this usually means that test suite is raising the same exception over and over).

See, for example log.txt.

This could be an issue for the interoperability test harness which now run 7500+ tests. The problem arises from the outputs produced by nosetests.

nosetests supports two flags controlling test output:

• -v ensures that nosetests is verbose.
  • If absent, nosetests prints . when test passes, F when test fails.
  • If present nosetests prints name of each test method as it is run along with either ok or FAIL.
  • These are printed to the standard error stream.
• --nocapture ensures that standard output is printed immediately (see capture).
  • If absent, standard output is captured and printed as part of nosetests test report. Only standard output for failed tests is printed.
  • If present, standard output is printed when each test method runs. It is not added to the nosetests test report.

To see the difference between these, compare and contrast their use with a simple Python test class that has one test that passes and one that fails:

import unittest

class FilesTestCase(unittest.TestCase):

  def test_one(self):
     print("Test one prints something");
     self.assertEqual(1, 1, msg="Test one assertEqual failed!")

  def test_two(self):
    print("Test two prints something else");
    self.assertEqual(2, 1, msg="Test two assertEqual failed!")

$ nosetests test_verbosity.py 
.F
======================================================================
FAIL: test_two (test_verbosity.FilesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/disk/ssi-dev0/home/mjj/test_verbosity.py", line 11, in test_two
    self.assertEqual(2, 1, msg="Test two assertEqual failed!")
AssertionError: Test two assertEqual failed!
-------------------- >> begin captured stdout << ---------------------
Test two prints something else

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

$ nosetests -v test_verbosity.py 
test_one (test_verbosity.FilesTestCase) ... ok
test_two (test_verbosity.FilesTestCase) ... FAIL

======================================================================
FAIL: test_two (test_verbosity.FilesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/disk/ssi-dev0/home/mjj/test_verbosity.py", line 11, in test_two
    self.assertEqual(2, 1, msg="Test two assertEqual failed!")
AssertionError: Test two assertEqual failed!
-------------------- >> begin captured stdout << ---------------------
Test two prints something else

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 2 tests in 0.002s

FAILED (failures=1)

$ nosetests --nocapture test_verbosity.py 
Test one prints something
.Test two prints something else
F
======================================================================
FAIL: test_two (test_verbosity.FilesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/disk/ssi-dev0/home/mjj/test_verbosity.py", line 11, in test_two
    self.assertEqual(2, 1, msg="Test two assertEqual failed!")
AssertionError: Test two assertEqual failed!

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

$ nosetests -v --nocapture test_verbosity.py 
test_one (test_verbosity.FilesTestCase) ... Test one prints something
ok
test_two (test_verbosity.FilesTestCase) ... Test two prints something else
FAIL

======================================================================
FAIL: test_two (test_verbosity.FilesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/disk/ssi-dev0/home/mjj/test_verbosity.py", line 11, in test_two
    self.assertEqual(2, 1, msg="Test two assertEqual failed!")
AssertionError: Test two assertEqual failed!

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

nosetests test report (the content from ===== onwards) is printed to the standard error stream.

At present, the test jobs are run as follows:

nosetests --nocapture -v ...

If many of the interoperability tests fail and/or the tests print lots of information (e.g. ProvToolbox provconvert, when used to compare documents, prints information to standard output on their differences) then this can cause the test output to quickly balloon. This can be the case even if both -v and --nocapture are omitted.

The use of HTML reporting (#8) (using nose-htmloutput and nose-html-reporting) does not avert this, since the HTML report is generated in addition to the nose test report.

One solution is to redirect the standard error stream to a file,

$ nosetests -v test_verbosity.py > test_report.txt 2>&1
$ more test_report.txt 
test_one (test_verbosity.FilesTestCase) ... ok
test_two (test_verbosity.FilesTestCase) ... FAIL

=============================================================
=========
FAIL: test_two (test_verbosity.FilesTestCase)
-------------------------------------------------------------
---------
Traceback (most recent call last):
  File "/disk/ssi-dev0/home/mjj/test_verbosity.py", line 11, 
in test_two
    self.assertEqual(2, 1, msg="Test two assertEqual failed!"
)
nose.proxy.AssertionError: 2 != 1 : Test two assertEqual fail
ed!
-------------------- >> begin captured stdout << ------------
---------
Test two prints something else

--------------------- >> end captured stdout << -------------
---------

-------------------------------------------------------------
---------
Ran 2 tests in 0.002s

FAILED (failures=1)

Adopting this solution would incur the need to push the test report file to a place where it could be browsed further e.g. see #8 and using Uploading Artifacts on Travis CI.

Note that I tried this in conjunction with --nocapture too but the standard output printed while tests are running appears after the nosetests test report:

$ more test_report.txt 
test_one (test_verbosity.FilesTestCase) ... ok
test_two (test_verbosity.FilesTestCase) ... FAIL

=============================================================
=========
FAIL: test_two (test_verbosity.FilesTestCase)
-------------------------------------------------------------
---------
Traceback (most recent call last):
  File "/disk/ssi-dev0/home/mjj/test_verbosity.py", line 11, 
in test_two
    self.assertEqual(2, 1, msg="Test two assertEqual failed!"
)
AssertionError: 2 != 1 : Test two assertEqual failed!

-------------------------------------------------------------
---------
Ran 2 tests in 0.001s

FAILED (failures=1)
Test one prints something
Test two prints something else

But, arguably, --nocapture isn't so useful as it isn't necessary to capture standard output from all tests, only those that fail, which is done regardless.

Another option to reducing the overall test output is to partition up the tests into smaller jobs (e.g. using the Travis build matrix for parallelisation as described in #9)

After an interop test job has finished, a HTML report is generated to give:

• a summary of how many test cases has been passed/failed/skipped, and
• a listing of failed test cases with details about the failure (with a link to the test case repo, if possible)

Please see the nose-htmloutput/nose-html-reporting package for examples.

Given we currently have over 7,500 test cases, the above report will be very useful in helping isolate the failed cases for further investigation, which is challenging with the default text outputs.

When running a set of tests, a Travis CI test run can terminate with the following warning, if the run takes longer than 2 hours:

Your test run exceeded 120 minutes. 

One possible solution is to split up your test run.

See, for example log.txt

As for #12, a possible solution is to partition up the tests into smaller jobs (e.g. using the Travis build matrix for parallelisation as described in #9).