patcpayne / regression_testing Goto Github PK

Regression Testing Management routines

regtest.py is the main regression testing script for AMReX-based
applications.  Typing './regtest.py -h' will give a verbose
description of usage and setup.  The regtest set of tools is 
fully documented in the AMReX Users Guide, but here's a quick
overview of usage.

Quickstart:

1. Each regression test suite is defined in a separate configuration
file.  AMReX is distributed with an example such file in 
Tools/RegressionTesting

2. With example-tests.ini as an example, which builds tests for the
Castro code.

 a. Castro depends on the following addition git repositories: AMReX,
    Microphysics, wdmerger

 b. In the config blocks of the .ini file, the locations of clones
    of these files to use for testing are set as "dir".  There are 
    blocks in the ini file for each; AMReX has its own special
    block, and Castro is listed as the [source] block, but any 
    additional dependent repos are listed as [extra-XXX].

    For each repo defined in these blocks, the tester script pulls
    the desired branch/SHA prior to building/running tests.  It is 
    probably best to have these as "extra" clones in a special area
    used solely for this purpose so that the tester can work on its
    own without confusing you.

    Thus...edit the paths in the config file to point to local clones
    of each of these repos.

 c. Parameters that define each regression test appear in labeled
    blocks below the repository blocks.  The number of options allowed
    for specifying these tests will certainly grow over time; the -h 
    option to the tester should list everything currently available.

 d. Each labeled test will be built, run and compared to "benchmark"
    results.  The results of these tasks will be formatted into a
    web page in a folder defined as "webTopDir" in the .ini file.
    Make sure that variable points to something you like and can access.

 e. You need benchmarks to exist before the tests can work.  Generate
    them manually with 

    ./regtest.py --make_benchmarks "<some useful comment>" example-tests.ini

 f. Afterward, tests are run by typing

    ./regtest.py example-tests.ini