Investigation group idea: testing about fims-planning HOT 3 CLOSED

Scope
This group will focus on investigating testing frameworks to be applied for FIMS. We aim to discuss how to compare FIMS model runs with previous FIMS model versions, other modeling platforms, and true values from operating models through various tests. We plan to discuss error checks for FIMS input and output (i.e., run-time testing) but will check with the input/output file and type structure group first to avoid overlap. More specifically, we will:

1) share testing experience and review test cases from various platforms (stock assessment platforms and other software projects). We can compile a list of examples and discuss how to set up a testing framework structure, how often to run each type of test, and how to automate acceptance tests. For automated acceptance tests, we can discuss how to create expectation and evaluation criteria (e.g., tolerance value).

2) discuss various types of tests that could be useful for FIMS development. For example, why, who, when, and how to conduct unit tests, functional tests, integrated tests, regression tests, compatibility tests, and one-off tests (e.g., for replicating and fixing user-reported bugs). We plan to provide a glossary of terms and simple examples for each type of test. We can work with @Andrea-Havron-NOAA and see how tests can be implemented across different software languages under consideration for FIMS.

3) discuss FIMS test procedures. For example, draft a test procedure template and discuss the steps for conducting individual tests.

from fims-planning.

I encourage feedback on this topic from @k-doering-NOAA as she has set up a highly functional test system using Jenkins for Stock Synthesis

from fims-planning.

@Rick-Methot-NOAA , luckily I asked to join this group with Bai already!

I added @Bai-Li-NOAA and myself to assignees to indicate we are planning on participating in this group (not sure if there is a better way to do this). Feel free to assign yourself, too if anyone else wanted to join this investigation group.

from fims-planning.