Python test script does not note occurrence of outputs that are not in the test dataset (Version check?) about cospv2.0 HOT 6 CLOSED

from cospv2.0.

This has been fixed by pull request #40, but only if the new python test script is used:

python cosp2_test_01.py data/outputs/UKMO/cosp2_output_um.nc data/outputs/UKMO/cosp2_output_um.ref.nc

You should see an output similar to the the step 'Compare output against KGO' in https://github.com/CFMIP/COSPv2.0/runs/645660411

@roj222 please can you confirm that you get that when you run the new script?

If this works as expected, then we should change the readme file to point users to the new script for their local tests. We also need to update the reference file cosp2_output_um.ref.nc so that it contains all the output variables. Given that we now have 2 reference datasets generated with 2 different compilers (ifort and gfortran), I propose to rename the ifort reference dataset to cosp2_output_um.ifort.kgo.nc.

from cospv2.0.

from cospv2.0.

@RobertPincus it's not entirely clear to me what you mean here. The new script exits with a failure code when the variables are not the same, i.e. it fails the continuous integration test, although it still prints the comparison table. I think we need this, because when you are adding a new diagnostic the test will fail, but you still want to know that the other variables are not broken.
Or are you proposing something different? Perhaps the output information needs to be more explicit so that it's easier to see when it's failed.

Regarding your second point, I like your suggestion. We could check for strict bit-comparability in the CI test, and relax tolerance in local tests. That would simplify things. It would only involve a small change in the readme file.

from cospv2.0.

from cospv2.0.

@RobertPincus Thanks for the clarification, I fully agree. I left the old script to avoid changing too many things at once, but that has obviously caused more confusion!
I'll create a pull request that addresses this.

from cospv2.0.