Need to add back and correct unit tests for the following stats:

• global mean
• min/max
• variance

need a utilities file for common functions (e.g., get_gridcell_area_data_path())

Monitoring of increment max, min, mean, and RMS/MSE from log file. Could also include S1 scores, other skill scores and scalar attributes of performance.

The daily mean precipitation harvester calculates from a list of input files a median time to export to the SQL database. The SQL database expects a datetime object, but the harvester currently exports a cftime datetime object, which is not compatible with the database.

The test data files in /score-hv/tests/data should instead be hosted on the AWS server along with the files listed in this script: get_unit_test_data.sh. The test data and the cached versions of it will need to be removed once that is set up. This should make cloning the repo smaller and faster.

harvester gets unwieldy after adding derived variables

need better method to maintain and update packages on monitoring cluster

current warning output from score-hv unit tests:

`=============================================== warnings summary ================================================
tests/test_harvester_daily_bfg_prateb.py::test_variable_names
/contrib/home/builder/UFS-RNR-stack/anaconda3/lib/python3.9/site-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.26.1
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html`

Need to add setup and usage information (perhaps from a sample unit test) to the README for better usability. It would also be good to document the harvester registry and its template for adding new harvesters.

add feature to harvest vertically resolved analysis increments for replay data (from the fv3_increment6.nc files)

Harvesters that calculate global mean values need to be weighted by grid cell areas.

Right now the following dependencies are required to build and test this repo from a clean conda environment:
pip
xarray
dask
pytest
The following are installed via the setup.cfg script:
netCDF4
numpy
pandas
pyyaml
scipy

To make things easier for users, setup.cfg should be updated to include the first section of packages, or a new environment.yml file should be created to contain all of the necessary packages.

update use of old style of string formatting, e.g.,

msg = ("'%s' is not a supported "
             "variable to harvest from the background forecast data. "
             "Please reconfigure the input dictionary using only the "
             "following variables: %r" % (var, VALID_VARIABLES))

to use "f strings", e.g.,

msg = (f"{var} is not a supported "
             "variable to harvest from the replay analysis " 
             "increments (fv3_increment6.nc). "
             "Please reconfigure the input dictionary using only the "
             f"following variables: {VALID_VARIABLES}")

Need to integrate support for regions into score-hv calculations

several fields in the setup.cfg field are outdated, providing bad information for new users

When using pip to build and install the package, you have to use editable mode. This installs the package in /home//score-hv/src. If a standard pip install is used, the package is installed in the conda site-packages directory (..../envs//lib/python3.12/site-packages. This file path difference means that the get_gridcell_area_data_path() function in daily_bfg.py retrieves the wrong data path.

Potential solutions:

1. Update get_gridcell_area_data_path() to pull in the correct filepath for either install method
2. Recommend users install with pip editable mode until package is more stable. Before package is released, update get_gridcell_area_data_path() to work with standard mode only.

The harvester app will take the following form.

1. harvester will be kicked off by either dict or yaml (just like eva)
2. the config file needs to at least contain the name of the registered harvester that the user desires
3. each new harvester will have 2 basic sections (a) a config handler, and (b) a file reader and parser
4. the config will tell the parser exactly where to find the file to be harvested and what data it should parse from that file.
   The config will also tell the parser portion of the harvester exactly what format to output the results (i.e. list of tuples or pandas dataframe)

Create a new harvester for diag_conv_uv_ges.2015120318_cntrl.nc4 type files (or one of the next most relevant analysis file types).
This would include both any required dict format as well as the reading, parsing, and packaging of the relevant data. The data should be packaged into a list of tuples or a pandas dataframe.

global constants specified in multiple files allow for ambiguous definitions

Context

One of the things we need to do is to inventory our data collections. One type of tool that we develop are harvesters that parse files in to a subset of tuples that can be passed upstream for insertion into a database. Attached to this issue is an example of the log file that inventories number of observations in one of the files. the numbers are aggregated by

• variable type (e.g. each of the 7 sections in the log files with headers like: {'PRESSURE', 'SPECIFIC HUMIDTY', 'TEMPERATURE', 'HEIGHT', 'WIND COMPONENTS', 'PRECIPITABLE H2O', 'RELATIVE HUMIDTY'}. Notice that there are 7 sections but only 6 tables since 'RELATIVE HUMIDTY' has no data entries.
• observation platform: integer index in the first column of each table (heading 'typ')
• number of observations in columns 2-11. Number of observations are binned in total number and number of observations that passes certain quality control procedures. We are only interested in the total number (second column with heading 'tot') and high quality obs (third column with heading '0-3').

Problem description

Add new harvester in this directory that will parse the attached log file into a list of tuples. Harvester should be controlled by a yaml file with the following information:

• Location of the log file on disk.
• Which of the variables should be parsed {'PRESSURE', 'SPECIFIC HUMIDTY', 'TEMPERATURE', 'HEIGHT', 'WIND COMPONENTS', 'PRECIPITABLE H2O', 'RELATIVE HUMIDTY'}

Harvester needs to inherent from the harvester base class. An example of the harvester implementation for a different problem is provided here

The named tuples for the output should have the following fields

HarvestedData = namedtuple(
    'HarvestedData',
    [
        'file_name', # name of the log file
        'cycletime', # parsed from 'DATA  VALID AT' on the first line of the log file
        'variable',    # name of the variable passed in the yaml file
        'insturment type', # first column of the table in the log file
        'number_obs', # second column of the table in the log file
        'number_obs_qc_0to3', # third column of the table in the log file
    ],
)

Delivery of results

Preferred way is for you to deliver the results is to (1) clone the repo to your computer and (2) sending us back a tarball of your local checkout. While not elegant, this will ensure that your contribution will remain private. You are welcome to ask questions through email.

Simplifications to the problem

You might notice that the full repo has :

• A registry to register the harvester classes.
• And the pytest directory.

Ideally you will deliver your results as another pytest in the test directory. However, we appreciate that we are requesting you to do this work on your own time. You might choose to deliver the results in some other way that demonstrates how you think about this parsing problem and how you structure your work in a sequence of commits.

test_log_cmpbqm.txt