monocongo / ingest_cmorph Goto Github PK

Create a process that downloads all files for a month, sums the data, and writes the monthly values in the appropriate month index for the precipitation variable. Create a range of time indices and run this process for each, with a process pool using the number of CPUs specified (command line option?) or all available. Follow the pattern used in indices_python/process_grid.py, etc.

Consolidate the duplicate code for NetCDF creation into a single function. Take care of any other duplicate code found in the same manner.

Create a new script that writes the NetCDF with daily values. Use a placeholder for February 29th which is filled with the average value from February 28 and March 1st.

Create command line option and supporting functions for download/unzip/delete of files from FTP location rather than the use of local files.

In order to facilitate SPI processing, it could be more convenient to be able to read data from the NetCDF as if each year contained 366 days, rather than making this conversion after reading the daily values from a variable where the time coordinate uses a normal Gregorian calendar.

• after reading binary data convert all missing values (-999.9 etc.) to numpy.NaN in the data arrays before writing to NetCDF
• modify the NetCDF variable's creation fill_value to reflect the above change

Add code to stop input processing on the actual final month of the final year, the code currently crashes at the end, file closes and writes OK but this is leading to other issues when the file is used later in the pipeline, I think, which are discussed on the NCO Help Forum.

We used lat_start/lat end as the range limits for full grid but these variables were reused in error as range indices when chopping out CONUS, and later used as such even if CONUS only was not selected, causing them to not be set with proper values when ingesting the full grid.

There is a version of this ingest specific to the ICDR verson of CMORPH. Add this to our repository.

Add the capability of ingesting the gauge adjusted data files, for use with a comparative analysis of SPI results. These are located in the same FTP location (more or less) as the raw files, we'll create an appropriate URL for downloads based on a "raw or adjusted" conditional.

Files are currently expected to be resident on the machine where the ingest is performed. Add code that instead goes to the FTP location for the file(s), downloads and unzips, reads, then (optionally?) deletes.

• Verify with Olivier that the FTP site is public
• Use urllib if possible (avoid third-party libs)