ciroh-ua / ngen-datastream Goto Github PK

Currently forcingprocessor contains a poor hack to calculate precip_rate for NextGen. RAINRATE should be converted in a physically consistent way.

nwm_variables = [
    "U2D",
    "V2D",
    "LWDOWN",
    "RAINRATE",
    "RAINRATE",
    "T2D",
    "Q2D",
    "PSFC",
    "SWDOWN",
]

global ngen_variables
ngen_variables = [
    "UGRD_10maboveground",
    "VGRD_10maboveground",
    "DLWRF_surface",
    "APCP_surface",
    "precip_rate",  # HACK RAINRATE * 3600
    "TMP_2maboveground",        
    "SPFH_2maboveground",
    "PRES_surface",
    "DSWRF_surface",
] 

the t-route ngen.yaml config file needs to have a look over. Perhaps this repo should have several configs depending on vpu, ncores, or any other axis we may want variation along.

the commit hashes of both NGIAB and the datastream should be tracked in metadata

Both t-route and NoahOWP require configuration files as input. Currently these are made from templates within the repository. Eventually, these should be created from pydantic models from ngen-cal (might be renamed to ngen-tools/utils).

the datastream should write out images of pre and post forcingprocessing.

In the current datastream aws state machine, each VPU independently calculates it's own forcings. Based on the internal algorithm to forcingprocessor, this is more efficient (cheaper) if done on a single instance. Then the individual vpu runs can be supplied the path to the forcings bucket in an argument to stream.sh.

This should be GPL-3.0 not GBL-3.0

Downloading this repository to a basic laptop or desktop and turning it on will, in the current configuration, either not work or will consume every resource available to the computer and make it unusable.

Let's update the documentation on the front to make that a little more clear and then, as an additional update, make sure it gives the user a little warning in the initial startup: "Hey, you are about to do something a little painful. You need to sign here on your RAM with this magic marker before we put you under."

Implement validation for forcings if provided as netcdf

By default we can just shove in the NGIAB defaults, then if CONUS is specified we can interpret the default CFE global config and let the user deal with any more specific configs.

Ideally we can build it so that the script will run a CONUS CFE with "DAILY" and CONUS specified.

It'll make it easier to spin up a stream and then customize, rather than have the user kind of walking blind until they get something functional.

Tests need to be written to ensure all file path options that can be produced by nwmurl can be processed without error.

TF can't create the bucket, just connect to it. Add conditional to check if bucket name exists, create if not. Will need to deal with the resources directory object somehow upon bucket creation.

Interactive NGIAB can (and should) be easily converted to an automatic run provided that the configs are married to the data stream configs in a way that makes the upper level config the only necessary config file (for the run, models etc. still need their own configs, calibration etc.).

Ideally the defaults for this will "just work" to make a CONUS run using NGIAB to parallelize to the size of the resident host.

The docker containers with NGIAB can be pulled from DockerHub. The 3 docker containers used in this repo should be in DockerHub as well so the installs are similar.

1. multiprocessing in validation (run_validator)
2. multiprocessing substitute for tar in forcingprocessor
3. save partition generator file for conus daily run and pull into resource_dir

ngen-datastream should be able to validate proper bmi module configuration within a realization file. i.e. make sure input/out variables are mapped correctly from one module to another.

a global process limit would allow users to run on common resources (non-hpc)

For Mac with:
⇒ uname -m
arm64

Getting "Unsupported architecture: arm64" while running install.sh. Please update the script to support this.

The python environment for the forcingprocessor can be generalized, cut down, and updated to the most recent python version. setup.cfg requirements.txt