hafs-community / hafs Goto Github PK

Issue:
On WCOSS, the 2020 tcvitals file under operational location:
/gpfs/dell1/nco/ops/com/gfs/prod/syndat
is incomplete. This will cause issues to run those storms on wcoss_cray and wcoss_dell_p3.

Solution:
Fixes in ush/tcutil_generate_vitals.py and ush/tcutil_multistorm_sort.py have been made and tested on wcoss_cray, so that they will use the SYNDAThafs environment variable when exists/defined by the HAFS workflow (for example, in parm/system.conf).

Description

Incorporate wave-coupling in the HAFS application/workflow by establishing the HAFS FV3-HYCOM-WW3 coupled system.

Related to

• associated with /pull/97

The inline post capability is already available in ufs-weather-model's (hafs_forecast.fd) write grid component. We need add the inline post option in the HAFS workflow as well.

Description

Enable online satellite bias correction capability.

Proposed solution

Add the online satellite bias correction capability in HAFS application/workflow.

Status

UMD collaborators (@jak5808 and others) have added this capability in HAFS workflow.

Related to

• /pull/105

Based on @TimMarchok-NOAA's recent research on the GFDL tracker, he made a suggestion to run TC tracker with the two thickness parameters turned off. Specifically, in the tracker namelist, change "y" to "n" for the following parameters:
user_wants_to_track_thick500850 = "n",
user_wants_to_track_thick200850 = "n",
The differences (w/ and w/o the thickness parameters) are small except for the TC locations at the initial time.

This issue is created based on an email from @ZhanZhang-NOAA.

Add a script under rocoto to check RT output

Description

Currently, in HAFS, fhcyc is set as 24 in input.nml. Propose to change it to 0, since HAFS focuses on hurricane weather forecasting.

Proposed solution

Change fhcyc in input.nml from 24 to 0.

Related to

• associated with hafs-community/HAFS/pull/<pr_number>

Sync HAFS submodules with their latest authoritative branches as of 202107.

Sync HAFS submodules (hafs_forecast.fd, hafs_gsi.fd, hafs_utils.fd) with their corresponding authoritative branches.

During the testing of the inline post capability in HAFS, we updated the HAFS post control files, which will need to be included in the HAFS workflow.

Explore using the hybens_info file to specify the vertical profile of horizontal and vertical localization length scales for HAFS EnVar and EnKF analysis.

Currently, the regional ESG grid is generated out of the HAFS workflow. So, we need to add the capability of generating regional ESG grid in the atm_prep job/task.

Import HAFS vortextracker component including source code and build system.

Description

Provide a clear and concise description of the requested feature/capability.
Add HAFS NHC ASCII products like:

• storm_id.yyyymmddhh.hafs.afos
• storm_id.yyyymmddhh.hafs.stats.short
• storm_id.yyyymmddhh}.hafs.stats.tpc

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.
Follow HMON's method to generate these files based on the atcfunix track file.

Status (optional)

Do you (or a colleague) plan to work on adding this feature?
@LinZhu-NOAA will explore to add this capability in HAFS application.

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by /pull/156

Description

For the HAFS application/workflow develop branch, switch to point to the develop branch of the authoritative ufs-weather-model directly for the sorc/hafs_forecast.fd submodule.

Proposed solution

Currently, the support/HAFS branch is identical to an early version (11/03/2021) of ufs-weather-model develop branch. When we plan our next round of update of the hafs_forecast.fd submodule inside HAFS, we can just directly point to the ufs-weather-model develop branch.

Note: The HAFS related developments will still be happening in the hafs-community fork of ufs-weather-model (e.g., the feature/hafs_nesting branch for the moving nesting development).

Related to (optional)

• associated with hafs-community/hafs/pull/<pr_number>

Check and compare HAFS grib2 output fields with those from HWRF as well as those from GFS and FV3CAM. Since HAFS is FV3 based, we may or may not be able to output every field as from HWRF. Also, we may try to follow GFS for those fields common to GFS and HAFS.

"atm_post" fails occasionally on Orion and Hera with the message:

**** FATAL ERROR: rd_grib2_msg_seq_file, missing end section ('7777') ****"

Normally this error is not reproducible, rewinding the relevant tasks will succeed the "atm_post".
It appears that the error occurs when the grib2 records were written, probably due to a memory issue and grib2 file is incomplete. Once the bad grib2 file is created, the error message can be duplicated by wgrib2 (grib2file).

In a grib file, there is a header that provides information about the length of the record, then the record, and a "7777" signals the end of the record. The error message "missing end section ('7777)" probably indicates that the grib2 file is incomplete.

On Hera, if you run the tasks with the "bigmem" partition (large memory available), such an error is not encountered so far.

Description

Provide a clear and concise description of the requested feature/capability.
As WCOSS2 replaced WCOSS Cray and WCOSS Dell_p3, need to remove the HAFS application support for WCOSS Cray and WCOSS Dell_p3.

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.
Update/remove the related module files and scripts to disable the support for WCOSS Cray and Dell_p3 (including for the related submodules).

Status (optional)

Do you (or a colleague) plan to work on adding this feature?

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

Enable using the restart_interval config option to write out more FV3ATM restart files
forecast.restart_interval='"0.025 3. 6. 9."'

Description

Provide a clear and concise description of the requested feature/capability.

• Add upoff as a namelist option for ufs-weather-model/fv3atm/fv3dycore, which can potentially alleviate the model numerical instability for the HAFS moving-nesting configuration (Suggested by GFDL collaborators and tested by HRD colleagues).
• Enable using upoff as an application/workflow-level config option.

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.
The upoff namelist option in FV3dycore has been added during a working session. Workflow level config option will need to be added.

Status (optional)

Do you (or a colleague) plan to work on adding this feature?

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

Description

Provide a clear and concise description of the requested feature/capability.
Upgrade to the new UGWP scheme (which can also reproduce the existing old ugwp scheme when setting namelist option properly).

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.
Workflow script changes are needed to generate GSL orography files.
Developments are needed to support the HAFS moving-nesting configuration properly.

Status (optional)

Do you (or a colleague) plan to work on adding this feature?

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

Description

Provide a clear and concise description of the requested feature/capability.
Convert modulefiles to .lua format on WCOSS2, hera, jet, orion, including for both HAFS application level and related submodule components (e.g., hafs_graphics.fd, hafs_tracker.fd).

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.
See examples from ufs-community/ufs-weather-model/pull/1399

Status (optional)

Do you (or a colleague) plan to work on adding this feature?

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

Due to two bugs #63 in the g2 library(prior to the latest 3.4.3 version), hafs subcomponents (hycom_utils and vortextracker) fail with g2 from hpc-stack on certain platforms. Currently, the workaround is, use g2 from HWRF build in the vortextracker component, and point to the specific g2/3.4.1 version for hycom_utils build. The two bugs in g2 were fixed in the 3.4.3 release. Accordingly, the HAFS script and modulefile files were updated to point g2/3.4.2 in hpc-stack in the PR (#77)

HAFS FV3ATM-HYCOM coupling forecast fails when there are sea ice grid points in the regional HAFS model domain. In the current ufs-weather-model forecast, when cplflx is set .true., it assumes there will be an active sea-ice model component, which is the case for the S2S application (FV3ATM-MOM6-CICE6 coupling). But, HAFS currently only has HYCOM coupling without a sea-ice model component.

Currently, a temporary fix (workaround) has been committed in the support/HAFS branch of https://github.com/hafs-community/ccpp-physics to deal with sea ice points properly in HAFS FV3-HYCOM coupling without the CICE6 component, following the discussions/suggestions among @junwang-noaa , @SMoorthi-emc, @ChunxiZhang-NOAA @danrosen25, @BinLiu-NOAA, etal. Detailed source code changes can be seen in this commit.

However, a better solution would be to separate the cplflx namelist option into cplocn and cplice (like the existing cplwav) options, and use these namelist options to control the related ocean and sea-ice coupling situations. This probably will need a unified/coordinated solution among all the related coupled UFS applications (e.g., S2S, HAFS).

Updating satinfo file to turn off metop-A IASI channels

Description

Add the fms2_io_nml namelist section with netcdf_default_format of netcdf4 in input*.nml
&fms2_io_nml
netcdf_default_format="netcdf4"
/
Otherwise, the default was the NetCDF "64-bit offset" version, which has issues to support large files.

Note: This information is from the RRFS application team (Eric Rogers) with the suggestion from @bensonr.

Proposed solution

Add the above-mentioned namelist section in the input*.nml files for the forecast job (ufs-weather-model).

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• addressed by hafs-community/hafs/pull/<pr_number>

Description

Enable running HAFS on WCOSS2

Proposed solution

For submodules of hafs_utils.fd (UFS_UTILS), hafs_gsi.fd, hafs_forecast.fd, and hafs_post.fd, wait for the porting from their own authoritative branches. For other HAFS submodules (hafs_tools.fd, hafs_hycom_utils.fd, hafs_graphics.fd, hafs_vortextracker.fd) as well as the application/workflow scripting system, will need to port them from the HAFS side.

Related to

Fixed by PR #136

Description

Provide a clear and concise description of what the bug is.
Also give a description of what behavior you expected to happen.
The atm_prep task will fail due to the typo if condition (introduced by this commit 7e56220):
if [[ -e ./hafs_sfc_climo_gen.x ]]; then
cp -p $SFCCLIMOEXEC ./hafs_sfc_climo_gen.x
fi
which should be
if [[ ! -e ./hafs_sfc_climo_gen.x ]]; then
cp -p $SFCCLIMOEXEC ./hafs_sfc_climo_gen.x
fi

To Reproduce:

The atm_prep task will fail for any HAFS application regression tests.

Additional context (optional)

Add any other context about the problem here.
Directly reference any issues or PRs in this or other repositories that this is related to, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull<pr_number>

Running Thompson microphysics in an automative way using the HAFS workflow.

Description

Update README.md as well as the issue and pull request templates.

Proposed solution

Discussed with code managers on making some further refinements for the issue and pull request templates.

Related to

• /pull/109

Description

Jobs keep failing on Orion when using GDAS ensemble in 3DEnVar. The error messages occur in random places due to out of memory related info.

Steps to reproduce

If someone unfamiliar with the bug follows these steps:

1. submit an analysis job with 3DEnVar using GDAS ensemble.

Potentially relevant details could include:
-It is likely due to the recent changes in the NOAA RDHPCS system.

Output

• Two examples of failure logs:

slurmstepd: error: Detected 1 oom-kill event(s) in step 3495064.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: Orion-01-01: task 4: Out Of Memory
srun: Terminating job step 3495064.0
slurmstepd: error: *** STEP 3495064.0 ON orion-01-01 CANCELLED AT 2021-10-27T19:35:36 ***

srun: error: Orion-18-69: task 289: Bus error
srun: Terminating job step 3495450.0
slurmstepd: error: *** STEP 3495450.0 ON orion-01-04 CANCELLED AT 2021-10-27T20:22:22 ***

Status

Bin Liu helped resolve the issue by changing the rocoto/sites/orion.ent file
from

into:

Description

Provide a clear and concise description of the requested feature/capability.
Sync HAFS submodules (hafs_forecast.fd, hafs_post.fd, hafs_utils.fd) with their latest authoritative branches as of 202201.

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.

• Sync HAFS submodules hafs_forecast.fd and hafs_utils.fd with their latest authoritative branches as of 202201.
• Symbolically link sorc/hafs_forecast.fd/FV3/upp into sorc/hafs_post.fd so that hafs_post.fd points to the same commit or UPP develop branch.

Related to

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

When choosing the disk archive option (e.g., on Orion), write out a done file after the tar file is successfully created. This is useful for easily detecting and transferring the tar files across platforms (saying from Orion to Niagara, and then to HPSS tape).

Add support to run global uniform or stretched configuration without a nest. Also the tile 6 can be configured to focus a TC basin, a specific storm, or any user specified lat-lon location.

Description

Add HAFS application/workflow level data assimilation related regression tests in the standard regression tests.

Proposed solution

How should the new feature be added? If you have thoughts on the implementation strategy, please share them here.
Add one or two regression tests with the 3DEnVar with GDAS ensembles and/or HAFS ensembles.

Status

Are you (or a colleague) working on adding this feature? If so, please assign yourself (or them) to this issue.

Related to

Directly reference any issues or PRs in this or other repositories that this is related to, and describe how they are related. Example:

• associated with /pull/130

The HAFSv0.2 configuration has been finalized, and the HFIP HAFSV0.2 series of real-time experiments are running on Jet and other platforms. So, it's time to merge hafsv0.2 configuration back to the HAFS develop branch.

I'm trying to
git clone --recursive https://github.com/hafs-community/HAFS

But I ran into following error, I wonder if someone could give me access to this repo, if this is the problem, thanks:

Cloning into '/work/noaa/marine/lliu/runs/deb/HAFS/sorc/hafs_gsi.fd/fix'...
ssh: Could not resolve hostname gerrit: Name or service not known
fatal: Could not read from remote repository.

Description

Sync HAFS submodules with their corresponding authoritative branches as of 09/09/2021.

Related to

• /pull/95

Description

Provide a clear and concise description of the requested feature/capability.
Sync HAFS submodules (hafs_forecast.fd, hafs_post.fd, hafs_utils.fd, hafs_gsi.fd, etc.) with their latest authoritative branches as of 202210.

Proposed solution

How should the new feature/capability be added? If you have thoughts on the implementation strategy, please share them here.
Sync HAFS submodules and conduct HAFS application level regression tests.

Status (optional)

Do you (or a colleague) plan to work on adding this feature?

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>

Description

Sync to use the same field_table as in ufs-weather-model HAFS regression tests:
sorc/hafs_forecast.fd/tests/parm/field_table/field_table_hafs

Proposed solution

Copy or link sorc/hafs_forecast.fd/tests/parm/field_table/field_table_hafs into the corresponding parm/forecast/[regional|globnest]/field_table files.

Status (optional)

Do you (or a colleague) plan to work on adding this feature?

Related to (optional)

• addressed by hafs-community/hafs/pull/<pr_number>

Description

The global offset predictor coefficient does not update when online_satbias config is set to yes. This is because in file parm/analysis/gsi/gsiparm.anl.tmp: upd_pred(1) equals 0 when it should equal _UPD_PRED_.

To Reproduce:

This bug can be seen by examining satbias_out files from model runs in which online_satbias=yes, and seeing that the first predictor coefficient remains constant.

Additional context

See the following commit for additional context on the purpose of the online_satbias option and the significance of the upd_pred variables:
b6172ee

Switch to use /bin/sh instead of /bin/ksh for Shell based scripts under jobs, scripts, and ush.

Please branch off the feautre/hafs_ensda branch for this work.

Sync HAFS submodules/subcomponents as of 05/12/2021

Description

Turn on the pressure tendency diagnostic, which is useful to monitor the model instability.

Proposed solution

Add print_diff_pgr = .true. to the &gfs_physics_nml section of the UFSATM namelist input.nml.

Related to

• related to hafs-community/hafs/pull/<pr_number>

G2 (GRIB2 encoder/decoder) from hpc-stack is not reading grib2 files. Specifically, the routine getgb2 from G2, signaling that there is an error (code 96) in reading GRIB2 files. This is happening on all platforms(Jet/Hera/Orion/Wcoss-Cray).

I have an example(on Jet) that G2(not from hpc-stack, g2/v3.1.0) successfully decoding a GRIB2 file:
/lfs1/HFIP/hwrfv3/Biju.Thomas/Test_G2_getgb2/G2_getgb2
However, when I use G2(from hpc-stack, g2/3.4.1 ) to read the same file, it throws an error(error code 96).
/lfs1/HFIP/hwrfv3/Biju.Thomas/Test_G2_getgb2/G2_getgb2_hpc-stack

Description

Sync HAFS submodules with their corresponding authoritative branches as of 11/02/2021.

Related to

• /pull/107

Description

As we now have the HAFSv0.3 configurations finalized for 2022 hurricane season real-time experiments, we would like to bring the HAFSv0.3 series developments back to HAFS develop branch.

Proposed solution

Merge the feature/hafs.v0.3.0 branch back to the develop branch.

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• fixed by hafs-community/hafs/pull/<pr_number>
• dependent upon ufs-community/ufs-weather-model/pull/<pr_number>

Description

The HAFSv0.3 phase2 configuration in the feature/hafsv0.3_phase2 branch contains the latest HAFS moving nesting, coupling, vortex initialization, data assimilation capabilities, and it points to the latest develop branch (as of 04/12/2022) of ufs-weather-model. Besides, the hafs_utils.fd and hafs_gsi.fd were also synced with their corresponding develop/master branches recently. With that, it would be beneficial to merge this feature branch back to HAFS develop branch.

Proposed solution

Merge the feature/hafsv0.3_phase2 branch back to HAFS develop branch.

Related to (optional)

Directly reference any related issues or PRs in this or other repositories, and describe how they are related. Examples:

• addressed by hafs-community/hafs/pull/<pr_number>

Currently, HAFS does not have satellite post products yet. Add satellite post capability and products in HAFS application workflow.

Description

Currently, all HAFS related changes/developments for UPP have been merged back to UPP develop after this PR (NOAA-EMC/UPP/pull/418). So HAFS can switch to point UPP develop branch directly.

Proposed solution

When we plan our next round of update of the hafs_post.fd submodule inside HAFS, we can just directly point to the UPP develop branch. Beside, UPP is also becoming a submodule of ufs-weather-model/FV3/upp (see NOAA-EMC/fv3atm/pull/421), so we can also consider to just symbolically link upp from sorc/hafs_forecast.fd/FV3/upp into sorc/hafs_post.fd.

Related to (optional)

• addressed by hafs-community/hafs/pull/<pr_number>