We are running gfs.v16.3.0 real-time parallel for testing a bug found in NSST.
The start date is 2022061918, and the first analysis is 2022062000.
We noticed that the VIIRS radiances are not assimilated as we expected.

                          read_in     keep  assim
o-g 01 rad  npp viirs-m  65887737      0      0    0.0000       0.0000       0.0000       0.0000

o-g 01 rad  j1  viirs-m  71580211      0      0    0.0000       0.0000       0.0000       0.0000

The VIIRS bufr file (gdas.t12z.sstvcw.tm00.bufr_d sstviirs) exists, but the observed data (TMBR) were filled with missing values. So, no data passed the read routine. We have about a week of assimilation results, and all VIIRS data are missing.

We also ran gfs.v16.3.0 in retrospective mode from 2021101518Z for about three weeks. TMBR has valid values and VIIRS radiances were assimilated.

We need help from the ObsProc team to find out the status of the VIIRS data. Do we expect that the TMBR values in the VIIRS bufr are all missing values?

Create two new files in the obsproc repo for the WC2 transition:
run.ver
build.ver
These will reside in a new dir called "versions".

I need to install the new WCOSS2 obsproc package (release/obsproc.v1.0.0) on Hera and Orion to support users running the new GFSv16.2.0 package there (and WCOSS2 in global space eventually too). I tried to build the release/obsproc.v1.0.0 branch on Hera/Orion and encountered issues on both. I ran the build script like this while under the build-obsproc folder:

INSTALL_TARGET=orion ./build.sh

Is that the correct way to build on Orion (and Hera with INSTALL_TARGET=hera)?

I made a couple changes on both Hera and Orion to get the modules loaded (nemsio wasn't loading) but am hitting a compilation errors on both machines still. See the changes I made on Orion (similar changes needed for Hera) and the errors below.

Changed module reset to module purge (module reset only works on WCOSS2, need a purge elsewhere) and added missing load("hpc-impi/2018.4") in obsproc_orion.lua to get nemsio loading:

Orion-login-1[68] /work/noaa/global/kfriedma/git/obsproc/obsproc.v1.0.0$ git diff
diff --git a/build-obsproc/build.sh b/build-obsproc/build.sh
index d496380..476f791 100755
--- a/build-obsproc/build.sh
+++ b/build-obsproc/build.sh
@@ -13,7 +13,8 @@ target=$(echo $INSTALL_TARGET | tr [:upper:] [:lower:])
 if [[ "$target" =~ ^(wcoss2|hera|orion)$ ]]; then
   source $pkg_root/versions/build.ver
   set +x
-  module reset
+  #module reset
+  module purge
   module use $pkg_root/modulefiles
   module load obsproc_$target
   module list
diff --git a/modulefiles/obsproc_orion.lua b/modulefiles/obsproc_orion.lua
index 093e5c7..81741e1 100644
--- a/modulefiles/obsproc_orion.lua
+++ b/modulefiles/obsproc_orion.lua
@@ -7,6 +7,7 @@ load("cmake/3.18.1")
 prepend_path("MODULEPATH", "/apps/contrib/NCEP/libs/hpc-stack/modulefiles/stack")
 load("hpc/1.1.0")
 load("hpc-intel/2018.4")
+load("hpc-impi/2018.4")

I had to also add the missing hpc-impi/2018.0.4 module load in obsproc_hera.lua.

Part of the error I am seeing on Orion is:

/work/noaa/global/kfriedma/git/obsproc/obsproc.v1.0.0/sorc/prepobs_oiqcbufr.fd/oiqcbufr.f(175): error #5102: Cannot open include file 'mpif.h'
      include "mpif.h"
--------------^

Here is a log with the Orion build output:
/work/noaa/global/kfriedma/git/obsproc/obsproc.v1.0.0/build-obsproc/build.log

Here is a log with the Hera build output (different error):
/scratch1/NCEPDEV/global/Kate.Friedman/git/obsproc/obsproc.v1.0.0/build-obsproc/build.log

Before post-go-live obsproc upgrade (for v16.3), i.e. early July 2022,

once all new dumps (Sentinel-6, snow, gmi, etc) generation is tested on wcoss2, revisit the distribution of the load between the dumping groups. (See also emails from Steven Earle, subjects: cdas obsproc and obsproc nam, Apr 20, 2020 )

There are 2 factors here:
1 - look into redistributing the load so that each dump group more or less takes the same amount of time, as best as can be balanced.
2 - reorder the threads so that the longest one runs first and so that it can end more or less as the others finish up.

That rebalancing needs to be explored for all networks, not just global.

Q. In exglobal_dump.sh, one can see: $ushscript_dump/bufr_dump_obs.sh $dumptime 3.0 1 avcspm esmhs 1bmhs
airsev atmsdb gome omi trkob gpsro $crisf4
airsev atmsdb gome omi trkob gpsro $crisf4
Why are some groups preceded by $ and some are not?
A. The $ preceding some dump names is to test first whether there are any tanks available for this data before attempting to dump it. You can see how the variable is used further up in the script, prior to the call to run bufr_dump_obs.sh.

In helping to get airnow processing running correctly on WCOSS2, it was discovered that airnow processing has a different bug when run by a non-NCO user.

Lines 215 and 286 of JOBSPROC_AIRNOW_DUMP need to be modified to not inherit the previous definition for COMIN.
Change from
export COMIN=${COMIN:-${COMIN_ROOT:?}/${obsNET}/${obsproc_ver}/${model}.${pdy}}
to
export COMIN=${COMIN_ROOT:?}/${obsNET}/${obsproc_ver}/${model}.${pdy}

Target this change to be included in the first obsproc upgrade following go-live.
This bug only impacts dev (non-NCO) users and so is not impactful to operations, but may impact development and check out.

To address RRFS model's observations needs, add a RRFS network to obsproc's capability, and generate desired bufr dump and prepbufr obs files.

Due December 2022

• Leverage off RAP network's scripts, ush scripts and triggers to set RRFS up and running (rrfs, rrfs_e, rrfs_p)
• Update RRFS to obsproc.v1.1.0 to generate gmi1cr, snocvr, saldrn and subpfl, add Sent-6 to gpsro, add Metop-B/C AMV to satwnd, add saldrn and subpfl to nsstbufr
• Split the large radiances dump group to speed up processing (for global, and may be later for rrfs)
• Replace -np xx with -np $NPROCS (see exrap_dump.sh as a good example) (IG)
• Replace mpiexec with launcher_DUMP (see exurma_dump.sh as a good example) (IG)
• Resolve BUFRLIB: CLOSBF warnings in dump jobs(http://www2.spa.ncep.noaa.gov/bugzilla/show_bug.cgi?id=1350) (IG - under testing)
• Unify time dump window - set _earliest to -1h, and _latest to +1h (confirm with @JacobCarley-NOAA ) ( /Utils/GO_CHKDAT (?)/go_chkdat dump_file_name)
• Generate CONUS 15min AMVs dumps (in separate "regional" satwnd dumps) (bufr-dump/fix/bufr_dumplist , satwhr.bufr_d, exrrfs_dump*.sh : 00505**) (SN - under testing & soon to add to Ashley's collecting branch
• Verify and maintain generation of Direct Broadcast radiances (es*** and ***db bufr dumps)
• Confirm all GPSRO observation streams - DO5, Eumetsat, any others?
• Confirm domain - see mask files

Add JPSSS-2 NOAA-21 products of interest. Due at NOAA-21 provisional readiness, early 2023

• L1B ATMS rads (IG - tanks are generated in NRT, need to start dumping)
• L1B CrISS rads (IG - tanks are generated in NRT, need to start dumping
• VIIRS (or AVHRR??) AMVs - (IG - waiting on NESDIS )
• SST - (IG - waiting on NESDIS )
• Land surface temperature - Need info from RRFS team , Waiting on NESDIS

Due March 2023 (May-June 2023 is more realistic)

• Add UPRAIR full resolution bufr dumps (need Chris Hill's T2B code to be completed)
• Add T2B prepbufr (need Chris Hill's T2B code to be completed)
  ~~~The next few products are *nc and will be ingested by DataFlow, tickets submitted or soon to be submitted (Matt P.)
• [ x] Add VIIRSS Fire radiative power (RAVE product sample due March 2023)
• [ x] Add GOES Fire radiative power (also in RAVE??)
• [ x] Add RAVE (uses VIIRSS add GOES Fire radiative power as inputs, see emails Shobha, chat with Ravan)
• [ x] Add AOD and ADP aerosol products - on PDA, format? Satingest or DF? see email Amanda
• [ x] Add surface pm2.5 air constituents , smoke and dust NESDIS products - EPA AirNow - email Amanda and Hongli
• [ x] Add GOES Lightning Mapper data (source data, format, routing, etc) - Amanda Black, NESDIS products, submit ticket to DF (data on LDM) see email with Amanda
  ~~~
• Add wind tower data - review tanks and update bufr_dump to include dumping it (bufr_dumplist, wndtow 002/020, 002/021, 000/021, exrrfs_dump.sh, add to smallest Dump group (least run time) )

Hopefully this is the right place to ask this.
In October Eumetsat have removed the Seviri clear sky radiance product and hence the SERVIRI bufr files on the nomads server have been blank (zero size) ever since.
Im guessing this means no geostationary satellite data is going into the GFS from Europe/Africa area?
Is there any plan to put the all sky radiances into the GFS and put these into the Seviri bufr file on the nomads server?

The obsproc package does not work outside of the operational environment (e.g. developers running on R&D HPCs like Hera and Orion). This issue will document needed updates to support developers running obsproc outside of operations.

Two branches were created to generate T2B uprair dump files on wcoss2:
bufr-dump: feature/dev.iss49636.TAC2BUFR_upper-air
obsproc: feature/dev.iss49636.TAC2BUFR_upper-air
but they generate 0 size uprair bufr dump files.

Suggestions:
1)review the cmake configuration flags and library use in the relevant cmake files (i.e. https://github.com/NOAA-EMC/bufr-dump/blob/feature/dev.iss49636.TAC2BUFR_upper-air/sorc/bufr_dupupr.fd/CMakeLists.txt)
2)compile code in debug mode (both bufr-dump and obsproc)
3)add print statements to isolate possible issues with BUFRLIB calls
4)abandon the branches above and start from scratch (from tested wcoss2 obsproc code)
5)redriect log files to /ptmp for more time to study them (change trigger and cycqsyb paths)
6)run code with 002/101 tanks composed of smaller in size messages (and hopefully subsets)

Add support for obsproc to build on RDHPCS Jet platform. Needed for supporting GFS/global-workflow experiments on that platform. Changes to support Jet will not impact operational functionality.

Companion prepobs issue: NOAA-EMC/prepobs#15
Companion bufr-dump issue: NOAA-EMC/bufr-dump#16

Changes are minimal and will include:

1. Add "jet" to target list in ush/build.sh
2. Create module file (modulefiles/obsproc_jet.lua) for Jet.

Changes have already been made and tested on Jet by @DavidHuber-NOAA within a clone of the obsproc.v1.1.2 tag.

• stop generation of 0 size bufr dumps
• remove 0 size files from pushing to nomad for next obsproc release
• fix msonet segfault, ABORT and warning messages
• revisit CLOSBF before openbf -
• "[Bug 1349] Change $NET to obsproc for all tasks" - bugzilla
• global dump group split
• see email "Re: optimization test results"
• dump CIMSS and CONUS winds
• T2B ???
• dependecy w3emc Jack Woolen
• Russ - error table as submodule
• marine data in situ???
• rrfs and 3d RTMA needed NRT or operationally?
• two bigzillas
• re=read Carlos's emails

The version on libraries between build.ver and run.ver are inconsistent.

Note, none of the obsproc code now depends on w3nco. They only depend on w3emc.

Here is a snapshot of build.ver and run.ver from develop at bd4211a.

$> cat build.ver
export hdf5_ver=1.10.6
export netcdf_ver=4.7.4
export bacio_ver=2.4.1
export w3emc_ver=2.9.1
export sp_ver=2.3.3
export sigio_ver=2.3.2
export nemsio_ver=2.5.2
export bufr_ver=11.5.0

$> cat run.ver
#########################################
#Export module version numbers
#########################################

export prod_util_ver=2.0.8
export prod_envir_ver=2.0.5

export grib_util_ver=1.2.2
export util_shared_ver=1.4.0
export hdf5_ver=1.10.6
export netcdf_ver=4.7.4
export w3nco_ver=2.4.1
export w3emc_ver=2.7.3
export bacio_ver=2.4.1
export bufr_dump_ver=1.0.0
export bufr_ver=11.5.0

export envvar_ver=1.0
export PrgEnv_intel_ver=8.1.0
export intel_ver=19.1.3.304
export craype_ver=2.7.8
export cray_mpich_ver=8.1.7
export cray_pals_ver=1.0.12
export cfp_ver=2.0.4

export gfs_ver=v16.2
export dictionaries_ver=v3.4.0
export omb_ver=v4.4

export PROCESS_MASTER_SHIP_STNLIST=NO
export BACK=off

Also, note that all NCEPLibs are built as static libraries, and linked statically in the obsproc codes, so it is very likely these libraries don't need to be loaded at runtime.

At a bare minimum, the library versions in build.ver and run.ver need to be consistent.

See the satingest repository, since it does this better.

• The RTMA_RU 0000Z cycle was failing intermittently when trying to read from a malformed ADPSFC BUFR tank, essentially owing to some read-write interference. The obsproc_rtma code was fixed with a temporary trap to prevent this clash from occurring.

• The purpose of this issue is to explore a robust fix and integrate it into the existing RTMA codes.

Notes and old work history:

• https://vlab.noaa.gov/redmine/issues/70153
• https://vlab.noaa.gov/redmine/issues/73355
• https://vlab.noaa.gov/redmine/issues/75717

Proposed plan:

• Disable the bug-fix and set an end-to-end cron to allow 8 minutes of processing
• Check output WCOSS2 data

See comments in PR #39 for instructions and clean up /obsproc/triggers in obsproc's first update post-WCOSS2 go-live

nam_prep processing has not been properly finding tcvitals files to run the syndat_syndata exec in operations for quite some time ... likely back to GFSv15 days, when the com dir structure for GFS did not include $COMPONENT and $cyc. NAM is at end of life and it is not worth trying to restore the SYNDATA processing, especially given that it hasn't been functioning for the better part of 2 years and no one ever knew.

NCO made some changes for WCOSS2 go-live to nam_prep processing to prevent script failure relating to unknown variable definitions. Those changes can be retained. But a better method to prevent failure is to prevent the set up to run syndat_syndata altogether.

jobs/JOBSPROC_NAM_PREP should be modified so that SYNDATA defaults to NO.
Line 292 should be changed to this:
export SYNDATA=${SYNDATA:-NO} # NEVER execute program SYNDAT_SYNDATA

Note1: while $COMINtcvital and $COMSPtcvital are defined in jobs/JOBSPROC_NAM_PREP and scripts/exnam_makeprepbufr.sh, they will never lead the processing to finding the tcvitals files. Those files only reside in the gfs and gdas com, not nam com. So even if SYNDATA accidentally is set to YES, the processing will never find the required tcvitals files w/ the current path variable definitions, which leads to syndat_syndata not being executed. I mention this only to say that one could correct $COMINtcvital to point to the correct path, but that would require pointing to gfs com and a can of worms not worth opening.

Note2: Another approach can be to exclude the $COMINtcvital and $COMSPtcvital definitions altogether. They are not needed when SYNDATA=NO. This will then allow versions/run.ver to have the nam_ver dependency removed.
imo, this is the cleanest approach ... remove $COMINtcvital from jobs/JOBSPROC_NAM_PREP, remove $COMSPtcvital from scripts/exnam_makeprepbufr.sh, and remove nam_ver from versions/run.ver.

To test run nam_prep trigger pointing to modified obsproc codes noted above. COMIN can point to production dump files. Look for the following things in the *.o* log file:

• normal completion
• SYNDATA=NO

The resultant prepbufr file should have no SYNDAT item when binv.x is run on it.

Direct questions to @ShelleyMelchior-NOAA

This cron reprocesses global aircft and aircar bufr_d and prepbufr files that are 48 hours old. Since the restriction expiration for these data are 48 hours, the reprocessed data files include the full/original data set in the operational bufr_d and prepbufr files as produced in NRT, but now un-restricted.

The reprocessed bufr_d and prepbufr files are archived in GDA.

Impacted code in obsproc:
sorc/bufr_remorest.fd/remorest.f
scripts/exdump_post.sh
jobs/JOBSPROC_GLOBAL_DUMP_POST
jobs/JOBSPROC_GLOBAL_PREP_POST

Impacted code in prepobs:
scripts/exprep_post.sh