hpc-stack on Hercules about hpc-stack HOT 19 OPEN

from hpc-stack.

Please try spack-stack installations. The hpc-stack do not have the compiler sets for hercules, unless it has similar compiler env settings to existing machines or you have to set it.

from hpc-stack.

Thanks. Could somebody point out the documentation on how to use spack-stack on Hercules?

from hpc-stack.

Hang on, the PR that updates the spack-stack documentation is still in the works ...

from hpc-stack.

How to use spack-stack on hercules?

ALWAYS DO THIS

module purge
module use /work/noaa/epic-ps/role-epic-ps/spack-stack/modulefiles
module load ecflow/5.8.4-hercules
module load mysql/8.0.31-hercules

FOR INTEL, DO THIS

module use /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1-hercules/envs/unified-env/install/modulefiles/Core
module load stack-intel/2021.7.1
module load stack-intel-oneapi-mpi/2021.7.1
module load stack-python/3.9.14
module available

FOR GNU, DO THIS

module use /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1-hercules/envs/unified-env/install/modulefiles/Core
module load stack-gcc/11.3.1
module load stack-openmpi/4.1.4
module load stack-python/3.9.14
module available

from hpc-stack.

@climbfuji Great! Thanks

from hpc-stack.

I shall warn you, we haven't done a lot of testing with this new software stack yet.

from hpc-stack.

module load stack-intel/2021.7.1
Lmod has detected the following error: The following module(s) are unknown: "stack-intel/2021.7.1"

from hpc-stack.

It's there, and I just ran those commands successfully ... but I forgot to add the most important line above (I'll edit the comment):

ls -la /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1-hercules/envs/unified-env/install/modulefiles/Core/stack-intel/2021.7.1.lua

module use /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1-hercules/envs/unified-env/install/modulefiles/Core
module load stack-...

from hpc-stack.

we have a draft pr in to add hercules support on the wm level. feel free to take a look at how i have the modulefile set for hercules here: ufs-community/ufs-weather-model#1733. the fv3_conf files for hercules may also be of some use. however, this is not thoroughly tested yet, and the WM does not yet successfully run in coupled mode on hercules (at least in my testing). i am not sure about the hafs configuration there.

let me know if you have any questions! happy to help if i can.

from hpc-stack.

@ulmononian Thank you. It is very helpful!

from hpc-stack.

A new hpc-stack built on Hercules with the following compilers:
intel-oneapi-compilers/2022.2.1
intel-oneapi-mpi/2021.7.1

The stack could be loaded as following:

module use /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_hrcs/modulefiles/stack
module load hpc
module load hpc-intel-oneapi-compilers
module load hpc-intel-oneapi-mpi

Please see below a the modules in the stack when inquired using "module list"
(UPDATED 05/16/2023, to include esmf/8.4.2, mapl/2.35.2-esmf/8.4.2, pio/2.5.10):

---- /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_hrcs/modulefiles/mpi/intel-oneapi-compilers/2022.2.1/intel-oneapi-mpi/2021.7.1 ----
   atlas/ecmwf-0.24.1        fms/2022.04                      nemsio/2.5.4    (D)
   crtm/2.4.0                hdf5/1.10.6               (D)    nemsiogfs/2.5.3
   eckit/ecmwf-1.16.0        madis/4.3                        netcdf/4.7.4    (D)
   esmf/8.3.0b09             mapl/2.22.0-esmf-8.3.0b09        pio/2.5.7
   esmf/8.4.2-debug          mapl/2.35.2-esmf-8.4.2    (D)    pio/2.5.10      (D)
   esmf/8.4.2         (D)    ncdiag/1.0.0                     upp/10.0.10
   fckit/ecmwf-0.9.2         ncio/1.1.2                       wrf_io/1.2.0

---- /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_hrcs/modulefiles/compiler/intel-oneapi-compilers/2022.2.1 ----
   bacio/2.4.1                          ip/3.3.3          (D)    prod_util/1.2.2
   bufr/11.7.0                          ip/4.0.0                 sfcio/1.4.1
   g2/3.4.5                             jasper/2.0.25     (D)    sigio/2.3.2
   g2c/1.6.4                            jpeg/9.1.0               sp/2.3.3
   g2tmpl/1.10.0                        landsfcutil/2.4.1        szip/2.1.1
   gfsio/1.4.1                          libpng/1.6.37            udunits/2.2.28  (D)
   gftl-shared/v1.5.0                   metplus/4.1.3            w3emc/2.9.2
   grib_util/1.2.4                      nccmp/1.8.9.0     (D)    w3nco/2.4.1     (D)
   gsl/2.7.1                     (D)    nco/5.0.6         (D)    yafyaml/v0.5.1
   hdf5/1.10.6                          nemsio/2.5.4             zlib/1.2.11     (D)
   hpc-intel-oneapi-mpi/2021.7.1 (L)    netcdf/4.7.4

Feel free to test it if possible. ESMF 8.4.2 likely needs to be used, not 8.3.0b09.

from hpc-stack.

The tar command failed on Hercules compute nodes with the following environment:

module use /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1-hercules/envs/unified-env/install/modulefiles/Core
module load stack-intel/2021.7.1                                                                     
module load stack-intel-oneapi-mpi/2021.7.1     

Here is the error message:
tar: Relink /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2022.2.1-z2sjni66fcyqcsamnoccgb7c77mn37qj/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libimf.so' with /usr/lib64/libm.so.6' for IFUNC symbol `sinf'
/var/spool/slurmd/job01525/slurm_script: line 23: 6267 Segmentation fault (core dumped)

from hpc-stack.

The tar command failed on Hercules compute nodes with the following environment:

module use /work/noaa/epic-ps/role-epic-ps/spack-stack/spack-stack-1.3.1-hercules/envs/unified-env/install/modulefiles/Core
module load stack-intel/2021.7.1                                                                     
module load stack-intel-oneapi-mpi/2021.7.1     

Here is the error message: tar: Relink /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2022.2.1-z2sjni66fcyqcsamnoccgb7c77mn37qj/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libimf.so' with /usr/lib64/libm.so.6' for IFUNC symbol `sinf' /var/spool/slurmd/job01525/slurm_script: line 23: 6267 Segmentation fault (core dumped)

Sorry about that! Can you check if spack installed tar itself? What does which tar say?

from hpc-stack.

which tar
/usr/bin/tar

from hpc-stack.

Re: hpc-stack installation
According to the ESMF support, ESMF v8.4.2 is needed when using newer intel compilers (such as intel-oneapi-compilers) .
The hpc-stack on Hercules has been updated to include esmf/8.4.2, mapl/2.35.2-esmf-8.4.2 module, as well as pio/2.5.10.
(Some tests on another machine fail to compile with pio/2.5.7 built with newer compilers.)

Updating the earlier comment with the modules shown from "module list" : #521 (comment)

from hpc-stack.

which tar /usr/bin/tar

We should probably create an issue in spack-stack for that so that we remember it.

from hpc-stack.

Ok done.
JCSDA/spack-stack#584

from hpc-stack.

Thanks very much!

from hpc-stack.