wdmapp / effis Goto Github PK

While I'm at it, here's another observation: Adapting a sample effis yaml file to use one's own spack-installed executables is cumbersome.

I don't think there's any real simple solution, in particular given that one may have multiple installs around. But one thing that'd help is if one could do something like

executable_path: $(spack location-i gene)

in the yaml file (maybe yaml itself already allows something like that -- I don't know.)

Similarly, it'd be nice if one could use $MEMBERWORK or something when specifying the run dir.

And, what, there's more ;) Instead of having to change the run dir manually, it'd also be nice if there were an option to say run_dir_base=/path/to, and effis made a new subdirectory every time you run effis-compose. (This is just an idea I just had, I haven't thought about it in detail, so there may be issues with it, but in particular during debugging it often happens that I'll submit one job after the other...)

So another issue we had today was that after spack load effis python-type=full, it didn't work because the correct version of python wasn't in the path.

However, i cannot reproduce this -- for me, the spack load does put the correct python into the path, so I don't know what happened -- I just figured I'd mention it, anyway.

In the composition file here:
https://github.com/SCOREC/testcases/blob/cplEffis/run_1/summit/run_1.yaml
how can I request that each process for the coupler has one GPU associated with it? I found the use-gpus logic

Note, if this is not supported yet we can make Kokkos optional in the coupler to avoid this.

When Sameer tried to build and run wdmapp today on Rhea, one of the many issues we hit was some confusion because some of the effis commands have the .py suffix, but others do not. (In particular, effis-compose.py vs effis-submit.) I can submit a PR to remove the .py from all, if @suchyta1 agrees?

Currently, it is not possible to specify command line options for the job submission command (at least for sbatch in case of SLURM) in the EFFIS input file. I would need this because some clusters I work with use a submission filter that rejects jobs if certain job constraints are not specified in the job submission command.

It would be great to have this capability in EFFIS.

Installing effis (spack install effis following the instructions here https://wdmapp.readthedocs.io/en/latest/build/build.html#build-wdmapp-label) on Rhea and running effis-compose.py appears to require job_setup: /path/to/script for the last line of .cheetah/<user>/campaign-env.sh to be generated correctly. Specifically, I created three yaml composition files, one with job_setup:, one without, and one with an empty job_setup field:

$ diff run_externalCpl_cws.yaml run_externalCpl_cws_noSetup.yaml                                    
10c10
< rundir: /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl
---
> rundir: /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_noSetup
16d15
<   job_setup: run_externalCpl_setup.sh

$ diff run_externalCpl_cws.yaml run_externalCpl_cws_emptySetup.yaml 
10c10
< rundir: /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl
---
> rundir: /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_emptySetup
16c16
<   job_setup: run_externalCpl_setup.sh
---
>   job_setup: 

Then ran effis-compose.py on the three yaml files (note, they output to different run directories):

$ effis-compose.py run_externalCpl_cws_noSetup.yaml 
$ effis-compose.py run_externalCpl_cws.yaml)
$ effis-compose.py run_externalCpl_cws_emptySetup.yaml

And then looking at the two campaign-env.sh files we see an empty 'source' for the cp command is in the 'noSetup' output

cwsmith@rhea-login6g: /gpfs/alpine/phy122/scratch/cwsmith $ grep -n cp testWdmCpl*/.cheetah/cwsmith/campaign-env.sh                                                                     
testWdmCpl/.cheetah/cwsmith/campaign-env.sh:12:cp "/gpfs/alpine/phy122/scratch/cwsmith/testcases/run_1/rhea/run_externalCpl_setup.sh" "/gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl/.cheetah/cwsmith/."
testWdmCpl_emptySetup/.cheetah/cwsmith/campaign-env.sh:12:cp "" "/gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_emptySetup/.cheetah/cwsmith/."
testWdmCpl_noSetup/.cheetah/cwsmith/campaign-env.sh:12:cp "" "/gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_noSetup/.cheetah/cwsmith/."

Running effis-submit on a yaml with the empty 'source' arg for the cp command produces the following error. Note, I appended -x to the #!/bin/bash at the top of effis-submit to see where it was failing:

$ effis-submit /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_noSetup/
+ '[' -z '' ']'
+ case "$-" in
+ __lmod_vx=x
+ '[' -n x ']'
+ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/sw/rhea/lmod/8.2.10/init/bash)
Shell debugging restarted
+ unset __lmod_vx
+ mydir=/gpfs/alpine/phy122/scratch/cwsmith/testcases/run_1/rhea
+ '[' -f /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_noSetup//run/.kittie-script.sh ']'
+ cd /gpfs/alpine/phy122/scratch/cwsmith/testWdmCpl_noSetup//.cheetah/cwsmith
+ ./run-all.sh
cp: cannot stat ââ: No such file or directory
Failed to load compaign-env.sh, aborting
+ cd /gpfs/alpine/phy122/scratch/cwsmith/testcases/run_1/rhea

It appears there is a bug in the argparse module with 2.7.5 that was resolved by 2.7.12. Specifically, on Summit using the default system python (/usr/bin/python) the command:

cd examples/simple/Fortran
../../../util/effis-cpp.py 'repo' -t ~/killme/effis .

exits silently as the args.which variable is set to top instead of repo. Running with python 2.7.12 or 2.7.18 works as expected (module load python/2.7.12); the source files are copied to the specified dir. I stumbled upon this when the spack install failed with the default summit python.

Assuming this is correct, I think we should add a spack dependency on 2.7.12 (not sure if a lower version will work).

It looks like effis puts a tab into the gene source code. It's just a warning at this point, but would be nice to fix.

/root/src/gene-dev/build-spack-ubuntu-coupling/src/gene_subroutine-effis.F90:141:1:

  call kittie_initialize(my_mpi_comm_world, adios_cfg, ngroups=6, numtimers=1)
 1
Warning: Nonconforming tab character at (1) [-Wtabs]

Please note that adios2_init() has removed the debug_mode boolean parameter and hence effis/src/Fortran/kittie.F90:647:80: needs to be updated

[  8%] Building Fortran object src/Fortran/CMakeFiles/kittie_f.dir/kittie.F90.o
/home/adios/Software/effis/effis/src/Fortran/kittie.F90:647:80:

  647 |      call adios2_init(kittie_adios, trim(xml), kittie_comm, adios2_debug_mode_on, iierr)
      |                                                                                1
Error: Symbol ‘adios2_debug_mode_on’ at (1) has no IMPLICIT type