wacl-york / geos-chem-schedule Goto Github PK
View Code? Open in Web Editor NEWSplit up GEOS-Chem runs into smaller chunks to schedule jobs on HPC. Also enables managing Meteorology & emissions input files.
License: The Unlicense
Split up GEOS-Chem runs into smaller chunks to schedule jobs on HPC. Also enables managing Meteorology & emissions input files.
License: The Unlicense
Testing suite not running with the standard command (py.test
) in Python3.
Document strings currently not present for the functions.
A recent update has stopped the functionality to email users once jobs are finished.
Currently, the functionality is to split runs by various numbers of days, weeks, and months. Originally this code was only used to split up runs by months. The name of the repository no longer reflects the main functionality/use so a new name may be better.
An example could be "GC_job_split".
Add option for the dependency of subsequent jobs on initial submission.
This could simply be done for SLURM using the --dependency=afterok:
flag.
This was the default action in the first interaction of monthly_run
Should be obsolete in V11 but this should be forced to something nice.
The main functionality is currently through a class structure. However, a dictionary class could be a simpler and adequate approach.
It used to be possible to call the scheduler script from the command line (e.g. with the command below).
python geos-chem-schedule.py --job-name=hello_world --step=day --queue-name=test --submit=yes
However, this now results in the error message below:
Invalid argument --submit=yes
Try --help for more info.
The README appears to be written for running this on earth0, so doesn't make for easy reading when thinking about how to use this on Viking, e.g.
The script has a UI to chose job name, queue name, priority, if you want to start the jobs outside of work hours, and if you want to have the script submit the job to the queue.
Refactor the code from the current setup where all functionality is in a single file, to functionality by file names with appropriate names (e.g. testing, core, scripts etc).
York's Viking HPC has a batch functionality option (linked below). Update the module to allow the use of this.
Currently, split jobs can be submitted with a dependency to proceed if the preceding job is completed successfully. However, when jobs exit with a model crash the rest of the queued jobs submitted together are proceeding.
This is [create_SLURM_run_script2submit_together](https://github.com/wacl-york/geos-chem-schedule/blob/main/core.py#L1033-L1065)
which uses the SLURM option --dependency=afterok
.
TODO: work out how to capture all the job/model fail codes via SLURM and then abort the following model runs in the queue.
def create_SLURM_run_script2submit_together(times):
"""
Create the script that can set the 1st scheduled job running
Parameters
-------
time (str): string time to run job script for in the format YYYYMMDD
Returns
-------
(None)
"""
print(times)
FileName = 'run_geos_SLURM_queue_all_jobs.sh'
run_script = open(FileName, 'w')
Line0 = "#!/bin/bash \n"
Line1 = """job_num_{time}=$(sbatch --parsable SLURM_queue_files/{time}.sbatch) \n"""
Line2 = """echo "$job_num_{time}" \n"""
Line3 = """job_num_{time2}=$(sbatch --parsable --dependency=afterok:"$job_num_{time1}" SLURM_queue_files/{time2}.sbatch) \n"""
for n_time, time in enumerate(times[:-1]):
#
if time == times[0]:
run_script.write(Line0)
run_script.write(Line1.format(time=time))
run_script.write(Line2.format(time=time))
else:
run_script.write(Line3.format(time1=times[n_time-1], time2=time))
run_script.write(Line2.format(time=time))
run_script.close()
# Change the permissions so it is executable
st = os.stat(FileName)
os.chmod(FileName, st.st_mode | stat.S_IEXEC)
return
Example emailed error codes are (1) and a model run abort output of (2).
(1) Slurm Job_id=17908287 Name=Iso.UnlimAll.2 Ended, Run time 1-09:03:48, COMPLETED, ExitCode 0
(2)
---> DATE: 2018/06/05 UTC: 09:30 X-HRS: 3729.500000
===============================================================================
WETDEP: ERROR at 42 23 71 for species 128 in area RESUSPENSION in middle levels
LS : T
PDOWN : 0.000000000000000E+000
QQ : 0.000000000000000E+000
ALPHA : 0.000000000000000E+000
ALPHA2 : 0.000000000000000E+000
RAINFRAC : 0.000000000000000E+000
WASHFRAC : 0.000000000000000E+000
MASS_WASH : 0.000000000000000E+000
MASS_NOWASH : 0.000000000000000E+000
WETLOSS : 0.000000000000000E+000
GAINED : 0.000000000000000E+000
LOST : 0.000000000000000E+000
DSpc(NW,:) : 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000
Spc(I,J,:N) : 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
0.000000000000000E+000 0.000000000000000E+000 -1.198463418495579E-013
-3.378856226737475E-014 -2.314320910996734E-015 -3.019816125666191E-017
-2.520870547180549E-019 -2.822882195877689E-020 -1.445442763521664E-020
-6.900126364332953E-022 -8.767709715535021E-024 -4.400381547465612E-025
-1.720033035554861E-026 -4.865030418189842E-028 -1.209928241506788E-029
-2.093866091759672E-031 -5.970450354644719E-033 -5.162779587092464E-035
-3.748736158226715E-037 -6.734895733485428E-040 -1.499747125810236E-039
-1.842168830893617E-038 -1.148225302626263E-037 -8.781446960823125E-037
-5.560642728382409E-034 -3.794567594400997E-031 -1.821084089043769E-029
-4.488053629617312E-028 -1.680343276761365E-026 -1.155873919999522E-024
-3.060986958260714E-022 -1.313308483959904E-021 -1.574449846171001E-021
-1.338100164273038E-021 -6.956426779262325E-022 -3.506180178581743E-022
-6.266069886329827E-022 -1.353555184532673E-021 -4.106777266211684E-021
-1.014640714706731E-020 -1.134631269064776E-020 -9.833174509975247E-021
-7.707150917708346E-021
===============================================================================
===============================================================================
GEOS-Chem ERROR: Error encountered in wet deposition!
-> at SAFETY (in module GeosCore/wetscav_mod.F90)
===============================================================================
===============================================================================
GEOS-Chem ERROR: Error encountered in "Safety"!
-> at Do_Complete_Reevap (in module GeosCore/wetscav_mod.F90)
===============================================================================
===============================================================================
GEOS-Chem ERROR:
-> at WetDep (in module GeosCore/wetscav_mod.F90)
===============================================================================
===============================================================================
GEOS-Chem ERROR: Error encountered in "Wetdep"!
-> at Do_WetDep (in module GeosCore/wetscav_mod.F90)
===============================================================================
===============================================================================
GEOS-CHEM ERROR: Error encountered in "Do_WetDep"!
STOP at -> at GEOS-Chem (in GeosCore/main.F90)
===============================================================================
srun: error: node112: task 0: Exited with exit code 159
The BASH setup option for geos-chem-schedule no longer works (BASH call of geos-chem-schedule
is not found). Work out why this is and fix it.
Lines 55 to 80 in f543a09
The current installation process for this package is quite invasive (see above). A rework to follow a simple package structure (see Packaging Python Projects) would go a long way to smoothing this out.
Example of lines currently output geos.log
pasted below. To aid reading of GEOS-Chem information, could theses lines be output to a separate file?
Also, the job submission template should be updated so that the the command not found
SLURM messages are not presented on reading BASH variables strings.
************** E N D O F G E O S -- C H E M **************
/var/spool/slurmdspool/job8817675/slurm_script: line 123: last_line: command not found
/var/spool/slurmdspool/job8817675/slurm_script: line 124: complete_last_line: command not found
Submitted batch job 8856961
============================
Job utilisation efficiency
============================
Job ID: 8817675
Cluster: viking
User/Group: ts551/clusterusers
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 20
CPU Utilized: 27-00:30:01
CPU Efficiency: 98.91% of 27-07:40:00 core-walltime
Job Wall-clock time: 1-08:47:00
Memory Utilized: 8.82 GB
Memory Efficiency: 22.04% of 40.00 GB
Requested wall clock time: 2-00:00:00
Actual wall clock time: 1-08:47:00
Wall clock time efficiency: 68.3%
Job queued time: 00:00:01
GCST uses a single year run twice, with the 1st as starting input for second, for the 1 year benchmarking. This would be a useful functionality for comparisons with benchmarks. It also may be a better approach in future to avoid the "is the atmosphere in equilibrium" question as a single year rather than contiguous years are used for spin-up/analysis.
Read a configuration file with limits for queue on a specific HPC. Use this to set defaults for job configuration, then check any user changes against this.
https://wiki.york.ac.uk/display/RHPC/VK12%29+Available+Resource+Partitions+and+their+limits
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.