Ability to write to file in hour increments and smaller about isca HOT 8 CLOSED

Hi @mcthreems - it is certainly possible to have outputs saved more than once per day etc. I regularly output variables every timestep, and that's worked alright.

Could you share with us the whole diag section of your experiment script? That will help us diagnose the problem.

My best guess without seeing the code is that in the code you sent:
diag.add_file('atmos_daily', 1, 'hours', time_units='hours')
has the file being called atmos_daily. If you've got another entry similar to this one with atmos_daily in, for example:
diag.add_file('atmos_daily', 1, 'days', time_units='days')
then your entry with 1-hour time increments will be overwritten by the subsequent call with daily time increments. The reason for this is that the diag.add_file command creates a dictionary of files to be output, and the key for that dictionary is the first argument:
https://github.com/ExeClim/Isca/blob/master/src/extra/python/isca/diagtable.py#L53
so if you have more than one call with the same first argument, it'll get overwritten.

Could you try:
diag.add_file('atmos_hourly', 1, 'hours', time_units='hours')
and see if that works?

from isca.

Sorry for such a long gap in the response, we ended up finding an unrelated bug that made our need to try short timesteps secondary. We're back to this now though, so I'll attach the runscript I'm using for this. I'm attempting to write data to a file every hour, with each file having 24 hours.
example.txt

from isca.

I think I realized that the confusion came from the fact that the model only printed out progress for each day, even though I had incremented the day into hours. As such, it was working fine as long as I had one file per day. However, when I tried to do one file per hour, incremented in minutes, the model crashed immediately. I've pasted in the output from the crash below:

2019-10-25 18:28:42,973 - isca - INFO - Beginning run 156
2019-10-25 18:28:42,986 - isca - INFO - process running as 11791
2019-10-25 18:28:42,998 - isca - DEBUG - loadmodules for emps-hoff machines
2019-10-25 18:28:43,068 - isca - DEBUG -
2019-10-25 18:28:43,068 - isca - DEBUG - The 'gcc/4.9.3' module is being loaded
2019-10-25 18:28:43,068 - isca - DEBUG -
2019-10-25 18:28:43,321 - isca - DEBUG - Unloading the conflicting module 'intel/18.0.3'
2019-10-25 18:28:43,321 - isca - DEBUG -
2019-10-25 18:28:44,884 - isca - DEBUG - NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 32768.
2019-10-25 18:28:44,885 - isca - DEBUG - &MPP_IO_NML
2019-10-25 18:28:44,885 - isca - DEBUG - HEADER_BUFFER_VAL = 16384,
2019-10-25 18:28:44,885 - isca - DEBUG - GLOBAL_FIELD_ON_ROOT_PE = T,
2019-10-25 18:28:44,885 - isca - DEBUG - IO_CLOCKS_ON = F,
2019-10-25 18:28:44,886 - isca - DEBUG - SHUFFLE = 0,
2019-10-25 18:28:44,886 - isca - DEBUG - DEFLATE_LEVEL = -1
2019-10-25 18:28:44,886 - isca - DEBUG - /
2019-10-25 18:28:44,888 - isca - DEBUG - NOTE from PE 0: MPP_IO_SET_STACK_SIZE: stack size set to 131072.
2019-10-25 18:28:44,900 - isca - DEBUG - NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 600000.
2019-10-25 18:28:44,917 - isca - DEBUG - starting 1 OpenMP threads per MPI-task
2019-10-25 18:28:44,919 - isca - DEBUG - ATMOS MODEL DOMAIN DECOMPOSITION
2019-10-25 18:28:44,919 - isca - DEBUG - X-AXIS = 64
2019-10-25 18:28:44,919 - isca - DEBUG - Y-AXIS = 4 4 4 4 4 4 4 4
2019-10-25 18:28:45,012 - isca - DEBUG - NOTE from PE 0: idealized_moist_phys: Using Frierson Quasi-Equilibrium convection scheme.
2019-10-25 18:28:45,054 - isca - DEBUG -
2019-10-25 18:28:45,055 - isca - WARNING - WARNING from PE 0: socrates_init: lw_hires_spectral_filename is unset, making equal to lw_spectral_filename
2019-10-25 18:28:45,055 - isca - DEBUG -
2019-10-25 18:28:45,055 - isca - DEBUG -
2019-10-25 18:28:45,055 - isca - WARNING - WARNING from PE 0: socrates_init: sw_hires_spectral_filename is unset, making equal to sw_spectral_filename
2019-10-25 18:28:45,055 - isca - DEBUG -
2019-10-25 18:28:45,056 - isca - DEBUG -
2019-10-25 18:28:45,056 - isca - WARNING - WARNING from PE 0: socrates_init: lw_hires_spectral_filename is unset, making equal to lw_spectral_filename
2019-10-25 18:28:45,056 - isca - DEBUG -
2019-10-25 18:28:45,056 - isca - DEBUG -
2019-10-25 18:28:45,056 - isca - WARNING - WARNING from PE 0: socrates_init: sw_hires_spectral_filename is unset, making equal to sw_spectral_filename
2019-10-25 18:28:45,057 - isca - DEBUG -
2019-10-25 18:28:45,169 - isca - DEBUG - NOTE from PE 0: interpolator_mod :ozone_1990.nc is a year-independent climatology file
2019-10-25 18:28:45,170 - isca - DEBUG - Initializing src field : ozone_1990
2019-10-25 18:28:45,170 - isca - DEBUG -
2019-10-25 18:28:45,170 - isca - DEBUG - -----------------------------------
2019-10-25 18:28:45,171 - isca - DEBUG - ____ _
2019-10-25 18:28:45,171 - isca - DEBUG - / | ___ ___ _ __ __ | | ___ ___
2019-10-25 18:28:45,171 - isca - DEBUG - _ \ / _ \ / __| __/ ` | __/ _ / __|
2019-10-25 18:28:45,171 - isca - DEBUG - ) | () | (| | | (| | || _/_
2019-10-25 18:28:45,171 - isca - DEBUG - |/ _/ _|| _,|__||/
2019-10-25 18:28:45,171 - isca - DEBUG -
2019-10-25 18:28:45,171 - isca - DEBUG - Initialised Socrates v17.03
2019-10-25 18:28:45,171 - isca - DEBUG - Stellar constant = 1370.00000000000
2019-10-25 18:28:45,171 - isca - DEBUG - ES0 = 2.00000000000000
2019-10-25 18:28:45,171 - isca - DEBUG - OMEGA = 9.115143750000000E-006
2019-10-25 18:28:45,171 - isca - DEBUG - Longwave spectral file =
2019-10-25 18:28:45,171 - isca - DEBUG - /u/home/m/mmckinne/hab_proj/Isca/src/atmos_param/socrates/src/trunk/data/spectr
2019-10-25 18:28:45,171 - isca - DEBUG - a/ga7/sp_lw_ga7 WITH 9 bands
2019-10-25 18:28:45,172 - isca - DEBUG - Longwave hires spectral file =
2019-10-25 18:28:45,172 - isca - DEBUG - /u/home/m/mmckinne/hab_proj/Isca/src/atmos_param/socrates/src/trunk/data/spectr
2019-10-25 18:28:45,172 - isca - DEBUG - a/ga7/sp_lw_ga7 WITH 9 bands
2019-10-25 18:28:45,172 - isca - DEBUG - Shortwave spectral file =
2019-10-25 18:28:45,172 - isca - DEBUG - /u/home/m/mmckinne/hab_proj/Isca/src/atmos_param/socrates/src/trunk/data/spectr
2019-10-25 18:28:45,172 - isca - DEBUG - a/ga7/sp_sw_ga7 WITH 6 bands
2019-10-25 18:28:45,172 - isca - DEBUG - Shortwave hires spectral file =
2019-10-25 18:28:45,172 - isca - DEBUG - /u/home/m/mmckinne/hab_proj/Isca/src/atmos_param/socrates/src/trunk/data/spectr
2019-10-25 18:28:45,172 - isca - DEBUG - a/ga7/sp_sw_ga7 WITH 6 bands
2019-10-25 18:28:45,172 - isca - DEBUG -
2019-10-25 18:28:45,172 - isca - DEBUG - -----------------------------------
2019-10-25 18:28:45,172 - isca - DEBUG -
2019-10-25 18:28:45,290 - isca - DEBUG -
2019-10-25 18:28:45,290 - isca - DEBUG - FATAL from PE 0: diag_manager_mod::send_data_3d: module/output_field socrates/soc_surf_flux_sw, write EMPTY buffer
2019-10-25 18:28:45,291 - isca - DEBUG -
2019-10-25 18:28:45,291 - isca - DEBUG -
2019-10-25 18:28:45,291 - isca - DEBUG - FATAL from PE 0: diag_manager_mod::send_data_3d: module/output_field socrates/soc_surf_flux_sw, write EMPTY buffer
2019-10-25 18:28:45,291 - isca - DEBUG -
2019-10-25 18:28:45,291 - isca - DEBUG - application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
Exception in thread background thread for pid 11791:
Traceback (most recent call last):
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/site-packages/sh.py", line 1540, in wrap
fn(*args, **kwargs)
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/site-packages/sh.py", line 2459, in background_thread
handle_exit_code(exit_code)
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/site-packages/sh.py", line 2157, in fn
return self.command.handle_command_exit_code(exit_code)
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/site-packages/sh.py", line 815, in handle_command_exit_code
raise exc
sh.ErrorReturnCode_1:

RAN: /bin/bash /u/flashscratch/m/mmckinne/isca_work/experiment/soc_bd1000m_es02_rot8d/run/run.sh

STDOUT:

STDERR:

2019-10-25 18:28:45,393 - isca - ERROR - Run 156 failed. See log for details.
2019-10-25 18:28:45,394 - isca - ERROR - Error: ErrorReturnCode_1('\n\n RAN: /bin/bash /u/flashscratch/m/mmckinne/isca_work/experiment/soc_bd1000m_es02_rot8d/run/run.sh\n\n STDOUT:\n\n\n STDERR:\n')
Traceback (most recent call last):
File "/u/home/m/mmckinne/hab_proj/Isca/src/extra/python/isca/experiment.py", line 299, in run
proc.wait()
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/site-packages/sh.py", line 792, in wait
self.handle_command_exit_code(exit_code)
File "/u/home/m/mmckinne/miniconda/envs/isca_env/lib/python3.7/site-packages/sh.py", line 815, in handle_command_exit_code
raise exc
sh.ErrorReturnCode_1:

RAN: /bin/bash /u/flashscratch/m/mmckinne/isca_work/experiment/soc_bd1000m_es02_rot8d/run/run.sh

STDOUT:

STDERR:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "soc_bd1000m_es02_rot8d_exp.py", line 44, in
exp.run(i, num_cores=NCORES,overwrite_data=OVERWRITE) # use the restart i-1 by default
File "/u/home/m/mmckinne/hab_proj/Isca/src/extra/python/isca/helpers.py", line 22, in _destructive
return fn(*args, **kwargs)
File "/u/home/m/mmckinne/hab_proj/Isca/src/extra/python/isca/helpers.py", line 38, in _useworkdir
return fn(*args, **kwargs)
File "/u/home/m/mmckinne/hab_proj/Isca/src/extra/python/isca/experiment.py", line 313, in run
raise FailedRunError()
isca.experiment.FailedRunError

from isca.

Hi @mcthreems ,
Looking forward to your visit. I'm just doing some tidying up - am I correct in thinking this issue has now been dealt with offline, with the solution being to reduce the timesteps to <= 1 minute?:

ie.
dt_atmos = 120. -> 60.
dt_rad = 2400. -> 60.

If you are happy, I will close this issue.

from isca.

Is there some sort of requirement for the timestep to be smaller than 1 minute, wouldn't reducing the timestep by that extent would greatly increase the integration time needed since more timesteps are needed to run the same amount of years?

from isca.

This is not a general requirement. It was needed in a very particular case when the output was required very frequently for debugging purposes.

from isca.

Yes, the issue was just that I had made my file timesteps shorter than my output timesteps. Once I made the dt_atmos and dt_rad equal to or shorter than my file timesteps, things ran fine. We wanted the short timesteps to get data as close as possible to the point of a crash, so we gradually reduced the timesteps down all the way to the second as we approached the moment of the crash.

from isca.

Noted. Thank you!

from isca.