Comments (32)
But Δt = clock.time - wta.previous_collection_time
is not the simulation time-step. It's the time increment between collections. We can change the symbol to T = clock.time - wta.previous_collection_time
to help clarify this.
Moreover, do you think that a small error in T
would cause the kind of issue that @liuchihl is seeing? It looks to me like the average is not calculated at all --- rather than having a small round-off error.
PS I updated your comment to show more of the code so that we could see explicitly the part of the code you were referring to.
EDIT: and just to clarify the time increment between collections can be more than one time-step because we support strided averages (eg computing the averaged quantity less frequently than every time step).
from oceananigans.jl.
The zeros are written down at iteration 38, 50, 74, 98, 119, 140, 146, 167, 188, 194, and 201 from what I can see. If we find a pattern regarding the parameters of the WindowedTimeAverage
we can probably find what the problem is
from oceananigans.jl.
I think this is essentially a duplicate of #3485, although @liuchihl found a MWE where similar issues pop up even in the initial period (without picking up from a checkpoint).
from oceananigans.jl.
I tested the simulation without picking up a checkpoint (running only the first simulation from 0-6s), but the error still unexpectedly persists. It seems the issue might be related to a bug in AveragedTimeInterval
and maybe not necessarily just related to the checkpoint. (Hence I change the name of this issue.)
from oceananigans.jl.
Let me see if I understand this. We want to run a simulation in which the output interval and the averaging window is between 2-3 time-steps, right? Maybe we need to write out the sequence of steps that we expect to occur and then we can look at the source code and see why that is not happening. Does the problem also happen when δt = 0.02
(eg output exactly every two time-steps)?
from oceananigans.jl.
It looks like the time step calculated here
Oceananigans.jl/src/OutputWriters/windowed_time_average.jl
Lines 211 to 213 in 406eb9c
does not agree with the time step used in the simulation because of finite precision. I don't think we need to recompute it, we can use the simulation time step size. Probably this will not solve this issue but it will help to make the averaging more consistent
from oceananigans.jl.
Yes! Make sure to work from main
because remember we changed how TimeInterval
works recently. TimeInterval
actuates on this criteria:
Oceananigans.jl/src/Utils/schedules.jl
Lines 65 to 79 in dc5dc28
when t
exceeds or is equal to next_actuation_time
:
Oceananigans.jl/src/Utils/schedules.jl
Lines 58 to 63 in dc5dc28
The first actuation time and number of actuations have to be initialized:
Oceananigans.jl/src/Utils/schedules.jl
Lines 50 to 56 in dc5dc28
I think WindowedTimeAverage
works differently and looks at previous_stop_time
, plus the interval
of the time-averaging
and previous_stop_time
is recorded here:
which in turn is called here:
So there's something about using the previous stop time, rather than the number of actuations, which incurs a floating point error issue.
I think AveragedTimeInterval
should somehow use a number of actuations as a criteria. Not sure actually how that would work, need to study the flow of information in detail. It will probably help to sketch out a diagram / make a cartoon on a piece of paper so that you understand precisely the flow of information.
from oceananigans.jl.
Thanks a lot for your hint, I wanted to make sure what you mean by number of actuations as a criteria. Do you mean
first_actuation_time::Numberand do you think this will potentially prevent the floating point issue?
Not quite.
TimeInterval
keeps a count of how many times it has been actuated. This is stored in the variable TimeInterval.actuations
. It's use is illustrated in the computation of the next_actuation_time
:
Oceananigans.jl/src/Utils/schedules.jl
Lines 58 to 63 in dc5dc28
To understand this consider an example. TimeInterval
has been actuated 2 times, with an interval of 1.1. This means that the next actuation time --- the third actuation_ -- will occur at the time
t = 3 * 1.1 = 3.3
Thus rather than storing the "previous actuation time" (a floating point number), we instead store the number of actuations (an integer, and exact), and compute the next actuation time with the formula above.
Does this make sense?
from oceananigans.jl.
The issue not only happens after picking up checkpoint, it also occurs before that. For example, with the same MWE, if I set:
Δt = .01 # timestep (s)
T1 = 6 # first simulation stop time (s)
T2 = 2T1 # second simulation stop time (s)
δt = .03 # progress message interval and output saving interval
The issue also occurs in the first run (e.g., 0-6 s):
from oceananigans.jl.
Hmm ok. Can you identify the difference between the cases where it works and where it fails?
I'd like some other devs to help out with this @navidcy @simone-silvestri
from oceananigans.jl.
Sure, I ran a simple test using the MWE above without picking up a checkpoint and set:
Δt = .01 # timestep (s)
T1 = 5 # first simulation stop time (s)
δt = .03 # progress message interval and output saving interval
# Run a simulation
simulation = test_simulation(T1, Δt, δt, true)
run!(simulation)
I've noticed that this strange behavior occurs when δt is smaller than or equal to 0.03, but it disappears when δt is greater than 0.03 (even at 0.031). It seems there is a cutoff value of δt below which the error emerges.
from oceananigans.jl.
By the way, we originally thought we might be able to solve this issue by setting the checkpointing interval to be an integer multiple of the averaging interval (and window) but that is evidently not the case since the bug shows up in the MWEs above that satisfy this condition (where the checkpoint time is T1 = 6
and the averaging interval is δt = 0.03
).
from oceananigans.jl.
We want to run a simulation in which the output interval and the averaging window is between 2-3 time-steps, right?
Yes, exactly. The error occurs in this range.
Maybe we need to write out the sequence of steps that we expect to occur and then we can look at the source code and see why that is not happening. Does the problem also happen when δt = 0.02 (eg output exactly every two time-steps)?
Yes, I agree, the problem still happens when δt = 0.02.
from oceananigans.jl.
It seems like the Δt
that is calculated in accumulate_result!
does not agree with the simulation Δt
because of finite precision arithmetic. Wouldn't it be better to pass the simulation Δt
to the schedule so everything becomes more consistent and we avoid discrepancies between schedule time and simulation time?
from oceananigans.jl.
@simone-silvestri can you spell out what you're saying more explicitly by referencing / linking into the source code? I can't figure out what you're saying (it seems I would have to dive into the source code to figure it out).
from oceananigans.jl.
I'm also doubtful small round off errors in the timestep would be related, unless there a place in the code where the scheduled times for outputting / checkpointing need to exactly match some value.
In the MWE, the problematic values look like they're all zeros. In our more complicated example, where the averaging interval is a large (decimal) multiple of the timestep, it's not clear if the values are underestimated because the velocities are of both signs so a biased average could go either way.
from oceananigans.jl.
I am not sure if this helps, but this might shed some light.
This is the same MWE with the progress function set as:
function progress_message(sim)
wta = sim.output_writers[:timeavg2].outputs[1]
wd = wta.window_start_time
ws = wta.window_start_iteration
pc = wta.previous_collection_time
fo = wta.fetch_operand
res = sum(wta.result)
interval = wta.schedule.interval
pis = wta.schedule.previous_interval_stop_time
cll = wta.schedule.collecting
@info string("Iter: ", iteration(sim), ", time: ", prettytime(sim), " , u-avg: ", res, ", window_start_time: ", wd, ", window_start_iteration: ", ws, ", previous_collection_time: ", pc, ", fetch_operand: ", fo, ", interval: ", interval, ", previous_interval_stop_time: ", pis, ", collecting: ", cll)
end
julia> include("test.jl")
[ Info: Initializing simulation...
[ Info: Iter: 0, time: 0 seconds , u-avg: 0.0, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.0, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: ... simulation initialization complete (257.729 ms)
[ Info: Executing initial time step...
[ Info: ... initial time step complete (1.212 seconds).
[ Info: Iter: 1, time: 10 ms , u-avg: 0.0031413629825035438, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.01, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: Iter: 2, time: 20 ms , u-avg: 0.007851828677677537, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.02, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: Iter: 3, time: 30 ms , u-avg: 0.014652384734839867, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.03, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: false
[ Info: Iter: 4, time: 40 ms , u-avg: 0.0, window_start_time: 0.04, window_start_iteration: 4, previous_collection_time: 0.04, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: true
[ Info: Iter: 5, time: 50 ms , u-avg: 0.07837860105159718, window_start_time: 0.04, window_start_iteration: 4, previous_collection_time: 0.05, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: true
[ Info: Iter: 6, time: 60 ms , u-avg: 0.09557081820157169, window_start_time: 0.04, window_start_iteration: 4, previous_collection_time: 0.06, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: false
[ Info: Iter: 7, time: 70.000 ms , u-avg: 0.0, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.06999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 8, time: 80.000 ms , u-avg: 0.20000603541011058, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.07999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 9, time: 90.000 ms , u-avg: 0.22639226913537686, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.08999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 10, time: 90 ms , u-avg: 0.22639226913537688, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.09, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: false
[ Info: Iter: 11, time: 100.000 ms , u-avg: 0.0, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.09999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: true
[ Info: Iter: 12, time: 110.000 ms , u-avg: 0.3763650237305524, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.10999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: true
[ Info: Iter: 13, time: 120.000 ms , u-avg: 0.41171106795258217, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.11999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: true
[ Info: Iter: 14, time: 120 ms , u-avg: 0.4117110679525822, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.12, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.12, collecting: false
[ Info: Iter: 15, time: 130 ms , u-avg: 0.0, window_start_time: 0.13, window_start_iteration: 15, previous_collection_time: 0.13, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.12, collecting: true
[ Info: Iter: 16, time: 140 ms , u-avg: 0.6058901911474607, window_start_time: 0.13, window_start_iteration: 15, previous_collection_time: 0.14, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.12, collecting: true
[ Info: Iter: 17, time: 150 ms , u-avg: 0.649882311875457, window_start_time: 0.13, window_start_iteration: 15, previous_collection_time: 0.15, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.15, collecting: false
[ Info: Iter: 18, time: 160 ms , u-avg: 0.0, window_start_time: 0.16, window_start_iteration: 18, previous_collection_time: 0.16, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.15, collecting: true
[ Info: Iter: 19, time: 170 ms , u-avg: 0.8865442560258744, window_start_time: 0.16, window_start_iteration: 18, previous_collection_time: 0.17, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.15, collecting: true
[ Info: Iter: 20, time: 180 ms , u-avg: 0.9387919760818845, window_start_time: 0.16, window_start_iteration: 18, previous_collection_time: 0.18, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.18, collecting: false
[ Info: Iter: 21, time: 190 ms , u-avg: 0.0, window_start_time: 0.19, window_start_iteration: 21, previous_collection_time: 0.19, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.18, collecting: true
[ Info: Iter: 22, time: 200 ms , u-avg: 1.2158361130173538, window_start_time: 0.19, window_start_iteration: 21, previous_collection_time: 0.2, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.18, collecting: true
[ Info: Iter: 23, time: 210 ms , u-avg: 1.2758756779393434, window_start_time: 0.19, window_start_iteration: 21, previous_collection_time: 0.21, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.21, collecting: false
[ Info: Iter: 24, time: 220 ms , u-avg: 0.0, window_start_time: 0.22, window_start_iteration: 24, previous_collection_time: 0.22, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.21, collecting: true
[ Info: Iter: 25, time: 230 ms , u-avg: 1.5908429442874084, window_start_time: 0.22, window_start_iteration: 24, previous_collection_time: 0.23, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.21, collecting: true
[ Info: Iter: 26, time: 240 ms , u-avg: 1.6581414386466988, window_start_time: 0.22, window_start_iteration: 24, previous_collection_time: 0.24, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.24, collecting: false
[ Info: Iter: 27, time: 250 ms , u-avg: 0.0, window_start_time: 0.25, window_start_iteration: 27, previous_collection_time: 0.25, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.24, collecting: true
[ Info: Iter: 28, time: 260 ms , u-avg: 2.008236162653511, window_start_time: 0.25, window_start_iteration: 27, previous_collection_time: 0.26, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.24, collecting: true
[ Info: Iter: 29, time: 270 ms , u-avg: 2.082196240249851, window_start_time: 0.25, window_start_iteration: 27, previous_collection_time: 0.27, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: false
[ Info: Iter: 30, time: 280 ms , u-avg: 0.0, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.28, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
[ Info: Iter: 31, time: 290.000 ms , u-avg: 2.464310956360575, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.29000000000000004, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
[ Info: Iter: 32, time: 300 ms , u-avg: 2.5442761423093008, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.3, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 33, time: 310 ms , u-avg: 2.6254722813218105, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.31, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: false
[ Info: Iter: 34, time: 320 ms , u-avg: 0.0, window_start_time: 0.32, window_start_iteration: 34, previous_collection_time: 0.32, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: true
[ Info: Iter: 35, time: 330.000 ms , u-avg: 3.1255402085515462, window_start_time: 0.32, window_start_iteration: 34, previous_collection_time: 0.32999999999999996, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: false
[ Info: Iter: 36, time: 340.000 ms , u-avg: 0.0, window_start_time: 0.33999999999999997, window_start_iteration: 36, previous_collection_time: 0.33999999999999997, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: true
[ Info: Iter: 37, time: 350 ms , u-avg: 3.476005252456648, window_start_time: 0.33999999999999997, window_start_iteration: 36, previous_collection_time: 0.35, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.32999999999999996, collecting: false
[ Info: Iter: 38, time: 360 ms , u-avg: 0.0, window_start_time: 0.36, window_start_iteration: 38, previous_collection_time: 0.36, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.32999999999999996, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 39, time: 370 ms , u-avg: 3.837876565646086, window_start_time: 0.36, window_start_iteration: 38, previous_collection_time: 0.37, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.36, collecting: false
[ Info: Iter: 40, time: 380 ms , u-avg: 0.0, window_start_time: 0.38, window_start_iteration: 40, previous_collection_time: 0.38, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.36, collecting: true
[ Info: Iter: 41, time: 390 ms , u-avg: 4.209726007369981, window_start_time: 0.38, window_start_iteration: 40, previous_collection_time: 0.39, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: false
[ Info: Iter: 42, time: 400 ms , u-avg: 0.0, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.4, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: true
[ Info: Iter: 43, time: 410.000 ms , u-avg: 4.590086057763697, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.41000000000000003, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: true
[ Info: Iter: 44, time: 420 ms , u-avg: 4.6865378368773, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.42, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 45, time: 430 ms , u-avg: 4.783510427744303, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.43, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: false
[ Info: Iter: 46, time: 440 ms , u-avg: 0.0, window_start_time: 0.44, window_start_iteration: 46, previous_collection_time: 0.44, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: true
[ Info: Iter: 47, time: 450.000 ms , u-avg: 5.370305891865513, window_start_time: 0.44, window_start_iteration: 46, previous_collection_time: 0.44999999999999996, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: false
[ Info: Iter: 48, time: 460.000 ms , u-avg: 0.0, window_start_time: 0.45999999999999996, window_start_iteration: 48, previous_collection_time: 0.45999999999999996, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: true
[ Info: Iter: 49, time: 470 ms , u-avg: 5.76708650433695, window_start_time: 0.45999999999999996, window_start_iteration: 48, previous_collection_time: 0.47, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.44999999999999996, collecting: false
[ Info: Iter: 50, time: 480 ms , u-avg: 0.0, window_start_time: 0.48, window_start_iteration: 50, previous_collection_time: 0.48, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.44999999999999996, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 51, time: 490 ms , u-avg: 6.166231535087016, window_start_time: 0.48, window_start_iteration: 50, previous_collection_time: 0.49, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.48, collecting: false
[ Info: Iter: 52, time: 500 ms , u-avg: 0.0, window_start_time: 0.5, window_start_iteration: 52, previous_collection_time: 0.5, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.48, collecting: true
[ Info: Iter: 53, time: 510 ms , u-avg: 6.566165741031355, window_start_time: 0.5, window_start_iteration: 52, previous_collection_time: 0.51, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.51, collecting: false
[ Info: Iter: 54, time: 520 ms , u-avg: 0.0, window_start_time: 0.52, window_start_iteration: 54, previous_collection_time: 0.52, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.51, collecting: true
[ Info: Iter: 55, time: 530 ms , u-avg: 6.965310764571667, window_start_time: 0.52, window_start_iteration: 54, previous_collection_time: 0.53, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.51, collecting: true
[ Info: Iter: 56, time: 540 ms , u-avg: 7.064702771106775, window_start_time: 0.52, window_start_iteration: 54, previous_collection_time: 0.54, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: false
[ Info: Iter: 57, time: 550 ms , u-avg: 0.0, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.55, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: true
[ Info: Iter: 58, time: 560 ms , u-avg: 7.559105120877197, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.56, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: true
[ Info: Iter: 59, time: 570 ms , u-avg: 7.657023372200447, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.57, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 60, time: 580 ms , u-avg: 7.7544847826693255, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.58, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.57, collecting: false
[ Info: Iter: 61, time: 590 ms , u-avg: 0.0, window_start_time: 0.59, window_start_iteration: 61, previous_collection_time: 0.59, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.57, collecting: true
[ Info: Iter: 62, time: 600 ms , u-avg: 8.333461880244451, window_start_time: 0.59, window_start_iteration: 61, previous_collection_time: 0.6, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6, collecting: false
[ Info: Iter: 63, time: 610 ms , u-avg: 0.0, window_start_time: 0.61, window_start_iteration: 63, previous_collection_time: 0.61, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6, collecting: true
[ Info: Iter: 64, time: 620 ms , u-avg: 8.709752264017125, window_start_time: 0.61, window_start_iteration: 63, previous_collection_time: 0.62, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6, collecting: true
[ Info: Iter: 65, time: 630 ms , u-avg: 8.802136412533468, window_start_time: 0.61, window_start_iteration: 63, previous_collection_time: 0.63, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: false
[ Info: Iter: 66, time: 640 ms , u-avg: 0.0, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.64, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: true
[ Info: Iter: 67, time: 650 ms , u-avg: 9.256391825350283, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.65, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: true
[ Info: Iter: 68, time: 660.000 ms , u-avg: 9.344764747359864, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.6599999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 69, time: 670.000 ms , u-avg: 9.432128769553014, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.6699999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: false
[ Info: Iter: 70, time: 680.000 ms , u-avg: 0.0, window_start_time: 0.6799999999999999, window_start_iteration: 70, previous_collection_time: 0.6799999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: true
[ Info: Iter: 71, time: 690 ms , u-avg: 9.944532401619153, window_start_time: 0.6799999999999999, window_start_iteration: 70, previous_collection_time: 0.69, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: false
[ Info: Iter: 72, time: 700 ms , u-avg: 0.0, window_start_time: 0.7, window_start_iteration: 72, previous_collection_time: 0.7, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: true
[ Info: Iter: 73, time: 710 ms , u-avg: 10.268085937114538, window_start_time: 0.7, window_start_iteration: 72, previous_collection_time: 0.71, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.69, collecting: false
[ Info: Iter: 74, time: 720 ms , u-avg: 0.0, window_start_time: 0.72, window_start_iteration: 74, previous_collection_time: 0.72, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.69, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 75, time: 730 ms , u-avg: 10.576240502438422, window_start_time: 0.72, window_start_iteration: 74, previous_collection_time: 0.73, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.72, collecting: false
[ Info: Iter: 76, time: 740 ms , u-avg: 0.0, window_start_time: 0.74, window_start_iteration: 76, previous_collection_time: 0.74, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.72, collecting: true
[ Info: Iter: 77, time: 750 ms , u-avg: 10.867779952303898, window_start_time: 0.74, window_start_iteration: 76, previous_collection_time: 0.75, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.75, collecting: false
[ Info: Iter: 78, time: 760 ms , u-avg: 0.0, window_start_time: 0.76, window_start_iteration: 78, previous_collection_time: 0.76, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.75, collecting: true
[ Info: Iter: 79, time: 770 ms , u-avg: 11.141553713694051, window_start_time: 0.76, window_start_iteration: 78, previous_collection_time: 0.77, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.75, collecting: true
[ Info: Iter: 80, time: 780 ms , u-avg: 11.206495836906438, window_start_time: 0.76, window_start_iteration: 78, previous_collection_time: 0.78, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: false
[ Info: Iter: 81, time: 790 ms , u-avg: 0.0, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.79, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: true
[ Info: Iter: 82, time: 800 ms , u-avg: 11.516560410575497, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.8, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: true
[ Info: Iter: 83, time: 810.000 ms , u-avg: 11.574058559459141, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.8099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 84, time: 820 ms , u-avg: 11.62982461713531, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.82, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8099999999999999, collecting: false
[ Info: Iter: 85, time: 830 ms , u-avg: 0.0, window_start_time: 0.83, window_start_iteration: 85, previous_collection_time: 0.83, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8099999999999999, collecting: true
[ Info: Iter: 86, time: 840 ms , u-avg: 11.944939757472355, window_start_time: 0.83, window_start_iteration: 85, previous_collection_time: 0.84, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.84, collecting: false
[ Info: Iter: 87, time: 850 ms , u-avg: 0.0, window_start_time: 0.85, window_start_iteration: 87, previous_collection_time: 0.85, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.84, collecting: true
[ Info: Iter: 88, time: 860 ms , u-avg: 12.126506036338247, window_start_time: 0.85, window_start_iteration: 87, previous_collection_time: 0.86, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.84, collecting: true
[ Info: Iter: 89, time: 870 ms , u-avg: 12.167655766858058, window_start_time: 0.85, window_start_iteration: 87, previous_collection_time: 0.87, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: false
[ Info: Iter: 90, time: 880 ms , u-avg: 0.0, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.88, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: true
[ Info: Iter: 91, time: 890 ms , u-avg: 12.356031044664883, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.89, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: true
[ Info: Iter: 92, time: 900.000 ms , u-avg: 12.388421440868372, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.8999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 93, time: 910.000 ms , u-avg: 12.418820112689675, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.9099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: false
[ Info: Iter: 94, time: 920.000 ms , u-avg: 0.0, window_start_time: 0.9199999999999999, window_start_iteration: 94, previous_collection_time: 0.9199999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: true
[ Info: Iter: 95, time: 930.000 ms , u-avg: 12.579076993274457, window_start_time: 0.9199999999999999, window_start_iteration: 94, previous_collection_time: 0.9299999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: false
[ Info: Iter: 96, time: 940 ms , u-avg: 0.0, window_start_time: 0.94, window_start_iteration: 96, previous_collection_time: 0.94, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: true
[ Info: Iter: 97, time: 950 ms , u-avg: 12.654017134137858, window_start_time: 0.94, window_start_iteration: 96, previous_collection_time: 0.95, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.9299999999999999, collecting: false
[ Info: Iter: 98, time: 960 ms , u-avg: 0.0, window_start_time: 0.96, window_start_iteration: 98, previous_collection_time: 0.96, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.9299999999999999, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 99, time: 970 ms , u-avg: 12.704142124423381, window_start_time: 0.96, window_start_iteration: 98, previous_collection_time: 0.97, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.96, collecting: false
[ Info: Iter: 100, time: 980 ms , u-avg: 0.0, window_start_time: 0.98, window_start_iteration: 100, previous_collection_time: 0.98, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.96, collecting: true
[ Info: Iter: 101, time: 990 ms , u-avg: 12.729254143694298, window_start_time: 0.98, window_start_iteration: 100, previous_collection_time: 0.99, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.99, collecting: false
[ Info: Iter: 102, time: 1 second , u-avg: 0.0, window_start_time: 1.0, window_start_iteration: 102, previous_collection_time: 1.0, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.99, collecting: true
[ Info: Iter: 103, time: 1.010 seconds , u-avg: 12.729254086283136, window_start_time: 1.0, window_start_iteration: 102, previous_collection_time: 1.01, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.99, collecting: true
[ Info: Iter: 104, time: 1.020 seconds , u-avg: 12.724543620587964, window_start_time: 1.0, window_start_iteration: 102, previous_collection_time: 1.02, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.02, collecting: false
[ Info: Iter: 105, time: 1.030 seconds , u-avg: 0.0, window_start_time: 1.03, window_start_iteration: 105, previous_collection_time: 1.03, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.02, collecting: true
[ Info: Iter: 106, time: 1.040 seconds , u-avg: 12.682195964176623, window_start_time: 1.03, window_start_iteration: 105, previous_collection_time: 1.04, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.02, collecting: true
[ Info: Iter: 107, time: 1.050 seconds , u-avg: 12.668106406195333, window_start_time: 1.03, window_start_iteration: 105, previous_collection_time: 1.05, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.05, collecting: false
[ Info: Iter: 108, time: 1.060 seconds , u-avg: 0.0, window_start_time: 1.06, window_start_iteration: 108, previous_collection_time: 1.06, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.05, collecting: true
[ Info: Iter: 109, time: 1.070 seconds , u-avg: 12.579076594562144, window_start_time: 1.06, window_start_iteration: 108, previous_collection_time: 1.07, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.05, collecting: true
[ Info: Iter: 110, time: 1.080 seconds , u-avg: 12.555733004208836, window_start_time: 1.06, window_start_iteration: 108, previous_collection_time: 1.08, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: false
[ Info: Iter: 111, time: 1.090 seconds , u-avg: 0.0, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.09, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: true
[ Info: Iter: 112, time: 1.100 seconds , u-avg: 12.420811272264622, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.1, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: true
[ Info: Iter: 113, time: 1.110 seconds , u-avg: 12.388420848899857, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.1099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 114, time: 1.120 seconds , u-avg: 12.354060011630247, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.1199999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: false
[ Info: Iter: 115, time: 1.130 seconds , u-avg: 0.0, window_start_time: 1.13, window_start_iteration: 115, previous_collection_time: 1.13, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: true
[ Info: Iter: 116, time: 1.140 seconds , u-avg: 12.126505258118181, window_start_time: 1.13, window_start_iteration: 115, previous_collection_time: 1.14, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: false
[ Info: Iter: 117, time: 1.150 seconds , u-avg: 0.0, window_start_time: 1.15, window_start_iteration: 117, previous_collection_time: 1.15, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: true
[ Info: Iter: 118, time: 1.160 seconds , u-avg: 11.944938876944853, window_start_time: 1.15, window_start_iteration: 117, previous_collection_time: 1.16, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.14, collecting: false
[ Info: Iter: 119, time: 1.170 seconds , u-avg: 0.0, window_start_time: 1.17, window_start_iteration: 119, previous_collection_time: 1.17, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.14, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 120, time: 1.180 seconds , u-avg: 11.741355753127747, window_start_time: 1.17, window_start_iteration: 119, previous_collection_time: 1.18, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.17, collecting: false
[ Info: Iter: 121, time: 1.190 seconds , u-avg: 0.0, window_start_time: 1.19, window_start_iteration: 121, previous_collection_time: 1.19, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.17, collecting: true
[ Info: Iter: 122, time: 1.200 seconds , u-avg: 11.51655933624829, window_start_time: 1.19, window_start_iteration: 121, previous_collection_time: 1.2, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.2, collecting: false
[ Info: Iter: 123, time: 1.210 seconds , u-avg: 0.0, window_start_time: 1.21, window_start_iteration: 123, previous_collection_time: 1.21, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.2, collecting: true
[ Info: Iter: 124, time: 1.220 seconds , u-avg: 11.271436795064185, window_start_time: 1.21, window_start_iteration: 123, previous_collection_time: 1.22, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.2, collecting: true
[ Info: Iter: 125, time: 1.230 seconds , u-avg: 11.206494650021211, window_start_time: 1.21, window_start_iteration: 123, previous_collection_time: 1.23, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.23, collecting: false
[ Info: Iter: 126, time: 1.240 seconds , u-avg: 0.0, window_start_time: 1.24, window_start_iteration: 126, previous_collection_time: 1.24, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.23, collecting: true
[ Info: Iter: 127, time: 1.250 seconds , u-avg: 10.867778659886214, window_start_time: 1.24, window_start_iteration: 126, previous_collection_time: 1.25, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.23, collecting: true
[ Info: Iter: 128, time: 1.260 seconds , u-avg: 10.795968973249174, window_start_time: 1.24, window_start_iteration: 126, previous_collection_time: 1.26, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.26, collecting: false
[ Info: Iter: 129, time: 1.270 seconds , u-avg: 0.0, window_start_time: 1.27, window_start_iteration: 129, previous_collection_time: 1.27, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.26, collecting: true
[ Info: Iter: 130, time: 1.280 seconds , u-avg: 10.424164173642733, window_start_time: 1.27, window_start_iteration: 129, previous_collection_time: 1.28, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.26, collecting: true
[ Info: Iter: 131, time: 1.290 seconds , u-avg: 10.3461243332739, window_start_time: 1.27, window_start_iteration: 129, previous_collection_time: 1.29, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: false
[ Info: Iter: 132, time: 1.300 seconds , u-avg: 0.0, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.3, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: true
[ Info: Iter: 133, time: 1.310 seconds , u-avg: 9.944530889918811, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.31, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: true
[ Info: Iter: 134, time: 1.320 seconds , u-avg: 9.860953582966044, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.3199999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 135, time: 1.330 seconds , u-avg: 9.776254135548728, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.3299999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: false
[ Info: Iter: 136, time: 1.340 seconds , u-avg: 0.0, window_start_time: 1.3399999999999999, window_start_iteration: 136, previous_collection_time: 1.3399999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: true
[ Info: Iter: 137, time: 1.350 seconds , u-avg: 9.256390196808994, window_start_time: 1.3399999999999999, window_start_iteration: 136, previous_collection_time: 1.3499999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: false
[ Info: Iter: 138, time: 1.360 seconds , u-avg: 0.0, window_start_time: 1.3599999999999999, window_start_iteration: 138, previous_collection_time: 1.3599999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: true
[ Info: Iter: 139, time: 1.370 seconds , u-avg: 8.894518883619556, window_start_time: 1.3599999999999999, window_start_iteration: 138, previous_collection_time: 1.3699999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3499999999999999, collecting: false
[ Info: Iter: 140, time: 1.380 seconds , u-avg: 0.0, window_start_time: 1.38, window_start_iteration: 140, previous_collection_time: 1.38, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3499999999999999, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 141, time: 1.390 seconds , u-avg: 8.522669441895662, window_start_time: 1.38, window_start_iteration: 140, previous_collection_time: 1.39, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: false
[ Info: Iter: 142, time: 1.400 seconds , u-avg: 0.0, window_start_time: 1.4, window_start_iteration: 142, previous_collection_time: 1.4, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: true
[ Info: Iter: 143, time: 1.410 seconds , u-avg: 8.142309391501945, window_start_time: 1.4, window_start_iteration: 142, previous_collection_time: 1.41, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: false
[ Info: Iter: 144, time: 1.420 seconds , u-avg: 0.0, window_start_time: 1.42, window_start_iteration: 144, previous_collection_time: 1.42, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: true
[ Info: Iter: 145, time: 1.430 seconds , u-avg: 7.754939839787331, window_start_time: 1.42, window_start_iteration: 144, previous_collection_time: 1.43, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.41, collecting: false
[ Info: Iter: 146, time: 1.440 seconds , u-avg: 0.0, window_start_time: 1.44, window_start_iteration: 146, previous_collection_time: 1.44, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.41, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 147, time: 1.450 seconds , u-avg: 7.362089557400126, window_start_time: 1.44, window_start_iteration: 146, previous_collection_time: 1.45, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.44, collecting: false
[ Info: Iter: 148, time: 1.460 seconds , u-avg: 0.0, window_start_time: 1.46, window_start_iteration: 148, previous_collection_time: 1.46, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.44, collecting: true
[ Info: Iter: 149, time: 1.470 seconds , u-avg: 6.965308944928689, window_start_time: 1.46, window_start_iteration: 148, previous_collection_time: 1.47, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.47, collecting: false
[ Info: Iter: 150, time: 1.480 seconds , u-avg: 0.0, window_start_time: 1.48, window_start_iteration: 150, previous_collection_time: 1.48, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.47, collecting: true
[ Info: Iter: 151, time: 1.490 seconds , u-avg: 6.566163914178623, window_start_time: 1.48, window_start_iteration: 150, previous_collection_time: 1.49, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.47, collecting: true
[ Info: Iter: 152, time: 1.500 seconds , u-avg: 6.466180362467067, window_start_time: 1.48, window_start_iteration: 150, previous_collection_time: 1.5, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5, collecting: false
[ Info: Iter: 153, time: 1.510 seconds , u-avg: 0.0, window_start_time: 1.51, window_start_iteration: 153, previous_collection_time: 1.51, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5, collecting: true
[ Info: Iter: 154, time: 1.520 seconds , u-avg: 5.966459950002811, window_start_time: 1.51, window_start_iteration: 153, previous_collection_time: 1.52, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5, collecting: true
[ Info: Iter: 155, time: 1.530 seconds , u-avg: 5.866772317348391, window_start_time: 1.51, window_start_iteration: 153, previous_collection_time: 1.53, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.53, collecting: false
[ Info: Iter: 156, time: 1.540 seconds , u-avg: 0.0, window_start_time: 1.54, window_start_iteration: 156, previous_collection_time: 1.54, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.53, collecting: true
[ Info: Iter: 157, time: 1.550 seconds , u-avg: 5.370304086613586, window_start_time: 1.54, window_start_iteration: 156, previous_collection_time: 1.55, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.53, collecting: true
[ Info: Iter: 158, time: 1.560 seconds , u-avg: 5.271797207501014, window_start_time: 1.54, window_start_iteration: 156, previous_collection_time: 1.56, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: false
[ Info: Iter: 159, time: 1.570 seconds , u-avg: 0.0, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.57, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: true
[ Info: Iter: 160, time: 1.580 seconds , u-avg: 4.782987845658554, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.58, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: true
[ Info: Iter: 161, time: 1.590 seconds , u-avg: 4.686536074120517, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.5899999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 162, time: 1.600 seconds , u-avg: 4.59066857242074, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.5999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: false
[ Info: Iter: 163, time: 1.610 seconds , u-avg: 0.0, window_start_time: 1.6099999999999999, window_start_iteration: 163, previous_collection_time: 1.6099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: true
[ Info: Iter: 164, time: 1.620 seconds , u-avg: 4.022643185248518, window_start_time: 1.6099999999999999, window_start_iteration: 163, previous_collection_time: 1.6199999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: false
[ Info: Iter: 165, time: 1.630 seconds , u-avg: 0.0, window_start_time: 1.63, window_start_iteration: 165, previous_collection_time: 1.63, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: true
[ Info: Iter: 166, time: 1.640 seconds , u-avg: 3.6556017405745704, window_start_time: 1.63, window_start_iteration: 165, previous_collection_time: 1.64, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.6199999999999999, collecting: false
[ Info: Iter: 167, time: 1.650 seconds , u-avg: 0.0, window_start_time: 1.65, window_start_iteration: 167, previous_collection_time: 1.65, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.6199999999999999, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 168, time: 1.660 seconds , u-avg: 3.2992577798961906, window_start_time: 1.65, window_start_iteration: 167, previous_collection_time: 1.66, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.65, collecting: false
[ Info: Iter: 169, time: 1.670 seconds , u-avg: 0.0, window_start_time: 1.67, window_start_iteration: 169, previous_collection_time: 1.67, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.65, collecting: true
[ Info: Iter: 170, time: 1.680 seconds , u-avg: 2.9550176300281055, window_start_time: 1.67, window_start_iteration: 169, previous_collection_time: 1.68, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.68, collecting: false
[ Info: Iter: 171, time: 1.690 seconds , u-avg: 0.0, window_start_time: 1.69, window_start_iteration: 171, previous_collection_time: 1.69, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.68, collecting: true
[ Info: Iter: 172, time: 1.700 seconds , u-avg: 2.6242398495734784, window_start_time: 1.69, window_start_iteration: 171, previous_collection_time: 1.7, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.68, collecting: true
[ Info: Iter: 173, time: 1.710 seconds , u-avg: 2.54427468086229, window_start_time: 1.69, window_start_iteration: 171, previous_collection_time: 1.71, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.71, collecting: false
[ Info: Iter: 174, time: 1.720 seconds , u-avg: 0.0, window_start_time: 1.72, window_start_iteration: 174, previous_collection_time: 1.72, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.71, collecting: true
[ Info: Iter: 175, time: 1.730 seconds , u-avg: 2.1561549468272188, window_start_time: 1.72, window_start_iteration: 174, previous_collection_time: 1.73, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.71, collecting: true
[ Info: Iter: 176, time: 1.740 seconds , u-avg: 2.0821948885524773, window_start_time: 1.72, window_start_iteration: 174, previous_collection_time: 1.74, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.74, collecting: false
[ Info: Iter: 177, time: 1.750 seconds , u-avg: 0.0, window_start_time: 1.75, window_start_iteration: 177, previous_collection_time: 1.75, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.74, collecting: true
[ Info: Iter: 178, time: 1.760 seconds , u-avg: 1.7254386818218563, window_start_time: 1.75, window_start_iteration: 177, previous_collection_time: 1.76, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.74, collecting: true
[ Info: Iter: 179, time: 1.770 seconds , u-avg: 1.6581402086967247, window_start_time: 1.75, window_start_iteration: 177, previous_collection_time: 1.77, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: false
[ Info: Iter: 180, time: 1.780 seconds , u-avg: 0.0, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.78, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: true
[ Info: Iter: 181, time: 1.790 seconds , u-avg: 1.3359141226176374, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.79, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: true
[ Info: Iter: 182, time: 1.800 seconds , u-avg: 1.2758745806538938, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.7999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 183, time: 1.810 seconds , u-avg: 1.217529300743549, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.8099999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: false
[ Info: Iter: 184, time: 1.820 seconds , u-avg: 0.0, window_start_time: 1.8199999999999998, window_start_iteration: 184, previous_collection_time: 1.8199999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: true
[ Info: Iter: 185, time: 1.830 seconds , u-avg: 0.8865433256230806, window_start_time: 1.8199999999999998, window_start_iteration: 184, previous_collection_time: 1.8299999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: false
[ Info: Iter: 186, time: 1.840 seconds , u-avg: 0.0, window_start_time: 1.8399999999999999, window_start_iteration: 186, previous_collection_time: 1.8399999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: true
[ Info: Iter: 187, time: 1.850 seconds , u-avg: 0.6938736028202257, window_start_time: 1.8399999999999999, window_start_iteration: 186, previous_collection_time: 1.8499999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8299999999999998, collecting: false
[ Info: Iter: 188, time: 1.860 seconds , u-avg: 0.0, window_start_time: 1.8599999999999999, window_start_iteration: 188, previous_collection_time: 1.8599999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8299999999999998, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 189, time: 1.870 seconds , u-avg: 0.5235899518877757, window_start_time: 1.8599999999999999, window_start_iteration: 188, previous_collection_time: 1.8699999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: false
[ Info: Iter: 190, time: 1.880 seconds , u-avg: 0.0, window_start_time: 1.88, window_start_iteration: 190, previous_collection_time: 1.88, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: true
[ Info: Iter: 191, time: 1.890 seconds , u-avg: 0.3763644046007603, window_start_time: 1.88, window_start_iteration: 190, previous_collection_time: 1.89, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: false
[ Info: Iter: 192, time: 1.900 seconds , u-avg: 0.0, window_start_time: 1.9, window_start_iteration: 192, previous_collection_time: 1.9, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: true
[ Info: Iter: 193, time: 1.910 seconds , u-avg: 0.252777992933367, window_start_time: 1.9, window_start_iteration: 192, previous_collection_time: 1.91, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.89, collecting: false
[ Info: Iter: 194, time: 1.920 seconds , u-avg: 0.0, window_start_time: 1.92, window_start_iteration: 194, previous_collection_time: 1.92, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.89, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 195, time: 1.930 seconds , u-avg: 0.15331845599118626, window_start_time: 1.92, window_start_iteration: 194, previous_collection_time: 1.93, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.92, collecting: false
[ Info: Iter: 196, time: 1.940 seconds , u-avg: 0.0, window_start_time: 1.94, window_start_iteration: 196, previous_collection_time: 1.94, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.92, collecting: true
[ Info: Iter: 197, time: 1.950 seconds , u-avg: 0.07837831512778862, window_start_time: 1.94, window_start_iteration: 196, previous_collection_time: 1.95, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.95, collecting: false
[ Info: Iter: 198, time: 1.960 seconds , u-avg: 0.0, window_start_time: 1.96, window_start_iteration: 198, previous_collection_time: 1.96, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.95, collecting: true
[ Info: Iter: 199, time: 1.970 seconds , u-avg: 0.0282533248422682, window_start_time: 1.96, window_start_iteration: 198, previous_collection_time: 1.97, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.95, collecting: true
[ Info: Iter: 200, time: 1.980 seconds , u-avg: 0.020407752224732935, window_start_time: 1.96, window_start_iteration: 198, previous_collection_time: 1.98, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.98, collecting: false
[ Info: Iter: 201, time: 1.990 seconds , u-avg: 0.0, window_start_time: 1.99, window_start_iteration: 201, previous_collection_time: 1.99, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.98, collecting: true
[ Info: Simulation is stopping after running for 2.041 seconds.
[ Info: Simulation time 2 seconds equals or exceeds stop time 2 seconds.
[ Info: Iter: 202, time: 2 seconds , u-avg: 8.929976983568233e-15, window_start_time: 1.99, window_start_iteration: 201, previous_collection_time: 2.0, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.98, collecting: true
from oceananigans.jl.
What is going on here? I don't understand it but it looks wrong...
[ Info: Iter: 9, time: 90.000 ms , u-avg: 0.22639226913537686, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.08999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 10, time: 90 ms , u-avg: 0.22639226913537688, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.09, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: false
Maybe the actuations
are being counted incorrectly?
from oceananigans.jl.
@simone-silvestri, I think the problems arise earlier than the iterations you highlight, where zeros are written.
At first the averages are correctly being collected over three iteration, which makes sense because the interval is equal to three times the timestep.
[ Info: Iter: 1, time: 10 ms , u-avg: 0.0031413629825035438, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.01, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: Iter: 2, time: 20 ms , u-avg: 0.007851828677677537, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.02, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: Iter: 3, time: 30 ms , u-avg: 0.014652384734839867, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.03, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: false
But after the warnings it goes out of whack:
[ Info: Iter: 29, time: 270 ms , ... collecting: false
[ Info: Iter: 30, time: 280 ms , ... collecting: true
[ Info: Iter: 31, time: 290.000 ms , ... collecting: true
[ Info: Iter: 32, time: 300 ms , ... collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 33, time: 310 ms , ... collecting: false
[ Info: Iter: 34, time: 320 ms , ... collecting: true
[ Info: Iter: 35, time: 330.000 ms , ... collecting: false
[ Info: Iter: 36, time: 340.000 ms , ... collecting: true
[ Info: Iter: 37, time: 350 ms , ... collecting: false
It looks like between Iter: 33
and Iter: 35
it only collects the time average for 20 ms (whereas the interval is supposed to be 30 ms) and then writes out this incomplete average. Maybe this missing 10 ms is where the zeros get written in? I don't know.
from oceananigans.jl.
Not sure if it helps but the result is zero'd out here:
Oceananigans.jl/src/OutputWriters/windowed_time_average.jl
Lines 255 to 268 in 406eb9c
so when collecting
flips from false
to true
, the result should be 0 after
from oceananigans.jl.
@tomchor uses this or did at some point. Not sure who else has used it but would nice to tag them here to gather a good consensus on tests to write up.
from oceananigans.jl.
so when
collecting
flips fromfalse
totrue
, the result should be 0 after
Yeah, I think the problem is that collecting
is being flipped at the wrong time.
from oceananigans.jl.
@liuchihl, you tested that this bug is reproduced in which versions of Oceananigans again?
from oceananigans.jl.
@hdrake I am using v0.91.5, but v0.90.11 already has this issue, so it doesn't seem like a new problem.
from oceananigans.jl.
The problem is this iteration:
[ Info: Iter: 32, time: 300 ms , u-avg: 2.5442761423093008, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.3, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
We should have collecting: false
.
That means end_of_window
spuriously returns false
.
So we want to look at model.clock.time
and sch.previous_interval_stop_time + sch.interval
. It does seem probable the issue is roundoff error.
Probably AveragedTimeInterval
needs to be updated to align more with the (new) criteria currently being used for TimeInterval
eg we need a similar criteria as used for next_actuation_time
:
Oceananigans.jl/src/Utils/schedules.jl
Lines 56 to 61 in 406eb9c
Would be helpful to have a test too.
from oceananigans.jl.
Probably
AveragedTimeInterval
needs to be updated to align more with the (new) criteria currently being used forTimeInterval
@glwagner, are you working on this or would you like me and @liuchihl to take it on?
from oceananigans.jl.
It would be great if you are able to take this up! I can help of course. It will also be useful to have fresh eyes on this algorithm.
from oceananigans.jl.
I will try to figure it out.
from oceananigans.jl.
So we want to look at model.clock.time and sch.previous_interval_stop_time + sch.interval. It does seem probable the issue is roundoff error.
It does seem like a roundoff error, below is the progress message. At iter:64, for example, model.clock.time
is slightly larger than sch.previous_interval_stop_time + sch.interval
(which I think they should've been equal):
[ Info: Initializing simulation...
[ Info: Iter: 0, time: 0 seconds, model clock time:0.0, previous_interval_stop_time + interval:0.1, u-avg: 0.0, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time:0.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.0, collecting: true
[ Info: ... simulation initialization complete (231.038 ms)
[ Info: Executing initial time step...
[ Info: ... initial time step complete (546.837 ms).
[ Info: Iter: 11, time: 100 ms, model clock time:0.1, previous_interval_stop_time + interval:0.2, u-avg: 0.1202985845671951, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time:0.1, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.1, collecting: false
[ Info: Iter: 22, time: 200 ms, model clock time:0.2, previous_interval_stop_time + interval:0.30000000000000004, u-avg: 0.8057920910530502, window_start_time: 0.11, window_start_iteration: 12, previous_collection_time:0.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.2, collecting: false
[ Info: Iter: 32, time: 300.000 ms, model clock time:0.30000000000000004, previous_interval_stop_time + interval:0.4, u-avg: 2.0225593735128355, window_start_time: 0.21, window_start_iteration: 23, previous_collection_time:0.30000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.30000000000000004, collecting: false
[ Info: Iter: 42, time: 400 ms, model clock time:0.4, previous_interval_stop_time + interval:0.5, u-avg: 3.664512241688186, window_start_time: 0.3100000000000001, window_start_iteration: 33, previous_collection_time:0.4, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.4, collecting: false
[ Info: Iter: 52, time: 500 ms, model clock time:0.5, previous_interval_stop_time + interval:0.5, u-avg: 5.570924908683772, window_start_time: 0.4100000000000001, window_start_iteration: 43, previous_collection_time:0.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.4, collecting: false
[ Info: Iter: 64, time: 600.000 ms, model clock time:0.6000000000000001, previous_interval_stop_time + interval:0.6, u-avg: 0.0, window_start_time: 0.6000000000000001, window_start_iteration: 64, previous_collection_time:0.6000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.5, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 75, time: 700.000 ms, model clock time:0.7000000000000001, previous_interval_stop_time + interval:0.8, u-avg: 9.512220799868613, window_start_time: 0.6199999999999999, window_start_iteration: 66, previous_collection_time:0.7000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.7000000000000001, collecting: false
[ Info: Iter: 86, time: 800 ms, model clock time:0.8, previous_interval_stop_time + interval:0.9, u-avg: 10.99170409268473, window_start_time: 0.71, window_start_iteration: 76, previous_collection_time:0.8, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.8, collecting: false
[ Info: Iter: 97, time: 900 ms, model clock time:0.9, previous_interval_stop_time + interval:0.9, u-avg: 12.107573765048185, window_start_time: 0.8099999999999999, window_start_iteration: 87, previous_collection_time:0.9, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.8, collecting: false
[ Info: Iter: 108, time: 1 second, model clock time:1.0, previous_interval_stop_time + interval:1.0, u-avg: 12.691638724405168, window_start_time: 0.9299999999999997, window_start_iteration: 100, previous_collection_time:1.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.9, collecting: false
[ Info: Iter: 118, time: 1.100 seconds, model clock time:1.1, previous_interval_stop_time + interval:1.2000000000000002, u-avg: 12.56681992191321, window_start_time: 1.0300000000000007, window_start_iteration: 111, previous_collection_time:1.1, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.1, collecting: false
[ Info: Iter: 128, time: 1.200 seconds, model clock time:1.2000000000000002, previous_interval_stop_time + interval:1.3000000000000003, u-avg: 11.926603358212661, window_start_time: 1.1100000000000003, window_start_iteration: 119, previous_collection_time:1.2000000000000002, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.2000000000000002, collecting: false
[ Info: Iter: 139, time: 1.300 seconds, model clock time:1.3, previous_interval_stop_time + interval:1.3000000000000003, u-avg: 10.709836075752904, window_start_time: 1.2100000000000004, window_start_iteration: 129, previous_collection_time:1.3, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.2000000000000002, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 149, time: 1.400 seconds, model clock time:1.4000000000000001, previous_interval_stop_time + interval:1.5000000000000002, u-avg: 8.9791965740231, window_start_time: 1.3200000000000005, window_start_iteration: 141, previous_collection_time:1.4000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.4000000000000001, collecting: false
[ Info: Iter: 160, time: 1.500 seconds, model clock time:1.5, previous_interval_stop_time + interval:1.5000000000000002, u-avg: 7.161470540582019, window_start_time: 1.4100000000000004, window_start_iteration: 150, previous_collection_time:1.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.4000000000000001, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 170, time: 1.600 seconds, model clock time:1.6, previous_interval_stop_time + interval:1.7000000000000002, u-avg: 5.078554914290692, window_start_time: 1.5200000000000005, window_start_iteration: 162, previous_collection_time:1.6, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.6, collecting: false
[ Info: Iter: 180, time: 1.700 seconds, model clock time:1.7000000000000002, previous_interval_stop_time + interval:1.8000000000000003, u-avg: 3.309337820047445, window_start_time: 1.6100000000000003, window_start_iteration: 171, previous_collection_time:1.7000000000000002, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.7000000000000002, collecting: false
[ Info: Iter: 191, time: 1.800 seconds, model clock time:1.8, previous_interval_stop_time + interval:1.8000000000000003, u-avg: 1.7406913565812152, window_start_time: 1.7100000000000004, window_start_iteration: 181, previous_collection_time:1.8, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.7000000000000002, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 201, time: 1.900 seconds, model clock time:1.9000000000000001, previous_interval_stop_time + interval:2.0, u-avg: 0.5790445551477348, window_start_time: 1.8200000000000005, window_start_iteration: 193, previous_collection_time:1.9000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.9000000000000001, collecting: false
[ Info: Iter: 212, time: 2 seconds, model clock time:2.0, previous_interval_stop_time + interval:2.0, u-avg: 0.07095790120929331, window_start_time: 1.9100000000000004, window_start_iteration: 202, previous_collection_time:2.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.9000000000000001, collecting: false
[ Info: Iter: 223, time: 2.100 seconds, model clock time:2.1000000000000005, previous_interval_stop_time + interval:2.2, u-avg: 0.16557552735280148, window_start_time: 2.0299999999999994, window_start_iteration: 215, previous_collection_time:2.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.1, collecting: false
[ Info: Iter: 234, time: 2.200 seconds, model clock time:2.2, previous_interval_stop_time + interval:2.3000000000000003, u-avg: 0.8057920910533722, window_start_time: 2.1100000000000003, window_start_iteration: 224, previous_collection_time:2.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.2, collecting: false
[ Info: Iter: 245, time: 2.300 seconds, model clock time:2.3000000000000007, previous_interval_stop_time + interval:2.4000000000000004, u-avg: 2.0225593735131833, window_start_time: 2.21, window_start_iteration: 235, previous_collection_time:2.3000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.3000000000000003, collecting: false
[ Info: Iter: 256, time: 2.400 seconds, model clock time:2.4000000000000004, previous_interval_stop_time + interval:2.5000000000000004, u-avg: 3.6645122416885725, window_start_time: 2.3100000000000005, window_start_iteration: 246, previous_collection_time:2.4000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.4000000000000004, collecting: false
[ Info: Iter: 267, time: 2.500 seconds, model clock time:2.5, previous_interval_stop_time + interval:2.5000000000000004, u-avg: 5.570924908684207, window_start_time: 2.41, window_start_iteration: 257, previous_collection_time:2.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.4000000000000004, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 278, time: 2.600 seconds, model clock time:2.6000000000000005, previous_interval_stop_time + interval:2.7, u-avg: 7.6538405349756085, window_start_time: 2.5199999999999996, window_start_iteration: 269, previous_collection_time:2.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.6, collecting: false
[ Info: Iter: 289, time: 2.700 seconds, model clock time:2.7, previous_interval_stop_time + interval:2.8000000000000003, u-avg: 9.42305762921893, window_start_time: 2.6100000000000003, window_start_iteration: 279, previous_collection_time:2.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.7, collecting: false
[ Info: Iter: 300, time: 2.800 seconds, model clock time:2.8000000000000007, previous_interval_stop_time + interval:2.9000000000000004, u-avg: 10.991704092685225, window_start_time: 2.71, window_start_iteration: 290, previous_collection_time:2.8000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.8000000000000003, collecting: false
[ Info: Iter: 311, time: 2.900 seconds, model clock time:2.9000000000000004, previous_interval_stop_time + interval:3.0000000000000004, u-avg: 12.107573765048683, window_start_time: 2.8100000000000005, window_start_iteration: 301, previous_collection_time:2.9000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.9000000000000004, collecting: false
[ Info: Iter: 322, time: 3 seconds, model clock time:3.0, previous_interval_stop_time + interval:3.0000000000000004, u-avg: 12.66143754805721, window_start_time: 2.91, window_start_iteration: 312, previous_collection_time:3.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.9000000000000004, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 333, time: 3.100 seconds, model clock time:3.1000000000000005, previous_interval_stop_time + interval:3.2, u-avg: 12.583985175726626, window_start_time: 3.0199999999999996, window_start_iteration: 324, previous_collection_time:3.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.1, collecting: false
[ Info: Iter: 344, time: 3.200 seconds, model clock time:3.2, previous_interval_stop_time + interval:3.3000000000000003, u-avg: 11.926603358213136, window_start_time: 3.1100000000000003, window_start_iteration: 334, previous_collection_time:3.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.2, collecting: false
[ Info: Iter: 355, time: 3.300 seconds, model clock time:3.3000000000000007, previous_interval_stop_time + interval:3.4000000000000004, u-avg: 10.709836075753326, window_start_time: 3.21, window_start_iteration: 345, previous_collection_time:3.3000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.3000000000000003, collecting: false
[ Info: Iter: 366, time: 3.400 seconds, model clock time:3.4000000000000004, previous_interval_stop_time + interval:3.5000000000000004, u-avg: 9.067883207577935, window_start_time: 3.3100000000000005, window_start_iteration: 356, previous_collection_time:3.4000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.4000000000000004, collecting: false
[ Info: Iter: 377, time: 3.500 seconds, model clock time:3.5, previous_interval_stop_time + interval:3.5000000000000004, u-avg: 7.161470540582303, window_start_time: 3.41, window_start_iteration: 367, previous_collection_time:3.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.4000000000000004, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 388, time: 3.600 seconds, model clock time:3.6000000000000005, previous_interval_stop_time + interval:3.7, u-avg: 5.078554914290898, window_start_time: 3.5199999999999996, window_start_iteration: 379, previous_collection_time:3.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.6, collecting: false
[ Info: Iter: 399, time: 3.700 seconds, model clock time:3.7, previous_interval_stop_time + interval:3.7, u-avg: 3.309337820047581, window_start_time: 3.6100000000000003, window_start_iteration: 389, previous_collection_time:3.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.6, collecting: false
[ Info: Iter: 410, time: 3.800 seconds, model clock time:3.8000000000000007, previous_interval_stop_time + interval:3.9000000000000004, u-avg: 1.6002624850128424, window_start_time: 3.7299999999999995, window_start_iteration: 402, previous_collection_time:3.8000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.8000000000000003, collecting: false
[ Info: Iter: 421, time: 3.900 seconds, model clock time:3.9000000000000004, previous_interval_stop_time + interval:4.0, u-avg: 0.6248216842178228, window_start_time: 3.8100000000000005, window_start_iteration: 411, previous_collection_time:3.9000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.9000000000000004, collecting: false
[ Info: Iter: 432, time: 4 seconds, model clock time:4.0, previous_interval_stop_time + interval:4.0, u-avg: 0.07095790120929706, window_start_time: 3.91, window_start_iteration: 422, previous_collection_time:4.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.9000000000000004, collecting: false
[ Info: Iter: 443, time: 4.100 seconds, model clock time:4.1000000000000005, previous_interval_stop_time + interval:4.2, u-avg: 0.16557552735280162, window_start_time: 4.029999999999999, window_start_iteration: 435, previous_collection_time:4.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.1000000000000005, collecting: false
[ Info: Iter: 454, time: 4.200 seconds, model clock time:4.2, previous_interval_stop_time + interval:4.2, u-avg: 0.8057920910533768, window_start_time: 4.11, window_start_iteration: 444, previous_collection_time:4.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.1000000000000005, collecting: false
[ Info: Iter: 465, time: 4.300 seconds, model clock time:4.3, previous_interval_stop_time + interval:4.3, u-avg: 2.164461823304153, window_start_time: 4.2299999999999995, window_start_iteration: 457, previous_collection_time:4.3, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.2, collecting: false
[ Info: Iter: 476, time: 4.400 seconds, model clock time:4.4, previous_interval_stop_time + interval:4.5, u-avg: 3.8428643990560714, window_start_time: 4.329999999999999, window_start_iteration: 468, previous_collection_time:4.4, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.4, collecting: false
[ Info: Iter: 487, time: 4.500 seconds, model clock time:4.5, previous_interval_stop_time + interval:4.5, u-avg: 5.57092490868422, window_start_time: 4.41, window_start_iteration: 477, previous_collection_time:4.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.4, collecting: false
[ Info: Iter: 498, time: 4.600 seconds, model clock time:4.6000000000000005, previous_interval_stop_time + interval:4.7, u-avg: 7.7522019307475505, window_start_time: 4.529999999999999, window_start_iteration: 490, previous_collection_time:4.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.6000000000000005, collecting: false
[ Info: Iter: 509, time: 4.700 seconds, model clock time:4.7, previous_interval_stop_time + interval:4.7, u-avg: 9.42305762921895, window_start_time: 4.61, window_start_iteration: 499, previous_collection_time:4.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.6000000000000005, collecting: false
[ Info: Iter: 520, time: 4.800 seconds, model clock time:4.800000000000001, previous_interval_stop_time + interval:4.9, u-avg: 11.132132964253687, window_start_time: 4.7299999999999995, window_start_iteration: 512, previous_collection_time:4.800000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.800000000000001, collecting: false
[ Info: Iter: 531, time: 4.900 seconds, model clock time:4.9, previous_interval_stop_time + interval:5.0, u-avg: 12.10757376504871, window_start_time: 4.8100000000000005, window_start_iteration: 521, previous_collection_time:4.9, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.9, collecting: false
[ Info: Iter: 542, time: 5 seconds, model clock time:5.0, previous_interval_stop_time + interval:5.0, u-avg: 12.661437548057236, window_start_time: 4.91, window_start_iteration: 532, previous_collection_time:5.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.9, collecting: false
[ Info: Iter: 553, time: 5.100 seconds, model clock time:5.1000000000000005, previous_interval_stop_time + interval:5.2, u-avg: 12.566819921913737, window_start_time: 5.029999999999999, window_start_iteration: 545, previous_collection_time:5.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.1000000000000005, collecting: false
[ Info: Iter: 564, time: 5.200 seconds, model clock time:5.2, previous_interval_stop_time + interval:5.2, u-avg: 11.926603358213159, window_start_time: 5.11, window_start_iteration: 554, previous_collection_time:5.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.1000000000000005, collecting: false
[ Info: Iter: 575, time: 5.300 seconds, model clock time:5.300000000000001, previous_interval_stop_time + interval:5.4, u-avg: 10.567933625962374, window_start_time: 5.2299999999999995, window_start_iteration: 567, previous_collection_time:5.300000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.300000000000001, collecting: false
[ Info: Iter: 586, time: 5.400 seconds, model clock time:5.4, previous_interval_stop_time + interval:5.5, u-avg: 9.067883207577948, window_start_time: 5.3100000000000005, window_start_iteration: 576, previous_collection_time:5.4, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.4, collecting: false
[ Info: Iter: 597, time: 5.500 seconds, model clock time:5.5, previous_interval_stop_time + interval:5.5, u-avg: 7.1614705405823145, window_start_time: 5.41, window_start_iteration: 587, previous_collection_time:5.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.4, collecting: false
[ Info: Iter: 608, time: 5.600 seconds, model clock time:5.6000000000000005, previous_interval_stop_time + interval:5.7, u-avg: 4.980193518518983, window_start_time: 5.529999999999999, window_start_iteration: 600, previous_collection_time:5.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.6000000000000005, collecting: false
[ Info: Iter: 619, time: 5.700 seconds, model clock time:5.7, previous_interval_stop_time + interval:5.7, u-avg: 3.309337820047582, window_start_time: 5.61, window_start_iteration: 609, previous_collection_time:5.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.6000000000000005, collecting: false
[ Info: Iter: 630, time: 5.800 seconds, model clock time:5.800000000000001, previous_interval_stop_time + interval:5.9, u-avg: 1.6002624850128442, window_start_time: 5.7299999999999995, window_start_iteration: 622, previous_collection_time:5.800000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.800000000000001, collecting: false
[ Info: Iter: 641, time: 5.900 seconds, model clock time:5.9, previous_interval_stop_time + interval:6.0, u-avg: 0.6248216842178208, window_start_time: 5.8100000000000005, window_start_iteration: 631, previous_collection_time:5.9, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.9, collecting: false
[ Info: Simulation is stopping after running for 1.413 seconds.
from oceananigans.jl.
Probably AveragedTimeInterval needs to be updated to align more with the (new) criteria currently being used for TimeInterval eg we need a similar criteria as used for next_actuation_time:
@glwagner Could you maybe provide some hints on where the source code in AveragedTimeInterval should be modified to align with the implementation in TimeInterval? So far I haven't been able to fix the bug. My guess is something along these lines, but I could be wrong:
Oceananigans.jl/src/OutputWriters/windowed_time_average.jl
Lines 130 to 142 in 406eb9c
from oceananigans.jl.
Thanks a lot for your hint, I wanted to make sure what you mean by number of actuations as a criteria. Do you mean
first_actuation_time::Number
and do you think this will potentially prevent the floating point issue?
from oceananigans.jl.
yes, it makes sense to me now! Thanks for the clear explanation.
from oceananigans.jl.
Related Issues (20)
- `Mem.free` is deprecated HOT 1
- Add in documentation for user defined vertical diffusivity profile in the hydrostatic model HOT 3
- Documentation 2.0 HOT 3
- Egregious errors in `NonhydrostaticModel` simulations on `ImmersedBoundaryGrid` when nonhydrostatic pressure is not separated HOT 21
- `cell_advection_timescale` (and CFL) does not account for `Flat` topologies HOT 1
- Support passing keyword arguments to `jldopen` when using `FieldTimeSeries` HOT 1
- `PartialCellBottom` needs to be adapted correctly for the GPU when using a `HydrostaticFreeSurfaceModel` HOT 4
- Performance benchmarks section of the README is super out of date HOT 3
- `view` fails on a `Field` indexed from `FieldTimeSeries`
- Ocenanigans license HOT 5
- Reducing parameter space usage by `LatitudeLongitudeGrid` HOT 3
- Why do we need 4 halo points in the vertical for `VectorInvariant(vertical_scheme=Centered())`? HOT 2
- `xspacings` is broken for `ImmersedBoundaryGrid`
- Change free surface displacement name from `η` to `displacement` for `HydrostaticFreeSurfaceModel`
- Docstring for `WENOVectorInvariant` is too terse and does not have examples
- Bottom drag on immersed boundaries may differ if implemented with `Forcing` vs `ImmersedBoundaryCondition`
- Convert code blocks documenting `ImmersedBoundaryCondition` to doctests HOT 10
- Re-licensing Oceananigans under Apache 2.0
- Do `intrinsic_vector` and `extrinsic_vector` belong in `Operators`?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oceananigans.jl.