Boundary condition ordering about oceananigans.jl HOT 7 OPEN

Another solution, slightly more complex but that allows us to keep the current execution model is to separate the sides only on communicating boundary conditions

from oceananigans.jl.

Ok, and to clarify, the issue is cropping up now because we are trying to distribute a model / grid along a Bounded direction. If we only distribute in Periodic directions, this problem never crops up. But if we distributed in a Bounded direction, then we run into the issue where one "side" may have a bounded condition like flux, value, gradient, and the other side may be communicating.

If we don't consider distributed problems, this issue never occurs: either both sides are Periodic, or bounded.

from oceananigans.jl.

exactly this is the issue with communication that pops up also in shared cases (multi-region)

from oceananigans.jl.

So how do you eliminate ordering requirements? (Why do we have ordering requirements in the first place?)

from oceananigans.jl.

ordering requirements are necessary for filling corner halos.
This is done by Periodic boundary conditions in non-distributed simulations.
Additionally, since communication boundary conditions can be asynchronous, distributed (and multi-region) BCS need to be filled last.

To remove order requirements we would need to fill the halo for flux, value, and gradient also in the corners.
I thought that might be a good idea but we hit a problem when having an AbstractArray boundary condition because we would need to construct the associated OffsetArray.

This can be prevented by wrapping the array in a Field and filling the halo regions but it seems like a heavy requirement to do it, and generally, a large API change that we might want to think about a little more. In #3338 I fixed the problem by separating out communicating boundary conditions which wasn't that complicated and maintained the current logic.

Maybe in the future, we might want to eliminate the order requirement though. So we can keep this issue open.

from oceananigans.jl.

Why don't we discontinue support for array boundary conditions and only support Field?

from oceananigans.jl.

It's better practice to use Field, because then user scripts are more likely to port to GPU and distributed architectures. I think we want to discourage using Array --- if it makes development easier, we might as well discontinue support altogether.

from oceananigans.jl.