OMP pragmas for higher core counts about nrpytutorial HOT 1 CLOSED

Hi Gonçalo!

Great question. In fact in our old (now deprecated) SENR code (https://bitbucket.org/zach_etienne/nrpy/src/master/; which didn't have reference metric precomputation [rfm_precompute]) I indeed found that moving the OMP pragma from the i2 loop to the i1 or i0 loops did improve the performance. However, as a warning it wasn't an earth-shattering performance boost! In other words it might be the case that the compiler/OpenMP is being cleverer than we think about parallelizing the outermost loop, or we're just not scaling as well as we hope.

I've had a few ideas that could be tried with the current version of NRPy+, I think the top one should be tried first in fact (though I'm split on whether 2 or 3 should be next):

1. Disable rfm_precompute and benchmark the performance with the omp pragma hand placed both in the outer loop and the inner loops. What is the speedup when we move to the inner loops? If it's <10% for example, then we might want to rethink proceeding further.
2. Create an option to permute the spherical or cylindrical coordinates. Other than in some diagnostics, there should be no hard-coded assumption that e.g., spherical is (xx0,xx1,xx2)=(r,theta,phi), instead of (xx0,xx1,xx2)=(theta,phi,r); notice the latter is also a right-handed coordinate system. A spherical coordinate system with such permuted coordinates may scale far better in OpenMP loops.
3. Fix the problem you are seeing with rfm_precompute. I think it may be possible to simply move the pragma from just before the i2 for() statement to just before the i1 for() statement by hand if necessary. If that works, we could add an option to loop.py to specify the placement of the #pragma omp parallel.

from nrpytutorial.