New enhancement idea about hcipy HOT 3 CLOSED

I know that OCTOPUS (developed by ESO) also takes several days for the largest systems (https://www.eso.org/sci/facilities/develop/ao/tecno/octopus.html) to simulate. And this a is custom piece of software that has been developed to take advantage of multi-processing on large clusters. Even in those cases it still takes some time to run simulations.

We check if there are parts of the code that can be speed up, but usually it is not easy to create higher performance code. This is mainly because hcipy has more of a focused on flexibility and correctness. And depending on how you setup things it chooses the best method available which may not be the fastest method.

What are you trying to simulate with which parameter set? And have you tried to profile your code? Then we have an indication which part is actually slowing down the calculations and we could provide some suggestions.

from hcipy.

Indeed, HCIPy is focused on correctness and flexibility/ease-of-use, rather than speed. That being said, we take care to not sacrifice performance unnecessarily. In a typical application, most of the time (think >80%) is spent inside numpy functions, primarily np.dot(), which are highly optimized. Some specific examples of potential speed improvements that are on my radar:

• For the FraunhoferPropagator propagations, there might be a speedup with numba of about 10-20% based on experiences of people developing other packages. I tried replicating their benchmarks but didn't see the same improvement, so I never looked into this further. A GPU implementation would improve performance by ~3-5x. I had one student working on an implementation which indicated that integration into HCIPy was not as simple as we expected. I'm currently interested in an implemention using TensorFlow or pyTorch, which is super interesting for other reasons as well.
• For the FresnelPropagator propagations, there is a speedup of ~3x by removing unnecessary (i)fftshifts. This would be a relatively easy implementation. There could be another independent ~3x improvement when transitioning to a GPU implementation, but I never benchmarked this sufficiently well.

My own simulations are generally limited by Fraunhofer propagations, so at maximum a 3-5x speed improvement can be obtained. Your ATLAS/BLAS implementation is ~10-20% typically. Using Anaconda should install this correctly. Tuning your sampling can yield more improvement than these. Multiprocessing is something to look at if you are doing broadband simulations, though most BLAS implementations already try to use multithreading to some degree for speedup. I never looked closely at numba so it might be that I missed something there.

You should really do a profile to see what part of your simulation is actually limiting you. That gives you an indication where the most gain can be made.

from hcipy.

Last week I saw a presentation of compass (https://anr-compass.github.io/compass/index.html). This is a GPU accelerated end-to-end AO simulation. There is a python interface for compass. And I learned that there is interest in adding HCIPy functionality with compass. So we could always write a HCIPy interface to communicate with compass. The downside of compass is that it only works on LINUX+NVIDIA.

from hcipy.