CPU-GPU copies instead of GPU-GPU MPI communications in cg_mpi ? about rocalution HOT 7 CLOSED

There is going to be a change soon, which will introduce multiple streams in order to overlap packing and sending with the interior computation. If you are using MPI, I guess you compile from source. I'll update this issue, once that change has been implemented, so you can re do your performance tests.

from rocalution.

Currently, rocALUTION does not support direct GPU-GPU communication. This is something we will be adding in a future release, however.

from rocalution.

Thanks for you quick answer.
Do you have an idea of when this feature will be supported ?

from rocalution.

I am not able to give you a specific release. May I ask why you are interested in such feature? I do not expect any performance improvement on most solvers, as time critical algorithms such as SpMV can fully hide communication by computation.

from rocalution.

I understand your point but the bench case we are working on shows that about 10% of the time of each rocALUTION solver call (1 per time step) is dedicated to hipMemcpy for these ghost cells exchange, with almost no computational kernels overlap. We use the CG solver with the c-amg preconditionner. It would have been nice to try this feature on AMD nodes specifically designed to do so. We will test it when it will be available :).

from rocalution.

Sounds a good feature to try, thanks !

from rocalution.

I have merged some major modifications to GlobalMatrix::Apply() in order to overlap communication and packing with interior matrix vector multiplication. This should slightly improve SpMV performance and scale linear with additional nodes / GPUs. Let me know, if this fixes the issue you have been observing.
It will still use the host for communication, but you should not see any communication related slow downs.
Note, that you will have to checkout develop branch, in order to see those changes.

from rocalution.