Comments (4)
"Recent versions of MPI include an alternate interface that uses MPI_Count as the type for numbers of elements. In C, these have a _c appended to the name. For example,
int MPI_Send_c(const void *buf, MPI_Count count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm)
The intent of this interface is to handle exactly this issue.", Bill Gropp
This solution worked seamlessly by replacing MPI_Isend -> MPI_Isend_c and MPI_Irecv -> MPI_Irecv_c.
Lines 700 to 703 in 6d51166
As a test, below shows micro-benchmarking of point-to-point bandwidth across two nodes of Aurora. Message size is chosen as 16 GB.
******************** MPI COMMUNICATOR IS CREATED
printid: 0 Create Bench 0 with 24 processors
Port: SYCL, Library: dummy
printid: 0 Create Bench 1 with 24 processors
Port: SYCL, Library: MPI
Bench 1 comm 0 (0->12) sendbuf 0xff00000000200000 sendoffset 0 recvbuf 0xff000003b9e00000 recvoffset 0 count 16000000000 (16.0000 GB) MPI
CommBench 1: MPI communication matrix (reciever x sender) nnz: 1
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
1 . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
send footprint: 16000000000 16.0000 GB
recv footprint: 16000000000 16.0000 GB
5 warmup iterations (in order):
startup 2.20e+04 warmup: 7.27e+05
startup 6.82e+01 warmup: 6.88e+05
startup 6.52e+01 warmup: 6.88e+05
startup 6.42e+01 warmup: 6.88e+05
startup 6.38e+01 warmup: 6.88e+05
10 measurement iterations (sorted):
start: 6.2896e+01 time: 6.8806e+05 -> min
start: 6.3082e+01 time: 6.8813e+05
start: 6.3186e+01 time: 6.8814e+05
start: 6.3288e+01 time: 6.8818e+05
start: 6.3303e+01 time: 6.8826e+05
start: 6.3381e+01 time: 6.8834e+05 -> median
start: 6.3384e+01 time: 6.8836e+05
start: 6.3557e+01 time: 6.8840e+05
start: 6.3594e+01 time: 6.8842e+05
start: 6.3855e+01 time: 6.8872e+05 -> max
data: 16.0000 GB
minTime: 6.8806e+05 us, 4.3004e+01 ms/GB, 2.3254e+01 GB/s
medTime: 6.8834e+05 us, 4.3021e+01 ms/GB, 2.3244e+01 GB/s
maxTime: 6.8872e+05 us, 4.3045e+01 ms/GB, 2.3231e+01 GB/s
avgTime: 6.8830e+05 us, 4.3019e+01 ms/GB, 2.3246e+01 GB/s
from commbench.
The same code with _c API did not compile with the default MPI version on Frontier.
merth@frontier08929:~/HiCCL/collectives> make
CC -std=c++14 -fopenmp -I/opt/rocm-5.3.0/include -D__HIP_ROCclr__ -D__HIP_ARCH_GFX90A__=1 -x hip -O3 main.cpp -c -o main.o -craype-verbose
clang++ -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_HCC__ --offload-arch=gfx90a -D__HIP_ARCH_GFX90A__=1 -dynamic -D__CRAY_X86_TRENTO -D__CRAY_AMD_GFX90A -D__CRAYXT_COMPUTE_LINUX_TARGET --gcc-toolchain=/opt/cray/pe/gcc/10.3.0/snos -isystem /opt/cray/pe/cce/15.0.0/cce-clang/x86_64/lib/clang/15.0.2/include -isystem /opt/cray/pe/cce/15.0.0/cce/x86_64/include/craylibs -std=c++14 -fopenmp -I/opt/rocm-5.3.0/include -D__HIP_ROCclr__ -D__HIP_ARCH_GFX90A__=1 -x hip -O3 main.cpp -c -o main.o -I/opt/cray/pe/libsci/22.12.1.1/CRAY/9.0/x86_64/include -I/opt/cray/pe/mpich/8.1.23/ofi/cray/10.0/include -I/opt/cray/pe/dsmml/0.2.2/dsmml//include -I/opt/cray/pe/pmi/6.1.8/include -I/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/include
In file included from main.cpp:19:
In file included from ./../hiccl.h:24:
In file included from ./../CommBench/commbench.h:143:
./../CommBench/comm.h:701:11: error: use of undeclared identifier 'MPI_Isend_c'
MPI_Isend_c(sendbuf[send] + sendoffset[send], sendcount[send] * sizeof(T), MPI_BYTE, sendproc[send], 0, comm_mpi, &sendrequest[send]);
^
./../source/comm.h:190:34: note: in instantiation of member function 'CommBench::Comm<unsigned long>::start' requested here
commandptr[i]->comm->start();
^
./../source/bench.h:17:10: note: in instantiation of member function 'HiCCL::Comm<unsigned long>::run' requested here
comm.run();
^
main.cpp:174:12: note: in instantiation of function template specialization 'HiCCL::measure<unsigned long>' requested here
HiCCL::measure<Type>(warmup, numiter, count * numproc, coll);
^
In file included from main.cpp:19:
In file included from ./../hiccl.h:24:
In file included from ./../CommBench/commbench.h:143:
./../CommBench/comm.h:703:11: error: use of undeclared identifier 'MPI_Irecv_c'
MPI_Irecv_c(recvbuf[recv] + recvoffset[recv], recvcount[recv] * sizeof(T), MPI_BYTE, recvproc[recv], 0, comm_mpi, &recvrequest[recv]);
^
2 errors generated when compiling for gfx90a.
make: *** [Makefile:17: main.o] Error 1
merth@frontier08929:~/HiCCL/collectives>
from commbench.
Temporary fix as below:
Lines 290 to 296 in 5fe5914
Test across nodes on Frontier:
******************** MPI COMMUNICATOR IS CREATED
printid: 0 Create Bench 0 with 16 processors
Port: HIP, Library: dummy
printid: 0 Create Bench 1 with 16 processors
Port: HIP, Library: MPI
Bench 1 comm 0 (0->8) sendbuf 0x7ff726c00000 sendoffset 0 recvbuf 0x7fefb3400000 recvoffset 0 count 500000000 (2.0000 GB) MPI
Bench 1 comm 1 (0->8) sendbuf 0x7ff726c00000 sendoffset 500000000 recvbuf 0x7fefb3400000 recvoffset 500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 2 (0->8) sendbuf 0x7ff726c00000 sendoffset 1000000000 recvbuf 0x7fefb3400000 recvoffset 1000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 3 (0->8) sendbuf 0x7ff726c00000 sendoffset 1500000000 recvbuf 0x7fefb3400000 recvoffset 1500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 4 (0->8) sendbuf 0x7ff726c00000 sendoffset 2000000000 recvbuf 0x7fefb3400000 recvoffset 2000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 5 (0->8) sendbuf 0x7ff726c00000 sendoffset 2500000000 recvbuf 0x7fefb3400000 recvoffset 2500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 6 (0->8) sendbuf 0x7ff726c00000 sendoffset 3000000000 recvbuf 0x7fefb3400000 recvoffset 3000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 7 (0->8) sendbuf 0x7ff726c00000 sendoffset 3500000000 recvbuf 0x7fefb3400000 recvoffset 3500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 8 (0->8) sendbuf 0x7ff726c00000 sendoffset 4000000000 recvbuf 0x7fefb3400000 recvoffset 4000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 9 (0->8) sendbuf 0x7ff726c00000 sendoffset 4500000000 recvbuf 0x7fefb3400000 recvoffset 4500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 10 (0->8) sendbuf 0x7ff726c00000 sendoffset 5000000000 recvbuf 0x7fefb3400000 recvoffset 5000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 11 (0->8) sendbuf 0x7ff726c00000 sendoffset 5500000000 recvbuf 0x7fefb3400000 recvoffset 5500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 12 (0->8) sendbuf 0x7ff726c00000 sendoffset 6000000000 recvbuf 0x7fefb3400000 recvoffset 6000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 13 (0->8) sendbuf 0x7ff726c00000 sendoffset 6500000000 recvbuf 0x7fefb3400000 recvoffset 6500000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 14 (0->8) sendbuf 0x7ff726c00000 sendoffset 7000000000 recvbuf 0x7fefb3400000 recvoffset 7000000000 count 500000000 (2.0000 GB) MPI
Bench 1 comm 15 (0->8) sendbuf 0x7ff726c00000 sendoffset 7500000000 recvbuf 0x7fefb3400000 recvoffset 7500000000 count 500000000 (2.0000 GB) MPI
CommBench 1: MPI communication matrix (reciever x sender) nnz: 16
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
16 . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
send footprint: 8000000000 32.0000 GB
recv footprint: 8000000000 32.0000 GB
5 warmup iterations (in order):
startup 1.23e+05 warmup: 1.45e+06
startup 1.52e+02 warmup: 1.33e+06
startup 1.32e+02 warmup: 1.33e+06
startup 1.32e+02 warmup: 1.33e+06
startup 1.30e+02 warmup: 1.33e+06
10 measurement iterations (sorted):
start: 1.2909e+02 time: 1.3293e+06 -> min
start: 1.3013e+02 time: 1.3293e+06
start: 1.3034e+02 time: 1.3293e+06
start: 1.3047e+02 time: 1.3293e+06
start: 1.3150e+02 time: 1.3294e+06
start: 1.3205e+02 time: 1.3294e+06 -> median
start: 1.3240e+02 time: 1.3294e+06
start: 1.3276e+02 time: 1.3294e+06
start: 1.3539e+02 time: 1.3295e+06
start: 1.3580e+02 time: 1.3295e+06 -> max
data: 32.0000 GB
minTime: 1.3293e+06 us, 4.1541e+01 ms/GB, 2.4073e+01 GB/s
medTime: 1.3294e+06 us, 4.1543e+01 ms/GB, 2.4071e+01 GB/s
maxTime: 1.3295e+06 us, 4.1546e+01 ms/GB, 2.4070e+01 GB/s
avgTime: 1.3294e+06 us, 4.1543e+01 ms/GB, 2.4071e+01 GB/s
from commbench.
If Frontier doesn't support MPI_Isend_c, it doesn't have a valid MPI 4.0 library.
from commbench.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from commbench.