Building the CombBLAS 2.0 release on Linux (Debian unstable) with OpenMPI 4.1.5, 4 out of 20 fail, running via ctest (/usr/bin/ctest --force-new-ctest-process --verbose -j8
),
test 8
Start 8: Indexing_Test
8: Test command: /usr/bin/mpiexec "-n" "4" "/projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest" "../TESTDATA" "B_100x100.txt" "B_10x30_Indexed.txt" "rand10outta100.txt" "rand30outta100.txt"
8: Working Directory: /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests
8: Test timeout computed to be: 1500
8: Indexing working correctly
8: Elements stored on proc 0: {(0,0.234), (1,0.829), (2,0.221), (3,0.454), (4,0.096), (5,0.399), (6,0.895), (7,0.156), (8,0.052), (9,0.709), (10,0.305), (11,0.669), (12,0.493), (13,0.619), (14,0.736), (15,0.615), (16,0.124), (17,0.831), (18,0.958), (19,0.284), (20,0.411), (21,0.473
8: Elements stored on proc 1: {(0,0.196), (1,0.571), (2,0.482), (3,0.09), (4,0.79), (5,0.939), (6,0.684), (7,0.465), (8,0.236), (9,0.713), (10,0.32), (11,0.748), (12,0.771), (13,0.123), (14,0.79), (15,0.06), (16,0.82), (17,0.506), (18,0.859), (19,0.268), (20,0.49), (21,0.01), (22,0
8: Elements stored on proc 2: {(0,0.159), (1,0.811), (2,0.198), (3,0.163), (4,0.779), (5,0.241), (6,0.623), (7,0.955), (8,0.258), (9,0.861), (10,0.104), (11,0.381), (12,0.657), (13,0.356), (14,0.083), (15,0.712), (16,0.413), (17,0.488), (18,0.646), (19,0.99), (20,0.523), (21,0.034)
8: Elements stored on proc 3: {(0,0.609), (1,0.557), (2,0.926), (3,0.481), (4,0.218), (5,0.92), (6,0.049), (7,0.052), (8,0.424), (9,0.214), (10,0.606), (11,0.385), (12,0.848), (13,0.583), (14,0.586), (15,0.615), (16,0.797), (17,0.48), (18,0.378), (19,0.66), (20,0.169), (21,0.258),
8: COMBBLAS Warning: It is dangerous to create (vector) objects without specifying the communicator, are you sure you want to create this object in MPI_COMM_WORLD?
8: COMBBLAS Warning: It is dangerous to create (vector) objects without specifying the communicator, are you sure you want to create this object in MPI_COMM_WORLD?
8: [sandy:2061479] *** An error occurred in MPI_Alltoallv
8: [sandy:2061479] *** reported by process [2022768641,1]
8: [sandy:2061479] *** on communicator MPI COMMUNICATOR 3 DUP FROM 0
8: [sandy:2061479] *** MPI_ERR_TRUNCATE: message truncated
8: [sandy:2061479] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
8: [sandy:2061479] *** and potentially your MPI job)
8: [sandy:2061523] *** Process received signal ***
8: [sandy:2061523] Signal: Segmentation fault (11)
8: [sandy:2061523] Signal code: Address not mapped (1)
8: [sandy:2061523] Failing at address: 0x78a63cc0
8: [sandy:2061523] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x3c510)[0x7fa666c5a510]
8: [sandy:2061523] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x152449)[0x7fa666d70449]
8: [sandy:2061523] [ 2] /usr/lib/x86_64-linux-gnu/libopen-pal.so.40(opal_convertor_pack+0xaf)[0x7fa6670ae0df]
8: [sandy:2061523] [ 3] /usr/lib/x86_64-linux-gnu/libmpi.so.40(ompi_datatype_sndrcv+0x1fe)[0x7fa66729055e]
8: [sandy:2061523] [ 4] /usr/lib/x86_64-linux-gnu/libmpi.so.40(ompi_coll_base_alltoallv_intra_basic_linear+0x2bf)[0x7fa6672decaf]
8: [sandy:2061523] [ 5] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_coll_tuned.so(ompi_coll_tuned_alltoallv_intra_dec_fixed+0x42)[0x7fa6649e6fa2]
8: [sandy:2061523] [ 6] /usr/lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Alltoallv+0x1b5)[0x7fa667293315]
8: [sandy:2061523] [ 7] /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest(+0x1b151)[0x562012e18151]
8: [sandy:2061523] [ 8] /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest(_ZN8combblas11SpParHelper13KeyValuePSortIdiiEESt6vectorISt4pairIT_T0_ESaIS6_EEPS6_T1_PSA_RKP19ompi_communicator_t+0x3bb)[0x562012e3ac7b]
8: [sandy:2061523] [ 9] /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest(_ZN8combblas14FullyDistSpVecIidE4sortEv+0x284)[0x562012e3af34]
8: [sandy:2061523] [10] /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest(_Z4TopKIidESt4pairIN8combblas12FullyDistVecIT_S3_EENS2_IS3_T0_EEERNS1_14FullyDistSpVecIS3_S5_EES3_+0x247)[0x562012e3b3a7]
8: [sandy:2061523] [11] /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest(main+0xa2c)[0x562012e12ebc]
8: [sandy:2061523] [12] /lib/x86_64-linux-gnu/libc.so.6(+0x276ca)[0x7fa666c456ca]
8: [sandy:2061523] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fa666c45785]
8: [sandy:2061523] [14] /projects/mathlibs/build/combblas/obj-x86_64-linux-gnu/ReleaseTests/IndexingTest(_start+0x21)[0x562012e13981]
8: [sandy:2061523] *** End of error message ***
8: [sandy:2061354] 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
8: [sandy:2061354] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
3/20 Test #8: Indexing_Test ....................***Failed 20.56 sec