Is there any documentation on what the following error codes mean, and what can cause them?
[g996eee][Sep 30 04:35:42 131457][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887003958052883] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
# mpirun -np 16 -map-by ppr:8:node -x UCX_TLS=dc,shm,self -x LD_LIBRARY_PATH -x NCCL_COLLNET_ENABLE=1 /opt/nccl_tests/build/all_reduce_perf -b 1G -e 2G -f 2 -g 1 -w 5 -n 1
Warning: Permanently added 'nccl-test-32-worker-0.nccl-test-32-worker.multus.svc,10.139.111.2' (ECDSA) to the list of known hosts.
Warning: Permanently added 'nccl-test-32-worker-3.nccl-test-32-worker.multus.svc,10.139.112.2' (ECDSA) to the list of known hosts.
Warning: Permanently added 'nccl-test-32-worker-1.nccl-test-32-worker.multus.svc,10.139.112.4' (ECDSA) to the list of known hosts.
Warning: Permanently added 'nccl-test-32-worker-2.nccl-test-32-worker.multus.svc,10.139.112.3' (ECDSA) to the list of known hosts.
# nThread 1 nGpus 1 minBytes 1073741824 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 1 validation: 1
#
# Using devices
# Rank 0 Pid 598 on g996eee device 0 [0x27] NVIDIA A100-SXM4-80GB
# Rank 1 Pid 599 on g996eee device 1 [0x2a] NVIDIA A100-SXM4-80GB
# Rank 2 Pid 600 on g996eee device 2 [0x51] NVIDIA A100-SXM4-80GB
# Rank 3 Pid 601 on g996eee device 3 [0x57] NVIDIA A100-SXM4-80GB
# Rank 4 Pid 602 on g996eee device 4 [0x9e] NVIDIA A100-SXM4-80GB
# Rank 5 Pid 603 on g996eee device 5 [0xa4] NVIDIA A100-SXM4-80GB
# Rank 6 Pid 604 on g996eee device 6 [0xc7] NVIDIA A100-SXM4-80GB
# Rank 7 Pid 605 on g996eee device 7 [0xca] NVIDIA A100-SXM4-80GB
# Rank 8 Pid 594 on g2f4e7c device 0 [0x27] NVIDIA A100-SXM4-80GB
# Rank 9 Pid 595 on g2f4e7c device 1 [0x2a] NVIDIA A100-SXM4-80GB
# Rank 10 Pid 596 on g2f4e7c device 2 [0x51] NVIDIA A100-SXM4-80GB
# Rank 11 Pid 597 on g2f4e7c device 3 [0x57] NVIDIA A100-SXM4-80GB
# Rank 12 Pid 598 on g2f4e7c device 4 [0x9e] NVIDIA A100-SXM4-80GB
# Rank 13 Pid 599 on g2f4e7c device 5 [0xa4] NVIDIA A100-SXM4-80GB
# Rank 14 Pid 600 on g2f4e7c device 6 [0xc7] NVIDIA A100-SXM4-80GB
# Rank 15 Pid 601 on g2f4e7c device 7 [0xca] NVIDIA A100-SXM4-80GB
[g996eee:0:598 - context.c:687] INFO job (ID: 35887902139160176) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:598 - context.c:875] INFO sharp_job_id:114 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:598 - context.c:886] INFO sharp_job_id:114 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee:0:598 - comm.c:392] INFO [group#:0] group id:0 tree idx:0 tree_type:LLT rail_idx:0 group size:2 quota: (osts:8 user_data_per_ost:1024) mgid: (subnet prefix:0xff12a01bfe800000 interface id:0x6f0000000000) mlid:c004
[g996eee:0:598 - comm.c:392] INFO [group#:1] group id:0 tree idx:1 tree_type:SAT rail_idx:0 group size:2 quota: (osts:64 user_data_per_ost:0) mgid: (subnet prefix:0x0 interface id:0x0) mlid:0
[g996eee:0:600 - context.c:687] INFO job (ID: 35887003958052883) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:115 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:115 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:42 131457][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887003958052883] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:42 125532][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887003958052883] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:602 - context.c:687] INFO job (ID: 35887114005571810) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:602 - context.c:875] INFO sharp_job_id:116 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:602 - context.c:886] INFO sharp_job_id:116 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee:0:602 - comm.c:392] INFO [group#:0] group id:0 tree idx:0 tree_type:LLT rail_idx:0 group size:2 quota: (osts:8 user_data_per_ost:1024) mgid: (subnet prefix:0xff12a01bfe800000 interface id:0x710000000000) mlid:c005
[g996eee:0:602 - comm.c:392] INFO [group#:1] group id:0 tree idx:1 tree_type:SAT rail_idx:0 group size:2 quota: (osts:64 user_data_per_ost:0) mgid: (subnet prefix:0x0 interface id:0x0) mlid:0
[g996eee:0:604 - context.c:687] INFO job (ID: 35887018084864365) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:604 - context.c:875] INFO sharp_job_id:117 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:604 - context.c:886] INFO sharp_job_id:117 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee:0:604 - comm.c:392] INFO [group#:0] group id:0 tree idx:0 tree_type:LLT rail_idx:0 group size:2 quota: (osts:8 user_data_per_ost:1024) mgid: (subnet prefix:0xff12a01bfe800000 interface id:0x720000000000) mlid:c006
[g996eee:0:604 - comm.c:392] INFO [group#:1] group id:0 tree idx:1 tree_type:SAT rail_idx:0 group size:2 quota: (osts:64 user_data_per_ost:0) mgid: (subnet prefix:0x0 interface id:0x0) mlid:0
[g996eee:0:600 - context.c:687] INFO job (ID: 35887004130201420) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:118 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:118 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:45 056431][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887004130201420] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:45 066747][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887004130201420] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887003132968255) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:119 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:119 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:46 307672][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887003132968255] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:46 318712][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887003132968255] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887003451118562) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:120 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:120 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:47 548419][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887003451118562] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:47 550832][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887003451118562] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887004604703053) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:121 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:121 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g2f4e7c][Sep 30 04:35:48 937799][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887004604703053] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee][Sep 30 04:35:48 955482][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887004604703053] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887003503035138) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:122 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:122 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:50 204167][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887003503035138] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:50 214760][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887003503035138] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887004402930063) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:123 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:123 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:51 519524][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887004402930063] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:51 530746][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887004402930063] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887004947744738) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:124 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:124 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:52 807063][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887004947744738] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:52 801270][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887004947744738] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887003900347484) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:125 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:125 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:54 048396][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c][Sep 30 04:35:54 042484][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887003900347484] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 unique id 35887003900347484] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887004364391853) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:126 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:126 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:55 305056][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887004364391853] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:55 299238][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887004364391853] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887004620413021) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:127 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:127 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:56 568313][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887004620413021] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:56 582732][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887004620413021] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:0:600 - context.c:687] INFO job (ID: 35887005092490615) resource request quota: ( osts:0 user_data_per_ost:0 max_groups:0 max_qps:1 max_group_channels:1, num_trees:1)
[g996eee:0:600 - context.c:875] INFO sharp_job_id:128 tree_type:LLT tree_idx:0 treeID:0x0 caps:0x6 quota:(osts:25 user_data_per_ost:1024 max_groups:25 max_qps:1 max_group_channels:1)
[g996eee:0:600 - context.c:886] INFO sharp_job_id:128 tree_type:SAT tree_idx:1 treeID:0x3f caps:0x16
[g996eee][Sep 30 04:35:57 916182][SD][703][error] - AM TreeConfig add children MAD response status 0x1c00
[g996eee:0:600 unique id 35887005092490615] ERROR AN MAD error in sharp_connect_tree.
[g996eee:0:600 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g2f4e7c][Sep 30 04:35:57 910354][SD][701][error] - AM TreeConfig add children MAD response status 0x1c00
[g2f4e7c:1:596 unique id 35887005092490615] ERROR AN MAD error in sharp_connect_tree.
[g2f4e7c:1:596 - comm.c:33] ERROR sharp_connect_tree failed: AN MAD error(-18)
[g996eee:600 :0:600] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x44b8)
#
# out-of-place in-place
# size count type redop time algbw busbw error time algbw busbw error
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
==== backtrace (tid: 600) ====
0 0x0000000000014420 __funlockfile() ???:0
1 0x000000000005ebc8 ncclGroupEnd() ???:0
2 0x000000000005ee95 ncclGroupEnd() ???:0
3 0x000000000005f33a ncclGroupEnd() ???:0
4 0x00000000000562e0 ncclGetUniqueId() ???:0
5 0x0000000000057eea ncclRedOpDestroy() ???:0
6 0x000000000005769c ncclRedOpDestroy() ???:0
7 0x000000000007fd93 ncclAllReduce() ???:0
8 0x00000000000052e6 AllReduceRunColl() /opt/nccl_tests/src/all_reduce.cu:57
9 0x0000000000007a51 startColl() /opt/nccl_tests/src/common.cu:563
10 0x0000000000009fb4 TimeTest() /opt/nccl_tests/src/common.cu:764
11 0x0000000000005174 AllReduceRunTest() /opt/nccl_tests/src/all_reduce.cu:103
12 0x0000000000005b2a threadRunTests() /opt/nccl_tests/src/common.cu:792
13 0x000000000000b6d3 run() /opt/nccl_tests/src/common.cu:1166
14 0x0000000000003fbf main() /opt/nccl_tests/src/common.cu:1007
15 0x0000000000024083 __libc_start_main() ???:0
16 0x0000000000004efe _start() ???:0
=================================
[g996eee:00600] *** Process received signal ***
[g996eee:00600] Signal: Segmentation fault (11)
[g996eee:00600] Signal code: (-6)
[g996eee:00600] Failing at address: 0x258
[g996eee:00600] [ 0] /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f0c8142f420]
[g996eee:00600] [ 1] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5ebc8)[0x7f0c8149cbc8]
[g996eee:00600] [ 2] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5ee95)[0x7f0c8149ce95]
[g996eee:00600] [ 3] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5f33a)[0x7f0c8149d33a]
[g996eee:00600] [ 4] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x562e0)[0x7f0c814942e0]
[g996eee:00600] [ 5] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x57eea)[0x7f0c81495eea]
[g996eee:00600] [ 6] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5769c)[0x7f0c8149569c]
[g996eee:00600] [ 7] /usr/lib/x86_64-linux-gnu/libnccl.so.2(ncclAllReduce+0xf3)[0x7f0c814bdd93]
[g996eee:00600] [ 8] /opt/nccl_tests/build/all_reduce_perf(+0x52e6)[0x55d4292c22e6]
[g996eee:00600] [ 9] /opt/nccl_tests/build/all_reduce_perf(+0x7a51)[0x55d4292c4a51]
[g996eee:00600] [10] /opt/nccl_tests/build/all_reduce_perf(+0x9fb4)[0x55d4292c6fb4]
[g996eee:00600] [11] /opt/nccl_tests/build/all_reduce_perf(+0x5174)[0x55d4292c2174]
[g996eee:00600] [12] /opt/nccl_tests/build/all_reduce_perf(+0x5b2a)[0x55d4292c2b2a]
[g996eee:00600] [13] /opt/nccl_tests/build/all_reduce_perf(+0xb6d3)[0x55d4292c86d3]
[g996eee:00600] [14] /opt/nccl_tests/build/all_reduce_perf(+0x3fbf)[0x55d4292c0fbf]
[g996eee:00600] [15] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f0c81069083]
[g996eee:00600] [16] /opt/nccl_tests/build/all_reduce_perf(+0x4efe)[0x55d4292c1efe]
[g996eee:00600] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[g2f4e7c:596 :0:596] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x44b8)
==== backtrace (tid: 596) ====
0 0x0000000000014420 __funlockfile() ???:0
1 0x000000000005ebc8 ncclGroupEnd() ???:0
2 0x000000000005ee95 ncclGroupEnd() ???:0
3 0x000000000005f33a ncclGroupEnd() ???:0
4 0x00000000000562e0 ncclGetUniqueId() ???:0
5 0x0000000000057eea ncclRedOpDestroy() ???:0
6 0x000000000005769c ncclRedOpDestroy() ???:0
7 0x000000000007fd93 ncclAllReduce() ???:0
8 0x00000000000052e6 AllReduceRunColl() /opt/nccl_tests/src/all_reduce.cu:57
9 0x0000000000007a51 startColl() /opt/nccl_tests/src/common.cu:563
10 0x0000000000009fb4 TimeTest() /opt/nccl_tests/src/common.cu:764
11 0x0000000000005174 AllReduceRunTest() /opt/nccl_tests/src/all_reduce.cu:103
12 0x0000000000005b2a threadRunTests() /opt/nccl_tests/src/common.cu:792
13 0x000000000000b6d3 run() /opt/nccl_tests/src/common.cu:1166
14 0x0000000000003fbf main() /opt/nccl_tests/src/common.cu:1007
15 0x0000000000024083 __libc_start_main() ???:0
16 0x0000000000004efe _start() ???:0
=================================
[g2f4e7c:00596] *** Process received signal ***
[g2f4e7c:00596] Signal: Segmentation fault (11)
[g2f4e7c:00596] Signal code: (-6)
[g2f4e7c:00596] Failing at address: 0x254
[g2f4e7c:00596] [ 0] /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fdf48964420]
[g2f4e7c:00596] [ 1] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5ebc8)[0x7fdf489d1bc8]
[g2f4e7c:00596] [ 2] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5ee95)[0x7fdf489d1e95]
[g2f4e7c:00596] [ 3] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5f33a)[0x7fdf489d233a]
[g2f4e7c:00596] [ 4] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x562e0)[0x7fdf489c92e0]
[g2f4e7c:00596] [ 5] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x57eea)[0x7fdf489caeea]
[g2f4e7c:00596] [ 6] /usr/lib/x86_64-linux-gnu/libnccl.so.2(+0x5769c)[0x7fdf489ca69c]
[g2f4e7c:00596] [ 7] /usr/lib/x86_64-linux-gnu/libnccl.so.2(ncclAllReduce+0xf3)[0x7fdf489f2d93]
[g2f4e7c:00596] [ 8] /opt/nccl_tests/build/all_reduce_perf(+0x52e6)[0x55bccdcad2e6]
[g2f4e7c:00596] [ 9] /opt/nccl_tests/build/all_reduce_perf(+0x7a51)[0x55bccdcafa51]
[g2f4e7c:00596] [10] /opt/nccl_tests/build/all_reduce_perf(+0x9fb4)[0x55bccdcb1fb4]
[g2f4e7c:00596] [11] /opt/nccl_tests/build/all_reduce_perf(+0x5174)[0x55bccdcad174]
[g2f4e7c:00596] [12] /opt/nccl_tests/build/all_reduce_perf(+0x5b2a)[0x55bccdcadb2a]
[g2f4e7c:00596] [13] /opt/nccl_tests/build/all_reduce_perf(+0xb6d3)[0x55bccdcb36d3]
[g2f4e7c:00596] [14] /opt/nccl_tests/build/all_reduce_perf(+0x3fbf)[0x55bccdcabfbf]
[g2f4e7c:00596] [15] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fdf4859e083]
[g2f4e7c:00596] [16] /opt/nccl_tests/build/all_reduce_perf(+0x4efe)[0x55bccdcacefe]
[g2f4e7c:00596] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 600 on node nccl-test-32-worker-0.nccl-test-32-worker.multus.svc exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------