Question about ring performance between intra-node and inter-node about nccl HOT 7 CLOSED

My understanding is that since the performance with one GPU per node, it's not due to network limitations but rather limitations within the node itself, correct?

Correct.

Why does the addition of network cards result in decreased performance? Are there any optimization methods available?

I didn't get that. Where did you add network cards and it degraded performance?

from nccl.

Thank you for your reply!

My point is, why is there a performance drop when there are two GPUs per node(20GB/s) compared to one GPU per node(21.4GB)? As we discussed earlier, it could be assumed that this degradation due to the limitations within the node. What exactly causes this limitation? possibly PCIe switching performance?

However, performance testing within the node with 2/3/4 GPUs does not exhibit any degradation. From the perspective of a single server, compared to intra-node, inter-node simply adds another data stream from the NIC. If the degradation is due to switching performance, it should also happen during intra-node test.

Or could there be other reasons?

from nccl.

Yes it could be the PCI switch degrading performance a bit, but it's not a big degradation; it's unlikely to matter much in real apps.

Can you see if NCCL_NET_GDR_READ=1 gets better results?

from nccl.

Alright, thank you for your advice.

Our servers are busy now, I will try once they are free.

from nccl.

Thank you very much for your advice! It's really helpful.
Actually, I tried NCCL_NET_GDR_READ=0 and get better results (22GB/s). It seems that the default value in our environment is 1, although there is no NVLINK in my setup. Our NCCL version is 2.19.4.

from nccl.

Ah, indeed GDR Read was disabled by default only on Volta and before. Since Ampere, it is enabled by default. Disabling it may improve performance, but it will cause more traffic to the CPU and back, so in a real application, that could slow down the CPU-GPU exchanges and NCCL could also be slowed down by those transfers. So I would advise to make sure real applications do see a benefit when using this environment variable before setting it.

from nccl.

Got it. Thanks again.

from nccl.