I had to comment out CheckDelta in the test files to make them work ... what are the implications of this? I am extremely new to C++ and building via Visual Studio (like, a few days of experience), so I do not have strong debugging skills! But please let me know your thoughts. Additionally, when I use task manager to track GPU activity, I only see my original GPU doing everything in terms of processing. The new, 2nd GPU is showing copying activity, and VRAM utilization increases, so I know something is working. I wonder if it is not doing any processing because of the commented-out code! I highly appreciate any help, I bought a 2nd 3090, a beefy PSU, an NVLink, and redid all my wiring for my Windows system without knowing that cross GPU communication is not straightforward on Windows! Your repo gives me hope though.
If there is anything I can provide, please let me know. There really isn't much of an error code, it just says:
(base) [User]\GitHub\NCCL\windows\test\single\x64\Debug>all_reduce_test 300000000
Using devices
Rank 0 uses device 0 [0x24] NVIDIA GeForce RTX 3090
Rank 1 uses device 1 [0x2d] NVIDIA GeForce RTX 3090
out-of-place in-place
bytes N type op time algbw busbw res time algbw busbw res
300000000 300000000 char sumCuda failure [User]\GitHub\NCCL\test\include\test_utilities.h:313 'invalid argument'
Which I traced back to the CheckDelta calls in the test codes.