Build Issue On Lassen about lbann HOT 9 CLOSED

Hi Joey,

A recent Spack update is causing this issue in our build script. To get you unstuck until we update the build script, you can add the following line right before this one:
spack config update packages -y

Hopefully that helps! Either way, we can leave this issue open until this is fixed.

-Pier

from lbann.

Okay. Thank you! I'll give that a shot.

How do I start the build after modifying 'build_lbann.sh'? Activate the spack environment then rerun the 'build_lbann.sh' script?

from lbann.

Yep, you should just need to re-run build_lbann.sh the same as you did the first time.

from lbann.

That took the build further, but now 'nccl' is failing to build through spack. Have y'all seen that error before?

from lbann.

yeah, NVIDIA/nccl#835 is the root cause of that. Short of convincing LC to install modern compilers on the system, the best fix is to just force NCCL to build at a lower version. For the LBANN build script, add ^[email protected] to the end of the spec you pass to the build script.

from lbann.

Haha. The compilers strike again. Thanks a ton!

from lbann.

Thanks again for the help y'all! I got things up and running on lassen. Closing the issue.

from lbann.

@pate7 Great! Just an FYI, NCCL 1.18.3 was recently released, which fixes this. Not sure if Spack has it yet.

from lbann.

Cool. I may take a look at that. Thanks for the heads up!

BTW, spack only has up to version 0.102 of LBANN available. I'm not sure if this is the place to raise that issue or if we need to wait for spack to catch up to v0.103

from lbann.