Cmake fails to find the installed lapack on Lassen about hiop HOT 19 CLOSED

@nychiang, could you please provide output from when you run make VERBOSE=1? Also, what branch are you using?

from hiop.

I started from the master branch.
Here I attached two log files, from commands CMake and make.
log1.txt
log2.txt

from hiop.

Have you tried building with newer version of GCC? You are building with 4.9.3 which is pretty old. I haven't tested HiOp with that.

from hiop.

from hiop.

@nychiang if I look at your Cmake configuration

-- Found LAPACK libraries: /usr/lib64/libessl.so;/usr/lib64/libblas.so;-lpthread;-lm;-ldl

and the linking line in your build log that fails

/usr/tcetmp/bin/c++ -L/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-2020.08.19/lib -pthread CMakeFiles/nlpDenseCons_ex1.exe.dir/nlpDenseCons_ex1.cpp.o CMakeFiles/nlpDenseCons_ex1.exe.dir/nlpDenseCons_ex1_driver.cpp.o -o nlpDenseCons_ex1.exe  -Wl,-rpath,/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib ../../libhiop.a /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/libmpiprofilesupport.so /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib/libmpi_ibm.so /usr/tce/packages/gcc/gcc-4.9.3/gnu/lib64/libgomp.so -lpthread /usr/lib64/libessl.so /usr/lib64/libblas.so -lpthread -lm -ldl /usr/lib64/libessl.so /usr/lib64/libblas.so -lpthread -lm -ldl /g/g92/chiang7/software/HSL/MA57/lib/libma57.a /g/g92/chiang7/software/METIS_4/lib/libmetis.a -lgfortran 
../../libhiop.a(hiopKKTLinSys.cpp.o): In function `hiop::hiopKKTLinSysLowRank::solveWithRefin(hiop::hiopMatrixDense&, hiop::hiopVector&)':
hiopKKTLinSys.cpp:(.text+0x726c): undefined reference to `dposvx_'
../../libhiop.a(hiopKKTLinSys.cpp.o): In function `hiop::hiopKKTLinSysLowRank::solve(hiop::hiopMatrixDense&, hiop::hiopVector&)':
hiopKKTLinSys.cpp:(.text+0x7c24): undefined reference to `dposvx_'
collect2: error: ld returned 1 exit status

I find that Lapack libraries CMake found seem to be correctly linked.

I am curious what linker flags did you hard wire to make this work?

CC @ashermancinelli

from hiop.

To make it work, I just add the path to lapack into CMakeList.txt:
set(LAPACK_LIBRARIES "-lgfortran;/usr/lib64/liblapack.so;-lpthread;-lm;-ldl")

from hiop.

It seems to me that the problem was in the ESSL module, which is the Lapack implementation you chose. You are hard-wiring to the Lapack version that was shipped with the OS on your machine. Try unloading any Lapack module, commenting out the line

set(LAPACK_LIBRARIES "-lgfortran;/usr/lib64/liblapack.so;-lpthread;-lm;-ldl")

and re-running CMake. That should give you the same linking options as those you specified manually.

from hiop.

No, it doesn't work. If I don't hard-wiring to liblapack.so, the CMake command
find_package(LAPACK REQUIRED)
will find ESSL as the default lapack module.
See the log files from a clean installation and the modules I used, i.e., the latest version of CMake and GCC on Lassen.
log2.txt
log1.txt

from hiop.

In that case, we just need to cache LAPACK_LIBRARIES variable, so that user can change it in case the system configuration leads CMake to an undesired library. I would think CMake can do very little if ESSL is on your system library path before the Lapack library you actually want to use.

You could also configure your build with -DLAPACK_LIBRARIES="-lgfortran;/usr/lib64/liblapack.so;-lpthread;-lm;-ldl"; I think that should work, as well.

from hiop.

from hiop.

apparently, dposvx is not present in libessl.so, at least not on lassen.llnl.gov. This is the root cause of the problem. It will compile with liblapack, but we may want to use libessl for (possible) better performance. The good news is that the GPU-based code does not need dposvx, so we have the option to use libessl (by not building the quasi-Newton solver, ex1, ex2, and ex3, and assemblying newton-ipm, mds, ex4, and ex5 objects separately; requires changes to the build system).

from hiop.

a temporary fix is as suggested by @pelesh: use cmake with -DLAPACK_LIBRARIES="-lgfortran;/usr/lib64/libblas.so;/usr/lib64/liblapack.so;-lpthread;-lm;-ldl" (make sure you include blas lib, otherwise won't link)

and use the fix for the cmake build system from ef195e0

from hiop.

Another solution: link the noconflict routines before LAPACK, i.e.,
-lesslnoconflict -llapack

This will use the optimized versions of the LAPACK routines that are in ESSL, and use liblapack.a for the ones which are not in ESSL.

from hiop.

interesting, that would be ideal, indeed.

can that be used with cmake with -DLAPACK_LIBRARIES="....." or CMakeLists.txt needs to be edited?

from hiop.

Defining with cmake as you've suggested would work, though it would override any discovery of lapack libraries cmake might otherwise do. If you're sure you can find all the libraries needed that would be a good solution 😄

from hiop.

Defining with cmake as you've suggested would work, though it would override any discovery of lapack libraries cmake might otherwise do. If you're sure you can find all the libraries needed that would be a good solution 😄

@ashermancinelli it might be a good idea to cache LAPACK_LIBRARIES so that user can fix it if user's input was incorrect or wrong library was found.

from hiop.

Defining with cmake as you've suggested would work, though it would override any discovery of lapack libraries cmake might otherwise do. If you're sure you can find all the libraries needed that would be a good solution 😄

this manual override is a last resort

from hiop.

Another solution: link the noconflict routines before LAPACK, i.e.,
-lesslnoconflict -llapack

This will use the optimized versions of the LAPACK routines that are in ESSL, and use liblapack.a for the ones which are not in ESSL.

but you need to unpack libessl manually, right? or is -lesslnoconflict available on lassen?

from hiop.

I tested it. It is NOT available on lassen, but this is the command given by LC.
However, I use the command -lessl -llapack (inspired by the strumpack user manual), and it works fine.
Passed all the tests, i.e., EX1 - EX5.

from hiop.