Build on windows (with mingw) about leela-zero HOT 30 CLOSED

Download all the dependencies, follow all their build instructions until you get everything compiled and end up with the required set of libraries and includes.

Then point the leela-zero Makefile to the relevant directories and libs you just built in the aforementioned step, and run make (mingw) or make a MSVC console project, add the source files and set up all the library and include directories in MSVC.

The latter part could be a bit easier if we'd add CMake support or so, but realistically the difficult part is the first.

from leela-zero.

I am using MinGW. I managed to build all other dependencies, but it seems difficult to build OpenCL ICD loader...(I can build OpenCL ICD loader using MSVC but I want to use MinGW)

from leela-zero.

That appears to be a known issue: KhronosGroup/OpenCL-ICD-Loader#11

from leela-zero.

Then how do you manage to build leelaz on windows? using MSVC?

from leela-zero.

I used MSVC to build the ICD Loader. I used mingw for some other parts (IIRC getting a good performance out of OpenBLAS requires mingw, although our use is not so performance critical).

Final binary can be built with MSVC or mingw, I used MSVC.

from leela-zero.

when i use mingw to build leelaz, at the link step, it claims that my OpenCL.dll which was built with MSVC is "incompatible"

from leela-zero.

You should be linking with the .lib (that matches the dll), not the dll itself.

from leela-zero.

the same
skipping incompatible lib/OpenCL.lib when searching for -lOpenCL

from leela-zero.

Some Googling suggests this is a common problem, there's a few proposals on the mingw wiki:
http://www.mingw.org/wiki/MSVC_and_MinGW_DLLs, i.e. add the .lib as an object file, not a library, or by using pexports + dlltool.

See also:
https://sourceforge.net/p/mingw-w64/wiki2/gendef/
https://sourceforge.net/p/mingw-w64/wiki2/Answer%20generation%20of%20DLL%20import%20library/

from leela-zero.

By the way, the issue filed against OpenCL ICD suggests that a 64-bit mingw build should work.

from leela-zero.

If someone finds a way to successfully build and run on windows, it would be really great if they could provide step by step instructions in the readme. I took a crack at it, but got lost.
I installed the latest version of mingw-w64 from http://mingw-w64.org/doku.php/download#mingw-builds. There are several versions for Windows. I guessed and took Msys2 because it seems to have the latest of everything. After running the commands to set it up, I tried gcc, but the command was not recognized.
I downloaded boost from https://dl.bintray.com/boostorg/release/1.65.1/source/
In the msys64 shell, I unpacked it to some location, then ran ./bootstrap.sh from that location but it did not work because it could not find cc.
Instead of saying that we should run make (mingw) or make a MSVC console project - please give one set of consistent instructions that work end to end.
I don't have much experience building c++ apps on windows, but that's where my nividia card is and I would really like to experiment with this project!

from leela-zero.

I don't have much experience building c++ apps on windows, but that's where my nividia card is and I would really like to experiment with this project!

There are pre-built binaries on the releases page: https://github.com/gcp/leela-zero/releases, there is no need to build the software yourself if you don't know how to do it. It is pretty complicated on Windows even if you understand what you are doing.

from leela-zero.

Thanks for the pointer to the pre-built binaries. I would still like to build on windows so I can make changes and perhaps contribute.
I got a little further. In the mingw console I installed and did the following
pacman -S mingw64/mingw-w64-x86_64-gcc
pacman -S mingw64/mingw-w64-x86_64-boost
pacman -S mingw64/mingw-w64-x86_64-openblas
pacman -S mingw64/mingw-w64-x86_64-opencl-headers
pacman -S mingw64/mingw-w64-x86_64-cmake
cmake -G "MSYS Makefiles"
-- The C compiler identification is GNU 7.2.0
-- The CXX compiler identification is GNU 7.2.0
pacman -S git
pacman -S make
pacman -S vim
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader.git
install Windows SDK version 8.1 from https://developer.microsoft.com/en-us/windows/downloads/windows-8-1-sdk
cd OpenCL-ICD-Loader
make
E:/apps/msys64/home/barry/OpenCL-ICD-Loader/icd_dispatch.h:334:38: error: expected '=', ',', ';', 'asm' or 'attribute' before 'CL_API_SUFFIX__VERSION_2_2'
const void* spec_value) CL_API_SUFFIX__VERSION_2_2;
<maybe there are source changes that need to be made before it can be built with the latest version of gcc?>
cd ..
git clone https://github.com/barrybecker4/leela-zero.git
cd leela-zero/src
make
Network.cpp:41:10: fatal error: cblas.h: No such file or directory

from leela-zero.

When building for Windows you cannot just build the loader by running "make", see the build instructions for the ICD loader. (The start of this thread also has some discussion that building the loader does not work with mingw, though a 64 bit build as you are doing might work anyway - I have never tried)

from leela-zero.

I have created a pull request to allow building under Visual Studio. Once this is merged there shouldn't be any trouble to build on Windows. You only need to install Visual Studio 2015 Community.

from leela-zero.

Thanks for adding that. I will give it a try once its merged.

from leela-zero.

It almost works for me, I can build Leela itself, but autogtp fails during linking. It seems to have something to do with Qt? I have Qt 5.9.3 as well as the Qt VS Tools, and the project is configured to use the msvc2017_64 version. As far as I can tell, that's all correct. I've put curl.exe and gzip.exe from the pre-built version into the msvc folder - before it failed because it was missing them.

1>------ Build started: Project: autogtp, Configuration: Debug x64 ------
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppBuild.targets(387,5): warning MSB8028: The intermediate directory (x64\Debug) contains files shared from another project (leela-zero.vcxproj). This can lead to incorrect clean and rebuild behavior.
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): warning MSB8017: A circular dependency has been detected while executing custom build commands for item "..\curl.exe". This may cause incremental build to work incorrectly.
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): warning MSB8017: A circular dependency has been detected while executing custom build commands for item "..\gzip.exe". This may cause incremental build to work incorrectly.
1>Moc'ing Production.h...
1>Moc'ing Validation.h...
1>Game.cpp
1>main.cpp
1>Production.cpp
1>SPRT.cpp
1>Validation.cpp
1>moc_Production.cpp
1>moc_Validation.cpp
1>Generating Code...
1>Validation.obj : error LNK2001: unresolved external symbol "public: void __cdecl Results::addGameResult(enum Sprt::GameResult,int)" (?addGameResult@Results@@QEAAXW4GameResult@Sprt@@h@Z)
1>Validation.obj : error LNK2001: unresolved external symbol "public: void __cdecl Results::printResults(class QString const &,class QString const &)const " (?printResults@Results@@QEBAXAEBVQString@@0@Z)
1>C:\Users\Christian\Documents\LZ_compile\leela-zero\msvc\x64\Debug\autogtp.exe : fatal error LNK1120: 2 unresolved externals
1>Done building project "autogtp.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

from leela-zero.

@odeint I will try a clean build to see if I can replicate. Meanwhile, can you try to physical clean (manually delete all output and intermediate folders and try? or this is your first build?

from leela-zero.

It was a clean build. I tried now with my other computer, completely clean build, Win10 instead of 7 though. Same result, I just left the earlier stuff in this time. Which version of Qt are you using?

1>------ Build started: Project: leela-zero, Configuration: Debug x64 ------
1>Zobrist.cpp
1>Utils.cpp
1>UCTSearch.cpp
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(71): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(72): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(82): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(85): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(107): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(110): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>UCTNode.cpp
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(71): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(72): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(82): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(85): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(107): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(110): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\uctnode.cpp(381): warning C4244: 'initializing': conversion from 'double' to 'float', possible loss of data
1>TTable.cpp
1>Training.cpp
1>c:\users\patro\downloads\leela-zero-next\src\training.cpp(70): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\training.cpp(136): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\training.cpp(138): warning C4244: '=': conversion from 'double' to 'float', possible loss of data
1>Timing.cpp
1>TimeControl.cpp
1>SMP.cpp
1>SGFTree.cpp
1>SGFParser.cpp
1>Random.cpp
1>OpenCL.cpp
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(71): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(72): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(82): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(85): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(107): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(110): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.cpp(738): warning C4244: '+=': conversion from 'float' to 'int', possible loss of data
1>Network.cpp
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(71): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(72): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(82): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(85): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(107): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\opencl.h(110): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\network.cpp(455): warning C4267: 'argument': conversion from 'size_t' to 'const int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\network.cpp(499): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\network.cpp(272): warning C4267: 'argument': conversion from 'size_t' to 'const int', possible loss of data
1>c:\users\patro\downloads\leela-zero-next\src\network.cpp(429): note: see reference to function template instantiation 'void convolve<1,2>(const std::vector<net_t,std::allocator<_Ty>> &,const std::vector<_Ty,std::allocator<_Ty>> &,const std::vector<_Ty,std::allocator<_Ty>> &,std::vector<_Ty,std::allocator<_Ty>> &)' being compiled
1> with
1> [
1> _Ty=net_t
1> ]
1>c:\users\patro\downloads\leela-zero-next\src\network.cpp(291): warning C4267: 'argument': conversion from 'size_t' to 'const blasint', possible loss of data
1>Leela.cpp
1>KoState.cpp
1>GTP.cpp
1>GameState.cpp
1>FullBoard.cpp
1>FastState.cpp
1>Generating Code...
1>Compiling...
1>FastBoard.cpp
1>Generating Code...
1>leela-zero.vcxproj -> C:\Users\patro\Downloads\leela-zero-next\msvc\x64\Debug\leela-zero.exe
1>leela-zero.vcxproj -> C:\Users\patro\Downloads\leela-zero-next\msvc\x64\Debug\leela-zero.pdb (Partial PDB)
1>Done building project "leela-zero.vcxproj".
2>------ Build started: Project: autogtp, Configuration: Debug x64 ------
2>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppBuild.targets(387,5): warning MSB8028: The intermediate directory (x64\Debug) contains files shared from another project (leela-zero.vcxproj). This can lead to incorrect clean and rebuild behavior.
2>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): warning MSB8017: A circular dependency has been detected while executing custom build commands for item "..\curl.exe". This may cause incremental build to work incorrectly.
2>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): warning MSB8017: A circular dependency has been detected while executing custom build commands for item "..\gzip.exe". This may cause incremental build to work incorrectly.
2>Moc'ing Production.h...
2>Moc'ing Validation.h...
2>Performing Custom Build Tools
2> 1 file(s) copied.
2>Performing Custom Build Tools
2> 1 file(s) copied.
2>Game.cpp
2>main.cpp
2>Production.cpp
2>SPRT.cpp
2>Validation.cpp
2>moc_Production.cpp
2>moc_Validation.cpp
2>Generating Code...
2>Validation.obj : error LNK2001: unresolved external symbol "public: void __cdecl Results::addGameResult(enum Sprt::GameResult,int)" (?addGameResult@Results@@QEAAXW4GameResult@Sprt@@h@Z)
2>Validation.obj : error LNK2001: unresolved external symbol "public: void __cdecl Results::printResults(class QString const &,class QString const &)const " (?printResults@Results@@QEBAXAEBVQString@@0@Z)
2>C:\Users\patro\Downloads\leela-zero-next\msvc\x64\Debug\autogtp.exe : fatal error LNK1120: 2 unresolved externals
2>Done building project "autogtp.vcxproj" -- FAILED.
========== Build: 1 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

from leela-zero.

I've tried building with Intel C++ 17.0 instead, that gets rid of most of the warnings, but linking is still done by MSVC and fails again. [ had to follow this guide to have Boost again, "vc141" instead of "iw", https://software.intel.com/en-us/articles/intel-c-compiler-for-windows-fatal-link-error-lnk1104-when-using-intel-c-compiler-with-boost-libraries ]. The error message is only slightly more verbose in giving the name of the function call that failed:

1>------ Build started: Project: leela-zero, Configuration: Debug x64 ------
1>FastBoard.cpp
1>FastState.cpp
1>FullBoard.cpp
1>GameState.cpp
1>GTP.cpp
1>KoState.cpp
1>Leela.cpp
1>Network.cpp
1>OpenCL.cpp
1>Random.cpp
1>SGFParser.cpp
1>SGFTree.cpp
1>SMP.cpp
1>TimeControl.cpp
1>Timing.cpp
1>Training.cpp
1>TTable.cpp
1>UCTNode.cpp
1>UCTSearch.cpp
1>Utils.cpp
1>Zobrist.cpp
2>------ Build started: Project: autogtp, Configuration: Debug x64 ------
2>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppBuild.targets(387,5): warning MSB8028: The intermediate directory (x64\Debug) contains files shared from another project (leela-zero.vcxproj). This can lead to incorrect clean and rebuild behavior.
2>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): warning MSB8017: A circular dependency has been detected while executing custom build commands for item "..\curl.exe". This may cause incremental build to work incorrectly.
2>C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): warning MSB8017: A circular dependency has been detected while executing custom build commands for item "..\gzip.exe". This may cause incremental build to work incorrectly.
2>Moc'ing Production.h...
2>Moc'ing Validation.h...
2>Performing Custom Build Tools
2> 1 file(s) copied.
2>Performing Custom Build Tools
2> 1 file(s) copied.
2>Game.cpp
2>main.cpp
2>Production.cpp
2>SPRT.cpp
2>Validation.cpp
2>moc_Production.cpp
2>moc_Validation.cpp
2>Validation.obj : error LNK2019: unresolved external symbol "public: void __cdecl Results::addGameResult(enum Sprt::GameResult,int)" (?addGameResult@Results@@QEAAXW4GameResult@Sprt@@h@Z) referenced in function "public: void __cdecl Validation::getResult(enum Sprt::GameResult,int)" (?getResult@Validation@@QEAAXW4GameResult@Sprt@@h@Z)
2>Validation.obj : error LNK2019: unresolved external symbol "public: void __cdecl Results::printResults(class QString const &,class QString const &)const " (?printResults@Results@@QEBAXAEBVQString@@0@Z) referenced in function "public: void __cdecl Validation::getResult(enum Sprt::GameResult,int)" (?getResult@Validation@@QEAAXW4GameResult@Sprt@@h@Z)
2>C:\Users\patro\Downloads\leela-zero-next\msvc\x64\Debug\autogtp.exe : fatal error LNK1120: 2 unresolved externals
2>Done building project "autogtp.vcxproj" -- FAILED.
========== Build: 1 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

from leela-zero.

Those warnings are normal and exists in my own build as well. The build command has been adjusted since my first build so some errors occurs here.

But I have no idea about the real errors you experiencing. I have to make my own clean build when I have time tonight.

Advises before I can look into it:

Figure out which source file contains the functions that were missing. It might suppose to be generated by Qt though. Or more likely, they are in a cpp file that is missing in the compile. In this case, simply add that cpp file to the project will solve the problem.

from leela-zero.

I can get it to compile (but probably doesn't do what it should) by commenting out these two lines in Validation.cpp:

Line 167:
//m_results.addGameResult(result, net_one_color);

Line 179:
//m_results.printResults(m_firstNet, m_secondNet);

So it seems to be some problem with that m_results object.

Edit: It actually does seem to work for normal autogtp, I guess it never needs to run the stuff from Validation.cpp - I'll just use it and see if anything's different.

from leela-zero.

Yesterday I looked into it and see that Result.cpp is missing in the project. So this is why you get this error. The build command to copy curl.exe and gzip.exe also have problems. I have fixed already but the commit still have something else to fix.

I am not available tonight, so wait for tomorrow. A new pull request will solve all these.

The proper workaround right now, is manually add Result.cpp to the project. It should compile since then.

from leela-zero.

Hey @earthengine , it is so obvious now! Didn't notice because the headers were also not in the project. Should have compared what's in the folder to whats in the MSVC project... Well that's why I'm not good at C++ ;-)

Edit: And yes I periodically have to copu curl and gzip into the folder again after a rebuild or clean. Not sure how to fix that. And copy a few dlls into the final folder, too.

from leela-zero.

Tried on my other machine, it also works there now. Both Visual C++ and ICC can compile. I don't see a speed difference. Let's see if there will be different results. Would it be worth it to recompile some of the dlls as well?

from leela-zero.

@odeint Because the custom action for curl.exe and gzip.exe have "Output"s to be "$(Identity)", which means they link to the same file, so the files get deleted before the action can be performed. That's my fault in the first place.

To solve it, go to the property of those two exe files, click "Custom Action" in the left, then change the output from "$(Identity)" to "$(OutDir)curl.exe" or "$(OutDir)gzip.exe", according to which file are you looking at.

from leela-zero.

Would it be worth it to recompile some of the dlls as well?

Not so much recompiling but replacing OpenBLAS by MKL will help. (see config.h)

from leela-zero.

@gcp Roger, did it, built it with ICC and #define USE_MKL instead of #define USE_OPENBLAS in leelaz's config.h

Benchmarking now, feels maybe 20% faster than the latest stock release but I'll measure it. The libraries used are
mkl_intel_thread.dll
mkl_core.dll
mkl_avx2.dll

Edit: Stock gets around 3000ms per move, IntelMKL around 2350ms per move, with two concurrent autogtps running. PC is a i5-7500 and a GTX 1060 6GB.

Edit2: Running only the new MKL version with -g 2 reduces it to 1075ms per move. I guess that's already summing the two instances, so comparable to 2150ms per move in the old scenario.

from leela-zero.

Probably this issue can be closed now that it is possible to build on Windows using Visual Studio 2015 or 2017.

from leela-zero.

closing old techsupport issue with proposed solution

from leela-zero.