Giter Club home page Giter Club logo

fishtest's Introduction

Overview

CI server CI worker posix CI worker msys2 CI lint

Fishtest is a distributed task queue for testing chess engines. The main instance for testing the chess engine Stockfish is at this web page https://tests.stockfishchess.org

Developers submit patches with new ideas and improvements, CPU contributors install a fishtest worker on their computers to play some chess games in the background to help the developers testing the patches.

The fishtest worker:

  • Automatically connects to the server to download a chess opening book, the cutechess-cli chess game manager and the chess engine sources (both for the current Stockfish and for the patch with the new idea). The sources will be compiled according to the type of the worker platform.
  • Starts a batch of games using cutechess-cli.
  • Uploads the games results on the server.

The fishtest server:

  • Manages the queue of the tests with customizable priorities.
  • Computes several probabilistic values from the game results sent by the workers.
  • Updates and publishes the results of ongoing tests.
  • Knows how to stop tests when they are statistically significant and publishes the final tests results.

To get more information, such as the worker/server install and configuration instructions, visit the Fishtest Wiki.

fishtest's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fishtest's Issues

ValueError: No JSON object could be decoded -> Too many failed update attempts -> Task exited

see the output below, my question is:

does that mean that I just wasted all those 762 calculations? or were they transmitted to your server ?

version used: Worker version 59 connecting to http://tests.stockfishchess.org:80

...
Started game 762 of 1000 (base vs stockfish)
Finished game 762 (base vs stockfish): 0-1 {Black wins by adjudication}
Score of stockfish vs base: 101 - 102 - 559 [0.499] 762

Exception from calling update_task:
Traceback (most recent call last):
File "/erkan_yilmaz/fishtest-master/worker/games.py", line 256, in run_game
req = requests.post(remote + '/api/update_task', data=json.dumps(result), headers={'Content-type': 'application/json'}, timeout=HTTP_TIMEOUT).json()
File "/erkan_yilmaz/fishtest-master/worker/requests/models.py", line 604, in json
return json.loads(self.text or self.content)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Started game 763 of 1000 (stockfish vs base)
Finished game 763 (stockfish vs base): 1-0 {White wins by adjudication}
Score of stockfish vs base: 102 - 102 - 559 [0.500] 763

Exception from calling update_task:
Traceback (most recent call last):
File "/erkan_yilmaz/fishtest-master/worker/games.py", line 256, in run_game
req = requests.post(remote + '/api/update_task', data=json.dumps(result), headers={'Content-type': 'application/json'}, timeout=HTTP_TIMEOUT).json()
File "/erkan_yilmaz/fishtest-master/worker/requests/models.py", line 604, in json
return json.loads(self.text or self.content)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Started game 764 of 1000 (base vs stockfish)
Finished game 764 (base vs stockfish): 1-0 {White wins by adjudication}
Score of stockfish vs base: 102 - 103 - 559 [0.499] 764

Exception from calling update_task:
Traceback (most recent call last):
File "/erkan_yilmaz/fishtest-master/worker/games.py", line 256, in run_game
req = requests.post(remote + '/api/update_task', data=json.dumps(result), headers={'Content-type': 'application/json'}, timeout=HTTP_TIMEOUT).json()
File "/erkan_yilmaz/fishtest-master/worker/requests/models.py", line 604, in json
return json.loads(self.text or self.content)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Started game 765 of 1000 (stockfish vs base)
Finished game 765 (stockfish vs base): 1/2-1/2 {Draw by adjudication}
Score of stockfish vs base: 102 - 103 - 560 [0.499] 765

Exception from calling update_task:
Traceback (most recent call last):
File "/erkan_yilmaz/fishtest-master/worker/games.py", line 256, in run_game
req = requests.post(remote + '/api/update_task', data=json.dumps(result), headers={'Content-type': 'application/json'}, timeout=HTTP_TIMEOUT).json()
File "/erkan_yilmaz/fishtest-master/worker/requests/models.py", line 604, in json
return json.loads(self.text or self.content)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Started game 766 of 1000 (base vs stockfish)
Finished game 766 (base vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs base: 102 - 103 - 561 [0.499] 766

Exception from calling update_task:
Traceback (most recent call last):
File "/erkan_yilmaz/fishtest-master/worker/games.py", line 256, in run_game
req = requests.post(remote + '/api/update_task', data=json.dumps(result), headers={'Content-type': 'application/json'}, timeout=HTTP_TIMEOUT).json()
File "/erkan_yilmaz/fishtest-master/worker/requests/models.py", line 604, in json
return json.loads(self.text or self.content)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Started game 767 of 1000 (stockfish vs base)
Finished game 767 (stockfish vs base): 1/2-1/2 {Draw by adjudication}
Score of stockfish vs base: 102 - 103 - 562 [0.499] 767

Exception from calling update_task:
Traceback (most recent call last):
File "/erkan_yilmaz/fishtest-master/worker/games.py", line 256, in run_game
req = requests.post(remote + '/api/update_task', data=json.dumps(result), headers={'Content-type': 'application/json'}, timeout=HTTP_TIMEOUT).json()
File "/erkan_yilmaz/fishtest-master/worker/requests/models.py", line 604, in json
return json.loads(self.text or self.content)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Too many failed update attempts
Task exited

Exception accessing host:
Traceback (most recent call last):
File "worker.py", line 57, in worker
req = json.loads(req.text)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

....
SNIP by me -- it compiles here new versions ---
....

make[1]: Leaving directory '/tmp/tmpdw50Te/snicolet-Stockfish-eccccba/src'
Verifying signature of stockfish ...
Verifying signature of base ...
CPU factor : 0.956448 - tc adjusted to 9.56+0.10
Running supported_passer vs master
['/erkan_yilmaz/fishtest-master/worker/testing/cutechess-cli', '-repeat', '-rounds', '1000', '-tournament', 'gauntlet', '-pgnout', 'results.pgn', '-srand', '3106097062', '-resign', 'movecount=3', 'score=400', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '1', '-openings', u'file=2moves_v1.pgn', u'format=pgn', 'order=random', 'plies=16', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=4', '-engine', 'name=base', 'cmd=base', u'option.Hash=4', '-each', 'proto=uci', 'tc=9.56+0.10', 'option.Threads=1']('TC limit', 47822.406710440104, 'End time:', datetime.datetime%282016, 10, 3, 9, 6, 7, 538090))
Indexing opening suite...
Started game 1 of 1000 (stockfish vs base)
Finished game 1 (stockfish vs base): 1-0 {White wins by adjudication}
Score of stockfish vs base: 1 - 0 - 0 [1.000] 1
Started game 2 of 1000 (base vs stockfish)
Finished game 2 (base vs stockfish): 1-0 {White wins by adjudication}
Score of stockfish vs base: 1 - 1 - 0 [0.500] 2

Minimum Nps?

I am just starting to help on Fishtest, but the more I look into how it is done, the less I understand why you need to calculate the speed of the processor (core) before allowing it to run.

Now I don't know much about chess programming, but I would like to understand why I read stuff like 'game quality' being less on slower processor vs faster ones (or also in the case of HT being On or Off).

Isn't the tests run on our machines there to validate that new committed code hasn't broke anything, and that those tests' results are already known in advance?

Thanks to clarify this for me. And if this is not the right place to ask this, please tell me where.

Regards

Docker Image

As I can see, many users have problems with setup the test suite, especially Windows Users. Thinking on simplicity, if this users had a Docker Image, it would be more easy to run with a single command (or click on Docker Toolbox with Kitematic) the fishtest suite.

I just opened this issue, but in fact I've done the image already and published at docker hub: https://hub.docker.com/r/marcelorodrigo/fishtest/

If you like, I can submit a pull request to update the README.md including instructions in a Docker section to run the tests on a container.

Windows compile error

misc.cpp:32:23: error: typedef 'fun1_t' is initialized (use decltype instead)
typedef bool(*fun1_t)(LOGICAL_PROCESSOR_RELATIONSHIP,

CPU rescaling of increment

Not rescaling the increment was supposed to be a feature, but I'm pretty sure it's a bug. Two reasons:

Reason 1:
machines who's speed is far away from the benchmark 1.6Mnps. Example:
http://tests.stockfishchess.org/tests/view/525d5a1f0ebc5951f87d2215

In this test it is clear from the chi2 results that Pyafue-1cores is the black sheap. And when I look at long TC tests, there is nothing indicating that this machine has more noise than the others. It behaves normally at long TC.

With 0.28 Mnps, it has a CPU scaling factor such that we get the equivalent of a 1.6 Mnps machine running at 15+0.0087 instead of 15+0.05. This makes a measurable difference at short tc, where the increment is not negligible compared to the base time.

Reason 2:
testing time management patches. when we do that we need to test the following scenarios, to cover the various possibilities:

  • big time small increment (eg. 60+0.05)
  • small time big increment (eg. 30+0.5)
  • tournament time control (eg. 40/40)

Because increment is rescaled and we have a whole blend of machines with different NPS, the above explanation should make it clear that it is not possible to do such tests reliably (time/inc ratio completely different between machines). Also it will introduce more discussions in the forum, as chi2 tests will fail convincingly on such tests: people will try to find culprits among the machines, instead of looking at the real bug which is that fishtest does not rescale the increment.

Test breaks verifying signature

I keep getting the error below when trying to run Stefano80-Stockfish-c3d2e6a

Verifying signature of stockfish ...

Exception running games:
Traceback (most recent call last):
  File "fishtest-master/worker/worker.py", line 88, in worker
  File "/home/fishtest/fishtest-master/worker/games.py", line 409, in run_games
    base_nps = verify_signature(new_engine, run['args']['new_signature'], remote, result, games_concurrency * threads)
  File "/home/fishtest/fishtest-master/worker/games.py", line 91, in verify_signature
    busy_process.stdin.write('quit\n')
IOError: [Errno 32] Broken pipe
Task exited

It is not the first time that it happens. Any thoughts?

Add checks to see if all required libs are installed.

I started running the fishtest on a fresh installed virtual machine, running debian 8.0

I noticed that under testing, i got:

fishtest-master/worker/testing/cutechess-cli: error while loading shared libraries: libQtCore.so.4: cannot open shared object file: No such file or directory
Task exited

After installing libqt4-core it seems to run fine.

I guess this should be handled somehow.

auto-purge is broken

The chi-square test indicator is the pvalue, not the max residual.

Using the max residual isn't equivalent, and it's dangerously biased towards over-purging tests with many workers.

With the current purge based on max residual > 2.7, you're purging far too much. I don't know if it's possible to see on the server side in the logs, but I bet half the tests are purged.

Urgent

I want to replace the repo in a test
All my attempts failed
I want you to solve

The old repo is still stuck in the column

I wish the honorable people (only) replied to the topic
I do not want people who do not have humanity

THANK'S

Wrong bench by 1 user stops test

This feature is fine except that if only 1 user has wrong bench, the test should not stop right away.

For months there is 2 or 3 workers which constantly stops tests because using the signature from the run they just finished !

Exception: Non-zero return code: 127

I recently updated my linux system to the latest (sorta unstable) version of Gnome 3.12 and I git-cloned a fresh version of fishtest and tried to run the worker but I get the following error message:

Verifying signature of stockfish ...
Verifying signature of base ...
CPU factor : 0.972763 - tc adjusted to 58.37+0.05
Running candidate_passed vs master
['/home/infinity/fishtest/worker/testing/cutechess-cli', '-repeat', '-rounds', '1000', '-tournament', 'gauntlet', '-pgnout', 'results.pgn', '-resign', 'movecount=3', 'score=400', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '6', '-openings', u'file=8moves_v3.pgn', u'format=pgn', 'order=random', 'plies=16', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=128', u'option.OwnBook=false', '-engine', 'name=base', 'cmd=base', u'option.Hash=128', u'option.OwnBook=false', '-each', 'proto=uci', 'option.Threads=1', 'tc=58.37+0.05']
/home/infinity/fishtest/worker/testing/cutechess-cli: error while loading shared libraries: libQtCore.so.4: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/home/infinity/fishtest/worker/games.py", line 317, in launch_cutechess
    raise Exception('Non-zero return code: %d' % (p.returncode))
Exception: Non-zero return code: 127
Task exited

Any ideas?

worker FAIL on Core2 DUO

Platform: Windows 7 64-bit (Core2 DUO 4GB)
I have followed very carefully the instructions for fishtest installation on a Windows machine

Worker is launched from batch file:

@echo on
SET PATH=C:\MinGW\mingw64\bin;C:\MinGW\msys\1.0\bin;%PATH%
C:\Python27\python.exe -i worker.py --concurrency 1 MyUsername MyPassword

Cmd.exe outputs the following:

C:\fishtest-master\worker>SET PATH=C:\MinGW\mingw64\bin;C:\MinGW\msys\1.0\bin;C:
\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\NVIDIA Corporation\Phys
X\Common;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\Syst
em32\WindowsPowerShell\v1.0\;C:\Program Files\Microsoft SQL Server\110\Tools\Bin
n\;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\;C:\Program Files\Micros
oft SQL Server\120\Tools\Binn\;C:\Program Files\AMI\DuOS\;C:\Program Files (x86)
\Skype\Phone\;C:\Program Files (x86)\GNU\GnuPG\pub

C:\fishtest-master\worker>C:\Python27\python.exe -i worker.py --concurrency 1 My
Username MyPassword
Worker version 59 connecting to http://tests.stockfishchess.org:80
mingw32-make ARCH=x86-64-modern COMP=mingw config-sanity
mingw32-make[1]: Entering directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcq
f/pb00068-Stockfish-f588826/src'

Config:
debug: 'no'
optimize: 'yes'
arch: 'x86_64'
bits: '64'
prefetch: 'yes'
popcnt: 'yes'
sse: 'yes'
pext: 'no'

Flags:
CXX: g++
CXXFLAGS: -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -Wextra -Wshad
ow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT
LDFLAGS: -static

Testing config sanity. If this fails, try 'make help' ...

mingw32-make[1]: Leaving directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcqf
/pb00068-Stockfish-f588826/src'

Step 0/4. Preparing for profile build.
mingw32-make ARCH=x86-64-modern COMP=mingw gcc-profile-prepare
mingw32-make[1]: Entering directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcq
f/pb00068-Stockfish-f588826/src'
mingw32-make ARCH=x86-64-modern COMP=mingw gcc-profile-clean
mingw32-make[2]: Entering directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcq
f/pb00068-Stockfish-f588826/src'
mingw32-make[2]: Leaving directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcqf
/pb00068-Stockfish-f588826/src'
mingw32-make[1]: Leaving directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcqf
/pb00068-Stockfish-f588826/src'

Step 1/4. Building executable for benchmark ...
mingw32-make ARCH=x86-64-modern COMP=mingw gcc-profile-make
mingw32-make[1]: Entering directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcq
f/pb00068-Stockfish-f588826/src'
mingw32-make ARCH=x86-64-modern COMP=mingw \
EXTRACXXFLAGS='-fprofile-generate' \
EXTRALDFLAGS='-lgcov' \
all
mingw32-make[2]: Entering directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcq
f/pb00068-Stockfish-f588826/src'
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o benchmark.o benchmark.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o bitbase.o bitbase.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o bitboard.o bitboard.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o endgame.o endgame.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o evaluate.o evaluate.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o main.o main.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o material.o material.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o misc.o misc.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o movegen.o movegen.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o movepick.o movepick.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o pawns.o pawns.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o position.o position.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o psqt.o psqt.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o search.o search.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o thread.o thread.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o timeman.o timeman.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o tt.o tt.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o uci.o uci.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o ucioption.o ucioption.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -W
extra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -c -
o syzygy/tbprobe.o syzygy/tbprobe.cpp
syzygy/tbprobe.cpp: In function 'int probe_ab(Position&, int, int, int*)':
syzygy/tbprobe.cpp:205:14: warning: array subscript is above array bounds [-Warr
ay-bounds]
p[i++] = pop_lsb(&bb) ^ mirror;
^
syzygy/tbprobe.cpp: In function 'int Tablebases::probe_dtz(Position&, int*)':
syzygy/tbprobe.cpp:315:14: warning: array subscript is above array bounds [-Warr
ay-bounds]
p[i++] = pop_lsb(&bb) ^ mirror;
^
g++ -o stockfish benchmark.o bitbase.o bitboard.o endgame.o evaluate.o main.o ma
terial.o misc.o movegen.o movepick.o pawns.o position.o psqt.o search.o thread.o
timeman.o tt.o uci.o ucioption.o syzygy/tbprobe.o -lgcov -static
mingw32-make[2]: Leaving directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcqf
/pb00068-Stockfish-f588826/src'
mingw32-make[1]: Leaving directory 'c:/users/9dec~1/appdata/local/temp/tmpp8rcqf
/pb00068-Stockfish-f588826/src'

Step 2/4. Running benchmark for pgo-build ...
./stockfish bench 16 1 1000 default time > /dev/null

THEN disaster window pops up: stockfish.exe has stopped working.
After I close it, the cmd.exe continues with the following output:

Makefile:403: recipe for target 'profile-build' failed
mingw32-make: *** [profile-build] Error 255

Exception running games:
Traceback (most recent call last):
File "worker.py", line 88, in worker
run_games(worker_info, password, remote, run, task_id)
File "C:\fishtest-master\worker\games.py", line 383, in run_games
setup_engine(new_engine, binaries_url, worker_dir, run['args']['resolved_new'], repo_url, worker_info['concurrency'])
File "C:\fishtest-master\worker\games.py", line 152, in setup_engine
build(worker_dir, sha, repo_url, destination, concurrency)
File "C:\fishtest-master\worker\games.py", line 131, in build
subprocess.check_call(MAKE_CMD + ' -j %s' % (concurrency), shell=True)
File "C:\Python27\lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'mingw32-make profile-build COMP=mingw ARCH=x86-64-modern -j1' returned non-zero exit status 2
Task exited

What can I do to start contributing with a little processing power to Stockfish?

Unknown cause of crash

Was just running tests and noticed the following

Finished game 115 (stockfish vs base): 1-0 {White wins by adjudication}
Score of stockfish vs base: 20 - 32 - 63  [0.448] 115
Traceback (most recent call last):
  File "/home/webmaster/fishtest/worker/games.py", line 293, in launch_cutechess
    raise Exception('Non-zero return code: %d' % (p.returncode))
Exception: Non-zero return code: -9
Task exited

I think it was running on the Multiple-Not-Improving tests. If there is a way to provide more info on what happened please let me know. Couldn't find any detailed logs or even a logfile for the stuff the testing framework is posting on the console.

simplify stat_util.py

Now that we use scipy, you can apply the following (non functional) patch in stat_util.py:

  • import scipy.stats
  • remove our home brew erf(), erf_inv(), phi(), phi_inv() functions
  • use scipy.stats.norm.cdf() instead of phi()
  • use scipy.stats.norm.ppf() instead of phi_inv()

worker sets wrong CPU factor

Worker takes a CPU factor that is lower than usual for my CPU.

see attached log file:
fishtest_worker_log_file.txt

I started the worker, it ran benchmarks which showed about 1570 Knodes/second both.
Then engine match started with

CPU factor : 1.022074 - tc adjusted to 10.22+0.10

The nodes number is way too low for my CPU which is a Intel Core i7-6800K @ 4GHz. Usual number is 2500Knodes. Local bench made by myself:
bench_test_locally.txt
I stopped the worker with Ctrl+C and started it again immediately.

$ ./stockfish_start_5.sh
Worker version 59 connecting to http://tests.stockfishchess.org:80
Verifying signature of stockfish ...
Verifying signature of base ...
CPU factor : 0.613244 - tc adjusted to 6.13+0.06

The new CPU factor looks correct and might be one that was used before (?). No benchmarks were rerun so I can't explain where it took the correct CPU factor from the second time I started it. It was the first time I started the worker today and all current tests on fishtest were created today.

When the 1000 game completed I started the worker again and received again bench numbers:

===========================
Total time (ms) : 3032
Nodes searched : 4735038
Nodes/second : 1561687

The match started with a correct CPU factor again:

Step 4/4. Deleting profile data ...
make ARCH=x86-64-modern COMP=gcc gcc-profile-clean
make[1]: Entering directory '/tmp/tmpPuEMnR/GuardianRM-Stockfish-876f07c/src'
make[1]: Leaving directory '/tmp/tmpPuEMnR/GuardianRM-Stockfish-876f07c/src'
Verifying signature of stockfish ...
Verifying signature of base ...
CPU factor : 0.609349 - tc adjusted to 6.09+0.06

Is this a bug or am I understanding something wrong?

Detecting CPU architecture

I have a pretty old desktop CPU. It's 64 bit it doesn't support POPCNT as far as I can tell. Fishtest tries to compile using ARCH=x86-64-modern and bench crashes with 'Illegal instruction'.

Would it be a good idea to check /proc/cpuinfo flags for popcnt and falling back to ARCH=x86-64 if it's not there?

guard against slow machines does not work

In worker/games.py, I see a lower limit of 0.7 Mnps, to guard against slow machines.

I don't really understand why, but in practice it does not work: Pyafue-1core is currently running at 0.28 Mnps, and has been for a while, yet he never gets kicked.

If there is no bug in the code, it means that this user hacked worker.py on his machine to remove that limit and allow his machine to run anyway. If people start to do things like that, we need to take action. With the same success, I can modify the code in games.py and hardcode that my machine runs at whatever Mnps I want...

Worker doesn't handle passwords with % characters in them

Running the worker using python2 worker.py --concurrency 3 Username "Password%" results in an incorrect formatting error. If the % is doubled, the formatting error does not occur, but the password is then invalid.

Traceback (most recent call last):
  File "worker.py", line 173, in <module>
    main()
  File "worker.py", line 136, in main
    config.set('login', 'password', args[1])
  File "/usr/lib/python2.7/ConfigParser.py", line 752, in set
    "position %d" % (value, tmp_value.find('%')))
ValueError: invalid interpolation syntax in 'Password%' at position 8

Internal Server Error on http://tests.stockfishchess.org

Since yesterday I got in client: (1), and the website (2) shows: (3)

(1)
Worker version 59 connecting to http://tests.stockfishchess.org:80
Exception accessing host:
Traceback (most recent call last):
File "worker.py", line 57, in worker
req = json.loads(req.text)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

(2) http://tests.stockfishchess.org

(3)
Internal Server Error

The server encountered an unexpected internal server error

(generated by waitress)

Update issue

I had the following error tonight, I think this is update-related because you have pushed the worker V13 client :

Exception from calling update_task:
Traceback (most recent call last):
  File "/home/jeff/git/fishtest/worker/games.py", line 216, in run_games
    status = requests.post(remote + '/api/update_task', data=json.dumps(result)).json()
  File "/home/jeff/git/fishtest/worker/requests/models.py", line 604, in json
    return json.loads(self.text or self.content)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 451, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 406, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 426, in raw_decode
    raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)
Started game 294 of 1000 (base vs stockfish)
Finished game 288 (base vs stockfish): 0-1 {Black wins by adjudication}
Score of stockfish vs base: 58 - 64 - 166  [0.490] 288
Exception from calling update_task:
Traceback (most recent call last):
  File "/home/jeff/git/fishtest/worker/games.py", line 216, in run_games
    status = requests.post(remote + '/api/update_task', data=json.dumps(result)).json()
  File "/home/jeff/git/fishtest/worker/requests/models.py", line 604, in json
    return json.loads(self.text or self.content)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 451, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 406, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 426, in raw_decode
    raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

Add variant option to for cutechess-cli so chess960 opening books can be used for testing

https://groups.google.com/forum/m/?fromgroups#!topic/fishcooking/3am9W7IazE8

At this topic I have provided the idea that chess960 opening book should be used to run tests on positions that don't occur often in normal chess, raising the Stockfish versatility. I have made a book that could be used in this regard, however, to run cutechess-cli, "-variant fischerandom" option needs to be added to fishtest, which is currently not available.

ERROR: The process not found.

Core 2 Duo / Windows 7
Worker running on Supporting1Tunes.

cmd.exe window output:

Indexing opening suite...
Started game 1 of 2 (stockfish vs base)
Finished game 1 (stockfish vs base): 1/2-1/2 {Draw by adjudication}
Score of stockfish vs base: 0 - 0 - 1  [0.500] 1
Started game 2 of 2 (base vs stockfish)
Finished game 2 (base vs stockfish): 1-0 {White wins by adjudication}
Score of stockfish vs base: 0 - 1 - 1  [0.250] 2
ELO difference: -191
Finished match
Finished match cleanly
ERROR: The process "8008" not found.
['C:\\fishtest-master\\worker\\testing\\cutechess-cli.exe', '-repeat', '-rounds', '2', '-tournament', 'gauntlet', '-srand', '4034126775', '-resign', 'movecount=3', 'score=400', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '1', '-openings', u'file=2moves_v1.pgn', u'format=pgn', 'order=random', 'plies=16', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=128', u'option.nodestime=600', u'option.mSupport=22', u'option.eSupport=19', '-engine', 'name=base', 'cmd=base', u'option.Hash=128', u'option.nodestime=600', u'option.mSupport=-18', u'option.eSupport=-21', '-each', 'proto=uci', 'tc=24.06+0.24', 'timemargin=10000', 'option.Threads=1']('TC limit', 240.63370887231525, 'End time:', datetime.datetime(2016, 11, 25, 8, 48, 50, 763709))
Indexing opening suite...
Started game 1 of 2 (stockfish vs base)
Finished game 1 (stockfish vs base): 0-1 {Black wins by adjudication}
Score of stockfish vs base: 0 - 1 - 0  [0.000] 1
Started game 2 of 2 (base vs stockfish)
Finished game 2 (base vs stockfish): 0-1 {Black wins by adjudication}
Score of stockfish vs base: 1 - 1 - 0  [0.500] 2
ELO difference: 0
Finished match
Finished match cleanly
ERROR: The process "5184" not found.
['C:\\fishtest-master\\worker\\testing\\cutechess-cli.exe', '-repeat', '-rounds', '2', '-tournament', 'gauntlet', '-srand', '1324307888', '-resign', 'movecount=3', 'score=400', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '1', '-openings', u'file=2moves_v1.pgn', u'format=pgn', 'order=random', 'plies=16', '-engine', 'name=stockfish', 'cmd=stockfish', u'option.Hash=128', u'option.nodestime=600', u'option.mSupport=-18', u'option.eSupport=19', '-engine', 'name=base', 'cmd=base', u'option.Hash=128', u'option.nodestime=600', u'option.mSupport=22', u'option.eSupport=-21', '-each', 'proto=uci', 'tc=24.06+0.24', 'timemargin=10000', 'option.Threads=1']('TC limit', 240.63370887231525, 'End time:', datetime.datetime(2016, 11, 25, 8, 50, 5, 77709))
Indexing opening suite...
Started game 1 of 2 (stockfish vs base)

Fishtest only using 1 x Xeon on multi Xeon server

I have access to some multi Xeon servers, and I would like to know how to setup Fishtest to use all the nodes (socket) and not just 1 node ?

I am using the Fishtest-Portable-Worker but when I run the RUN_FIRST_PROFILER, it only detects 1 Xeon and I can only set concurrency to Max-1 core of that single Xeon.

Anyone can help with this?
Thx!

Using hyperthreaded cores

The README states that: "Option concurrency refers to the number of available cores in your system (not including Hyperthreaded cores!), leaving one core for the OS." I have 4 cores x 2 threads per core on Windows 7. When I set concurrency to 7, it simulates 7 games at a time with no problem and seems to run faster than if I just use concurrency 4. In the task manager concurrency 4 runs at 50% CPU while concurrency 7 is at roughly 90%. What am I missing?

exception running games

Hi everyone,

I am trying to contribute a few cores to the tests, using

OS X 10.9.2 (13C64) 2.3 GHz Intel Core i7 (Retina MacBook Pro)

This is what I get:

Exception running games:
Traceback (most recent call last):
File "worker.py", line 85, in worker
run_games(worker_info, password, remote, run, task_id)
File "/Users/idahabreh/Downloads/fishtest-master/worker/games.py", line 477, in run_games
launch_cutechess(*payload)
File "/Users/idahabreh/Downloads/fishtest-master/worker/games.py", line 310, in launch_cutechess
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True, bufsize=1, close_fds=not IS_WINDOWS)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
Task exited

Any help would be appreciated.

Exception running games

Hi all,
Tried to run tests on XP SP3 core 2 duo T5800 cpu, but got these error:

Worker version 55 connecting to http://tests.stockfishchess.org:80
Verifying signature of stockfish.exe ...

Exception running games:
Traceback (most recent call last):
File "D:\Documents and Settings\Ivan Trajkov\fishtest-master\worker\worker.py"
, line 86, in worker
run_games(worker_info, password, remote, run, task_id)
File "D:\Documents and Settings\Ivan Trajkov\fishtest-master\worker\games.py",
line 399, in run_games
base_nps = verify_signature(new_engine, run['args']['new_signature'], remote
, result, games_concurrency)
File "D:\Documents and Settings\Ivan Trajkov\fishtest-master\worker\games.py",
line 69, in verify_signature
p = subprocess.Popen([engine, 'bench'], stderr=subprocess.PIPE, stdout=f, un
iversal_newlines=True)
File "D:\Python27\lib\subprocess.py", line 711, in init
errread, errwrite)
File "D:\Python27\lib\subprocess.py", line 948, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
Task exited

All help appreciated.

Strange "stockfish doesn't have option [space]Hash" message

Hello,

I just had at the Terminal messages while my computer was connected to fishtest running worker.py, and this is the output :

Running tune_newmob vs tune_newmob
['/Users/stephane/Programmation/fishtest-master/worker/testing/cutechess-cli', '-repeat', '-rounds', '6', '-tournament', 'gauntlet', '-resign', 'movecount=3', 'score=400', '-draw', 'movenumber=34', 'movecount=8', 'score=20', '-concurrency', '3', '-openings', u'file=2moves_v1.pgn', u'format=pgn', 'order=random', 'plies=16', '-engine', 'name=stockfish', 'cmd=stockfish', u'option. Hash=128', u'option.nodestime=600', u'option.mMobTune_0_0=7', u'option.eMobTune_0_0=8', u'option.mMobTune_0_1=-10', u'option.eMobTune_0_1=11', u'option.mMobTune_0_2=12', u'option.eMobTune_0_2=12', u'option.mMobTune_0_3=10', u'option.eMobTune_0_3=-11', u'option.mMobTune_1_0=7', u'option.eMobTune_1_0=7', u'option.mMobTune_1_1=-8', u'option.eMobTune_1_1=4', u'option.mMobTune_1_2=-12', u'option.eMobTune_1_2=-4', u'option.mMobTune_1_3=10', u'option.eMobTune_1_3=-7', u'option.mMobTune_2_0=-10', u'option.eMobTune_2_0=-10', u'option.mMobTune_2_1=7', u'option.eMobTune_2_1=-8', u'option.mMobTune_2_2=-11', u'option.eMobTune_2_2=7', u'option.mMobTune_2_3=-5', u'option.eMobTune_2_3=-9', u'option.mMobTune_3_0=-11', u'option.eMobTune_3_0=7', u'option.mMobTune_3_1=-3', u'option.eMobTune_3_1=-11', u'option.mMobTune_3_2=-11', u'option.eMobTune_3_2=-7', u'option.mMobTune_3_3=7', u'option.eMobTune_3_3=-6', '-engine', 'name=base', 'cmd=base', u'option. Hash=128', u'option.nodestime=600', u'option.mMobTune_0_0=-11', u'option.eMobTune_0_0=-10', u'option.mMobTune_0_1=8', u'option.eMobTune_0_1=-7', u'option.mMobTune_0_2=-6', u'option.eMobTune_0_2=-6', u'option.mMobTune_0_3=-8', u'option.eMobTune_0_3=7', u'option.mMobTune_1_0=-11', u'option.eMobTune_1_0=-11', u'option.mMobTune_1_1=10', u'option.eMobTune_1_1=-14', u'option.mMobTune_1_2=6', u'option.eMobTune_1_2=14', u'option.mMobTune_1_3=-8', u'option.eMobTune_1_3=11', u'option.mMobTune_2_0=8', u'option.eMobTune_2_0=8', u'option.mMobTune_2_1=-11', u'option.eMobTune_2_1=10', u'option.mMobTune_2_2=8', u'option.eMobTune_2_2=-11', u'option.mMobTune_2_3=13', u'option.eMobTune_2_3=9', u'option.mMobTune_3_0=7', u'option.eMobTune_3_0=-11', u'option.mMobTune_3_1=15', u'option.eMobTune_3_1=7', u'option.mMobTune_3_2=7', u'option.eMobTune_3_2=11', u'option.mMobTune_3_3=-11', u'option.eMobTune_3_3=12', '-each', 'proto=uci', 'tc=28.78+0.05', 'timemargin=10000', 'option.Threads=1']
Indexing opening suite...
stockfish doesn't have option Hash
stockfish doesn't have option Hash

What is a little bit strange/suspect (to me) is the part which sets the first option of each engine (both stockfish and base) :

 u'option. Hash=128

It looks like there is a parasite space just before "Hash=128", so that both stockfish and base complain afterwards with the error messages :

stockfish doesn't have option  Hash    (note the invisible space)

I'm not fluent with the fishtest Python code, but as far as I can say, the error could be related to the block following line 338 in games.py :

Format options according to cutechess syntax

def parse_options(s):
results = []
chunks = s.split('=')
if len(chunks) == 0:
return results
param = chunks[0]
for c in chunks[1:]:
val = c.split()
results.append('option.%s=%s' % (param, val[0]))
param = ' '.join(val[1:])
return results

Can anybody confirm this on his/her computer ?

Hope this helps,
Stéphane

worker.py fails to clean up processes

I guess this is Windows specific.

Twice today I've had worker hung where it couldn't delete the test executable from the previous run.

it appears to happen when the test stops early

http://54.235.120.254:6543/tests/view/5150927e8f0c3e51b7bb3e87
http://54.235.120.254:6543/tests/view/515045578f0c3e51b7bb3e82

The cutechess-cli process (I assume) was stopped but the stockfish processes were running so the executables can't be deleted and replaced for the next test. I had to manual kill the processes, delete the testing directory and restart worker.py.

If this was native Windows code I'd use a job object to ensure clean up, but I've no idea if that is even feasible in python.

Exception: Bench exited with non-zero code -4

I've just downloaded the git repository and started working on tests, and on my laptop with i5 core, 64bit, Arch Linux system works fine. But another computer (4 cores Q6600), on 64bit ubuntu I get this error:

Exception running games: Traceback (most recent call last): File "worker.py", line 85, in worker run_games(worker_info, password, remote, run, task_id) File "/home/ed/fishtest/worker/games.py", line 423, in run_games base_nps = verify_signature(new_engine, run['args']['new_signature'], remote, result, games_concurrency) File "/home/ed/fishtest/worker/games.py", line 87, in verify_signature raise Exception('Bench exited with non-zero code %d' % (p.returncode)) Exception: Bench exited with non-zero code -4 Task exited Verifying signature of stockfish ...

Could someone help?

problem with Marco's chi2 iterative elimination patch

The way it is now, is very misleading. We have the max(WLD adjusted residual) in front of each worker, but the p-value, chi2 and df are computed after the process of iterative elimination.

What we want is to see the normal chi2, p-value, df, before any elimination. And the max(WLD adj. res.) in front of each worker.

For example, on the following test:
http://tests.stockfishchess.org/tests/view/5259018d0ebc5933b5469364

We should see:

  • chi2 = 91.31
  • df = 54
  • p-value = 0.00% (5.17e-5)

Iterative process to find the polluters:

  • polluter is "Thomas A. Anderson-7cores", with max(WLD adj res) = 3.658
  • we remove it and start again, polluter is "BrunoBanani-17cores", with max(WLD adj res) = 3.380
  • we remove it and start again, polluter is now "Thomas A. Anderson-23cores" with max(WLD adj res) = 3.591
  • we remove it and start again, polluter is now "DragonLord-3cores" with max(WLD adj res) = 3.278
  • stop: no more polluters (not one with adj res > 3).

So what we want to see is the list of polluters in order (while showing the stats of the initial test without removing anyone), like so:

Hall of shame
1/ Thomas A. Anderson-7cores
2/ BrunoBanani-17cores
3/ Thomas A. Anderson-23cores
4/ DragonLord-3cores

chi-square implementation

You need to aggregate the rows by worker before doing the chi-square test.

Example:
http://tests.stockfishchess.org/tests/view/52545c850ebc593ba5ae7104

Once aggregated by worker, only 23 lines left (23 different machines). So we should have df = (23-1)*(3-1) = 44, instead of 78.

Here are my results for this test:

  • chi2 = 73.72
  • df = 44
  • p-value = 99.67%

PS: This run fails the test convincingly, and the culprit is BrunoBanani-17cores.

Fishtest fails to verify signature

Running fishtest on my Ubuntu 13.04 64-bit machine with python 2.7.4 rises the following exception:

Verifying signature of stockfish ...

Exception running games:
Traceback (most recent call last):
File "worker.py", line 78, in worker
run_games(worker_info, password, remote, run, task_id)
File "/home/jonas/fishtest/worker/games.py", line 391, in run_games
base_nps = verify_signature(new_engine, run['args']['new_signature'], remote, result)
File "/home/jonas/fishtest/worker/games.py", line 81, in verify_signature
raise Exception('Bench exited with non-zero code %d' % (p.returncode))
Exception: Bench exited with non-zero code -4
Task exited

Offline testing support for fishtest?

It would be nice to support the use of fishtest offline (i.e. don't report back test results at all online, log them locally instead). This way, it would effectively be a wrapper for cutechess-cli that could used by a dev to do runs of 100's/1000's games offline more easily. Are there any such docs for current offline testing anyway - I didn't see them in the fishtest docs.

stockfish fatal compile error

Makefile:473: recipe for target 'stockfish' failed
g++ -o stockfish benchmark.o bitbase.o bitboard.o endgame.o evaluate.o main.o material.o misc.o movegen.o movepick.o pawns.o position.o psqt.o search.o thread.o timeman.o tt.o uci.o ucioption.o syzygy/tbprobe.o -lgcov -Wl,--no-as-needed -lpthread -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-use -fno-peel-loops -fno-tracer -pedantic -Wextra -Wshadow -m64 -DNDEBUG -O3 -DIS_64BIT -msse -flto
lto1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See file:///usr/share/doc/gcc-5/README.Bugs for instructions.
lto-wrapper: fatal error: g++ returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
Makefile:473: recipe for target 'stockfish' failed

NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu116.04.2' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1
16.04.2)

CUDA / OpenCL

Is it yet possible to calculate the tests with GPU?

GUI and Windows Installer

Stockfish has a massive user base, but there are only 34 active machines currently running fishnet. A huge barrier to contributing to the project is the lack of windows installer and GUI. If there is interest in incorporating a GUI into the project, I would be willing to help work on it with some help.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.