alexlandau / gdl-perf Goto Github PK

The tested engine should be able to report an error of invalid goals in a terminal state, no legal moves, etc. and have the reference engine confirm the error. If this is confirmed by the reference engine, this should be recorded and interpreted in a way different from an engine error.

Code for dealing with processes in particular, maybe code for figuring out which games to deal with based on the contents of a CSV file, code for computerName.txt, etc.

It would be nice to have these pairwise comparison pages where we can see a list of games that both engines support, sorted by the performance ratio between them. Currently it's hard to find the extrema games on that list, which can be interesting case studies.

We can also show the errors for those engines, in a table to show error messages in engine 1 and error messages in engine 2. (Sort by the category of which engine(s) have errors at all.)

The former is greater than the number of games listed.

This might have to do with the invalid games list filtering one but not the other?

After my last run of cadiaplayer, I had a leftover process that was consuming most of the memory on the box, causing significant swapping. I didn't see which it was, so it pretty much invalidated the collected statistics.

This suggests that kill -kill -1234 (for script PID 1234) might work for POSIX-reliant scripts like cadiaplayer's, if we can somehow figure out the script PID from Java: http://askubuntu.com/questions/520107/how-to-kill-a-script-running-in-terminal-without-closing-terminal-ctrl-c-doe

EDIT: From further research and testing, it seems like the issue is not the reliability of killing processes, but just the ability to kill process subtrees instead of an individual process. That will be available in Java 9, which sounds like a good long-term solution.

The current one is too finicky about where ECLiPSe Prolog is installed. This is maybe less important now that the full Fluxplayer prolog implementation is available, but it would be nice to have if we can track down the sources and make those modifications.

I expect "# of games with errors" to be >= "only we fail", but currently this isn't the case. We're probably including too many cases when doing this kind of comparison, or not weeding out invalid games.

Let's get rid of some of these infinities.

Assuming most people will be running these on 64-bit Linux machines, we can probably script an installation procedure for the versions of Prolog we rely on, or at least provide reliable installation instructions.

Should this be an "unknown" column instead?

Currently we specify it in the Java codebase. This approach isn't going to work well for non-Java executables, i.e. it would have to be manually updated whenever something independently-versioned changes, which is more error-prone than updating it in the executable as the engine changes.

Java:
Sancho
Palamedes BasicPlayer: http://palamedes-ide.sourceforge.net/start.html
https://github.com/parthy/ggp (? Might use Fluxplayer's Prolog as well?)
Rekkura: https://github.com/ptpham/rekkura
JavaEclipse (from GIGA-13 paper; either GameController or Moller et al, 2011/Centurio)
JavaProver (from GIGA-13 paper; Halderman et al 2006 and Gunther et al 2013) -- is this just an earlier version of GGP-Base prover? http://games.stanford.edu/resources/reference/java/java.html

• Link from Gunther et al citation: http://ggpserver.sourceforge.net/
• This might be the same as what's used in Palamedes BasicPlayer (JavaProver.jar)
• Palamedes code itself seems to include jars for JavaProver, Jocular, and a reference to a "prologprover"
  Jocular (mentioned in GIGA-13 paper; joc 2007) http://games.stanford.edu/resources/reference/jocular/jocular.html
  Centurio (compiles code into Java; mentioned in GIGA-13 paper; Moller et al, 2011)

.NET:
NGGP Base: https://github.com/druzil/nggp-base

C/C++:
Informafiosi: http://www.general-game-playing.de/downloads.html (C++Reasoner from GIGA-13 paper? points to that link)
https://github.com/thomasmarsh/ggp (?)
https://github.com/sumedhghaisas/libgdl
GDLCC (mentioned in GIGA-13 paper; Waugh 2009)
https://github.com/ptpham/piecemeal

Prolog-based:
FluxPlayer: http://www.general-game-playing.de/downloads.html
CadiaPlayer: ??? (B/F 2008, 2009) - based on YAP Prolog (Costa et al 2012)
https://github.com/muupan/ggpe: uses YAP Prolog (check if same as CadiaPlayer)
https://github.com/tflaherty/GeneralGamePlaying (?)
https://github.com/jvandenbroeck/The-Turk (Java and/or Prolog?)
Ary http://www.ai.univ-paris8.fr/~jm/ggp/ (C/Prolog)

Haskell:
https://github.com/ian-ross/ggp

Rust:
https://github.com/gsingh93/ggp-rs
https://github.com/strout/worthy-opponent

Clojure:
https://github.com/ramirez7/ggp-mcts (?)

OCAML:
GaDeLaC (mentioned in GIGA-13 paper; Saffidine/Cazenave 2011)

???:
Toss (mentioned in GIGA-13 paper; Kaiser/Stafiniak, 2011/2013) toss.sourceforge.net
Gamer (mentioned in GIGA-13 paper; Kissmann/Edelkamp, 2011)