We should use auto-formatters where possible. For python, there are a couple options. I'll probably go with autopep8, since that's pretty standard.

This is a discussion issue. Currently our SGFs have a huge amount of debug junk. Ultimately, this means that most of our GCS usage is really big SGFs. For our 9s run, our GCS bucket contained 130gb of data, 100gb of that was SGFs. just having the moves would reduce the size by probably 1/10 or 1/100

We should add custom compiling of TensorFlow for the CPU Image[s]. The problem is that the default CPU pip package is not optimized. Compiling from source can give a 30% CPU-usage boost.

Noticed while running tests

WARNING:tensorflow:From /Users/jhoak/inprogress/minigo/dual_net.py:274: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

As is standard with all OSS projects, we should add a CodeOfConduct. Since we're part of TensorFlow, we'll want to use the TF COC.

Hi, I know this is a very naive question. I am new to Go. I played a game using my model against gnu go following the command in the readme (the twogtp command) of this repository. Anyone can tell me how to check who wins?

The gnugo.sgf is below:

(;FF[4]CA[UTF-8]AP[gogui-twogtp:1.4.9]SZ[9]
KM[6.5]PB[GNU Go]PW[Somebot-000001-model]DT[2018-02-15]
C[Black command: gnugo --mode gtp
White command: python3 main.py gtp -l models/000001-model
Black version: 3.8
White version: 0.1
Result[Black]: B+78.5
Result[White]: ?
Host: xliu-linux3 (Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz)
Date: February 15, 2018 4:09:07 PM PST]
;B[ee];W[dg];B[cf];W[di];B[cd];W[fa];B[gb];W[ig];B[gg];W[ei]
;B[ff];W[gi];B[hh];W[fc];B[fb];W[eh];B[cg];W[if];B[he];W[cc]
;B[bc];W[ie];B[hd];W[da];B[dc];W[ih];B[ch];W[ga];B[ha];W[ad]
;B[ed];W[be];B[bd];W[de];B[df];W[fg];B[hi];W[eb];B[hb];W[gd]
;B[db];W[ef];B[gh];W[bh];B[ci];W[ca];B[cb];W[ac];B[ab];W[af]
;B[ce];W[id];B[ic];W[ag];B[ba];W[gf];B[fe];W[ah];B[fh];W[ec]
;B[hf];W[ii];B[hg];W[ae];B[hc];W[ig];B[dd];W[if];B[];W[ge]
;B[];W[ie];B[];W[id];B[];W[ib];B[];W[])

And there is a gnugo.dat file:

Black: GNU Go

BlackCommand: gnugo --mode gtp

BlackLabel: GNU Go

BlackVersion: 3.8

Date: February 15, 2018 4:08:51 PM PST

Host: xliu-linux3 (Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz)

Komi: 6.5

Referee: -

Size: 9

White: Somebot-000001-model

WhiteCommand: python3 main.py gtp -l models/000001-model

WhiteLabel: Somebot-000001-model

WhiteVersion: 0.1

Xml: 0

#GAME RES_B RES_W RES_R ALT DUP LEN TIME_B TIME_W CPU_B CPU_W ERR ERR_MSG
0 B+78.5 ? ? 0 - 78 10.1 4.8 10.2 0 0

Hi,

I used the bootstrap model to test the main.evaluation() function. The black and white players are the same model. However, during running, an error happened:

File "./try.py", line 13, in
main.evaluate(black_model = black_model, white_model = white_model, output_dir = out_dir, readouts = 10, games = 1, verbose = 3)
File "/home/xliu/minigo/main.py", line 113, in evaluate
black_net, white_net, games, readouts, verbose)
File "/home/xliu/minigo/evaluation.py", line 66, in play_match
prob, val, up_to=pair[num_moves % 2].root)
File "/home/xliu/minigo/mcts.py", line 197, in incorporate_results
assert not self.position.is_game_over()
AssertionError

I can not figure out the reason. Can anyone help me? Before this error, there was another error at the end of evaluation.py, that says the MCT object does not have the is_done() attribute. I fixed it by changing p[0].is_done() and p[1].is_done() to p[0].root.is_done() and p[1].root.is_done().

Thank you for your help.

Before we can accept outside contributions, we need to hook up the Google CLA bot.

Our GTP code has a nice place to put in some GTP extensions in gtp_extensions.py
It should be pretty easy to hook up some code to dump things in the format gogui/sabaki like to see for variations or heatmaps or whatever.

it'd be a good starter project for someone interested in getting acquainted with the codebase.

One of the symptoms of value net overfitting that we found was that the 8 board symmetries would yield wildly different results when put through the value net. So instead of the value net returning roughly consistent estimations, stdev=0.1 or so, it would instead disagree completely, with 7 symmetries saying that B+ 0.99, and the 8th symmetry saying W+ 0.99.

It'd be nice to log the average stdev of the 8 symmetries (or some other characterization of how divergent the symmetries are.)

We have multiple hyperparams. Here's a short list:

• go.N = 9, 19. Needs to stay constant through the entire training run. Currently being set via env variables
• MCTS playout parameters: resign threshold, number of readouts, virtual loss parallelism, game depth limit, move threshold for softpick, probability to disable resign, tree reuse. Can be changed with each selfplay worker. Currently being set by hardcoded constants.
• NN parameters: depth, conv layer width, fully connected layer width. Needs to stay constant through the entire training run. Currently being set via a makeshift hyperparams dict.
• Training parameters: l2 loss, learning rate, momentum, training window size, shuffle buffer size, number of concurrent files, example select rate. Can be changed with each run of train() to get next gen. Set in a variety of places
• Rate of training vs rate of game generation (aka how many self-play workers). This one is slightly linked to number of readouts; since training on an example is roughly 2x as slow as eval'ing an example, then to balance position generation rate, 1600 readouts would necessitate ~800 selfplay workers. Or if we wanted to generate twice as many positions as we would ever use, then we would double the number of selfplay workers.

Would help catch lots of bugs and help ensure style conformance.

It's easy to grab a bunch of SGFs with gsutil cp ... After that, we're looking at statistics largely by doing 'grep'... which is getting slower as we add more stuff to our SGF comments.

It'd be nice to:
a. See opening move stats per generation, with reflection/symmetry taken into account. This could be made pretty fast by only reading the head of the file until the 1st move, etc.
b. Figure out what moves the network is moving towards/away. This involves looking at how the visit counts after search match with the priors -- if a move (on the whole) is getting more visits than the priors suggested, after d-noise, then we can expect that move to be more represented in the next generation, & vice versa.
c. It'd be nice to look for the 'most surprising moves' in a game -- e.g., greatest difference from policy prior and final exploration rate.
d. Similarly, there are some nice questions about the times in a game when it has many options vs when it only has one or two; some nice ways to explore these would be good.

We've used tf.train.exponential_decay(1e-2, global_step, 10 ** 7, 0.1). This isn't quite right, and it's possible it's the source of some of our overfitting woes. Let's check our math to get an equivalent to the AGZ paper.

In the readme.md
python rl_loop.py selfplay --readouts=$READOUTS -v 2
should be
python3 rl_loop.py selfplay --readouts=$READOUTS -v 2

Still not sure how to do it using local file

Right now, we're relying on a single project for images, GKE clusters, GCS buckets etc. There's lots of problems with collision, quota, etc.

It should be easy to spin up a new project (get quota, which isn't terribly easy), and then run the system.

Current plan is to use the cloud project minigo-pub, but there's still some setup that's needed.

We should choose a code-formatter and use it! It helps a lot to ease the burden of day-to-day style considerations. Preferably something that implements the Pep8 guide.

https://pypi.python.org/pypi/autopep8 is a good candidate since that's hosted on pypi. And, selfishly, we have builtin support as part of our Vim CodeFmt plugin.

We should have a zoo where self-play variants can play against each other to establish ratings. This could just mean figuring out the infrastructure to automatically hook new training checkpoints into CGOS, ideally without overwhelming CGOS.

Is minigo ready for windows 10 ?

(from brilee)
Currently, our self-play and tournament mode both allocate a fixed number of seconds and/or reads.

Both modes suffer from the same problem, which is that they spend their readouts inefficiently. If a response is obvious and has 800 reads compared to its siblings which have 10 reads each, then, there's no point to doing another 600 reads to reach our set point of 1600 reads per move. On the other hand, if the current leading candidate has 800 reads but the second place candidate discovered a new response and is rapidly catching up, we don't give the new candidate enough time to overtake the first place candidate. Ideally, we should truncate search at a point when all moves look sort of equally uninteresting to invest more time into (i.e. they have very similar action score.)

I don't care so much about tournament mode, but self-play could potentially achieve stronger play and using fewer reads, which should speed up the RL feedback loop. That our tournament mode would get stronger is just a nice bonus.

I was reading the code in mtcs,py and here

How should we handle self-play balance with training?

AGZ 20 block
5M games, 700 checkpoints = ~7k games/checkpoint

AGZ 40 block
29M games, 3.1M batch = 3100 checkpoints. = ~10k games/batch???

but also:
"In each iteration, αθ∗ plays 25,000 games of self-play" (AGZ)
So perhaps: the evaluator passes approximately 1/3-1/4 of all checkpoints.

AZ: 700k steps of double size (4098)
21M games. no 'checkpoints'. Would be equivalent to 15k games/checkpoint in AGZ reckoning

These seem to suggest that having more games per generation is preferable, and suggests there's not as much concern about the number of generations represented in the training window.

So, i'd like this issue to resolve:

• how we should target our worker/playout balance
• what changes we'll need to make to achieve it. In particular, if our training/worker jobs are subject to pre-emption or are otherwise, uh, not robust, i'd like to brainstorm what we can do to keep the two jobs in balance.

• upgrade to the python client
• integrate opponent name & game end results (new extensions?)

This would greatly clarify the differences between local execution vs. GCS execution.

Apparently shuffle queues will round-robin over the files it's reading. So if it's reading 64 parallel files that are exactly 10,000 examples each, then it will finish reading all 64 at exactly the same time. That could result in some issues. To get rid of that, jitter the chunk sizes to be +/-25% of the desired size.

• We've got a script that calculates our current false positive rate for a given threshold
• we can parametrize the threshold for our workers by editing player_wrapper.sh
• We don't have a way to automatically calculate the correct false positive rate, edit the value in player_wrapper.sh, and then rebuild the image.

Alternate idea: We could also automatically calculate the correct false positive rate, stick it in a file, and have the workers read the file.

For some small percent of self-play games, we should set them aside, never train on them, and check the value net loss on these games. This should help us figure out if we're overfitting.

A variant on this is to create a validation set of pro games. Self-play validation sets are nice because there's not a very large library of pro 9x9 games. But for 19x19, a pro validation set is doable.

Currently, our repo is where we track discussions (via Issues). We could create a Google Group (i.e., a mailing list) or use some other discussion forum (Slack). We don't want to fragment discussion, but it depends on what the community wants and how much participation we have (we don't want empty rooms).

This is a discussion issue -- feel free to chime in. Currently we don't have any plans here, but that could change based on community feedback.

We should use a test-runner that tests PRs and maybe even does continuous testing. TensorFlow tends to use Jenkins. However, K8S has produced an awesome runner called Prow. Since this is a K8S + TF project, I think it would be a great fit.

I'll take a look at doing that integration.

Our naming scheme for coord type to coord type is a mess. Some homogenization would be nice.

with vlosses the 'terminate after too many moves' check got taken out. Needs to go back in before our v4 run.

instead of having our selfplay workers constantly look for the .data-00000-of-00001, .index, .meta files, we should properly export our graph. Not only would this make a lot of our scripts cleaner, it'd be a good way to run some optimizers etc. prior to farming it all out.

We use random symmetries for MCTS selfplay, but we don't use random symmetries during training. We should add that to help stabilize training.

doesn't work for TF 1.5
might not be correct to begin with :)
So GPUs on kubernetes are changing how they are used and consumed in 1.8 (https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/)

Our current daemonset job also installs the drivers AND exposes them for scheduling -- this seems to conflict or otherwise trample on the drivers in the base NVIDIA image, and means that the cuda 9 images don't actually work.

So the work here is to:

figure out what is needed to get a working device-plugin for k8s 1.8 that exposes our GPU
get that device-plugin sorted with its nvidia drivers
get TF1.5 working
update the docs to reflect this final solution.

When running the tests, I've noticed some DeprecationWarnings. This look like they're in the core tensorflow libraries. I wonder if we need to update or if the TF team needs to update their clients (probably the latter).

/Users/jhoak/.virtualenvs/mgz/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py:539: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
  return np.fromstring(tensor.tensor_content, dtype=dtype).reshape(shape)
/Users/jhoak/.virtualenvs/mgz/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))

Cloud is introducing TPUs: https://cloud.google.com/tpu/. When these come out, we'd like to use them. This is a forward looking bug to use TPUs when they're ready for general consumption

Currently only the basic ko is implemented; I thought superko would never arise in a real game, but it apparently comes up during endgame sequences where one side and then the other side passes. http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2017/12/31/342211.sgf

Running BOARD_SIZE=19 python3 /home/aolsen/projects/minigo/minigo/main.py gtp -l /home/aolsen/minigo-models/000199-lightning -r 100 -v 3, nvidia-smi shows it uses 2558MiB of GPU memory. After running genmove b it goes to 2662MiB. This is preventing me from running two minigo instances to test changes since my GPU only has 4GB.

LZ only uses 483MB for a larger 20x256 network (note LZ computes policy and value heads on the CPU).

@amj suggested this link https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory

Right now, we train the AI to play with one Komi setting (usually 7.5 or 6.5). Then, the AI must play with that Komi forever or it'll get confused about the score. That's a bit too bad because it means that even if the AI gets/gives a handicap, Komi still needs to be set at 7.5 for white.

I don't see this as a huge problem, but I think it's worth tracking this feature. It also might be fun to play around with this on a 9x9-trained AI.

We should monitor the norm of the gradient update, the norm of the weights, and the ratio. (If the norm of the gradient gets too small.)

Hi,
Right now the selfplay function only plays 1 game per call. I changed it to the following to make it play many games in one call. I am wondering can we just load the model once and let it play many games like this code? This is a lot faster than loading the model every time we play a game.

def selfplay(
load_file: "The path to the network model files",
output_dir: "Where to write the games"="data/selfplay",
holdout_dir: "Where to write the games"="data/holdout",
output_sgf: "Where to write the sgfs"="sgf/",
readouts: 'How many simulations to run per move'=100,
verbose: '>=2 will print debug info, >=3 will print boards' = 1,
resign_threshold: 'absolute value of threshold to resign at' = 0.95,
holdout_pct: 'how many games to hold out for evaluation' = 0.05):
_ensure_dir_exists(output_sgf)
_ensure_dir_exists(output_dir)

Only load model once

with timer("Loading weights from %s ... " % load_file):
    network = dual_net.DualNetwork(load_file)
    network.name = os.path.basename(load_file)

Let the model play many games

for i in range(5000):
  with timer("Playing game"):
      player = selfplay_mcts.play(
        network, readouts, resign_threshold, verbose)

  output_name = '{}-{}'.format(int(time.time()), socket.gethostname())
  game_data = player.extract_data()
  with gfile.GFile(os.path.join(output_sgf, '{}.sgf'.format(output_name)), 'w') as f:
     f.write(player.to_sgf())

  tf_examples = preprocessing.make_dataset_from_selfplay(game_data)

  # Hold out 5% of games for evaluation.
  if random.random() < holdout_pct:
    fname = os.path.join(holdout_dir, "{}.tfrecord.zz".format(output_name))
  else:
    fname = os.path.join(output_dir, "{}.tfrecord.zz".format(output_name))

  preprocessing.write_tf_examples(fname, tf_examples)

in particular:

• final_score
• final_status_list

maybe some of the time commands to make the CGOS player a little more robust.

We use random symmetries for MCTS selfplay, but we don't use random symmetries during training. We should add that to help stabilize training.

This is a discussion issue to track a discussion we were having. Should the python code at the top-level live at the top level or in a sub-dir? I'm of the opinion it would be cleaner and more standard for it to live in something like model/ or src/, which would keep tho top level cleaner. Thoughts @brilee @amj ?

A validation of value-error on pro games or kgs games would be nice to have

Nothing actionable here, just recording some thoughts:

Training from the last ~50 generations seems like an awfully long window. Early on in the training process, this seems like it would keep crappy games around for too long; later on in the training, as the RL process starts to asymptote, it seems like it might matter less. In both cases, it seems like the long history window is to prevent the network from forgetting how to play simpler moves.

All of this seems somewhat related to the "catastrophic forgetting" problem. Any solutions to that problem may be applicable somehow to the RL process.

We're using preemptible GPUs. And that means that sometimes, we get preempted!
We should have better insights / monitoring around the preemption rate.

I've chatted in passing with @amj about this and I wanted to discuss it more formally. Self-play is, as I understand it, the most important part to speed up since more-games-faster means you can train the go-bot faster.

Here are a couple of options that I would consider:

• C++
• Go
• Rust

Some factors:

Currently, we're leaning towards a rewrite in Go (maybe), since we believe that would be simple / easy to follow. Also, I've at least used Go quite a bit on Kubernetes, so there's an experience factor too.