Different ELO level about leela-zero HOT 7 CLOSED

In the AlphaGo Zero paper, where they show the Elo scale progression, the network starts at -3500.
Since it's the very beginning of the training (pre-training), and is (I assume) also starting from random weights, it seems unlikely that it would be that much different from leela zero's initial random state.

While the 20 block network is likely to have a lot higher potential than starter LeelaZ, I don't think it is too likely that anything far from LeelaZ elo is common enough in the search space to just be the initial random state of AGZ...

But then again, "not too far" isn't a quantitatively useful measure... So we don't really know.

Then, to make things more problematic, the 40 block network (where they compared GnuGo, among other programs) appears to start at a bit below -2000.
So, did the network start at such different ratings, despite both starting points being random, did they use a different scale, or was the initial "leap" so high and fast that they just removed it from the graph to avoid clutter? etc.

I have no idea, and I don't believe @gcp does as well.

@gcp chose to start on a different scale, which is also fine.
At any point he, or anyone else for that matter, can rebase that to whatever start elo we want to call the initial "random" state.

That said, As we start getting better results, it might be more convenient for people to see approximately how close/far we are from AGZ, but it definitely seems to not be an issue.

I would personally have thought it preferable to start on a similar scale (or definitely not a higher one) but it's not an issue at all since as stated, it can always be rebased to whatever is chosen, and until we have anything concrete to compare it to (that doesn't win 100% of the time), it probably is a moot point.

(even programs I dug up at that are around 17k seem to win 100% against it for now).

from leela-zero.

A tournament between networks can also be distributed modifying a bit the autogtp application

from leela-zero.

As an additional thought: I saw one ranking graph that included the strength of just the raw alpha-go-zero network. It was (to me) also surprisingly strong. It might also be a good idea to snapshot a couple of these weights at specific strengths, as I'd imagine these would be more useful on resource contained devices.

from leela-zero.

The networks will all be listed here: http://zero-test.sjeng.org/networks/ (minus the -test when we go live...)
This is linked to the server DB that sends out the best one so far. So there will be a recording of every "epoch".

For playing games between networks you can use any GTP interface, as Leela Zero is GTP compatible. I use the "ringmaster" tool from "gomill" (a Python package) for this.

I don't expect evaluating the networks to be a bottleneck.

from leela-zero.

@gcp According to AlphaGo Zero elo rating scale, what is your estimation of the beginning elo rating of the Leela Zero random network?

from leela-zero.

I have no idea how to even measure this. IIRC GnuGo is like a few hundred Elo on their rating scale, and GnuGo beast the current networks probably 100% of the time. So how can you calculate an Elo off of that?

If they had actually published any of their data, this would be easy...but then we wouldn't need to do this effort either.

from leela-zero.

closing very old issue with no clear owner

from leela-zero.