Gene pool merging and gene exchange about ga-bitbot HOT 7 OPEN

The gene strings don't actually carry all the information required to decode 
them. This was a design decision to eliminate redundant static data from being 
stored in the genes. 

The gene server trusts that the clients are all using the same configuration. 
This will cause problems when reloading the database after the gene_def.json 
file has been modified in any way (which I regularly do on the repo). If an old 
gene database is loaded after a configuration change, the genes will not be 
decoded properly and will result in errors if the length of the gene changes.

In order to correct this issue (and satisfy the enhancement request at the same 
time) the gene_server will need to be modified to require that clients send a 
copy of the gene_conf.json before accepting any gene submissions. The contents 
of the configuration can then be stored into the database along with a MD5 hash 
to fingerprint the population. 

This approach would have a few benefits: 
One, it would allow the gene server to manage multiple gene configurations at 
once. 
Two, it would prevent any cross contamination between different population 
configurations. 
Three, since the database would now carry both the genes and the configuration, 
users could share their files. A simple tool could be created to merge/prune 
multiple databases.

Once implemented, a new gts client option could be created to cycle through the 
available gene configurations instead of using the local configuration file. 
This would allow resources to be split across multiple independent populations.

Ok, so far so good; Except that this change would break the rest of the system:

report_gen.py and bcbookie.py currently assume only one configuration

All of the html reports (gene visualization, charts and order book) are setup 
for only one configuration

This would be a big undertaking. The good news is that it's already on my todo 
list, although for an entirely different reason... By supporting completely 
independent populations, a low scoring low markup population could can survive 
along side of a high scoring high markup population. Think one population 
focused on low markup high frequency trading and another that would focus on 
large price swings and high markups; both actively trading at the same time.

I guess this was a long winded way of saying yes I'm interested in making the 
requested change.

Thank you for the clarification and the interest. Can't wait to test out the 
changes =)

In the meantime, maybe you want to add this clarification to the ga-bitbot FAQ 
so other people do not try to update their instances from the repository and 
hope to keep their old populations.

The FAQ has been updated.

Work has started on implementation.

gene_server has been updated to implement the gene_library (db collection). 
Backwards compatibility has been maintained but upon the first db reload, the 
old db archive files will be converted to the new format before being deleted.  
The converted data will be stored in the UNDEFINED library. Before it can be 
used, I'll need to create a separate utility script to allow merging & deleting 
of databases. This will be needed to link the gene_def.json file to the db. 
Only after the converted db has been linked will it become active and shareable.

unregistered gts clients (if somehow the old gts.py script is used in the new 
system) will be captured in the UNDEFINED library. 

gts has been updated to register the local gene_def on startup. gts clients are 
also now registered and linked to a gene_def. This allows gts clients to work 
on different databases in the library. 

A new command line option was added to gts 'get_config'. If set, gts will pull 
a random gene_def from the gene server. If no configurations are available gts 
will fall back to the local gene_def.json file.

gal has been updated to launch all gts clients, except for one, with the 
get_config option. This will ensure any new gene_def.json files will be loaded 
into the db.

gene_def has been updated to add a name, description and version variable. This 
is to allow users to tag their gene_def files.

A MD5 hash of the gene_def file is used to register it. Any changes to the file 
at all, even adding spaces, will change the hash and result in a fork in the 
gene library.

When starting the system, the first client to register with the gene server 
sets the default db. The default db is what is used throughout the rest of the 
system (gene vis,charts,bcbookie). This is only temporary until I decide if and 
how I want to automate the switching of default databases from within 
report_gen.py.

Sounds great! =)

Tried updating my current instance [git pull] and it seems to work correctly. 
Also, ran the empty, new, instances for a few hours and it also seemed to work 
as intended. Can't wait for the the opportunity to merge my populations, 
exchange/mate them with other users and see what the results will be =)

The only weird issue I noticed is that bcbookie reports:
main: warning - ignoring invalid target order:
{u'status': u'active', u'buy': 4.69841, u'actual': 0, u'target': 4.87918, 
u'sell_period': 0, u'age': 0, u'stop': 4.646258164388718, u'shares': 
5.238379999999999, u'cost': 24.813316301508454, u'master_index': 2, 
u'trade_pos': 976.220180903862, u'score': 0, u'stop_age': 13020.0, 
u'buy_period': 99999}

...which I presume is only because it does not have a valid buy order at this 
moment due to pretty stagnant market conditions.

Please, feel free to ask for my help if you think I could help in any way.

Market moved, "system monitor" reports "Buy Order Trigger" > 0, but bcbookie 
still reports "ignoring invalid target order":

{u'status': u'active', u'buy': 4.7116, u'actual': 0, u'target': 4.8911, 
u'sell_period': 0, u'age': 0, u'stop': 4.22110970233263, u'shares': 
5.238379999999999, u'cost': 24.8828278873523, u'master_index': 97, 
u'trade_pos': 979.1069574997002, u'score': 0, u'stop_age': 491460.0, 
u'buy_period': 99999}

Seems the bcbookie does not want to play nice with the new version after all =(

regarding comment #6, the issue was caused by a recent change to genetic.py 
which unintentionally caused integers to be generated as floats. This was 
changed back. 

bcbookie has been changed to validate orders with either type. The issue was 
specifically with the stop_age variable; stop_age is now recast as in int 
durring order verification.