install_data.py requires more than 30 GiB of memory about basset HOT 2 OPEN

Thanks for pointing this out. The DNA sequences stored as float16's are responsible for the high memory usage. I originally made that choice to allow for storing N's as 0.25 vectors, but I haven't found that to be very helpful in the years since. I just pushed a new commit to store them as booleans, which cuts the memory usage in half. (Python unfortunately uses 8 bits for a boolean.)

I also removed the "-r" option, which permutes the sequences before dividing into train/valid/test according to the current order. I believe the script should always permute, since input BED files will often be nonrandomly ordered. I recommend that you regenerate your training dataset with the new commit.

from basset.

Thank you, @davek44 ! Let me play around with the new commit

from basset.