train/test split is inefficient about mmvec HOT 4 OPEN

I think this is related to biocore/biom-format#808 —until recently, the DataFrame produced by biom.Table.to_dataframe() was effectively dense. This caused some silly scaling problems with rankratioviz, also.

For rankratioviz, what I ended up doing to address this problem was extracting the scipy.sparse.csr_matrix data from the BIOM table and using that to construct an actually-sparse DataFrame. Here's the code that does this (it's only ~4 lines of work between creating the DF and then populating its IDs). This should work even on BIOM versions with the biocore/biom-format#808 bug.

from mmvec.

from mmvec.

what about only performing train/test split on the metadata? And then you subsample the biom tables accordingly. This is what is done in q2-sample-classifier, and could be applied here.

from mmvec.

@nbokulich that makes a lot of sense. And the refactored solution will ultimately require a solution something like that. Furthermore, getting this right will ultimately help with scalability.

Currently looking into interleaving this with pytorch's DataLoader class - this will help parallel processing so that the GPUs will be saturated (should speed things up at least 2x fold).

from mmvec.