check genome_updater for genbank/refseq downloads about databases HOT 8 CLOSED

I used it to create a mirrored subset of genbank and refseq (archaea, bacteria, fungi, viral, protozoa), and really liked it. I don't have time to make a pull request now, but some blueprints:

• genome_updater.sh supports recovering files from a previously obtained assembly summary file with the -e option. This is really useful for reproducing how a database was built, and we can stored it somewhere (osf? zenodo?)
• Instead of finding all files in a dir, use the assembly_summary.txt to select files (from a domain like 'archaea' or 'fungi').
• From previous item: build separate DBs for each domain. This is something that the euk SBTs already do (see https://github.com/dib-lab/2018-euk-SBTs and https://osf.io/a46zr/)
• Another benefit of using the assembly_summary.txt: We don't depend on acc2taxid to get the taxonomy of each dataset. This can also be a mapping generated during database construction, and saved in the zipped SBT (or whatever taxonomy format we end up adopting in sourmash: sourmash-bio/sourmash#969

I think most of these items fit sourmash-bio/sourmash#970 too.

from databases.

cool, I'll look into using it - is there a seed on farm that I can start from, or shall I just re-download the whole shebang?

from databases.

Re-download, I did this in my machine.

from databases.

Very good experience so far --

# make group directory
cd /group/ctbrowngrp
mkdir ncbi-genomes
cd ncbi-genomes

# install genome_updater
13820  conda create -n genupd -y genome_updater 
13821  conda activate genupd

# grab all shewanella genomes - dry run
genome_updater.sh -g "taxids:22" -k -m -o shew

# with one thread
genome_updater.sh -g "taxids:22" -m -o shew

# with four threads, + fix directory (-i)
genome_updater.sh -g "taxids:22" -m -o shew -i -t 4

from databases.

Another benefit of using the assembly_summary.txt: We don't depend on acc2taxid to get the taxonomy of each dataset. This can also be a mapping generated during database construction, and saved in the zipped SBT (or whatever taxonomy format we end up adopting in sourmash: dib-lab/sourmash#969

Also: instead of using --name-from-first, set the --name based on the assembly_summary.txt. This gives us better control on the right accession number, or even using the GCF/GCA identifiers instead. More info: luizirber/2020-cami#1

from databases.

(yep, that's how the GTDB snakemake does it)

from databases.

note to self, put things in /group/ctbrowngrp/

from databases.

closing in favor of our modern database build practices.

from databases.