pplacer OOM about drep HOT 3 CLOSED

Hey Bubba,

1. With regards to checkM filtering, there are a couple of options:

• Run with --checkM_method taxonomy_wf. This will skip pplacer entirely. The only downside is that it's not robust to archaea (uses universal bacterial single copy genes)

• Filter the genomes with checkM on your own using whatever parameters are best for your system, and then just run dRep with the `--skipCheckM`` option. This won't work if you want dRep to pick the best genomes.

• Run checkM on your own using whatever parameters are best for your system, make a tab_table of your results, and import them to dRep using --tab_table. This will make allow dRep to do the filtering and choose the best genomes based on the checkM results for you.

2. That seems like a problem with centrifuge, not checkM. It looks like a lot of the genomes have a _hits.tsv file, but not a report.tsv file. I suspect that in cases where there is no _report.tsv file, the corresponding _hits.tsv file is empty. Could you confirm that for me? If so, I'll just write a patch into the program that's able to handle empty files.

Thanks!
-Matt

from drep.

Hey Matt,

1. I tried your first suggestion since I'm not too concerned with archaea at the moment...

In a fresh dir, I --checkM_method taxonomy_wf:

nice dRep dereplicate_wf . -g genomes/* -o -p 16 --checkM_method taxonomy_wf --run_tax --S_algorithm gANI -sa 0.99 --cent_index /home/ubuntu/scripts/github/centrifuge/indices/b+h+v &> drep.log

It ended up terminating at the gANI step:

$ tail cmd_drep_gANI_gt99.log
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/ubuntu/miniconda3/envs/irep/lib/python3.6/site-packages/pandas/io/parsers.py", line 730, in __init__
    self._make_engine(self.engine)
  File "/home/ubuntu/miniconda3/envs/irep/lib/python3.6/site-packages/pandas/io/parsers.py", line 923, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/home/ubuntu/miniconda3/envs/irep/lib/python3.6/site-packages/pandas/io/parsers.py", line 1390, in __init__
    self._reader = _parser.TextReader(src, **kwds)
  File "pandas/parser.pyx", line 373, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:4184)
  File "pandas/parser.pyx", line 667, in pandas.parser.TextReader._setup_parser_source (pandas/parser.c:8449)
FileNotFoundError: File b'/home/ubuntu/db/ncbi_Blautia_20170418/dRep/gANI_gt99/test_mem1/data/gANI_files/GCF_001404535.1_13414_6_21_genomic.fna_vs_GCF_000153905.1_ASM15390v1_genomic.fna.gANI' does not exist

Indeed the file does not exist, but the two genomes gANI is trying to compare do exist. Nothing seems odd about the files (i.e. not empty or named weird). Is there there an issue with the --checkM_method taxonomy_wf flag? Attached is the logger.log in case that's helpful.

2. Yeah, this is really odd - but that's what's going on. In the previously mentioned run there are 22 hits:

$ l ../data/centrifuge/*fna_hits.tsv | wc -l
22

And 10 reports:

$ l ../data/centrifuge/*fna_report.tsv | wc -l
10

The centrifuge index I'm using wasn't built on this particular AMI, but I don't think that should matter...

Ok, this is going in all sorts of directions. Sorry!

logger.txt

from drep.

Hey Bubba,

1. Try the following command:

/usr/local/bin/ANIcalculator -genome1fna /home/ubuntu/db/ncbi_Blautia_20170418/dRep/gANI_gt99/test_mem1/data/prodigal/GCF_001404535.1_13414_6_21_genomic.fna.fna -genome2fna /home/ubuntu/db/ncbi_Blautia_20170418/dRep/gANI_gt99/test_mem1/data/prodigal/GCF_000153905.1_ASM15390v1_genomic.fna.fna -outfile /home/ubuntu/db/ncbi_Blautia_20170418/dRep/gANI_gt99/test_mem1/data/gANI_files/GCF_001404535.1_13414_6_21_genomic.fna_vs_GCF_000153905.1_ASM15390v1_genomic.fna.gANI -outdir /home/ubuntu/db/ncbi_Blautia_20170418/dRep/gANI_gt99/test_mem1/data/gANI_files/GCF_001404535.1_13414_6_21_genomic.fna_vs_GCF_000153905.1_ASM15390v1_genomic.fna.gANITEMP

Any errors? Are the output files made? That was what dRep tried to run to make that file that doesn't exist

2. Could you upload three files for me:

1 + 2) For a genome that has both a _hits.tsv and a _report.tsv file, upload both of those

3. For a genome that has only a _hits.tsv file, upload it.

Best,
-Matt

from drep.