Quast Running times. about quast HOT 8 CLOSED

Hi!
Normally, the analysis should finish much faster than 3 days for such reference length. So, probably something is going wrong. Could you send us the log files for the detailed analysis (quast.log and contigs_reports/*.stdout and *.stderr files under <quast_output_dir>)?

Our support email is [email protected]

from quast.

Thanks, Alexei. I sent an email with the log files.

from quast.

Do you have any answer for this issue?
My quast has already run for over a month. The newest file (contigs_report_**.mis_contigs.info) was generated about 20 days ago. I have not seen any newer results since then. My code was like:

quast.py contigs.fasta -R reference.fasta -G reference.gff3 -e --threads 8 -o out

The assembled file was ~400Mbp with ~370Mbp reference genome. No error reported in quast.log, only find a warning in contigs_reports/contigs_report_*.stderr

WARNING: Unable to run 'gnuplot contigs_reports/nucmer_output/contigs-quiver.gp', Inappropriate ioctl for device

No error found in stderr_part** files either, and quast is still running :-)
I pasted several log files here, any suggestion is appreciated. Thanks!

quast.log

/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/EGG-INFO/scripts/quast.py contigs.fasta -R reference.fasta -G reference.gff3 -e --threads 8 -o out

Version: 4.5

System information:
OS: Linux-2.6.32-642.13.1.el6.x86_64-x86_64-with-centos-6.8-Final (linux_64)
Python version: 2.7.9
CPUs number: 32

Started: 2017-03-20 19:30:13

Logging to out/quast.log
NOTICE: Output directory already exists. Existing Nucmer alignments can be used

CWD: out
Main parameters:
Threads: 8, eukaryotic: true, minimum contig length: 500, ambiguity: one, threshold for extensive misassembly size: 1000

Reference:
reference.fasta ==> reference

Contigs:
Pre-processing...
contigs.fasta ==> contigs

2017-03-20 19:30:45
Running Basic statistics processor...
Reference genome:
reference.fasta, length = 373245519, num fragments = 12, GC % = 43.56
Contig files:
contigs
Calculating N50 and L50...
contigs, N50 = 927700, L50 = 132, Total length = 399862497, GC % = 43.72, # N's per 100 kbp = 0.00
Drawing Nx plot...
saved to out/basic_stats/Nx_plot.pdf
Drawing NGx plot...
saved to out/basic_stats/NGx_plot.pdf
Drawing cumulative plot...
saved to out/basic_stats/cumulative_plot.pdf
Drawing GC content plot...
saved to out/basic_stats/GC_content_plot.pdf
Drawing contigs GC content plot...
saved to out/basic_stats/contigs_GC_content_plot.pdf
Done.

2017-03-20 19:31:20
Running Contig analyzer...
contigs
Logging to files out/contigs_reports/contigs_report_contigs.stdout and contigs_report_contigs.stderr...
Aligning contigs to the reference
Aligning to different chromosomes in parallel (8 threads)
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_1.fasta
--emem -t 1 quast_corrected_input/split_ref/part_1.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part1
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_2.fasta
--emem -t 1 quast_corrected_input/split_ref/part_2.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part2
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_3.fasta
--emem -t 1 quast_corrected_input/split_ref/part_3.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part3
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_4.fasta
--emem -t 1 quast_corrected_input/split_ref/part_4.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part4
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_5.fasta
--emem -t 1 quast_corrected_input/split_ref/part_5.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part5
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_8.fasta
--emem -t 1 quast_corrected_input/split_ref/part_8.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part8
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_7.fasta
--emem -t 1 quast_corrected_input/split_ref/part_7.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part7
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_6.fasta
--emem -t 1 quast_corrected_input/split_ref/part_6.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part6
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_9.fasta
--emem -t 1 quast_corrected_input/split_ref/part_9.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part9
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_10.fasta
--emem -t 1 quast_corrected_input/split_ref/part_10.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part10
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/nucmer
-c 65 -l 65 --maxmatch -p contigs_reports/nucmer_output/contigs_part_11.fasta
--emem -t 1 quast_corrected_input/split_ref/part_11.fasta quast_corrected_input/contigs.fasta
>> contigs_reports/contigs_report_contigs.stdout
2>> contigs_reports/contigs_report_contigs.stderr_part11
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/delta-filter
-i 95.0 -l 0 contigs_reports/nucmer_output/contigs.delta
> contigs_reports/nucmer_output/contigs.fdelta
2> contigs_reports/contigs_report_contigs.stderr
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/mummerplot
--html --layout -p contigs_reports/nucmer_output/contigs
contigs_reports/nucmer_output/contigs.delta > contigs_reports/contigs_report_contigs.stdout
2> contigs_reports/contigs_report_contigs.stderr
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/gnuplot/src/gnuplot
contigs_reports/nucmer_output/contigs.gp > /dev/null
2> contigs_reports/contigs_report_contigs.stderr
MUMmer plot saved to out/contigs_reports/contigs_mummerplot.html
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/show-coords
contigs_reports/nucmer_output/contigs.delta > contigs_reports/nucmer_output/contigs.coords_tmp
2> contigs_reports/contigs_report_contigs.stderr
/home/wl/lib/python2.7/site-packages/quast-4.5-py2.7.egg/quast_libs/MUMmer/show-snps
-S -T -H contigs_reports/nucmer_output/contigs.delta
< contigs_reports/nucmer_output/contigs.coords.headless
> contigs_reports/nucmer_output/contigs.all_snps
2> contigs_reports/contigs_report_contigs.stderr

contigs_report_contigs.stderr:

Writing filtered delta file contigs_reports/nucmer_output/contigs.filter
Reading delta file contigs_reports/nucmer_output/contigs.delta
Writing plot files contigs_reports/nucmer_output/contigs.fplot, contigs_reports/nucmer_output/contigs.rplot
Writing gnuplot script contigs_reports/nucmer_output/contigs.gp
Rendering plot contigs_reports/nucmer_output/contigs.html

set terminal canvas jsdir ""
^
"contigs_reports/nucmer_output/contigs.gp", line 1: unknown or ambiguous terminal type; type just 'set terminal' for a list

WARNING:
Unable to run 'gnuplot contigs_reports/nucmer_output/contigs.gp', Inappropriate ioctl for device
Stderr outputs for reference parts are in:
contigs_reports/contigs_report_contigs.stderr_part1
contigs_reports/contigs_report_contigs.stderr_part2
contigs_reports/contigs_report_contigs.stderr_part3
...
...
contigs_reports/contigs_report_contigs.stderr_part11

content of the stderr_part files:

1: PREPARING DATA
2,3: RUNNING e-mem AND CREATING CLUSTERS
4: FINISHING DATA

from quast.

Here is the CPU and memory usage. It just cost over 100G memory and seems run forever ...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2386 wl 20 0 114g 113g 2028 R 99.4 15.0 49110:48 quast.py

from quast.

Hi Wang,

The problem may be specific for you particular dataset. I will copy here part of my answer to Miquel (the author of this issue) which I wrote after analysing his logs and data. The Miquel problem was in very fragmented reference genome which is not your case (your reference has just 12 fragments, probably chromosomes). However, the following piece of advice may be appropriate for your case too:

"You can use more strict alignment settings to focus on longer and accurate seeds, this also result in faster processing. For doing it, you should use --min-cluster 200 (or another number). Default value for minimal exact match is 65 bp, the aligner finds all such seed and combine them into the full scale alignment. In your case default 65 bp size results in too many matches and slow processing. 200 bp should be fine."

Let me know how it goes. You could also write to our support email ([email protected])

from quast.

Many thanks Alexei! I have re-run quast with "--min-cluster 200" and "--min-cluster 500". It has now been run for a whole day. I'll return to you once any job is finished.

from quast.

Hi! I am wondering if I am having a similar problem or something different. Quast has been hanging at 'Running contig analyzer' for ~ 2 days (last logged output and modified files 2 days ago). I'm running on 2 megahit assemblies of metagenomic communities (both 1.1G) and my combined reference is 2.1G. There were no obvious errors in the stderr files (attached below ).

I did get a warning that my reference is fragmented, Quast no longer appears to have --min-cluster option (threw an error when I tried to use it), but I tried rerunning with --fragmented (skipping metaquast steps, and just rerunning from quast .py with the combined reference). Now, I'm getting a Key Error (KeyError: 'Streptococcus_mitis_AP023349.1' ERROR! exception caught!) in contigs_analyzer.py when calling pool.py (log files attached below )

Thanks in advance for any help or suggestions!

Original code:
metaquast -o MegahitQuastResults -l megahitdefault,megahitlarge megahit_default/final.contigs.fa megahit_large/final.contigs.fa
Original log & error files:
RunMetaQuast-19208132.log.txt
contigs_report_megahitlarge.stderr.txt
quast.log.txt

Code with fragmented option:
/home/blostein/.conda/envs/meta_assembly/opt/quast-5.0.2/quast.py --combined-ref --fragmented --ambiguity-usage all megahit_default/final.contigs.fa megahit_large/final.contigs.fa -R /gpfs/accounts/bfoxman_root/bfoxman/blostein/meta_CAVITIES/Files/newPipeline/MegahitQuastResults/quast_corrected_input/combined_reference.fasta -o /gpfs/accounts/bfoxman_root/bfoxman/blostein/meta_CAVITIES/Files/newPipeline/MegahitQuastResults/combined_reference_fragmented --labels "megahitdefault_f, megahitlarge_f”

Log & error files w/fragmented options
RunQuastFrag-19293196.log.txt
fragemented_quast.log.txt
contigs_report_megahitlarge_f.stderr.txt

from quast.

@blostein i know your comment was old but I was also having a similar issue. From the manual I think --min-cluster was replaced with --min alignment. Here is the description of --min-alignmnet and it seems to match the 65bp mentioned above.

--min-alignment (or -i)
Minimum length of alignment (in bp). Alignments shorter than this value will be filtered. Note that all alignments shorter than 65 bp will be filtered regardless of this threshold.

from quast.