Quast error about internet connection about quast HOT 18 CLOSED

I realize this issue is marked closed, but I've just encountered this error, and thought it would better to post here, rather than open a new issue. Anyway, I definitely have internet access, so I don't have any idea why that error's being thrown. Any suggestions on where to look or how to proceed would be greatly appreciated.

Here're the contents of the metaquast.log:

/home/sam/programs/quast-5.0.2/metaquast.py /home/sam/data/metagenomics/P_generosa/final.contigs.fa --threads 22

Version: 5.0.2

System information:
  OS: Linux-4.4.0-139-generic-x86_64-with-debian-stretch-sid (linux_64)
  Python version: 3.7.2
  CPUs number: 24

Started: 2019-01-10 11:10:58

Logging to /home/sam/analyses/20190110_geo_metagnomics_metaquast/quast_results/results_2019_01_10_11_10_58/metaquast.log

Contigs:
  Pre-processing...
  /home/sam/data/metagenomics/P_generosa/final.contigs.fa ==> final.contigs

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2019-01-10 11:12:29
Running BlastN..
  processing final.contigs
/home/sam/programs/quast-5.0.2/quast_libs/blast/blastn -query /home/sam/data/metagenomics/P_generosa/final.contigs.fa \
-db /home/sam/programs/quast-5.0.2/quast_libs/silva/silva.123.db -outfmt 7 -num_threads \
22 > quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.res_final-contigs \
2>> quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.err
  BLAST results for final.contigs are saved to /home/sam/analyses/20190110_geo_metagnomics_metaquast/quast_results/results_2019_01_10_11_10_58/quast_downloaded_references/blast.res_final-contigs...


2019-01-10 11:56:47
Trying to download found references from NCBI. Totally 50 organisms to try.
  unidentified_proteobacterium                                          | not found in the NCBI database
  unidentified_proteobacterium was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

In case you have troubles running QUAST, you can write to [email protected]
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.

from quast.

I fixed the problem by adding "time.sleep(1)" to the source code quast_lib/search_references_meta.py method "def try_send_request(url)".
NCBI recommends that users post no more than three URL requests per second; the time.sleep(1) in the method try_send_request(url) slows the speed of sending requests.

from quast.

import time at the beginning of the file

from __future__ import with_statement
import os
import shlex
import shutil
import re
from collections import defaultdict
import time
from os.path import isdir, isfile, join

from quast.

Thanks @ChunfangZheng, @kubu4 and @holtjma -- we added import time and time.sleep(1) solution to the master branch (will be available in the next release).

By the way, if you find a problem related to a closed issue feel free to reopen it!

from quast.

Hi,
Unfortunately, we didn't release a new version of Quast for a while, so the current latest release (version 5.0.2) is not yet affected by the fix mentioned here (I mean using time.sleep(1)).
So, could you please try to apply the fix locally to your copy of 5.0.2 and check whether it works for you?
To do so, please download the fixed copy of search_references_meta.py (use Right-Click and "Save as" menu), and put it in <your_quast_installation_dir>/quast_libs/ dir to replace the old version. After that, try to restart Quast.

from quast.

I am having the same issue with a newer version of quast when I try to run it over a series of assemblies using a slurm array.

quast 5.2.0 py39pl5321h2add14b_2 bioconda

This is how I try to run it:

conda activate quast
echo "version of `quast --version`" 

metaquast.py \
	--threads $cores \
	--no-icarus \
	$project_dir/outputs/b02_assembly_spades/${sampleID}/scaffolds_hq.fasta \
	-o $project_dir/outputs/b05_evaluateAssembly_metaquast/${sampleID}/	

conda deactivate

And this is the error I got

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2023-04-05 16:48:16
Running BlastN..
  processing scaffolds_hq
  BLAST results for scaffolds_hq are saved to /mnt/research/EvansLab/Benucci/metaGAAP/outputs/b05_evaluateAssembly_metaquast/09_REX_JJ_S5_L003/quast_downloaded_references/blast.res_scaffolds_hq...


2023-04-05 16:49:16
Trying to download found references from NCBI. Totally 50 organisms to try.
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
  delta_proteobacterium_WY32                      | not found in the NCBI database
  delta_proteobacterium_WY32 was not found in NCBI database, trying to download the next best match
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
  delta_proteobacterium_LX33                      | not found in the NCBI database
  delta_proteobacterium_LX33 was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0" to disable reference search in the NCBI database.

In case you have troubles running QUAST, you can write to [email protected]
or report an issue on our GitHub repository https://github.com/ablab/quast/issues
Please provide us with quast.log file from the output directory.

Thank you,
Gian

from quast.

Hello @MaryoHg,

no, I still have that issue. @alexeigurevich any solve?

Thanks,

G.

from quast.

Looks like this was actually a network problem.

from quast.

I'm having the same issue. I'm running metaquast on an AWS EC2 instance which has internet access. I have been running it successfully until about last week. Could there be a change to the NCBI API?

from quast.

Was also wondering if government shutdown was possibly impacting access?

from quast.

I fixed the problem by adding "time.sleep(1)" to the source code quast_lib/search_references_meta.py method "def try_send_request(url)".
NCBI recommends that users post no more than three URL requests per second; the time.sleep(1) in the method try_send_request(url) slows the speed of sending requests.

Thanks for this! This alleviated the issue for me.

from quast.

Thanks for this! This alleviated the issue for me.

Never mind. Got too excited, too quickly! Although I didn't get an NCBI connection error, I did get a NameError: name 'time' is not defined. Is there another line I need to add to define time()?

from quast.

In order to alleviate the NameError: name 'time' is not defined error, I think you also need to import the time module (e.g.):

import time

Insert that line somewhere at the top of the script with the other import commands should fix it.

from quast.

I randomly happened to run into this issue at the same time as you guys; the time import and insertion of time.sleep(1) worked for me as well.

from quast.

Hi, sorry if I'm missing the resolution to this elsewhere but I'm running quest 5.0.2 in metaquast mode and still getting this issue downloading references from NCBI. My network isn't having any issues that I can find, is there another workaround that you might suggest? Thanks!

Version: 5.0.2

System information:
OS: Linux-3.10.0-693.21.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core (linux_64)
Python version: 3.6.7
CPUs number: 32

Started: 2019-11-20 08:35:15

Logging to /home/mwoodwo/prj/PREMIX/mg/03.quast_output/metaquast.log
NOTICE: Output directory already exists and looks like a QUAST output dir. Existing results can be reused (e.g. previously generated alignments)!

Contigs:
Pre-processing...
1 ./PM1-S1/contigs.fasta ==> PM1-S1_contigs
2 ./PM1-S2/contigs.fasta ==> PM1-S2_contigs
3 ./PM1-S3/contigs.fasta ==> PM1-S3_contigs
4 ./PM1-C1D01/contigs.fasta ==> PM1-C1D01_contigs
5 ./PM1-C1D02/contigs.fasta ==> PM1-C1D02_contigs
6 ./PM1-C1D15/contigs.fasta ==> PM1-C1D15_contigs
7 ./PM1-C1D36/contigs.fasta ==> PM1-C1D36_contigs
8 ./PM3-S1/contigs.fasta ==> PM3-S1_contigs
9 ./PM3-C0D15/contigs.fasta ==> PM3-C0D15_contigs
10 ./PM3-C0D1/contigs.fasta ==> PM3-C0D1_contigs
11 ./PM3-C0D2/contigs.fasta ==> PM3-C0D2_contigs
12 ./PM3-C0D36/contigs.fasta ==> PM3-C0D36_contigs
13 ./SD-01/contigs.fasta ==> SD-01_contigs

No references are provided, starting to search for reference genomes in SILVA 16S rRNA database and to download them from NCBI...

2019-11-20 08:35:28
Using existing BLAST alignments for PM1-S1_contigs...
Using existing BLAST alignments for PM1-S2_contigs...
Using existing BLAST alignments for PM1-S3_contigs...
Using existing BLAST alignments for PM1-C1D01_contigs...
Using existing BLAST alignments for PM1-C1D02_contigs...
Using existing BLAST alignments for PM1-C1D15_contigs...
Using existing BLAST alignments for PM1-C1D36_contigs...
Using existing BLAST alignments for PM3-S1_contigs...
Using existing BLAST alignments for PM3-C0D15_contigs...
Using existing BLAST alignments for PM3-C0D1_contigs...
Using existing BLAST alignments for PM3-C0D2_contigs...
Using existing BLAST alignments for PM3-C0D36_contigs...
Using existing BLAST alignments for SD-01_contigs...

2019-11-20 08:35:36
Trying to download found references from NCBI. Totally 164 organisms to try.

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0".

from quast.

from quast.

Dear @Gian77

How is it going?
Were you able to fix the problem?
I'm facing the same error while using metaquast.py in a server with full access to the internet.

The option time.sleep(1) is already added to quast_libs/search_references_meta.py script, so I'm not sure how to proceed. I'll run the code in a personal computer to double check.

Dear @alexeigurevich, any advice will be appreciated.

Saludos.

from quast.

I am also facing the same issue with the latest version (5.2.0)

Trying to download found references from NCBI. Totally 35 organisms to try.
human gut metagenome has too fragmented reference genome! It will not be downloaded.
human_gut_metagenome | not found in the NCBI database
human_gut_metagenome was not found in NCBI database, trying to download the next best match
Trichuris_trichiura_(human_whipworm) | not found in the NCBI database
Trichuris_trichiura_(human_whipworm) was not found in NCBI database, trying to download the next best match
Prevotella_copri_DSM_18205 | successfully downloaded (total 1, 34 more to go)
Veillonella_parvula | successfully downloaded (total 2, 33 more to go)
Tyzzerella_sp._Marseille-P3062 | successfully downloaded (total 3, 32 more to go)
Sutterellaceae_bacterium_Marseille-P2968 | successfully downloaded (total 4, 31 more to go)
Streptococcus_thermophilus_ND03 | successfully downloaded (total 5, 30 more to go)
Streptococcus_salivarius_subsp._thermophilus | successfully downloaded (total 6, 29 more to go)
Streptococcus_pneumoniae | successfully downloaded (total 7, 28 more to go)
Streptococcus_phage_2096.2 | not found in the NCBI database
Streptococcus_phage_2096.2 was not found in NCBI database, trying to download the next best match
Streptococcus_pyogenes_phage_6180.4 | not found in the NCBI database
Streptococcus_pyogenes_phage_6180.4 was not found in NCBI database, trying to download the next best match
Streptococcus_phage_10750.4 | not found in the NCBI database
Streptococcus_infantarius | successfully downloaded (total 8, 26 more to go)
Streptococcus_agalactiae_SS1219 | successfully downloaded (total 9, 25 more to go)
Serratia_marcescens | successfully downloaded (total 10, 24 more to go)
Neisseria_zalophi | successfully downloaded (total 11, 23 more to go)
Myroides_odoratimimus_CCUG_12700 | successfully downloaded (total 12, 22 more to go)
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
Lactobacillus_sp._C33LV5 | not found in the NCBI database
Lactobacillus_sp._C33LV5 was not found in NCBI database, trying to download the next best match
WARNING: Empty/No response from NCBI. Could be an Internet connection issue!
Lactobacillus_ruminis | not found in the NCBI database
Lactobacillus_ruminis was not found in NCBI database, trying to download the next best match

ERROR! Cannot established internet connection to download reference genomes! Check internet connection or run MetaQUAST with option "--max-ref-number 0" to disable reference search in the NCBI database.

from quast.