Dear all

We have been trying to set up the Mantis database, but we have been encountering the following errors:

 python  mantis/  check_installation
Executing command:
 mantis/ check_installation
Checking installation
#  External data folders:
------------------------------------------
Default hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/
Custom hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/custom_hmms/
NCBI dump folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/NCBI/
Gene ontology folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/Gene_Ontology/
Uniprot dump folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/Uniprot/
TAX NOG hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/
Global NOG hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGG/
Pfam hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/pfam/
KOfam hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/kofam/
TIGRFAM hmms folder:
/disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/tigrfam/
------------------------------------------
Cython correctly compiled!
Checking extra files
Checking extra files
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/essential_genes
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/NCBI/
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/Uniprot/
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/Resources/Gene_Ontology/
Checking HMM installation
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/10/
10_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/104264/
104264_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/1028384/
.h3m; .h3p; .hmm; .h3i; .h3f; 1028384_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/10656/
10656_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/1161/
1161_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/119043/
119043_sql_annotations.tsv
................................

.h3m; .h3p; .hmm; 57723_sql_annotations.tsv; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/5794/
.h3m; .h3p; 5794_sql_annotations.tsv; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/5796/
5796_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/5809/
.h3m; .h3p; .hmm; .h3i; .h3f; 5809_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/5819/
.h3m; .h3p; .hmm; .h3i; .h3f; 5819_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/583/
583_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/586/
586_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/5863/
5863_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/5878/
5878_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/58840/
58840_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/590/
590_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/59732/
.h3m; .h3p; .hmm; .h3i; .h3f; 59732_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/60136/
.h3m; .h3p; .hmm; .h3i; .h3f; 60136_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/613/
.h3m; .h3p; .hmm; .h3i; .h3f; 613_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/61432/
61432_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/622450/
622450_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/6231/
6231_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/6236/
.h3m; .h3p; .hmm; 6236_sql_annotations.tsv; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/629/
.h3m; .h3p; .hmm; .h3i; .h3f; 629_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/629295/
629295_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/639021/
639021_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/651137/
651137_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/6656/
.h3m; .h3p; .hmm; .h3i; .h3f; 6656_sql_annotations.tsv
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/671232/
.h3m; .h3p; .hmm; .h3i; 671232_sql_annotations.tsv; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGT/675063/
675063_sql_annotations.tsv; .h3m; .h3p; .hmm; .h3i; .h3f
Failed installation check on [files missing]: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/NOGG/
.h3m; .h3p; .hmm; .h3i; NOGG_sql_annotations.tsv; .h3f
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/pfam/
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/kofam/
Passed installation check on: /disk/rdisk08/zhiyshen/zhiyshen/tools/mantis/mantis/hmm/tigrfam/
Installation check failed on: 303
------------------------------------------
--------INSTALLATION CHECK FAILED!--------
------------------------------------------

I am not entire sure why some of the files are missing in these NOGG/NOGT subdirectories. If anyone can point me in the right direction that would be great! Thanks

Regards

Marc

Hi,

I am trying to setup the software.
When I clone the repo from the master branch I got a Resouces/NOG_sql.tar.gz which is a text file with the next content:

version https://git-lfs.github.com/spec/v1
oid sha256:cecd31773a7cc729723d298b158d6cdc98187a143de2255095e73cb81f1ea394
size 176132348

When I use the setup or the check command, I got errors, apparently because of this:

raceback (most recent call last):
File "miniconda3/envs/mantis_env/lib/python3.8/shutil.py", line 1155, in _unpack_tarfile
tarobj = tarfile.open(filename)
File "miniconda3/envs/mantis_env/lib/python3.8/tarfile.py", line 1606, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "miniconda3/envs/mantis_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "miniconda3/envs/mantis_env/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "mantis/main.py", line 114, in
setup_databases(force_download=force_download,chunk_size=chunk_size,mantis_config=mantis_config)
File "mantis/source/MANTIS_Assembler.py", line 28, in setup_databases
mantis.setup_databases(force_download)
File "mantis/source/utils.py", line 417, in wrapper
res = f(self,*args, **kwargs)
File "mantis/source/MANTIS_DB.py", line 57, in setup_databases
self.prepare_queue_extract_metadata()
File "mantis/source/MANTIS_DB.py", line 97, in prepare_queue_extract_metadata
self.unpack_NOG_sql(stdout_file=stdout_file)
File "mantis/source/MANTIS_DB.py", line 430, in unpack_NOG_sql
uncompress_archive(source_filepath=NOG_sql_path,extract_path=resources_path,stdout_file=stdout_file,remove_source=False)
File "mantis/source/utils.py", line 428, in uncompress_archive
unpack_archive(source_file=source_filepath, extract_dir=extract_path, remove_source=remove_source, stdout_file=None)
File "mantis/source/utils.py", line 458, in unpack_archive
shutil.unpack_archive(source_file,extract_dir=extract_dir)
File "miniconda3/envs/mantis_env/lib/python3.8/shutil.py", line 1227, in unpack_archive
func(filename, extract_dir, **kwargs)
File "miniconda3/envs/mantis_env/lib/python3.8/shutil.py", line 1157, in _unpack_tarfile
raise ReadError(
shutil.ReadError: mantis/mantis/Resources/NOG_sql.tar.gz is not a compressed or uncompressed tar file

I first thought that Resources/NOG_sql/ would contain the data of NOG_sql.tar.gz, so I removed NOG_sql.tar.gz and created it again using NOG_sql. However, doing this I am getting:

Traceback (most recent call last):
File "miniconda3/envs/mantis_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "miniconda3/envs/mantis_env/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "mantis/main.py", line 114, in
setup_databases(force_download=force_download,chunk_size=chunk_size,mantis_config=mantis_config)
File "mantis/source/MANTIS_Assembler.py", line 28, in setup_databases
mantis.setup_databases(force_download)
File "mantis/source/utils.py", line 417, in wrapper
res = f(self,*args, **kwargs)
File "mantis/source/MANTIS_DB.py", line 57, in setup_databases
self.prepare_queue_extract_metadata()
File "mantis/source/MANTIS_DB.py", line 97, in prepare_queue_extract_metadata
self.unpack_NOG_sql(stdout_file=stdout_file)
File "mantis/source/MANTIS_DB.py", line 434, in unpack_NOG_sql
for f in os.listdir(NOGG_sql_path):
FileNotFoundError: [Errno 2] No such file or directory: 'mantis/mantis/Resources/NOG_sql/NOGG/'

Therefore, I guess that both NOG_sql and NOG_sql.tar.gz are required and contain different data??

Best,
Carlos

Hi,

I use six-frame translated sequences as input. But I found that many * symbl in translated sequences, will these * affect results?
HMMER doesn't particularly like * characters according to EddyRivasLab/hmmer#205.

Best,
Kun

Many translated sequences like followings
unigene_faa.txt

Allow use of multiple physical computers and implement communication of processes within Mantis
Perhaps we could spawn workers in other physical nodes and then collect data in the initial physical node?

Hey @PedroMTQ,

I think that the GFF format of the output files is not completely correct (which I have unfortunately not noticed before):

• the sequence-regions in the header should be contig IDs and not IDs of the features
• the 1st column ("seqid") should contain the contig ID and not the feature ID
• the 9th column ("attributes") should contain the feature ID, i.e. ID=<feature ID>;...

For reference: GFF3 format convention

Could you have a look at this?

Dear Pedro,

I was wondering if there was a possibility to output the results in a GFF instead of the .tsv files for the consensus annotation?

Best regards,

Massimo

http://www.essentialgene.org/

Let's say we have 3 proteins that get clustered together in an ortholog analysis. They all have very similar but slightly different annotations. Can this tool pick a consensus annotation? If so, what is the best way to do this on the program?

    Taxonomy_SQLITE_Connector.__init__(self,resources_folder=self.mantis_paths['resources'])

should come after reading config

right before:
if self.use_taxonomy:
if nogt_line:
if self.launch_taxonomy_connector():
self.set_nogt_line(nogt_line)

I have installed mantis on a compute server running Ubuntu 18.04.1 LTS using the command 'git clone https://github.com/PedroMTQ/mantis.git' . The git HEAD of the installed version is d983816
I have tried three times to run mantis setup_databases. In each attempt the program has aborted before completion with the error message:
Traceback (most recent call last):
File "/home/ian_reid/conda/envs/mantis_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/ian_reid/conda/envs/mantis_env/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "./mantis/source/MANTIS_DB.py", line 135, in worker_setup_databases
self.run_download_and_unzip(database,force_download,taxon_id,stdout_path)
File "./mantis/source/MANTIS_DB.py", line 162, in run_download_and_unzip
elif database=='uniprot_nlp': self.download_uniprot_reference(force_download=force_download,stdout_file=stdout_file)
File "./mantis/source/MANTIS_DB.py", line 233, in download_uniprot_reference
uncompress_archive(source_filepath=self.mantis_paths['uniprot_nlp'] + file, stdout_file=stdout_file, remove_source=False)
File "./mantis/source/utils.py", line 431, in uncompress_archive
gunzip(source_filepath=source_filepath,dest_filepath=extract_path,block_size=block_size,remove_source=remove_source,stdout_file=stdout_file)
File "./mantis/source/utils.py", line 448, in gunzip
block = s_file.read(block_size)
File "/home/ian_reid/conda/envs/mantis_env/lib/python3.8/gzip.py", line 292, in read
return self._buffer.read(size)
File "/home/ian_reid/conda/envs/mantis_env/lib/python3.8/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/home/ian_reid/conda/envs/mantis_env/lib/python3.8/gzip.py", line 479, in read
if not self._read_gzip_header():
File "/home/ian_reid/conda/envs/mantis_env/lib/python3.8/gzip.py", line 427, in _read_gzip_header
raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b've')

Hi,

Sorry to trouble you again.

And I want to know where the source of pfam metadata come from, it seems that not all GO terms in the metadata.tsv match those in http://current.geneontology.org/ontology/external2go/pfam2go file.

Best,
Kun

'resources' is not being used for taxonomy.db so setup doesnt finish.

v1.4.5
reproducible when running with -mc mantis.cfg and config file has uncommented resources_folder=folder_path

Check for intermediate outputs (output,integrated) and start from there

On two very different systems I'm getting stuck at the same place in the setup.

mantis setup
...
Merging profiles in  /lizardfs/erikg/miniconda3/lib/python3.8/site-packages/References/NCBI/986/to_merge/
Concatenating files into  /lizardfs/erikg/miniconda3/lib/python3.8/site-packages/References/NCBI/986/986_merged.hmm

The setup process simply hangs. On my laptop, I had to kill it. But, on a remote server I'll wait to see if it progresses. Nothing is running as far as htop says and no data is being written.

I have a local copy of uniprot in DMND format at path/to/uniprot.dmnd. I'm trying to get mantis to annotate with it using this config

#This file is meant to be edited in your own distribution of MANTIS

#######DEFAULT PATHS#######
#You can add custom paths here, otherwise they will be set to default values
#resources_folder=
#default_ref_folder=
#####<custom_ref_folder> defines the folder where Mantis will look for MULTIPLE custom HMMs or DMND. Each HMM should be contained within a folder, such that the path will look something like this: /path/to/custom_ref_folder/custom1/custom1.hmm
#custom_ref_folder=
nog_ref=dmnd # dmnd or hmm
#nog_ref_folder=
#pfam_ref_folder=
#kofam_ref_folder=
#ncbi_ref_folder=
#tcdb_ref_folder=
#nog_tax=NCBI_ID

#####Make sure the  names of the weights correspond to the path of the hmms
#####example: <NOGT>_ref_folder should be <NOGT>_weight
#####example: custom_ref=path/to/customHMM.hmm should be customHMM_weight
nog_weight=0.8
pfam_weight=0.9


#######CUSTOM REFERENCES#######
#####<custom_ref> defines the folder or file where Mantis will look for a SINGLE custom reference (HMM or DMND). The path will look something like this: /path/to/custom1/custom1.hmm or /path/to/custom1/
custom_ref=path/to/uniprot.dmnd
#custom_ref=path/to/hmm/custom2.dmnd
#to set the weight, simply add a line with the hmm file name followed by _weight , like so:
path/to/uniprot_weight=0.5

Running python mantis run_mantis -i ann_paper/genes_trimmed.fasta -o ann_paper/mantis_genomes -c 15 gets me

------------------------------------------
#  Setting up Mantis!
------------------------------------------
#  Reading config file and setting up paths
User configuration:
------------------------------------------
Output folder:                  ann_paper/mantis_genomes_2022-01-05T105119_hex_32ac65a9e1/
Target path:                    ann_paper/genes_trimmed.fasta
E-value threshold:              0.001
Overlap value:                  0.1
User cores:                     15
HMMER threads:                  1
Algorithm:                      dfs
------------------------------------------
#  This MANTIS process started running at 2022-01-05 10:51:19
Cores allocated: 15
Memory allocated: 115.29
Workers per core: 1
Traceback (most recent call last):
  File "/home/jsequeira/anaconda3/envs/mantis_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jsequeira/anaconda3/envs/mantis_env/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "mantis/__main__.py", line 166, in <module>
    run_mantis(input_path=input_path,
  File "mantis/source/MANTIS.py", line 94, in run_mantis
    mantis.run_mantis()
  File "mantis/source/utils.py", line 587, in wrapper
    res = f(self, *args, **kwargs)
  File "mantis/source/MANTIS.py", line 496, in run_mantis
    self.check_installation(verbose=True)
  File "mantis/source/Assembler.py", line 523, in check_installation
    self.compile_sql_metadata()
  File "mantis/source/Assembler.py", line 496, in compile_sql_metadata
    cursor = Metadata_SQLITE_Connector(metadata_file)
  File "mantis/source/Metadata_SQLITE_Connector.py", line 12, in __init__
    kill_switch(MissingMetadataFile)
NameError: name 'MissingMetadataFile' is not defined

Hi @PedroMTQ ,

We still need to connect eggnog website when run_mantis ?

cat nohup.out
Executing command:
/state/partition1/WORK/Bio/Project/xiek/software/mantis/ run_mantis -t /state/partition1/WORK/Bio/Project/xiek/pipe/divtime/bamboo/ref/annot/bbu.blast/all.fa -o bamboo -mc /state/partition1/WORK/Bio/Project/xiek/pipe/annot/mantis/MANTIS.config

Setting up Mantis!

Reading config file and setting up paths
Using custom MANTIS.config: /state/partition1/WORK/Bio/Project/xiek/pipe/annot/mantis/MANTIS.config
User configuration:

Output folder: bamboo/
Mantis config: /state/partition1/WORK/Bio/Project/xiek/pipe/annot/mantis/MANTIS.config
Target path: /state/partition1/WORK/Bio/Project/xiek/pipe/divtime/bamboo/ref/annot/bbu.blast/all.fa
E-value threshold: 0.001
Overlap value: 0.1
HMMER threads: 1
Algorithm: dfs

This MANTIS process started running at 2021-05-22 10:06:52
Traceback (most recent call last):
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/http/client.py", line 950, in send
self.connect()
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/connection.py", line 187, in connect
conn = self._new_conn()
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6773cb12e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='eggnog5.embl.de', port=80): Max retries exceeded with url: /download/latest/per_tax_level/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6773cb12e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/main.py", line 150, in
run_mantis(target_path=target_path,
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS.py", line 70, in run_mantis
mantis.run_mantis()
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/utils.py", line 537, in wrapper
res = f(self, *args, **kwargs)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS.py", line 455, in run_mantis
self.check_installation(verbose=False)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS_Assembler.py", line 520, in check_installation
tax_hmms = self.get_taxon_hmms(db='NOG', folder=True)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS_Assembler.py", line 646, in get_taxon_hmms
taxon_ids = self.get_hmm_taxon_ids(db)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS_Assembler.py", line 629, in get_hmm_taxon_ids
available_taxon_ids = self.get_taxon_ids_NOGT()
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS_DB.py", line 1017, in get_taxon_ids_NOGT
req = requests.get(url)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='eggnog5.embl.de', port=80): Max retries exceeded with url: /download/latest/per_tax_level/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6773cb12e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Cores allocated: 32
Memory allocated: 119.78
Workers per core: 1

Best,
Kun

Hi,

I am trying to install mantis on my labs server. The conda install went fine but the setup step did not work. Could you help me with this?

I run: mantis setup

Then get the following string of text. It looks like it's getting hung up at the downloading eggNOG step. I was able to download eggnog.db.gz separately, none of the other references were downloaded. Is it possible to set this up outside of setup or is it a bug?

I appreciate the help,
Alfredo

Executing command:
/tools/miniconda3/envs/MANTIS-Annotation/bin/mantis setup -c 20 -m 100

Setting up databases

ESC[32mUsing default MANTIS.cfg: /tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/config/MANTIS.cfgESC[0m
Default references folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/
Resources folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/Resources/
Custom references folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/Custom_references/
TAX NOG references folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/NOG/
TAX NCBI references folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/NCBI/
Pfam reference folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/pfam/
KOfam reference folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/kofam/
TCDB reference folder:
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/tcdb/
------------------------------------------# Weights:
nog:0.8, pfam:0.9

Will check data for the following NOGT:
995019,119603,7147,526524,1506553,68525,91061,228398,200643,1313,147541,1303,11632,2836,506,203682,33958,59732,118969,1100069,9397,651137,33154,50557,97050,81852,11989,38820,1164882,167375,85018,58840,69657,1307,119060,135625,200940,206389,713636,33183,400634,35301,40674,7148,40117,85016,766,11157,5125,186822,155619,135614,204458,1129,53335,29,85008,1268,183925,4890,72273,186807,33554,119089,1090,583,183963,267888,28211,10239,8782,265,35268,10699,46205,213118,204457,157897,117743,976,9989,90964,267893,7399,85012,586,91835,2,135624,68298,135618,326319,8459,10,285107,544448,5042,548681,119043,572511,314146,772,7711,76831,561,72275,5878,52959,671232,6231,1297,33342,186928,5863,200930,213481,1212,43988,7898,28221,81850,439488,85019,136846,265975,225057,121069,29000,267890,136845,7214,200918,5653,5148,204441,3699,85026,363408,186827,35237,85004,34397,4776,28890,85013,150247,237,35278,186828,622450,1,34383,33208,203494,85021,267889,35325,2433,766764,119065,191028,7088,186821,85005,110618,4891,45404,61432,10656,204428,28216,355688,68892,539002,112252,302485,1142,1570339,10841,57723,1239,10404,186813,186824,311790,76804,33090,290174,183967,68295,267894,35493,189330,244698,45401,60136,541000,45667,245186,1386,256005,224756,118884,213115,4751,75682,508458,1762,142182,33213,201174,186804,5151,2063,84998,451870,629,147548,85010,206350,283735,301297,613,5204,830,71274,675063,1060,186823,558415,1357,135613,252356,118882,203691,32061,289201,547,544,1236,85017,10860,136849,186820,171551,41294,909932,84406,213113,1653,136843,119069,6236,85020,7742,252301,32199,93682,6656,246874,135623,414999,1305,216572,551,2157,34037,422676,5338,74030,69277,136841,213462,468,186818,451867,5819,1511857,768503,117747,84992,92860,538999,31979,326457,74385,53433,85014,83612,206351,85009,815,1161,85025,186801,255475,183980,5794,82986,639021,34384,29258,4893,84995,2759,32003,2323,28883,91561,104264,147550,189775,191675,464095,82117,629295,1117,200795,1189,5796,10474,9263,80864,314294,423358,1150,52604,1016,358033,135619,451866,5809,119066,39782,85023,74201,308865,171550,10744,5234,4447,204037,33867,28037,5139,590,1283313,186806,29547,145357,554915,183939,147545,5129,452284,335928,10662,3041,9443,122277,129337,182709,200783,32066,1028384,31993,119045,69541,28889,35718,9604,34008,82115,10477,1224,183968,178469,204432,52018
/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/NOG/eggnog.db.gz
Downloading from http://eggnogdb.embl.de/download/emapperdb-5.0.2/eggnog.db.gz. The file will be kept in /tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/References/NOG/
Did not manage to download the following url correctly:
http://eggnogdb.embl.de/download/emapperdb-5.0.2/eggnog.db.gz
Traceback (most recent call last):
File "/tools/miniconda3/envs/MANTIS-Annotation/bin/mantis", line 11, in
sys.exit(main())
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/main.py", line 184, in main
setup_databases(chunk_size=chunk_size, no_taxonomy=no_taxonomy,
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/Assembler.py", line 17, in setup_databases
mantis.setup_databases()
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/utils.py", line 588, in wrapper
res = f(self, *args, **kwargs)
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/Database_generator.py", line 42, in setup_databases
passed_tax_check=self.prepare_queue_setup_databases_tax()
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/Database_generator.py", line 136, in prepare_queue_setup_databases_tax
self.download_and_unzip_eggnogdb()
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/Database_generator.py", line 853, in download_and_unzip_eggnogdb
download_file(url, output_folder=self.mantis_paths['NOG'], stdout_file=stdout_file)
File "/tools/miniconda3/envs/MANTIS-Annotation/lib/python3.9/site-packages/mantis/utils.py", line 725, in download_file
raise Exception
Exception
~

Mantis seems to spawn more processes than available cores and to run out of memory during the setup step.
I ran it with 5 cores and 20 Gb, and during the metadata extraction with 5 works the job spawned even more sub-processes and crashed. The error message from the slurm job was "out of memory".

I will try to run the job with 12 cores and 48 Gb to see whether the memory issue will appear again.

Used version: 14f75ac

CMD:

python submodules/mantis/ setup_databases --mantis_config mantis/mantis.config

Config:

nog_dmnd_ref_folder=/work/projects/ecosystem_biology/data/mantis_references/NOG/
pfam_ref_folder=/work/projects/ecosystem_biology/data/mantis_references/pfam/
kofam_ref_folder=/work/projects/ecosystem_biology/data/mantis_references/kofam/
ncbi_ref_folder=/work/projects/ecosystem_biology/data/mantis_references/NCBI/
tcdb_ref_folder=/work/projects/ecosystem_biology/data/mantis_references/tcdb/
ncbi_weight=0.9
nog_weight=0.8
pfam_weight=0.9
uniprot_ec_weight=0.9

Conda YAML:

channels:
  - anaconda
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - cython=0.29.21
  - hmmer=3.3.1
  - nltk=3.5
  - numpy=1.19.1
  - psutil=5.7.2
  - python=3.8.5
  - requests=2.24.0
  - sqlite=3.33.0

Log file

Screenshots:

• notify if metadata.tsv is not available but dont fail check
• remove download of taxonomy dbs

Greetings!

I have not found these informations anywhere, so will ask here:

1. How can I setup a NOG database for use with DIAMOND inside mantis' workflow? By default it seems to get HMMs only, but my system cannot handle the HMM version of NOG because of the size. I have tried using the refdb_generator, but it only has documentation for HMMs.
2. 1e-3 is the default e-value when using HMMs, am I right? Scaled e-value is only used in homology-based annotation with DIAMOND?
3. In the supplementary PDF of the mantis paper it says Wilcoxon signed rank test were performed using the python package Scipy [12], with the function "scipy.stats.wilcoxon" . Are the commands used for this analysis available anywhere?

Thank you for your attention

E.g., if using a tigrfam custom DB then if the custom db has tigrfam in its name then issues will arise:
elif 'tigrfam' in hmm_file:
self.get_link_tigrfam(dict_hmms)
self.is_essential(dict_hmms)

Dear all

I first performed the run_test option of mantis and it worked fine. Then I set up python mantis run_mantis with test_sample.faa and it also worked fine. However, when I tried to run my own test samples (also .faa) of around 30,000 contig amino acid sequences, it gave me the following error:

Traceback (most recent call last):
  File "/mypath/miniconda3/envs/mantis_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/mypath/miniconda3/envs/mantis_env/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "mantis/source/Multiprocessing.py", line 579, in worker_interpret_output
    self.generate_integrated_output(output_annotation_tsv, interpreted_annotation_tsv)
  File "mantis/source/Metadata.py", line 281, in generate_integrated_output
    output_annotation = self.read_and_interpret_output_annotation(output_annotation_tsv)
  File "mantis/source/Metadata.py", line 193, in read_and_interpret_output_annotation
    ref_file_links = self.get_hit_links(links_to_get[ref_file], ref_file)
  File "mantis/source/Metadata.py", line 125, in get_hit_links
    self.get_common_links(hit, dict_hits[hit])
  File "mantis/source/Metadata.py", line 66, in get_common_links
    self.add_to_dict(res, 'pfam', pfam)
  File "mantis/source/Metadata.py", line 39, in add_to_dict
    dict_hits['link'][dict_key].append(i)
AttributeError: 'set' object has no attribute 'append'
Ran into an issue, check the log for details. Exitting!

I notice two things that are different between my .faa file and the test_sample.faa:

1. the fasta IDs for the test_sample.faa do not contain special characters, whereas my .faa file would have sequences containing IDs like that >seq1 # 4012 # 4944 # 1 # ID=1_6;partial=00;start_type=ATG;rbs_motif=None;rbs_spacer=None;gc_cont=0.666

2. my fasta sequences contain an asterisk at the end of the sequence, indicating a stop codon

Could either one of these two factors contributed to the error? If not, what could be giving this error? Thanks

Marcus

As pointed out by @hbckleikamp Diamond needs to be setup differently in order to annotate peptides.
See bbuchfink/diamond#469 for details from author.

Hi,

I found that the latest TIGRFAMs versions is hosted in ncbi genome database, is the mantis pipeline use this latest version https://www.ncbi.nlm.nih.gov/genome/annotation_prok/tigrfams/ ?

Best,
Kun

Hi,

There are too many works activated when doing setup_databases even -c 1.

python mantis/ setup_databases -c 1
ps xf|grep python|wc -l
131

Is it possible to limit the jobs number when doing setup_databases ?

Best,
Kun

Greetings!

I attempted to install mantis on my system, following these commands:

git clone https://github.com/PedroMTQ/mantis.git # with git clone [email protected]:PedroMTQ/mantis.git it didn't work
cd mantis
conda env create -f mantis_env.yml
conda activate mantis_env
cd ..
python mantis setup_databases

however, my disk ran out of space after downloading 500 Gb! How large are the reference databases of mantis?

Hi, when building the database there is this error during the process (python mantis setup_databases).
It does not kill the job as this is still running, but it seems it will have an impact.

Traceback (most recent call last):                                                           
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap        
    self.run()                                                                               
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run               
    self._target(*self._args, **self._kwargs)                                                
  File "mantis/source/MANTIS_DB.py", line 183, in worker_setup_databases                     
    self.download_database(database, force_download, taxon_id, stdout_path)
  File "mantis/source/MANTIS_DB.py", line 218, in download_database
    elif database == 'kofam':       self.download_kofam(force_download=force_download, stdout_file=stdout_file)
  File "mantis/source/MANTIS_DB.py", line 546, in download_kofam
    self.compile_kofam_metadata()                                                            
  File "mantis/source/MANTIS_DB.py", line 412, in compile_kofam_metadata
    self.get_link_kofam_ko_to_pathway(metadata_to_write)                                     
  File "mantis/source/MANTIS_DB.py", line 462, in get_link_kofam_ko_to_pathway
    map_description = self.get_kofam_pathway_description()                                   
  File "mantis/source/MANTIS_DB.py", line 513, in get_kofam_pathway_description
    res[kegg_map] = {'description': description, 'parent': sub_title, 'grand_parent': main_tile}
UnboundLocalError: local variable 'main_tile' referenced before assignment

Hi,

I am running into an issue and unfortunately I can't pinpoint where the problem is as there is no useful error message in that case.

The mantis setup is ending before the end with an error.
Here is the last two lines of the log (I cut the full path):

Unpacking .../References/kofam/profiles.tar.gz to .../References/kofam/
Ran into an issue, check the log for details. Exitting!

Using Mantis v1.5.4 installed through bioconda

There is no obvious fault, space on disk is fine, memory too.
Let me know how I can try to help pinpointing the issue.
Thanks

Hey @PedroMTQ

The current installation instructions might need to be updated. Pulling the tool using conda install -c bioconda mantis_pfa was throwing Unsatisfiable errors.

Replacing it with conda install -c conda-forge -c bioconda mantis_pfa seems to work.

Thanks!

Hi Pedro -
I can't get the conda install to work. If I create new environment and let try to install conda, I get

conda install -c bioconda mantis_pfa
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                   

UnsatisfiableError: 

If I create the the conda environment with the dependencies in mantis.yaml, I get

conda install -c bioconda mantis_pfa
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                   

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.17=0
  - feature:|@/linux-64::__glibc==2.17=0

Your installed version is: 2.17

Any pointers?
Thanks -AHB

Hi,

I installed mantis using conda install which went fine:
conda create -p /tools/Miniconda3/envs/mantis_annotate -c conda-forge -c bioconda mantis_pfa

However, when I run:
mantis setup

I get the following error:
Traceback (most recent call last):
File "/tools/Miniconda3/envs/mantis_annotate/bin/mantis", line 7, in
from mantis.main import main
File "/tools/Miniconda3/envs/mantis_annotate/lib/python3.10/site-packages/mantis/main.py", line 7, in
from mantis.mantis import run_mantis, run_mantis_test, print_citation_mantis, print_version
File "/tools/Miniconda3/envs/mantis_annotate/lib/python3.10/site-packages/mantis/mantis.py", line 110, in
class MANTIS(Multiprocessing):
NameError: name 'Multiprocessing' is not defined

I am unsure what the issue is here and would appreciate some help! Thank you!

Hi,

setup_databases still need to run long long time. So I was downloaded eggnog hmm files and tried to use merge_hmm_folder to merge hmm files, but it didn't worked normally, do you why?
Here are the hmm files
ls 548681/|head
4R16V.hmm
4R16W.hmm
4R16X.hmm
4R16Y.hmm
4R16Z.hmm
4R170.hmm
4R171.hmm
4R172.hmm
4R173.hmm
4R174.hmm
python /state/partition1/WORK/Bio/Project/xiek/software/mantis/ merge_hmm_folder -t 548681/

Here are the error logs
Weights: nog:0.8, pfam:0.9, tigrfam:0.5

Merging hmm folder:
548681/
Merging hmm folder: 548681/
[: missing `]'
Traceback (most recent call last):
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/main.py", line 187, in
merge_hmm_folder(target_folder=target)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS_Assembler.py", line 31, in merge_hmm_folder
mantis.merge_hmm_folder(target_folder)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/MANTIS_Assembler.py", line 335, in merge_hmm_folder
run_command(
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/utils.py", line 942, in run_command
return run_command_simple(command=command_list, get_output=get_output, stdout_file=stdout_file)
File "/state/partition1/WORK/Bio/Project/xiek/software/mantis/source/utils.py", line 740, in run_command_simple
process = subprocess.run(command)
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Bio/User/kxie/software/anaconda3/envs/mantis/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: 'for'

Best,
Kun

improve estimate_chunk_size , especially when annotating multiple samples

Hey @PedroMTQ,

I'm running MANTIS like so
python run_mantis -t {input.TSV} --output_folder $(dirname {output}) --mantis_config {input.config}

and ran into the following issue

Could not find taxa ID for MAGS
#  Splitting samples into chunks!
#  Samples will be split into 9651 chunks with 112 workers
#  Running HMMER with 112 workers.

Can one provide GTDBtk data or is this pulled from NCBI? It looked like it failed due to connection issues. My TSV files like so.

GL_R10_GL11_UP_3_G2.2.1.1       /mnt/data/sbusi/rock_mantis/results/prokka/GL_R10_GL11_UP_3_G2.2.1.1/GL_R10_GL11_UP_3_G2.2.1.1.faa      MAGS
GL_R10_GL11_UP_3_maxbin_res.008.fasta_sub       /mnt/data/sbusi/rock_mantis/results/prokka/GL_R10_GL11_UP_3_maxbin_res.008.fasta_sub/GL_R10_GL11_UP_3_maxbin_res.008.fasta_sub.faa      MAGS
GL_R10_GL11_UP_3_maxbin_res.019.fasta_sub       /mnt/data/sbusi/rock_mantis/results/prokka/GL_R10_GL11_UP_3_maxbin_res.019.fasta_sub/GL_R10_GL11_UP_3_maxbin_res.019.fasta_sub.faa      MAGS
GL_R10_GL11_UP_3_O7     /mnt/data/sbusi/rock_mantis/results/prokka/GL_R10_GL11_UP_3_O7/GL_R10_GL11_UP_3_O7.faa  MAGS
GL_R11_GL15_UP_1_B1.1.1.1.1.1   /mnt/data/sbusi/rock_mantis/results/prokka/GL_R11_GL15_UP_1_B1.1.1.1.1.1/GL_R11_GL15_UP_1_B1.1.1.1.1.1.faa      MAGS

Thanks for your help!

Add putative operon prediction by coordinate and functional clustering

NOG metadata is consuming too much RAM.
Try to make it more efficient or reduce the number of workers (check how much RAM is needed per worker)

I installed Mantis as adviced and was checking if the databases were installed properly using mantis check_sql

mantis's latest release is: 1.5.5

Cython correctly compiled!
Checking extra files
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/Resources/essential_genes
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/Resources/NCBI/gc.prt.dmp
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/Resources/Taxonomy.db
Checking references installation
Failed installation check on [path unavailable]: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/NOG/
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/NCBI/
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/pfam/
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/kofam/
Failed installation check on [invalid referecence type]: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/tcdb/
Installation check failed on:
/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/NOG/
/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/tcdb/

------------------------------------------
--------INSTALLATION CHECK FAILED!--------
------------------------------------------
Traceback (most recent call last):
  File "/home/subudhak/miniconda3/envs/mantis/bin/mantis", line 11, in <module>
    sys.exit(main())
  File "/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/mantis/__main__.py", line 218, in main
    check_installation(mantis_config=mantis_config, check_sql=True, no_taxonomy=no_taxonomy)
  File "/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/mantis/assembler.py", line 27, in check_installation
    mantis.check_installation(check_sql=check_sql)
  File "/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/mantis/assembler.py", line 628, in check_installation
    raise InstallationCheckNotPassed
mantis.exceptions.InstallationCheckNotPassed: Installation check not passed! Make sure you've setup the databases

Recently I'm trying to annotate sequence with mantis (default databse). However, two sequences were annotated abnormally (seq1 and seq2)

When I search them in NCBI, they were annotated as "beta-propeller fold lactonase family protein" and "YncE family protein". However, with mantis, i got:

seq1	NOGG_merged;kofam_merged	397945.Aave_1437;K20932	2	5	|	cog:COG3391	description:hydrazine	synthase	subunit	eggnog:1P862	eggnog:2VKXA	eggnog:4ACF0	eggnog:COG3391	enzyme_ec:1.7.2.7	kegg_ko:K20932	pfam:PF05694
seq2	NOGG_merged;Pfam-A;kofam_merged	399795.CtesDRAFT_PD1384;Lactonase;K20932	3	5	|	cog:COG3391	description:Lactonase,	7-bladed	beta-propeller	description:hydrazine	synthase	subunit	eggnog:1P862	eggnog:2VKXA	eggnog:4ACF0	eggnog:COG3391	enzyme_ec:1.7.2.7	kegg_ko:K20932	pfam:Lactonase	pfam:PF02239	pfam:PF10282

In brief, while nogg potentially returned correct results, the annotation from kofam was totally wrong!
I also looked for detailed results in integrated_annotation.tsv:

Query	Ref_file	Ref_hit	Ref_hit_accession	evalue	bitscore	Direction	Query_length	Query_hit_start	Query_hit_end	Ref_hit_start	Ref_hit_end	Ref_length
seq1	kofam_merged	K07404	-	3.0007185000000003e-09	42.7	+	362	129	271	331	476	530
seq1	kofam_merged	K20932	-	1.0771810000000001e-05	30.8	+	362	264	359	45	127	377
seq2	kofam_merged	K07404	-	6.155319999999999e-14	58.2	+	389	115	288	281	460	530
seq2	kofam_merged	K20932	-	0.0002769894	26.1	+	389	285	387	44	127	377

It can be found that the evalue of these hits were pretty low, and the query hit length is quiet short. Is it possible to filter these low-quality annotations from cousensus results?

There sequences are list here

>seq1
VAGERAFARFGLSRLRDALVGIRATISFIKRSFVGSFLSSRFLFFFWAFVALSSSAFASQTTPLFVLNSLDANVSVINPVTWEEIRRIPTGKEPHHLYLTPDERSIIIANALSDTLTFIDPRTAEVQRTVRDIVDPYHLRFTPDMKWLVTAANRLNHVDFYKWDGKELTLAKRVATSRTPSHIWIDSKSTTAYVSMQDSDELVAMDIATQSIRWRTKTGAMPADIYGSPDDKRLFVGLTGSDSVEVFDVSGAQPSSIKRIKTGSGAHAFRAAGDKRHLYVSNRVANTISKLDMQRQEVVDTYPVPGGPDCMDVSADGRFIFVSSRWAKKMSVVDTVEKKVIKQVPVGKSPHGVWTLQHAPR*
>seq2
MDSEKVVVVGRLSIGRTGAAVALVVGVGAAAWLGASALPGFKSAKAAAPAVAATPVVAPAAAVATGAPVSPAAVKPAQAARAVQGPTPIFVLNSLDASISVIDPQTWKEQSRIPTGKEPHHLYLTPDEKSLVVANALGDSLTLVDPRTGAVQRVIRDIVDPYHLRFSPDMKWFVTAANRLNHIDFYRWDAATQTPTLVKRVSTGKTPSHLFIDAQSKTLYSTMQDSDALVAIDIATQTIKARVPTGPMPADVYGSPDGKKLFVGLTGGDGVEVFDITGPEPRSLGQIKTAAGAHAFRAAGDDRHLYVSNRVANTISKIDMVSSQVIANYPAPGGPDCMDVSADGRYIFVASRWARKLSVIDTVEKKVVNQVNVGKSPHGVWTLSHAAR*

Hey @PedroMTQ,

I'm running MANTIS via snakemake with the following command: snakemake -s Snakefile --use-conda --cores 64 -rp

The log file, however, states the 112 cores were allocated

#  This MANTIS process started running at 2021-04-28 23:23:51
Cores allocated: 112
Memory allocated: 967.37
Workers per core: 1

Is there a flag to `limit the number of cores it uses?

Thank you!

Allow for the input of CDS besides protein sequences

Dear Pedro

I have an error while running mantis.
This happens when a run mantis in a slurm job. From the terminal, it works but not in a slurm job.

Executing command:
mantis run_mantis -t GC_00001818_OG.fasta -o test

Setting up Mantis!

Reading config file and setting up paths

User configuration:

Output folder: test/
Target path: GC_00001818_OG.fasta
E-value threshold: 0.001
Overlap value: 0.1
HMMER threads: 1
Algorithm: dfs

This MANTIS process started running at 2021-06-29 21:06:21

Traceback (most recent call last):
File "/scratch/ulg/bioec/lcornet/miniconda/envs/mantis_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/scratch/ulg/bioec/lcornet/miniconda/envs/mantis_env/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "mantis/main.py", line 150, in
run_mantis(target_path=target_path,
File "mantis/source/MANTIS.py", line 78, in run_mantis
mantis.run_mantis()
File "mantis/source/utils.py", line 550, in wrapper
res = f(self, *args, **kwargs)
File "mantis/source/MANTIS.py", line 463, in run_mantis
self.check_installation(verbose=False)
File "mantis/source/MANTIS_Assembler.py", line 513, in check_installation
tax_hmms = self.get_taxon_refs(db='NOG', folder=True)
File "mantis/source/MANTIS_Assembler.py", line 658, in get_taxon_refs
taxon_ids=self.get_ref_taxon_ids(db)
File "mantis/source/MANTIS_Assembler.py", line 630, in get_ref_taxon_ids
available_taxon_ids = self.get_taxon_ids_eggNOG()
File "mantis/source/MANTIS_DB.py", line 1016, in get_taxon_ids_eggNOG
taxons_search = re.findall('href="\d+/"', webpage)
File "/scratch/ulg/bioec/lcornet/miniconda/envs/mantis_env/lib/python3.8/re.py", line 241, in findall
return _compile(pattern, flags).findall(string)
TypeError: expected string or bytes-like object
Cores allocated: 10
Memory allocated: 1998.68
Workers per core: 1

Can you help me with this problem ? Is there any connection on internet when running mantis (not allowed from of computing node) ?

Thanks
Luc

I have an error during the database setup:

Cloning into '/media/vol2/home/galaxy/mantis/Resources/UniFunc'...
Process Process-3:
Traceback (most recent call last):
File "/media/vol2/home/galaxy/miniconda/envs/mantis_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/media/vol2/home/galaxy/miniconda/envs/mantis_env/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "mantis/source/MANTIS_DB.py", line 178, in worker_setup_databases
self.download_database(database, force_download, taxon_id, stdout_path)
File "mantis/source/MANTIS_DB.py", line 212, in download_database
elif database == 'pfam': self.download_pfam(force_download=force_download, stdout_file=stdout_file)
File "mantis/source/MANTIS_DB.py", line 390, in download_pfam
download_file(url, output_folder=self.mantis_paths['pfam'], stdout_file=stdout_file)
File "mantis/source/utils.py", line 669, in download_file
if os.stat(file_path).st_size == target_size: return
FileNotFoundError: [Errno 2] No such file or directory: '/media/vol2/home/galaxy/mantis/References/pfam/Pfam-A.hmm.gz'

Could you help me to solve this ?