koszullab / metator Goto Github PK

Dear developers, I got the following error at the very last step of the pipeline:

metator binning --n-bins 300
Drawing enrichment vs. size plots...
Drawing distribution violinplot...
Extracting bin subnetworks and matrices...
INFO :: Loading partition...
INFO :: Loading network...
Extracting bin FASTA files...
WARNING :: /home/guritsk1/miniconda2/envs/metator-env/lib/python3.7/site-packages/metator/bin/../scripts/bins.py:216: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
  zip(*np.genfromtxt(partition_file, usecols=(0, 1), dtype=None))

Traceback (most recent call last):
  File "/home/guritsk1/miniconda2/envs/metator-env/lib/python3.7/site-packages/metator/bin/../scripts/bins.py", line 448, in <module>
    main()
  File "/home/guritsk1/miniconda2/envs/metator-env/lib/python3.7/site-packages/metator/bin/../scripts/bins.py", line 439, in main
    chunk_size=chunk_size,
  File "/home/guritsk1/miniconda2/envs/metator-env/lib/python3.7/site-packages/metator/bin/../scripts/bins.py", line 231, in extract_fasta
    fields = name.split("_")
TypeError: a bytes-like object is required, not 'str'

Could this be a syntax issue, or is this specific to my data? The chunkname_core_size_300.txt looks ok:

k141_8_0  208074 1
k141_11_0  164531 1
k141_12_0  119734 1
k141_15_0  74519 1
k141_17_0  26 1479
k141_18_0  2 16318
k141_20_0  2 16318
k141_21_0  361 30
k141_24_0  17 2418
k141_25_0  864 16

By the way, what do the two columns in chunkname_core_size_300.txt represent? Which column is the cluster number? If I know this I can just make a script to split the bin fasta files myself.

Hi Developer,

I follow the instructions from your tutorial to do the recursive Louvain clustering as:

'''
metator partition --network-file $subnetwork_dir/contaminated_subnetwork.dat --partition-dir subnetwork_partition_folder
metator binning
'''

However, I find that it will still run 300 times Louvain clustering for each contaminated bin, which consumes a long time and a large space. I change the parameter '--iterations' in the partition action but it still ran 300 times.

Moreover, I find a round of recursive Louvain binning can already drop the contamination to a low level. So will you run the recursive Louvain binning multiple times?

Thanks!

Hi,

I guess the HMM URL in metator.sh doesn't work. When I run ./metator.sh dependencies, I still cannot download such databases.

Best

Dear developers,

It looks like the link http://dl.pasteur.fr/fop/gzLz1lG8/hmm_databases.tgz is broken, at least from my location (US). Is it possible to have it updated so I can download the HMM model? Alternatively, can you direct me to the file I need to download to place and extract into python3.7/site-packages/metator/bin/HMM_databases?

Just would like to report a dead link in https://github.com/koszullab/metaTOR/blob/master/doc/TUTORIAL.md

metaTOR manual

redirects me to a 404

Hello,

I've been trying to implement this tool and cannot get past the first alignment segment.

If I run $ ./meta3c.sh align -1 fastq_r1 -2 fastq_r2 -a assembly -q 20 -c 10000 -s 10000 -p new_project
after merging bams
I get this error:

Traceback (most recent call last):
File "/mnt/nas3/cory/home_dir/metaTOR-master/network.py", line 749, in
parameters=parameters)
File "/mnt/nas3/cory/home_dir/metaTOR-master/network.py", line 97, in alignment_to_contacts
with pysam.AlignmentFile(sam_merged, "rb") as alignment_merged_handle:
File "calignmentfile.pyx", line 302, in pysam.calignmentfile.AlignmentFile.cinit (pysam/calignmentfile.c:4834)
File "calignmentfile.pyx", line 372, in pysam.calignmentfile.AlignmentFile._open (pysam/calignmentfile.c:5769)
File "calignmentfile.pyx", line 58, in pysam.calignmentfile._encodeFilename (pysam/calignmentfile.c:3527)
TypeError: Argument must be string or unicode.

Any thoughts? Thanks!

Hi Lyam,

I'm trying to run metaTOR from a dedicated conda environment with all required dependencies installed.

metaTOR.sh align starts running but exit with a Python error at the network creation step:
any idea on how to fix that?

Python 3.6.6

Mapping reads...
70604753 reads; of these:
  70604753 (100.00%) were unpaired; of these:
    65283672 (92.46%) aligned 0 times
    4041938 (5.72%) aligned exactly 1 time
    1279143 (1.81%) aligned >1 times
7.54% overall alignment rate
70604753 reads; of these:
  70604753 (100.00%) were unpaired; of these:
    64590673 (91.48%) aligned 0 times
    4480279 (6.35%) aligned exactly 1 time
    1533801 (2.17%) aligned >1 times
8.52% overall alignment rate
[bam_sort_core] merging from 1 files and 1 in-memory blocks...
[bam_sort_core] merging from 1 files and 1 in-memory blocks...
Alignment done.
Merging bam files...
Done.
Generating network from alignments...
Traceback (most recent call last):
  File "/mibi/users/jnesme/Bin/metaTOR/network.py", line 821, in <module>
    main()
  File "/mibi/users/jnesme/Bin/metaTOR/network.py", line 810, in main
    my_assembly, = reference_file
ValueError: too many values to unpack (expected 1)

Pyton 3.6.6 and all dependencies installed:

Requirement already up-to-date: numpy in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.15.3)
Requirement already up-to-date: scipy in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.1.0)
Requirement already up-to-date: matplotlib in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (3.0.0)
Requirement already up-to-date: biopython in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (1.72)
Requirement already up-to-date: pysam in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (0.15.1)
Requirement already up-to-date: seaborn in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (0.9.0)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 3)) (0.10.0)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 3)) (1.0.1)
Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 3)) (2.2.2)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from matplotlib->-r requirements.txt (line 3)) (2.7.3)
Requirement already satisfied, skipping upgrade: pandas>=0.15.2 in /usr/local/home/jnesme/.local/lib/python3.6/site-packages (from seaborn->-r requirements.txt (line 6)) (0.20.1)
Requirement already satisfied, skipping upgrade: six in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from cycler>=0.10->matplotlib->-r requirements.txt (line 3)) (1.11.0)
Requirement already satisfied, skipping upgrade: setuptools in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from kiwisolver>=1.0.1->matplotlib->-r requirements.txt (line 3)) (40.4.3)
Requirement already satisfied, skipping upgrade: pytz>=2011k in /mibi/users/jnesme/miniconda2/envs/metator_env/lib/python3.6/site-packages (from pandas>=0.15.2->seaborn->-r requirements.txt (line 6)) (2018.5)

Here is a report of some difficulties / indications that could be taken into account. I apologize in advance if they seem too obvious:

1. To properly get louvain working, the user should move to the working directory (i.e. the folder with all meta3cbox dependencies) and create a folder named tools, where louvain should be downloaded and compiled manually:

mkdir tools
cd tools
wget https://sourceforge.net/projects/louvain/files/louvain-generic.tar.gz
tar -xzvf louvain-generic.tar.gz
# Important to change the default name of the folder, otherwise it will not be recognized in the 2nd step (partition)
mv louvain-generic louvain
cd louvain
make
cd ..
rm louvain-generic.tar.gz

2. A similar procedure should be followed to allocate the HMMs that will be used in the 3rd step (annotation). From the working directory:

wget dl.pasteur.fr/fop/5eHgTGww/modele_HMM.tar.gz
tar -xzvf modele_HMM.tar.gz
rm modele_HMM.tar.gz

3. I encountered a permission error when accessing the distance file. Maybe this was just my case, but a full access to this file should be granted (chmod) before running the 2nd step (partition)

4. I encountered some errors when trying to re-run some of the individual steps, having to delete some of the previously generated files/folders, such as config_current.sh, config.sh or output. I did not really understand what was happening here, maybe I was not doing something right.

And I think this is it. I hope this is useful!

Best,
Juanma

Dear metaTOR Developers,

I hope all is well. We are​ trying to run the metaTOR and we have been running into the HMM databases issue. More specifically the HMM_databases.hmm file seems not to exist:

We know that the link to databases in metator.py has been broken as mentioned in issue #6 therefore we forked metaTOR to our github and updated the link in metator.py like seen in metator.sh:

Our current databases folder extracted from metaTOR after updating the link looks like this:

Should the .hmm file(s) be missing in the HMM_databases folder but built in the process of running metator? We used the command 'metator dependencies' to fetch dependencies. Could this be the issue of the 'hmmer' dependency? Or maybe the issue with how we install the forked metator from our github? HMM_databases folder ended up being in src path instead of lib path so we copied over the databases from src path to a brand new environment that had the original metator from koszullab installed but empty HMM_databased folder ~/metator0513/lib/python3.6/site-packages/metator/bin/HMM_databsases .

Good morning!

I am currently running the fourth step of the meta3box pipeline, i.e. the binning process, with these parameters:
bash meta3c.sh binning --n-bins 100 --iter 300 -p metabox_project

However, I ran into some troubles which I think are related to the arguments associated to read-in functions from Biopython

I hope you can help me out here! The version of Biopython that I am currently using is 1.68
If you need further information, I will be happy to provide it.

Best,
Juanma

Here is the full error report:

Drawing enrichment vs. size plots...
Drawing distribution violinplot...
Extracting bin subnetworks and matrices...
Loading partition...
Loading network...
Extracting bin FASTA files...

/Bin/meta3Cbox/bins.py:155: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
dtype=None))
Traceback (most recent call last):
File "/Bin/meta3Cbox/bins.py", line 305, in
chunk_size=chunk_size)
File "/Bin/meta3Cbox/bins.py", line 177, in extract_fasta
sequence = str(genome[header_name][pos_start:pos_end])
File "/usr/local/home/anaconda3/envs/meta3cbox/lib/python2.7/site-packages/Bio/Seq.py", line 235, in getitem
return Seq(self._data[index], self.alphabet)
TypeError: slice indices must be integers or None or have an index method