asntech / intervene Goto Github PK

View Code? Open in Web Editor NEW

127.0 6.0 28.0 20.83 MB

Intervene: a tool for intersection and visualization of multiple genomic region and gene sets

Home Page: http://intervene.rtfd.io/

License: Other

Python 92.45% R 7.55%

visualization genome-analysis dna-sequences bedtools pyton venn-diagram heatmaps intersection

intervene's Introduction

Intervene

a tool for intersection and visualization of multiple gene or genomic region sets

Documentation

A detailed documentation is available in different formats: HTML | PDF | ePUB

Installation

Quick installation using Conda

conda install -c bioconda intervene

This will install all the dependencies and you are ready to use Intervene.

Install using pip

You can install Intervene from PyPi using pip.

Install from PyPi:

pip install intervene

Note: If you install using pip, make sure to install BEDTools and R packages listed below.

Intervene requires the following Python modules and R packages:

Python (=> 3.3 ): https://www.python.org/

BedTools (Latest version): https://github.com/arq5x/bedtools2

pybedtools (>= 0.7.9): https://daler.github.io/pybedtools/

Pandas (>= 0.16.0): http://pandas.pydata.org/

Seaborn (>= 0.7.1): http://seaborn.pydata.org/

R (>= 3.0): https://www.r-project.org/

R packages including UpSetR (v1.4.0), corrplot

Install BEDTools

We are using pybedtools, which is Python wrapper for BEDTools. So, BEDTools should be installed before using Intervene. It's recomended to have a latest version, but if you have an older version already install, it should be fine.

A quick installation, if you have conda installed.

conda install -c bioconda bedtools

Please read the instructions at https://github.com/arq5x/bedtools2 to install BEDTools, and make sure it is on your path and you are able to call bedtools from any directory.

Install required R packages

Intervene rquires three R packages, UpSetR , corrplot for visualization and Cairo to generate high-quality vector and bitmap figures.

install.packages(c("UpSetR", "corrplot", "Cairo"))

Install Intervene from source

You can install a development version by using git from GitHub or Bitbucket.

Install development version from Bitbucket

If you have git installed, use this:

git clone https://bitbucket.org/CBGR/intervene.git
cd intervene
python setup.py sdist install

Install development version from GitHub

If you have git installed, use this:

git clone https://github.com/asntech/intervene.git
cd intervene
python setup.py sdist install

How to use Intervene

Once you have installed Intervene, you can type:

intervene --help

This will show the following help message.

usage: intervene <subcommand> [options]

positional arguments <subcommand>:
  {venn,upset,pairwise}
                        List of subcommands
    venn                Venn diagram of intersection of genomic regions or list sets (upto 6-way).
    upset               UpSet diagram of intersection of genomic regions or list sets.
    pairwise            Pairwise intersection and heatmap of N genomic region sets in <BED/GTF/GFF> format.

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

to see the help for the three subcommands pairwise, venn and upset type:

intervene pairwise --help

intervene venn --help

intervene upset --help

Run Intervene on test data

To run Intervene using example data, use the following commands. To access the test data make sure you have sudo or root access.

intervene pairwise --test

intervene venn --test

intervene upset --test

If you have installed Intervene locally from the source code, you may have problem to find test data. You can download the test data here https://github.com/asntech/intervene/tree/master/intervene/example_data and point to it using -i instead of --test.

./intervene/intervene venn -i intervene/example_data/ENCODE_hESC/*.bed       
./intervene/intervene upset -i intervene/example_data/ENCODE_hESC/*.bed      
./intervene/intervene pairwise -i intervene/example_data/dbSUPER_mm9/*.bed

The above three test commands will generate the following three figures (a, b and c).

By default your results will stored in the current working directory with a folder named Intervene_results. If you wish to save the results in a specific folder, you can type:

intervene upset --test --output ~/path/to/your/folder

Interactive Shiny App

Intervene Shiny App is freely available at https://asntech.shinyapps.io/intervene or https://intervene.shinyapps.io/intervene

The source code for the Shiny app is available at https://github.com/asntech/intervene-shiny

Support

If you have questions, or found any bug in the program, please write to us at azez.khan[at]gmail.com

Cite Us

If you use Intervene please cite us: Khan A, Mathelier A. Intervene: a tool for intersection and visualization of multiple gene or genomic region sets. BMC Bioinformatics. 2017;18:287. doi: 10.1186/s12859-017-1708-7

intervene's People

Contributors

Stargazers

Watchers

intervene's Issues

update intervene's bedtools

Hello,

I am having the exact issue below. When i update my bedtools, it solves the problem.

Is there a way to intagrate more updated version of the bedtools in intervene?

Thank you for developing and maintaining such an useful software.

Best,

arq5x/bedtools#153

Pairwise plot seems to have issues with recent seaborn

I'm seeing the following traceback if I set htype to dendrogram:

Traceback (most recent call last):
  File "/Users/mvandenb/miniconda3/envs/[email protected]/bin/intervene", line 606, in <module>
    main()
  File "/Users/mvandenb/miniconda3/envs/[email protected]/bin/intervene", line 426, in main
    pairwise.pairwise_intersection(label_names, options)
  File "/Users/mvandenb/miniconda3/envs/[email protected]/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 478, in pairwise_intersection
    heatmap_dendrogram(matrix,outfile, options)
  File "/Users/mvandenb/miniconda3/envs/[email protected]/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 304, in heatmap_dendrogram
    sns.plt.setp(sns_plot.ax_heatmap.yaxis.get_majorticklabels(), rotation=0)
AttributeError: module 'seaborn' has no attribute 'plt'

I have seen that sns.plt is not part of the seaborn API anymore (https://stackoverflow.com/questions/45070959/atributeerror-module-object-has-no-attribute-plt-seaborn)

pairwise function have problems on --names and --sort

Sorry, the --names is fine.
But, it automatically sort the label by name and I can't change the label order.
I'm using intervene version 0.6.4

cmd=paste("intervene pairwise -i", peak.file, "--output ../graph/Peak_without_control --filenames --compute frac --htype color --figsize 10 10 --project marker_pairwise_heatmap --diagonal", --names my.name, sep=""))

Support for proportional venn diagrams?

Thanks for developing a really useful package for quickly intersecting genesets and regions. If possible, one nice addition would be to support scaling of the circles of the venn diagrams to be proportional to the size of the sets.

conda install not working with existing environment python=3.7

Thank you for building this incredibly useful tool! It has greatly improved how we are able to look at set overlap data.

I ran into a problem trying to install to an existing conda environment, seems to be the issue is that it is not compatible with python 3.7.7 - here is the error message I received:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                      

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - intervene -> python[version='2.7.*|3.5.*|3.6.*']

Your python: python==3.7.7=hf48f09d_4

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

As a work around I just created a separate environment, but I wanted to share in case this was just a quick fix in conda to specify that intervene does in fact work with python 3.7.

I'm using conda 4.9.2 (installed from miniconda), on Mac OS X.

Intervene venn –save-overlaps not working

Hi,

Running the command intervene venn -I <bed files> --names=<> generated a Venn diagram but when I add --save-overlaps to get the intersections I get an error regardless if I do the default (False) or True to get the intersections.

intervene venn -I --names=<> --save-overlaps False
usage: intervene [options]
intervene: error: unrecognized arguments: False

intervene venn -I --names=<> --save-overlaps True
usage: intervene [options]
intervene: error: unrecognized arguments: True

bedtools intersect options

Feature suggestion as discussed via email with credits to @asntech .

It seems from looking at the source code that the genomic regions overlap is calculated using pybedtools intersect default settings which counts as intersection any regions that overlap by at least 1 bp. Could please consider

adding a note to the Docs, and/or
consider adding an option to, for instance, consider an overlap only regions matching 100 %? The perfect solution would be to an option, where user will be able to list any of the arguments available for bedtool’s intersect command. Something like this: --bedtool-options f=0.8,r=true,etc

The default is perfectly sensible, but it might lead to unexpected results for those not aware of it.

How can I turn the type 3 font off in the out.pdf？

Hey man, when I draw venn diagram based on bed files, intervene works well, but when I tried to modify the font of the texts in the out.pdf, it shows "Edit Text Tool doesn't support the type 3 font".how can i deal with it? Or how can I turn the font of out.pdf from type 3 font to type 1 font?
And, one more question, how can make the outputs of intervene with both out.pdf and Rscripts?
Many thanks!

total number diff in venn plot

intervene venn -i ./*.bed --output .  --bedtools-options  f=1,s,r --figtype png --title "precision(100%)" 
mv Intervene_venn.png  compare_1.png
sleep 3 

intervene venn -i ./*.bed --output .  --bedtools-options  f=0.75,s,r --figtype png --title "precision(75%)"  
mv Intervene_venn.png  compare_0.75.png
sleep 3 

intervene venn -i ./*.bed --output .  --bedtools-options  f=0.5,s,r  --figtype png --title "precision(50%)"  
mv Intervene_venn.png  compare_0.5.png
sleep 3 

intervene venn -i ./*.bed --output .  --bedtools-options  f=0.25,s,r  --figtype png --title "precision(25%)" 
mv Intervene_venn.png  compare_0.25.png
sleep 3 

intervene venn -i ./*.bed --output .  --bedtools-options  f=0.1,s,r  --figtype png --title "precision(10%)"  
mv Intervene_venn.png  compare_0.1.png
sleep 3

Hi dear author,

I have three bed files for intervene to draw the venn plot, using the various fraction (0.1, 0.25, 0.5, 0.75, 1); and manually add the total numbers in the venn plot for each bed file, while A, B total number almost the same under fraction 0.1 and 1, the C bed file total number added from the venn plot has near 3000 difference (f=1 vs f=0.1); is this normal, how to adjust it?

Thank you so much!

intersect result is wrong compare to bedtools intersect

I compare R function intersect in GenomicRanges package(terms RI), linux bedtools intersect(LI) and your tools, find that RI is the same as LI, but some wrong in your tools.
I can not send you an example pic in here.

search not working in readthedocs

Hi I'm not sure if this should be reported here but when I try to use the search bar in readthedocs for example:
https://intervene.readthedocs.io/en/latest/search.html?q=venn&check_keywords=yes&area=default

it returns 0 results, even though I'm searching for something that should be there i.e. "venn"

Here's a screenshot:

I'm using Mac OS X, Chrome Version 88.0.4324.182 (Official Build) (x86_64)

After upgrade to pandas 1.0.0, DataFrame.ix is removed

Hi,

I installed the intervene package with pip command, this is a real nice and useful tool.

However, I find that if the users try to generate a pairwise plot with pandas version 1.0.0, the software will throw an error message AttributeError: 'DataFrame' object has no attribute 'ix'.

.ix is deprecated in the newest pandas package.

Could you mind updating the intervene package with the new attribute:
use .iloc or .loc instead of .ix at https://github.com/asntech/intervene/blob/master/intervene/modules/pairwise/pairwise.py#L150 and
https://github.com/asntech/intervene/blob/master/intervene/modules/pairwise/pairwise.py#L214

D = D.iloc[cluster_order, cluster_order] is for number indexing
series = series.loc[order] is for str indexing

Thanks,
Lijia

how to display the "--reldist" result?

I'm trying to use the "--reldist" calculation.
intervene pairwise --compute reldist --sort --test

Then it returns error:
Running Intervene with test data.

Performing a pairwise intersection analysis. Please wait...

Traceback (most recent call last):
File "/export/data2/anaconda2/bin/intervene", line 4, in
import('pkg_resources').run_script('intervene==0.6.4', 'intervene')
File "/export/data2/anaconda2/lib/python2.7/site-packages/pkg_resources/init.py", line 750, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/export/data2/anaconda2/lib/python2.7/site-packages/pkg_resources/init.py", line 1527, in run_script
exec(code, namespace, namespace)
File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene-0.6.4-py2.7.egg/EGG-INFO/scripts/intervene", line 606, in
main()
File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene-0.6.4-py2.7.egg/EGG-INFO/scripts/intervene", line 426, in main
pairwise.pairwise_intersection(label_names, options)
File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene-0.6.4-py2.7.egg/intervene/modules/pairwise/pairwise.py", line 412, in pairwise_intersection
matrix, bed_sizes = create_matrix(beds=options.input, bed_names=label_names, func=FUNC, verbose=False, sort_bed=options.sort, **kwoptions)
File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene-0.6.4-py2.7.egg/intervene/modules/pairwise/pairwise.py", line 74, in create_matrix
matrix[bed_names[ia]][bed_names[ib]] = func(a, b, **kwoptions)
File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene-0.6.4-py2.7.egg/intervene/modules/pairwise/pairwise.py", line 41, in reldist_of_a
return float(sum(a.reldist(b)['reldist'])) / max(len(a.reldist(b)['reldist']), 1)
File "/export/data2/anaconda2/lib/python2.7/site-packages/pybedtools/bedtool.py", line 349, in wrapped
return does_not_return_bedtool(stream, **kwargs)
File "/export/data2/anaconda2/lib/python2.7/site-packages/pybedtools/bedtool.py", line 69, in _reldist_output_handler
header = six.advance_iterator(iterable).split()
StopIteration

Banana Venn diagrams

This is a feature request. Can you please add a feature to generate banana-style Venn diagrams like the one in https://doi.org/10.1038/nature11241

ipython-notebook not installing in conda

Dear all, would you kindly advise on how to go about this. Thanks

(base) C:\Users\Admin>conda install -c free ipython-notebook
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort./
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

ipython-notebook -> python[version='2.6.|2.7.|3.4.|3.5.|3.3.*']

Your python: python=3.9

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

error when attempting to use pairwise

Hi, I'm trying to use pairwise with lists and I'm getting an error message. Note the everything works fine when using venn instead.

Here's the command I used:
intervene pairwise -i a.grp b.grp c.grp --type list

Here's the output / error message:

Performing a pairwise intersection analysis. Please wait...

Traceback (most recent call last):
  File "/Users/dlahr/miniconda3/envs/intervene/bin/intervene", line 4, in <module>
    __import__('pkg_resources').run_script('intervene==0.5.8', 'intervene')
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/intervene-0.5.8-py3.6.egg-info/scripts/intervene", line 543, in <module>
    main()
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/intervene-0.5.8-py3.6.egg-info/scripts/intervene", line 355, in main
    pairwise.pairwise_intersection(options)
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 476, in pairwise_intersection
    barplot(series, matrix, outfile, options, max_size=max(bed_sizes))
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 132, in barplot
    cax, order = heatmap_triangle(matrix, ax, options)
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 211, in heatmap_triangle
    D = D.ix[cluster_order, cluster_order]
  File "/Users/dlahr/miniconda3/envs/intervene/lib/python3.6/site-packages/pandas/core/generic.py", line 5141, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'ix'

it does produce an output Intervene_pairwise_frac_matrix.txt before the error occurs that contains:

Here's the contents of the input files:

==> a.grp <==
a
b
c
d
e

==> b.grp <==
e
f
g
h
i

==> c.grp <==
i
j
k
l

Any help getting this working would be greatly appreciated, thank you.

Letting the User choose the "degree" of intersect [enhancement]

Hello! Thanks for the amazing tool you created.

I'm suggesting an improvement for your tool, which I'd actually need.

I've several ChIP-seq bed I want to visualise with your tool but I am not interested in no more than 2 degree of intersection.

I've already tried to not showing them with –ninter but obviously the plot, doesn't recalculate the peaks and I know I'm underestimating my peaks.

Install intervene latest version from conda

Hey Aziz,
I'm trying to install intervene from bioconda cloud, and I'm having issues with getting the latest version. I even started a new miniconda2 from scratch, and I'm still getting the error:

conda install -c bioconda intervene=0.6.4
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:

intervene=0.6.4
Use "conda info " to see the dependencies for each package.

When I type:
conda install -c bioconda intervene
it works fine, but only gets me version 0.5.8

Any ideas on how to fix this? Or can I more explicitly refer to a specific version of intervene?

Cheers,
Phil

upsetr script not plotting

Hi,

I was using the intervene UpSetR module to graph the intersect between bed files but the produced R script does not seem to plot. I get the following message when I try to plot:
geom_path: Each group consists of only one observation. Do you need to adjust the group aesthetic?
Any help resolving this problem would be appreciated!

Unable to install using conda

Hello. I am trying to install Intervene using conda, but haven't had any luck so far. I have tried some of the recommendations mentioned in some of the other issues, but they haven't worked either.

I tried forcing conda to install version 0.6.5:

conda create -n intervene  intervene=0.6.5 -c bioconda
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                     

UnsatisfiableError:

That's the full error message by the way. Nothing is printed after UnsatisfiableError.

I also tried specifying the Python version:

conda create -n intervene python=3.6 intervene=0.6.5 -c bioconda
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                     

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
python=3.6
intervene=0.6.5 -> matplotlib-base[version='<3.4'] -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.10,<3.11.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0|>=3.7.1,<3.8.0a0|3.4.*|>=3.6']
intervene=0.6.5 -> pythonThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.31=0
  - feature:|@/linux-64::__glibc==2.31=0

Your installed version is: 2.31

I also tried using mamba, but it can't find the needed version of upsetr:

mamba create -n intervene intervene=0.6.5 -c bioconda
 
Looking for: ['python=3.6', 'intervene=0.6.5']

bioconda/linux-64                                           Using cache
bioconda/noarch                                             Using cache
pkgs/r/noarch                                                 No change
pkgs/main/noarch                                              No change
pkgs/main/linux-64                                            No change
pkgs/r/linux-64                                               No change
Encountered problems while solving:
  - nothing provides r-upsetr >=1.4.0 needed by intervene-0.6.5-pyh3252c3a_0

I would appreciate any help. Thanks in advance!

suggestion on color options

Hi,

Thank you for the tool. My PI loves venn diagrams and it was very painful to draw them manually.

I have couple minor suggestions.

I tried to get one of the known sets (ex: Set1) but current form of the tool only accepts spesific colors rather than the defined colormaps. Implementing this could be useful.
https://matplotlib.org/3.1.0/gallery/color/colormap_reference.html#sphx-glr-gallery-color-colormap-reference-py
Labels of the plots should be independent from the venns color or maybe in default it is the same color as venn color. However, i would like to make the label just black.

I can give this a try if you guys are busy.

Cheers,

Intervene Docker

Need to create a Docker/Singularity image of Intervene

shiny app code open source

Thanks for the very useful intervene app. Is the shiny app code available somewhere?

pairwise is not working even on --test

Hi
install latest version of intervene using git but its not working for pairwise. and by the way, using conda, its installing 0.5.8 which is not latest and does not work either for pairwise.

intervene pairwise --test

Running Intervene with test data.


Performing a pairwise intersection analysis. Please wait...

Traceback (most recent call last):
  File "/Users/rupeshk/miniconda3/bin/intervene", line 4, in <module>
    __import__('pkg_resources').run_script('intervene==0.6.4', 'intervene')
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 665, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1463, in run_script
    exec(code, namespace, namespace)
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/intervene-0.6.4-py3.7.egg/EGG-INFO/scripts/intervene", line 606, in <module>
    main()
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/intervene-0.6.4-py3.7.egg/EGG-INFO/scripts/intervene", line 426, in main
    pairwise.pairwise_intersection(label_names, options)
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/intervene-0.6.4-py3.7.egg/intervene/modules/pairwise/pairwise.py", line 480, in pairwise_intersection
    barplot(series, matrix, outfile, options, max_size=max(bed_sizes))
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/intervene-0.6.4-py3.7.egg/intervene/modules/pairwise/pairwise.py", line 135, in barplot
    cax, order = heatmap_triangle(matrix, ax, options)
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/intervene-0.6.4-py3.7.egg/intervene/modules/pairwise/pairwise.py", line 214, in heatmap_triangle
    D = D.ix[cluster_order, cluster_order]
  File "/Users/rupeshk/miniconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5274, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'ix'

--bedtools-options doesn't work

intervene venn with lists does not work

Hi - I am getting odd errors when running venn on several gene lists. I tried 2 ways:

One list (.csv)
On your readthedocs:

List type data: List data is a correctly formatted csv/text file, with lists of names. Each column represents a set, and each row represents an element (names/gene/SNPs). Header names (first row) will be used as set names.

Thus, with test.csv which looks like:

l1,l2,l3
a,b,c
c,a,a
s,d,g

I run:

intervene venn -i test.csv --type list, and

error output is Input should have at least two files.

So I tried:
2) Two lists:

test.txt has

l1
a
b
c

and

test1.txt has

l2
b
a
d

I run:
intervene venn -i test.txt test1.txt --type list, and

error output is:

Generating a 2-way "venn" diagram. Please wait...
Traceback (most recent call last):
  File "/Users/ashleymaeconard/anaconda2/envs/timeor_env2/bin/intervene", line 4, in <module>
    __import__('pkg_resources').run_script('intervene==0.5.8', 'intervene')
  File "/Users/ashleymaeconard/anaconda2/envs/timeor_env2/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/ashleymaeconard/anaconda2/envs/timeor_env2/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1462, in run_script
    exec(code, namespace, namespace)
  File "/Users/ashleymaeconard/anaconda2/envs/timeor_env2/lib/python2.7/site-packages/intervene-0.5.8-py2.7.egg-info/scripts/intervene", line 543, in <module>
    main()
  File "/Users/ashleymaeconard/anaconda2/envs/timeor_env2/lib/python2.7/site-packages/intervene-0.5.8-py2.7.egg-info/scripts/intervene", line 417, in main
    a = open(options.a, 'r').read().splitlines()
AttributeError: 'Namespace' object has no attribute 'a'

how to order the files in pairwise heatmap

intervene pairwise -i a.bed c.bed b.bed --output outdir --compute jaccard --sort --htype color --names a,c,b

By using command as above, I want my sample ordered as "a,c,b". But it automatically order them as "a,b,c". How can I set the order of my file?

Thanks!

ImportError: cannot import name 'get_backend'

Dear @asntech,

When running the example code for intervene, I am getting:

intervene venn -i intervene/example_data/ENCODE_hESC/*.bed
Traceback (most recent call last):
  File "/anaconda3/bin/intervene", line 4, in <module>
    _import_('pkg_resources').run_script('intervene==0.6.4', 'intervene')
  File "/anaconda3/lib/python3.6/site-packages/pkg_resources/_init_.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/anaconda3/lib/python3.6/site-packages/pkg_resources/_init_.py", line 1446, in run_script
    exec(code, namespace, namespace)
  File "/anaconda3/lib/python3.6/site-packages/intervene-0.6.4-py3.6.egg/EGG-INFO/scripts/intervene", line 13, in <module>
    from matplotlib import use as mpl_use
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/_init_.py", line 1111, in <module>
    rcParamsOrig = RcParams(rcParams.copy())
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/_init.py", line 891, in __getitem_
    from matplotlib import pyplot as plt
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 32, in <module>
    import matplotlib.colorbar
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/colorbar.py", line 40, in <module>
    import matplotlib._constrained_layout as constrained_layout
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/_constrained_layout.py", line 52, in <module>
    from matplotlib.legend import Legend
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/legend.py", line 43, in <module>
    from matplotlib.offsetbox import HPacker, VPacker, TextArea, DrawingArea
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/offsetbox.py", line 33, in <module>
    from matplotlib.image import BboxImage
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/image.py", line 19, in <module>
    from matplotlib.backend_bases import FigureCanvasBase
  File "/anaconda3/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 46, in <module>
    from matplotlib import (
ImportError: cannot import name 'get_backend'

I have matplotlib '3.0.3' and python 3.7

Any advice would be much appreciated.

Many thanks,
Dimitris

Is there anyway to set the value range in "heatmap" and "pieplot"?

As shown, the value range in the plot is not from (0,1). So is there any way to set the zMin and zMax to give the minimum and maximum value in "heatmap" and "pieplot"?

Thanks!

no tmp dir created when generating 3-way "venn" diagram

I get this traceback when trying to run intervene venn with 3 vcf files:

Traceback (most recent call last):
  File "/opt/anaconda3/bin/intervene", line 606, in <module>
    main()
  File "/opt/anaconda3/bin/intervene", line 522, in main
    labels = upset.genomic_upset(options, label_names)
  File "/opt/anaconda3/lib/python3.6/site-packages/intervene/modules/upset/upset.py", line 50, in genomic_upset
    x = x.intersect(bed, v=True, **kwargs)
  File "/opt/anaconda3/lib/python3.6/site-packages/pybedtools/bedtool.py", line 806, in decorated
    result = method(self, *args, **kwargs)
  File "/opt/anaconda3/lib/python3.6/site-packages/pybedtools/bedtool.py", line 337, in wrapped
    decode_output=decode_output,
  File "/opt/anaconda3/lib/python3.6/site-packages/pybedtools/helpers.py", line 356, in call_bedtools
    raise BEDToolsError(subprocess.list2cmdline(cmds), stderr)
pybedtools.helpers.BEDToolsError: 
Command was:

        bedtools intersect -v -b path/to/Flex200_svaba_converted.vcf -a /tmp/pybedtools.uqris9wu.tmp

Error message was:
Error: unable to open file or unable to determine types for file /tmp/pybedtools.uqris9wu.tmp

- Please ensure that your file is TAB delimited (e.g., cat -t FILE).
- Also ensure that your file has integer chromosome coordinates in the 
  expected columns (e.g., cols 2 and 3 for BED).

Is Intervene supposed to create temporary file / folder for bedtools to work in? Or is this error related to pybedtools? The vcf files are ok, as I've tested them all in 2-way "venn" diagram

Bedtools intersect error

Hello GitHub Users, I am running a shell script and during the run I get this error although the script goes on to finish but its not generating my intersect bed.
the portion of the script the error is coming from is

bedtools intersect -wa -a TE.bed -b "$i".allMap.bed | \
		awk '{print $1":"$2"-"$3}' | sort | uniq -c | awk '$1 > 50' | \
		awk '{print $2"\t"$1}' > "$i"TEtargets.txt
		bedtools intersect -wa -a mRNA.bed -b "$i".allMap.bed | \
		awk '{print $1":"$2"-"$3}' | sort | uniq -c | awk '$1 > 50' | \
		awk '{print $2"\t"$1}' > "$i"mRNAtargets.txt

The Error is below

ERROR: illegal character ' ' found in integer conversion of string "7 556442". Exiting...

Help running intervene?

Hello! I'm a novice user of python and condas, so please bear with me:

I was able to install intervene using condas and can see it in my environment using list:

conda list

packages in environment at /anaconda3/envs/snakes:

However, when I try to run $intervene --help or $intervene I get the following message:
Traceback (most recent call last):
File "/anaconda3/envs/snakes/bin/intervene", line 4, in
import('pkg_resources').run_script('intervene==0.5.8', 'intervene')
File "/anaconda3/envs/snakes/lib/python3.5/site-packages/pkg_resources/init.py", line 3105, in
@_call_aside
File "/anaconda3/envs/snakes/lib/python3.5/site-packages/pkg_resources/init.py", line 3089, in _call_aside
f(*args, **kwargs)
File "/anaconda3/envs/snakes/lib/python3.5/site-packages/pkg_resources/init.py", line 3118, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/anaconda3/envs/snakes/lib/python3.5/site-packages/pkg_resources/init.py", line 578, in _build_master
ws.require(requires)
File "/anaconda3/envs/snakes/lib/python3.5/site-packages/pkg_resources/init.py", line 895, in require
needed = self.resolve(parse_requirements(requirements))
File "/anaconda3/envs/snakes/lib/python3.5/site-packages/pkg_resources/init.py", line 781, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'intervene==0.5.8' distribution was not found and is required by the application

If I try to run bedtools for example, I have no issues.

Has it not installed properly? Am I missing something? Any help would be greatly appreciated. Thanks so much!!

Error using upset due to old version of UpSetR module in conda installation

Hi,

I installed intervene using conda, but when running intervene upset, I ran into the following error:

Error in upset(fromExpression(expressionInput), nsets = 4, nintersects = 30,  : 
  unused arguments (mainbar.y.label = "No. of Intersections", sets.x.label = "Set size")
Execution halted

I checked, and the version of UpSetR that shipped with the conda installation was 1.3.0. I installed version 1.4.0 from source and it fixed this error. I think this dependency should be updated.

Bed files

Could you explain how you come up with the bedfiles/how do I generate them?

Intervene works well with 3-way lists, but not 2?

Hello,
I am having trouble running intervene with two files, but with 3 it works easily...

a = open(options.a, 'r').read().splitlines()

AttributeError: 'Namespace' object has no attribute 'a'

Thanks,
D.

AdobePiStd when loading UpSet PDFs in Illustrator

Dear Aziz,

Thank you for developing intervene.

When exporting PDFs and polishing them with Illustrator, I get this error. Because of this then circles in the matrix don't look ok.
https://stackoverflow.com/questions/9992275/ggplot2-pdf-import-in-adobe-illustrator-missing-font-adobepistd

Any chance of updating this to include this feature and avoid this problem while exporting plots? Alternative solutions are also welcome.

Thank you,
Ignacio

Support *.bedpe format

Hi,
intervene is a very nice tool.
I have used this tool before for comparing a list of *.bed file. And I wonder that maybe you can also add the file with *.bedpe format. and it also only uses bedtools.
if done, I think intervene will attract more and more people for its convenient.
Lu Zhang

Support for zero interval files

I use Intervene in a computational pipeline to check overlaps from multiple BED files. The command I use is:

intervene pairwise --input <INPUT> --compute jaccard --output <OUTPUT>

Sometimes one of the BED files contains zero intervals. When this occurs Intervene throws the following error:

Performing a pairwise intersection analysis. Please wait...

Traceback (most recent call last):
  File "/home/s1437643/conda/bin/intervene", line 606, in <module>
    main()
  File "/home/s1437643/conda/bin/intervene", line 426, in main
    pairwise.pairwise_intersection(label_names, options)
  File "/home/s1437643/conda/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 480, in pairwise_intersection
    barplot(series, matrix, outfile, options, max_size=max(bed_sizes))
  File "/home/s1437643/conda/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 135, in barplot
    cax, order = heatmap_triangle(matrix, ax, options)
  File "/home/s1437643/conda/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 211, in heatmap_triangle
    Z = sch.linkage(D, method='average')
  File "/home/s1437643/conda/lib/python3.6/site-packages/scipy/cluster/hierarchy.py", line 713, in linkage
    raise ValueError("The condensed distance matrix must contain only "
ValueError: The condensed distance matrix must contain only finite values.

Would you be able to provide support for zero interval files? Preferably, the file is still included in all the analysis results rather than just not using the file.

Unexpected (wrong?) results in terms of total length for the intersection of two sets

Hello,

I've been testing Intervene on two of the sample tests. I've generated an UpSet plot and saved the corresponding BED files for the intersection and set differences and then computed the total length of the features from each of the files.

I would expect that the summed total lengths would equal that of the initial files, but this is not the case. Any assistance in clarifying this issue would be much appreciated, as I was planning on using Intervene on a larger number of files.

I've also manually computed the intersection and set differences using bedops and, this time, the summed total lengths were as I'd expect. Finally, I've plotted a Venn diagram of the intersections using the R package bedr and its results match those obtained with bedops. I'm including the diagram below.

Below is a minimal reproducible example for what I did:


## set up a temporary directory
temp_dir=/path/to/temp_dir
mkdir $temp_dir
cd $temp_dir

wget https://raw.githubusercontent.com/asntech/intervene/master/intervene/example_data/ENCODE_hESC/H3K27me3.bed
wget https://raw.githubusercontent.com/asntech/intervene/master/intervene/example_data/ENCODE_hESC/H3K4me1.bed

## run Intervene and save the BED files for the overlaps
intervene upset --output overlap_counts_2 --save-overlaps -i H3K27me3.bed H3K4me1.bed

## compute the length of the UpSet diagram elements (subsets)
cd $temp_dir/overlap_counts_2/sets

ln -s $temp_dir/H3K4me1.bed H3K4me1.bed
ln -s $temp_dir/H3K27me3.bed H3K27me3.bed

## manually compute the intersection and set differences using BEDOPS
bedops --intersect H3K27me3.bed H3K4me1.bed > BEDOPS_11_H3K27me3_H3K4me1.bed
bedops --difference H3K27me3.bed BEDOPS_11_H3K27me3_H3K4me1.bed > BEDOPS_10_H3K27me3.bed
bedops --difference H3K4me1.bed BEDOPS_11_H3K27me3_H3K4me1.bed > BEDOPS_01_H3K4me1.bed

## compute the total length (in bp) for each BED file and write these to a text file
echo -e "BED file\t$Total length (bp)" > lengths.txt

for file_name in *.bed; do
  subset_length=$(cat $file_name | bedops --merge - | awk -F'\t' 'BEGIN{SUM=0}{ SUM+=$3-$2 }END{print SUM}')
  echo -e "${file_name}\t${subset_length}" >> lengths.txt
done

cat lengths.txt

These are the results I got:

BED file	 length (bp)
01_H3K4me1.bed	202168680
10_H3K27me3.bed	11936247
11_H3K27me3_H3K4me1.bed	208156989
BEDOPS_01_H3K4me1.bed	323296156
BEDOPS_10_H3K27me3.bed	135056952
BEDOPS_11_H3K27me3_H3K4me1.bed	83578607
H3K27me3.bed	218635559
H3K4me1.bed	406874763

Here, I would expect that the total length of e.g. H3K27me3.bed would be equal to the summed total lengths of 10_H3K27me3.bed and 11_H3K27me3_H3K4me1.bed, but this wasn't the case. However, the results produced with bedops did match this expectation.

I have used the following R code to produce a Venn diagram using the R package bedr:

install.packages("bedr")
library(bedr)

## read the BED files
regions.a = read.table("/path/to/H3K4me1.bed", 
               header = FALSE, stringsAsFactors = FALSE)

regions.b = read.table("/path/to/H3K27me3.bed", 
               header = FALSE, stringsAsFactors = FALSE)

bedr.plot.region(
  input = list(
    a = regions.a,
    b = regions.b
  ),
  feature = "bp",
  fraction.overlap = 0.000000001
)

Below is the resulting Venn diagram. The total bp length values are identical to those computed above by bedops.

Bedtools-options does not produce different results

I am trying to get the bedtools option 'f' to work with intervene.

I run the following command:

singularity run intervene_0.6.5--pyh3252c3a_1.sif intervene upset -i mm*.bed --bedtools-options f=0.1 -o BedFOn

However, the results do not differ from the base command.

Is my syntax incorrect?

intervene venn doesn't work with "--bedtools-option s=True"

Can intervene be complemented with "bedtools intersect -s "? (that is intersecting peaks by strand)

The intervene venn diagram is misleading

I found that The intervene venn diagram is misleading.
I use the peak summits file and narrowPeak file of the same sample to do intervene venn,
All of the summits are in narrowPeaks:

intervene venn -i macs2_2/DoxCD31mKATEpos3_summits.bed \
macs2_2/DoxCD31mKATEpos3_peaks.narrowPeak \
--names summits,narrowPeak \
--bedtools-options f=1bp \
--title DoxCD31mKATEpos3 \
-o venn \
--figsize 12 12 \
--fontsize 14 \
--dpi 300 \
--project DoxCD31mKATEpos3

When A overlap with B:
The left area label (no overlap area) is total of A subtract overlap
The right area label (no overlap area) is total of B.

It is better to show right area label with total of B subtract overlap.

I am wondering whether I can use the fisher test in pairwise module about my species(Arabidopsis thaliana)

Hi, Dr Khan
I am wondering whether I can use the fisher test in pairwise module about my species(Arabidopsis thaliana). I have made it when I use the bedtools

$ bedtools fisher -a test.narrowPeak -b merge_peak.bed -g Athaliana.genome 
# Number of query intervals: 15732
# Number of db intervals: 25530
# Number of overlaps: 13743
# Number of possible intervals (estimated): 78334
# phyper(13743 - 1, 15732, 78334 - 15732, 25530, lower.tail=F)
# Contingency Table Of Counts
#_________________________________________
#           |  in -b       | not in -b    |
#     in -a | 13743        | 1989         |
# not in -a | 11787        | 50815        |
#_________________________________________
# p-values for fisher's exact test
left	right	two-tail	ratio
1	0	0	29.788

-----------------------------------------------------------
Athaliana.genome 

Chr1    30427671
Chr2    19698289
Chr3    23459830
Chr4    18585056
Chr5    26975502
ChrM    366924
ChrC    154478

--------------------------------------------------

But when I use the Athaliana.genome in intervene pairwise.It all failed
(My intervene is installed by pip. The version is 0.6.4)

$ intervene pairwise -i * --compute=fisher --genome Athaliana.genome
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '.chromInfo' at line 1

-------------------------------------------------------------------------------------------------------
$ intervene pairwise -i * --compute=fisher --bedtools-option -g Athaliana.genome
usage: intervene pairwise [options]
intervene <subcommand> [options] pairwise: error: argument --bedtools-options: expected one argument
(intervene_module)

--------------------------------------------------------------------------------------------------------
$ intervene pairwise -i * --compute=fisher --bedtools-option g=Athaliana.genome
Fisher test requires genome name. Please set the --genome argument.
Type --help or read documentation for more details - http://intervene.readthedocs.io
(intervene_module)

It seems the intervene pairwise will get the genome info from uscs sql. But My species is not in USCS(it is a plant :D). So I am wondering if there is anotehr solution.

Guandong Shang

small but confused bug for reading error

Hi,
When the intervene read the peak file, it seems to report a error as below:

Performing a pairwise intersection analysis. Please wait...

Traceback (most recent call last):
  File "/export/data2/anaconda2/bin/intervene", line 606, in <module>
    main()
  File "/export/data2/anaconda2/bin/intervene", line 426, in main
    pairwise.pairwise_intersection(label_names, options)
  File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene/modules/pairwise/pairwise.py", line 412, in pairwise_intersection
    matrix, bed_sizes = create_matrix(beds=options.input, bed_names=label_names, func=FUNC, verbose=False, sort_bed=options.sort, **kwoptions)
  File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene/modules/pairwise/pairwise.py", line 74, in create_matrix
    matrix[bed_names[ia]][bed_names[ib]] = func(a, b, **kwoptions)
  File "/export/data2/anaconda2/lib/python2.7/site-packages/intervene/modules/pairwise/pairwise.py", line 32, in jaccard_of_a
    return a.jaccard(b,u=True, **kwoptions)['jaccard']
  File "/export/data2/anaconda2/lib/python2.7/site-packages/pybedtools/bedtool.py", line 405, in wrapped
    return does_not_return_bedtool(stream, **kwargs)
  File "/export/data2/anaconda2/lib/python2.7/site-packages/pybedtools/bedtool.py", line 63, in _jaccard_output_to_dict
    header, data = _s.splitlines()
ValueError: need more than 0 values to unpack

I found out that when the input peak file has a last line as blank one, it will report. If I remove the last blank line, it is fine.

Error running intervene upset method

Hi all,

Intervene is such a good program. Thank you letting it as open source.

On the other hand I was trying to run intervene upset -i /*gff and got the following massages:

So, I was wondering if you can give some lights to solve this issue. By the way, I installed intervene via conda

Cheers,
Camilo

Questions: Why half of pairwise is not a exact mirror of the other half? And about negative values..

Good morning,

Many thanks for developing this amazing tool.
I have a doubt about the pairwise mode that may seem naive, but I could not figure out.
I notice in some of my dataset and also here in this example:

intervene pairwise -i ~/dbSUPER/mm9/*.bed --filenames --compute frac --htype color
https://intervene.readthedocs.io/en/latest/_images/pairwise_color.png

In that example, half of the data is not mirrored on the other half. As the combination is the same, why the values are not mirrored?
For example, "Bone_Marrow" row and "Spleen" column should not have the same value as "Spleen" row and "Bone_Marrow" column?

I assume that this is the reason why "tribar" mode is not recommended for "count" or "frac", but why this happen?
Would you mind to clarify? And how to interpret this in the correct way?

If I plot correlation , let's say "pearson", this doesn't happen and I get mirrored values as I would expected. Would this be a solution?
If so, what negative values are saying to me in this case? Would that be that instead of being negative correlate (variable A increases while B decreases or vice-versa) it would be close to zero (-1 = 0 overlaps)?

Thank you for your time

Feature request to add an option to label the output file

Thanks for creating Intervene, it is very nice. I have a feature request. Would it be possible to add an option to label the output file, rather than only specify the output folder? Something like

--basename -b Base of output file name. Default is Intervene.

./intervene/intervene venn -i intervene/example_data/ENCODE_hESC/*.bed –-basename awesomeExperiment
Generates a file named awesomeExperiment_venn.pdf

fail to use pairwise comands

Hi
I install intervene using conda and its working with plotting commands but not the pairwise. even with test its failing, please can you tell me whats wrong here? or bugs ??

intervene pairwise --test

Running Intervene with test data.


Performing a pairwise intersection analysis. Please wait...

Traceback (most recent call last):
  File "/Users/rupeshk/miniconda3/envs/intervane/bin/intervene", line 4, in <module>
    __import__('pkg_resources').run_script('intervene==0.5.8', 'intervene')
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/pkg_resources/__init__.py", line 665, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1463, in run_script
    exec(code, namespace, namespace)
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/intervene-0.5.8-py3.6.egg-info/scripts/intervene", line 543, in <module>
    main()
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/intervene-0.5.8-py3.6.egg-info/scripts/intervene", line 355, in main
    pairwise.pairwise_intersection(options)
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/intervene/modules/pairwise/pairwise.py", line 454, in pairwise_intersection
    matrix = pd.read_table(matrix_file,index_col=0, delim_whitespace=True)
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/pandas/io/parsers.py", line 755, in read_table
    return read_csv(**locals())
  File "/Users/rupeshk/miniconda3/envs/intervane/lib/python3.6/site-packages/pandas/io/parsers.py", line 625, in read_csv
    "Specified a delimiter with both sep and "
ValueError: Specified a delimiter with both sep and delim_whitespace=True; you can only specify one.

intervene pairwise module bug

intervene's pairwise module doesn't take --names argument and if the file names are same it gives below error.
ValueError: Linkage must be computed on at least two observations.

provide an option to output the overlapping regions/sets

The current version of interven doesn't have the option to output the overlapping regions. We are working to release an updated version soon, which will have this option.

P.S. Many users asked to provide this feature.