If there is not sufficient reads in the given region, bamCorrelate reports an error:
File "bamCorrelate", line 447, in main num_reads_per_bin[:, col])[0]
ValueError: setting an array element with a sequence.
It would be good to print a (warning) message saying that not enough data in the selected region instead of this error.

to be filled up :)

• computeMatrix: how Galaxy handles output with multiple clusters
• profiler: color option

With the current setup.py script it is not possible to use:

• python setup.py --home /foo/bar or
• python setup.py --install-lib

Hi guys,

I was wondering if there is a way to plot several profiles in one plot using the profiler tool. It is always easier to compare the profiles if they are in one plot.

All best,

I have not figured out a quick and painless way to remove or add "chr" for chromosome names with the tools that are present currently. Perhaps we should add a small python script.

at the moment, only sorting "descendingly", "ascendingly" and "no sort" are available within Galaxy while for the command line, region-length-sorting is available, too

--sortUsing region_length

Dear developers,

• I came across an issue in bamCorrelate. For a reason, I was unable to make it run.
  It is always prompting the error:

"bamCorrelate bins: error: argument --numberOfProcessors/-p: lot is not a valid number of processors"

Specifying the number of processors did not make any difference.

• Also, I noticed differences in the arguments that bamCorrelate takes between the example run:
  "An example usage is: bamCorrelate bins -b treatment.bam input.bam -plot
  correlation.png -f 200 -method pearson" and the list of arguments from the help menu.
• Also, python-2.7 $KH/bin/deepTools/bin/bamCorrelate -h is not detailed enough to know the parameters for bamCorrelate

Sorry for all that,
Hope that it will help though and many thanks for the tool,

P.K

Currently, the wiki pages are a bit more elaborate than the explanations that users see when they select a tool within Galaxy.
My questions:

• Should we include the long descriptions? (I lean towards no)
• Should we include the links to the wiki pages (I lean towards yes)
• Where could I modify these information, i.e. if we really migrate the wiki once more to the deepTools website eventually, all these links need to be updated etc. I could do it if I knew where to look for it.

When a bed-file contains something else than a plus or minus sign in the 6th column, an error is produced. This should be changed to a warning. (a warning should also be raised when the bed-file does not contain any information about the strand)

with correctGCBias 1.5.6-45-g7190dd0 I get the following message when trying to output a bedgraph file:

format: bedgraph, database: dm3
applying correction genome partition size for multiprocessing: 362078 using region 4 sh: open: No such file or directory

it's a warning, but the file empty.

with output = bigWig I get a similar message:

end (1321) before start (1321484) line 23342 of /tmp/tmplJWLcp

This behavior is OK, but should proceed when the bam chromosome names are a subset of the 2bit chromosome names.

This is related to the issue I first raised for the Galaxy implementation, but Fidel and I agreed that it would be meaningful to make deepTools capable of processing BAM and BED files even if their chromosome naming conventions do not match 100%. After all, IGV and other tools can do it, too.

Discrepancies should still be reported as to make the user aware of the fact the chromosomes are labelled differently, but this should not break the code.

from Fidel:
The problem with the normalization (either RPKM or normalizeTo1X) is that they are based on the total number of mapped reads and not on the total number of reads of quality > minMappingQuality.

That means that people still need to filter the bamFile before running bamCoverage (which is contrary to our intention)

bamCompare should not be heavily affected by this.

The pearson correlation run by bamCorrelate is heavily affected by outliers that commonly occur on satellite regions that accumulate large numbers of reads.

• no duplicate removal

1 wiki page should contain all the options for all the programs

everytime the help within the python scripts is updated, one should just run this script that should generate a wiki-compatible markdown page

The issue is caused by the lack of # lines in the output of heatmapper nowadays. To make heatmapper etc. work with Galaxy, Bjoern added the feature that the cluster ID is indicated in the 7th column of a bed-file, so that users can easily separate the file into separate data sets and supply them individually to computeMatrix. On the command line, computeMatrix expects just one BED file where the groups are separated by #. That means that currently the user needs to know that he has to turn the file with the format:

chr1 10 12 Cluster1
chr1 20  22 Cluster2

into a file like this:

chr1 10 12
# cluster1
chr1 20  22
# cluster2 

Perhaps we can come up with a more elegant solution in the future.

Hi guys,

I appreciate that computeMatrix is telling me all the problems that it encounters (I know, it's a tough job, poor little bugger), but I would appreciate it even more if it would stop littering my stderror when I set the --quiet option. In short: I don't think the -q option works completely properly :)

I think we should add this tools to Galaxy at some point. Probably a more meaningful name for PE_fragment_size is required.

I saw in the supplement of the paper several small mistakes/inconsistencies with the commandline interface of heatmapper. I tried to fixed most of them. If anyone can have a brief look ... that would be nice. We can update the paper, during the proof read I hope.

1. a tool to generate count tables of unnormalized read numbers would be nice (very similar to what bamCorrelate is already doing) --> this could be useful to generate the count tables required by DESeq etc.
2. calculate the normalized read coverages (typically FPKM) per genes

The installation is not as easy as it could be. Especially more verbose warnings/errors would be useful, to assist the user.

from distutils import spawn
spawn.find_executable('samtools')

That can be used to determine binaries at runtime and fail with a meaningful and helpful message.

at the moment, labels are only put into the profiles

Hi,
I came across the following bug in bamCompare. Specifying the "--scaleFactors" in the command line in the --verbose mode throws the error below:
Hope this will help,
P.

Traceback (most recent call last):
File "/g/furlong1/khoueiry/bin/deepTools/bin/bamCompare", line 315, in
main(args)
File "/g/furlong1/khoueiry/bin/deepTools/bin/bamCompare", line 279, in main
"RPKM is {0}".format(scaleFactor)
UnboundLocalError: local variable 'scaleFactor' referenced before assignment

at the moment, --outRawCounts returns a matrix without row names. if a BED file is given for bamCorrelate, it makes sense to output the name of the regions in the BED file as the rowname. this matrix could then be directly plugged into DESeq and other downstream applications that require an unnormalized read count matrix

in line 124/125, comma missing before 'default=1000'

another thing pointed out by our proof-readers: the way users are asked to supply genome or effective genome sizes is currently not consistent between the different Galaxy tools

bamCompare

when "normalize to 1x sequencing depth" is selected, the following header and an empty field appear:

Report normalized coverage to 1x sequenceing depth:

--> this entry should probably be named "Effective genome size" instead

bamCompare, bamCoverage

There's an explanation for the genome size entry:

Enter the genome size to normalize the reads counts. Sequencing depth is defined as the total number of mapped reads * fragment length / effective genome size. To use this option, the effective genome size has to be given. Common values are: mm9: 2150570000, hg19:2451960000, dm3:121400000 and ce10:93260000.

This should read instead:

Enter the effective genome size to normalize the reads counts (the part of the genome that can be mapped, i.e. without undefined bases and highly repetitive regions). Sequencing depth is defined as the total number of mapped reads * fragment length / effective genome size. Common values are: mm9: 2150570000, hg19:2451960000, dm3:121400000 and ce10:93260000.

in correctGCbias and computeGCbias
there is a very elaborate description of the effective genome size including a drop down box (while in the other tools, it's an empty field where the user has to enter the value himself)

I think, we should make it consistent. Actually, I like the way it's done for the GCbias tools better than the empty field. The empty field, however, is consistent with how MACS has it...

@diehlsa & @bgruening : what is needed to directly load data from Galaxy into IGV browser?

I'm aiming at something like this:
https://github.com/snowplow/snowplow/wiki

Structure should be something along these lines:
0. Installation (this should be kept in the README.md so that it's still the first thing people see)

1. How we use deepTools
2. Documentation of the tools (QC, normalization, visualization)
3. Recipes (Galaxy and command line)
4. FAQ (Galaxy and command line)

Suggestions welcome.
Cheers,
Friederike

The option --ignoreDuplicates appears in bamCorrelate but it has no effect. This is because such option is not passed to the countReadsPerBin.getCoverageOfRegion function.

Fabian asked for recommendations here

i.e. when to use Spearman and Pearson

I think, we, ourselves, haven't fully understood this issue given the current plots we produced, right?

For example, for computeMatrix only true BED or INTERVAL files should be shown in the drop down menu for the regions while only true BIGWIG files should be shown in the drop down menu for the scores.

At the moment, almost all files in a history tend to be shown, very often with the comment ("as BED" or "as BIGWIG") which, in most cases, will not work and will lead to confusion.

Is this something that needs to changed every time the tools are updated or (re-)installed in the Galaxy? And where does it need to be changed?

Just as a reminder: currently, --missingDataAsZero {yes, no} only exists in bamCompare. There is no reason why it should not be available in bamCoverage (since 1.5.7., the default in bamCoverage is set to "yes")

The effective genome size (mappable portion of a genome) numbers for bamCoverage and bamCorrelate are for uniquely map reads. However, nowadays is common to have larger effective genome sizes when using random mapping of multi-reads. Furthermore, depending on the read length used the mappable portion of the genome changes.

Maybe we should add all options to the tools or point to Table 2 of this paper: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0030377

perhaps just add the link to the wiki page (https://github.com/fidelram/deepTools/wiki/Galaxy#wiki-dataup) within the TIP section underneath the selection box

From: https://www.biostars.org/p/101413/

I'm quite interested if it is possible to integrate side tools to your framework. Actually when working with exome data I've come across the alternative coverage measure that uses distinct reads, i.e. reads that cover a given position and have different offsets. It is also referred as molecular coverage. The inspiration comes from this paper http://genomebiology.com/content/12/1/R6, Fig1. Such measure could be more prone to sequencing artifacts. If you're interested the basic implementation using Picard API is here https://github.com/mikessh/exome-misc/blob/master/src/molcount/MolCountExome.groovy and here https://github.com/mikessh/exome-misc/releases/download/v1.0.0/exome-tools-v1.0.0.jar compiled as jar.

Now that we have deepTools being tolerant towards chromosome naming, I totally forgot that other tools are not that forgiving. I just put the error here so I eventually make an FAQ entry out of it:

BedGraph to bigWig on data 76
An error occurred with this dataset: 2L is not found in chromosome sizes file

and another thing pointed out by innocent users: they would like to have the abbreviation that we use in the overview table to be represented , i.e.:

• bamCorrelate (QC) correlates pairs..
• bamFingerprint (QC) plots profiles...
• computeGCbias (QC) ...
• correctGCbias (N) ...
• bamCoverage (N) ...
• bamCompare (N) ...
• computeMatrix (V) ...
• heatmapper (V) ...
• profiler (V)

perhaps a glossary would be meaningful? Fabian reports that he's struggling with the terms and them not being consistently used...(which is most likely very true)

any opinion on whether one should use the term NGS or HTS? I tried to use HTS because Thomas preferrred it, but I have the feeling that NGS is much more commonly used

last, but not least: ANY idea on how to unify the terms in the wiki, the help texts in the python scripts and the galaxy help texts most efficiently? perhaps a "help-text-a-thon" is needed where we edit all 3 texts simultaneously? I honestly cannot bring myself to go through these massive texts all on my own, but I think it would tremendously decrease the frustration potential if we did it

Is it possible to simply not add any number (0 in the example) when no additional regions are selected?

I think we cannot expect that everybody knows the term "effective genome size"

I propose to add the question (plus answer) "What do you mean by effective genome size?" to the FAQ section "General deepTools-related questions "

https://github.com/fidelram/deepTools/wiki/General-deepTools-FAQs

As discussed with @friedue it would be nice to have that option available in bamCoverage.