nextflow-io / nextflow Goto Github PK

A DSL for data-driven computational pipelines

License: Apache License 2.0

Shell 0.81% Groovy 87.07% Java 6.88% HTML 3.74% Smarty 0.01% Makefile 0.07% JavaScript 0.21% Nextflow 1.04% Dockerfile 0.03% Perl 0.14%

bioinformatics workflow-engine pipeline pipeline-framework nextflow cloud groovy sge slurm aws

nextflow's Introduction

"Dataflow variables are spectacularly expressive in concurrent programming"
Henri E. Bal , Jennifer G. Steiner , Andrew S. Tanenbaum

Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, AWS Batch, Azure Batch, Google Cloud Batch, and Kubernetes. Additionally, it supports many ways to manage your software dependencies, including Conda, Spack, Docker, Podman, Singularity, and more.

Quick start

Install Nextflow with a single command:

curl -fsSL https://get.nextflow.io | bash

It creates the nextflow executable file in the current directory. You can then move it to a directory in your $PATH to run it from anywhere.

Nextflow can also be installed from Bioconda:

conda install -c bioconda nextflow

Documentation

The Nextflow documentation is available for the latest stable and edge releases.

Community

You can post questions and get help in the Nextflow community forum or the Nextflow Slack. Bugs and feature requests should be reported as GitHub issues.

The Nextflow community is highly active with regular community meetings, events, a podcast and more. You can view much of this material on the Nextflow and nf-core YouTube channels.

The nf-core project is a community effort aggregating high quality Nextflow workflows which can be used by everyone.

Contributing

Contributions are more than welcome. See the CONTRIBUTING file for details.

License

Nextflow is released under the Apache 2.0 license. Nextflow is a registered trademark.

Citations

If you use Nextflow in your work, please cite:

P. Di Tommaso, et al. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 316–319 (2017) doi:10.1038/nbt.3820

Credits

Nextflow is built on two *great* open-source software projects, Groovy and GPars.

YourKit is kindly supporting Nextflow with its fully-featured Java Profiler.

nextflow's People

Contributors

Stargazers

Watchers

Forkers

jmchangtw jgonzale noctarius mustaang melix mariach msgendev robsyme gitter-badger lazymike holtgrewe heuermh yesimon campagnelaboratory imclab joecarlson ousamg biologghe boratonaj zengfengbo cbcrg wangzhenfei qqss88 mes5k cerebis codeaudit kgutwin samuell shihabhasan patia013 qingpeng madhukarreddy simexin hy714335634 nagyistge nh13 resurgo-genetics raonyguimaraes rbarnard shellfishgene pfern biocodings lucacozzuto evanfloden proycon fredericlemoine ingkebil mreddy8182 baffelli pprieto ishwarvh wikiselev kevinsayers djtzou andreas-wilm wuch tdudgeon melakbet swidnikk fmorency odiogosilva fstrozzi surendrk bio-dragon bmbbms leonardj09 edgano kindlyops ljxu1 scchess oliversi buhijs faithman raymondzhaoy matijasos emi80 opensciviz hadoop835 davidcotter willfurnass rohanshah yamad bzizou stevekm abremges sb10 wuxi-nextcode noelnamai martinsos kunju-pitt lukegoodsell srikanth9859 charleshenryhugo sahu0957 pipemg yarikoptic pditommaso wleepang izzykayu alneberg

nextflow's Issues

Pipeline hangs when all processes execution are skipped

See the log file in the above link and the example at #41

https://gist.github.com/yesimon/cf59053d90ee3ea6c700

Resuming a process that declared an parametric output value throws an exception

For example

#!/bin/env nextflow

params.in = 'test.txt'

test_file = file(params.in)
name = test_file.getBaseName()

process test {
    input:
    file test_file
    val x from name

    output:
    file "${x}.tsv" into out

    """
    echo "${test_file}" > ${x}.tsv
    """
}

out.subscribe {
    println "File name: $it.name"
    println "File content: $it.text"
}

When resumed throws the following exception

Jun-05 10:55:59.004 [main] DEBUG nextflow.script.CliRunner - nextflow test.nf -resume
Jun-05 10:55:59.108 [main] DEBUG nextflow.script.CliRunner - Parsing config file: /Users/pditommaso/.nextflow/config
Jun-05 10:55:59.455 [main] DEBUG nextflow.Session - Executor pool size: 9
Jun-05 10:55:59.477 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/pditommaso/workspace/nextflow/bin
Jun-05 10:55:59.478 [main] DEBUG nextflow.script.CliRunner - Script bin dir: null
Jun-05 10:55:59.515 [main] DEBUG nextflow.script.CliRunner - 
      Version: 0.8.4-b1 build 1838
      Last modified: 04-06-2014 12:59 UTC (14:59 CEST)
      Os: Mac OS X 10.9.3
      Groovy: 2.3.0
      Jvm: Oracle Corporation 1.7.0_51-b13
      Opts: -Djava.awt.headless=true -Xms512m
      Encoding: UTF-8 (UTF-8)
      Address: Retina.local [172.17.4.167]
      File systems: file,jar
Jun-05 10:55:59.526 [main] DEBUG nextflow.Session - > Session start -- phaser register (session) 
Jun-05 10:55:59.534 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Jun-05 10:55:59.534 [main] DEBUG nextflow.script.CliRunner - > Script parsing
Jun-05 10:55:59.598 [main] DEBUG nextflow.Session - Using default localLib path: /Users/pditommaso/workspace/nextflow/lib
Jun-05 10:55:59.605 [main] DEBUG nextflow.script.CliRunner - Adding to the classpath library: /Users/pditommaso/workspace/nextflow/lib
Jun-05 10:55:59.606 [main] DEBUG nextflow.script.CliRunner - Adding to the classpath library: /Users/pditommaso/workspace/nextflow/lib/commons-cli-1.2.jar
Jun-05 10:55:59.607 [main] DEBUG nextflow.script.CliRunner - Adding to the classpath library: /Users/pditommaso/workspace/nextflow/lib/guava-cache-16.0.1.jar
Jun-05 10:55:59.767 [main] DEBUG nextflow.script.CliRunner - > Launching execution
Jun-05 10:55:59.887 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
Jun-05 10:55:59.887 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-05 10:55:59.904 [main] INFO  nextflow.executor.Executor - [warm up] executor > local
Jun-05 10:55:59.940 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'local' > capacity: 7; pollInterval: 100ms; dumpInterval: 5min 
Jun-05 10:55:59.952 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: TaskPollingMonitor
Jun-05 10:55:59.953 [main] DEBUG n.processor.TaskPollingMonitor - >>> phaser register (scheduler)
Jun-05 10:56:00.032 [main] DEBUG nextflow.script.BaseParam - output > channel unknown: out -- creating a new instance
Jun-05 10:56:00.065 [main] DEBUG n.processor.ParallelTaskProcessor - Creating operator > test -- maxForks: 9
Jun-05 10:56:00.082 [main] DEBUG nextflow.Session - >>> phaser register (process)
Jun-05 10:56:00.102 [main] DEBUG nextflow.script.CliRunner - > Await termination 
Jun-05 10:56:00.200 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'test (1)'

Caused by:
  x

Tip: check the log file '.nextflow.log' for more details
Jun-05 10:56:00.204 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Process test raise the following exception:
groovy.lang.MissingPropertyException: x
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_51]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.script.BaseParam.resolveName(ParamsIn.groovy:200) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.script.FileOutParam$_getFilePatterns_closure1.doCall(ParamsOut.groovy:233) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.script.TokenGString$_resolve_closure1.doCall(ScriptTokens.groovy:130) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2342) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2312) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.dgm$63.invoke(Unknown Source) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.script.TokenGString.resolve(ScriptTokens.groovy:130) ~[main.jar:na]
    at nextflow.script.TokenGString$resolve.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.script.FileOutParam.getFilePatterns(ParamsOut.groovy:233) ~[main.jar:na]
    at nextflow.script.FileOutParam$getFilePatterns$0.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.processor.TaskProcessor.collectOutFiles(TaskProcessor.groovy:1008) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.processor.TaskProcessor$_collectOutputs_closure18.doCall(TaskProcessor.groovy:944) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1373) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1366) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.dgm$149.invoke(Unknown Source) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.processor.TaskProcessor.collectOutputs(TaskProcessor.groovy:936) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.processor.TaskProcessor.checkCachedOutput(TaskProcessor.groovy:617) ~[main.jar:na]
    at nextflow.processor.TaskProcessor$checkCachedOutput$2.callCurrent(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:486) ~[main.jar:na]
    at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.0.jar:2.3.0]
    at nextflow.processor.ParallelTaskProcessor.invokeTask(ParallelTaskProcessor.groovy:336) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1085) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:952) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1131) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1087) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:730) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.Script.invokeMethod(Script.java:78) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:407) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:346) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) ~[groovy-2.3.0.jar:2.3.0]
    at Script1$_run_closure1.doCall(Script1.groovy:1) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.0.jar:2.3.0]
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120) ~[gpars-1.2.1.jar:1.2.1]
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35) ~[gpars-1.2.1.jar:1.2.1]
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58) ~[gpars-1.2.1.jar:1.2.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
Jun-05 10:56:00.205 [Actor Thread 3] DEBUG nextflow.Session - Session abort -- terminating all processors
Jun-05 10:56:00.206 [Actor Thread 1] DEBUG n.processor.ParallelTaskProcessor - <test> After stop -- shareObjs [:]
Jun-05 10:56:00.207 [main] DEBUG nextflow.Session - <<< phaser deregister (session)

Documentation code example not working

The code example in the output files section on the process page does not work.

The code xample in question:

process randomNum {

   output:
   file 'result.txt' into numbers

   '''
   echo $RANDOM > result
   '''
}

numbers.subscribe { println "Received: " + it.text }

When executed, I get:

[14:45:53] ~/code/test/nextflow $ nextflow run randomnum.nf 
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=lcd_hrgb -Dswing.aatext=true -Dsun.java2d.xrender=true
N E X T F L O W  ~  version 0.9.0
[warm up] executor > local
[47/583d32] Submitted process > randomNum (1)
ERROR: Error executing process > 'randomNum (1)'

Caused by:
  Missing output file: 'result.txt' expected by process: randomNum (1)

Command executed:

  echo $RANDOM > result

Command exit status:
  0

Command output:
  (empty)

Work dir:
  /home/samuel/code/test/nextflow/work/47/583d329642c9b78d65fadaabf6637a

Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'

how pass linux commandline data to nextflow globe variable

I have file a.txt, it has 20 lines.
then I want use variable with value 20 in nextflow process.
I tried using nextflow script below, but failed.
How should I do?

1 // initialization
2 x=5
3
4 // setting again
5 x=10
6
7 // setting again in process
8 process set{
9
10 script:
11 x = " wc -l a.txt | awk '{print $1}' "
12 }
13
14
15 process get{
16 echo true
17
18
19 script:
20 """
21 echo "variable x in use : "${x}
22 """
23 }

Do not limit retries for re-sumbissions with maxErrors

Currently, if maxErrors is set to a value lesser than maxRetries, only up to maxErrrors re-submissions are attempted for a process. From a user's perspective, this is a bit confusing, especially if only maxRetries is set and maxErrors is left to its default value (1). maxErrors should not limit the number of possible resubmissions.

From outside the box, I see to alternative ways to go:

a conventional value (e.g. -1) for maxErrors that tells nextflow to ignore the errors limit. The risk with this option is that a process execution could go on forever with re-submissions. To avoid that, it should be allowed only when maxRetries is also set.
when maxRetries is set, maxErrors is ignored, regardless its value. This could be seen as a sort of priority between them. The drawback is that the maximum number of re-submission is determined by maxRetries * number of execution instances.

Support for executable Docker images

Docker images encapsulating individual applications are generally set up with an ENTRYPOINT instruction, which makes the image executable. In many deployments, it is considered good practice to set up images and containers in this way.

In these cases, the appropriate way to interact with the image is to pass a list of arguments to the executable to docker run, rather than to call it with a command to run a Bash script.

It would be much easier to interact with and use executable images if this support were available.

Long living channel

Recently, I have been working on a nextflow pipeline where I need to log all my output of 3 processes into a single file and I have to save the resulting files.

To attend this goal, I have build two Channels:

One to save the log
One to copy the files

    qc_logger.subscribe { logger(it) }
    qc_copy.subscribe { copyFile(it, read_folder) }

I have to repeat the same code for every process.

I have also tried to make a long living channel, but after the first process completed the channel was soon closed ignoring the output of others process.

Is there any work around? or include support for a long living Channel?

Cannot get Integer index for file list

a = Channel.fromPath('f/*.txt')                                                                                                                                                                                                                                                                                                                                             
b = Channel.fromPath('f/*.txt')                                                                                                                                                                                                                                                                                                                                             
c = a.merge(b) { one, two -> [one, two]}                                                                                                                                                                                                                                                                                                                                    

process handle_paired {                                                                                                                                                                                                                                                                                                                                                     
  echo true                                                                                                                                                                                                                                                                                                                                                                 

  input:                                                                                                                                                                                                                                                                                                                                                                    
  file c                                                                                                                                                                                                                                                                                                                                                                    

  script:                                                                                                                                                                                                                                                                                                                                                                   
  c_0 = c[0]                                                                                                                                                                                                                                                                                                                                                                
  """                                                                                                                                                                                                                                                                                                                                                                       
  echo $c_0                                                                                                                                                                                                                                                                                                                                                                 
  """                                                                                                                                                                                                                                                                                                                                                                       
}

N E X T F L O W  ~  version 0.14.2
Launching integer.nf
[warm up] executor > local
Error executing process > 'handle_paired (1)'

Caused by:
  No signature of method: nextflow.util.BlankSeparatedList.getAt() is applicable for argument types: (java.lang.Integer) values: [0]
Possible solutions: getAt(java.lang.String), get(int), set(int, java.lang.Object), putAt(java.lang.String, java.lang.Object), wait(), grep()


Source block:
  c_0 = c[0]
  """
  echo $c_0
  """

Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'

 -- Check script 'integer.nf' at line: 12 or see '.nextflow.log' file for more details

feature request: Enable S3 work directory outside of cirrus

Currently pipelines running on cirrus (or is it gridgain?) have the option of using S3 for the nextflow workdir. This kind of works outside of cirrus by setting 'nextflow run -w s3://bucket', with nextflow indeed using the designated S3 object as the work directory, except that for some reason pipelines are unable to finish(?). Why this is the case was discussed briefly in the gitter chat but I forget the cause.

Anyway, this would be a really nice feature to have to complement using S3 for inputs/outputs directly within Nextflow. An example use-case would be running nextflow on an EC2 cluster (via StarCluster), and running a nextflow pipeline such that all inputs, outputs and contents of the workdir were stored in S3.

Handle suspended SLURM jobs

I haven't experienced this myself, so this might be invalid, but SLURM can suspend jobs on a cluster if a high-priority job comes in. This might become a real situation for us in the future, which is why I am concerned about this. The job then goes into the suspended state, indicates by the S flag, as indicated here https://computing.llnl.gov/linux/slurm/squeue.html.

I looked into the code, and there's no handling of the S flag.

I cannot test a fix myself, unfortunately, but to me it seems to be safe to treat the S flag same as running, meaning Nextflow should keep waiting for it to finish?

Process with input from non-existing channel doesn't give warning

I just had a situation where I removed some code, removing one of the input channels for a process.
What happened next is that my nextflow script would just sit there at the stage where it would normally launch said process, not giving any warning (that I can find) and not doing anything at all.

I have experienced similar issues before, so I know I had to check my channels, but for a person new to Nextflow such a thing can be quite confusing, or even a dealbreaker.

What I'm talking about is

process someProcess {
   input:
    file bam_file from bam_input
    file vcf_file from vcf_input
    file idx_file from idx_input  <-- channel doesn't exist anymore

    etc...

Can you confirm this behaviour? Is it possible to check consistency upon initialization and raise an exception in those cases?

Add leftJoin operator

While it should be possible to perform a left outer join operation using map and reduce operators, it might be useful to include a leftJoin operator in the DataflowExtensions groovy class.

Here are some simple tests

  def testLeftJoin() {
    when:
    def ch1 = Channel.from( 1,2,3 )
    def ch2 = Channel.from( 1,0,0,2,7,8,9,3 )

    // forward
    def result = ch1.leftJoin(ch2)

    then:
    result.val == [1,1]
    result.val == [2,2]
    result.val == [3,3]
    result.val == Channel.STOP
  }

  def testLeftJoin2() {
    when:
    def ch1 = Channel.from( 1,2,3 )
    def ch2 = Channel.from( 1,0,0,2,7,8,9,3 )

    // reverse
    def result = ch2.leftJoin(ch1)

    then:
    result.val == [1,1]
    result.val == [0,null]
    result.val == [0,null]
    result.val == [2,2]
    result.val == [7,null]
    result.val == [8,null]
    result.val == [9,null]
    result.val == [3,3]
    result.val == Channel.STOP

    // alternatively, it would be ok if order was not preserved
    /*
    result.val == [1,1]
    result.val == [2,2]
    result.val == [3,3]
    result.val == [0,null]
    result.val == [0,null]
    result.val == [7,null]
    result.val == [8,null]
    result.val == [9,null]
    result.val == Channel.STOP
    */
  }

The filter operator crash when using a regex pattern

For example:

Channel
     .path('misc/sample.fa')
     .splitFasta( record: [id: true, seqString: true ])
     .filter { record -> record.id =~ /.*_B$/ }
     .subscribe { record -> println record.seqString }

It returns

The dataflow processor experienced an unhandled exception and is about to terminate. java.lang.ClassCastException: java.util.regex.Matcher cannot be cast to java.lang.Boolean

Moreover the application hung

Nextflow installer does not work when running behind a proxy server

Nextflow installer download required dependencies on-demand by using Capsule project.

Unfortunately it does not support proxy server, thus when running behind a proxy server the installer in unable to download required dependencies.

only one process execute in two parallel channels

Dear Sir,
I have two parallel process, utr corelatted by the miRNA and lncRNA corelatted by the miRNA.
But only corelation of utr and miRNA executted.
== please see the zip attachment "example_nextflow.zip.png" ==

[llwang@fat01 example_nextflow]$ ./corelation.nf
== messages == N E X T F L O W  ~  version 0.12.0
== messages == [warm up] executor > local
== messages == [3a/f8e09d] Submitted process > val_mir (1)
== messages == [98/77d5d4] Submitted process > val_lnc (1)
== messages == [ce/1d8363] Submitted process > val_utr (1)
== messages == [55/5bc0ca] Submitted process > cor_utr (5)
== messages == [df/44885b] Submitted process > cor_utr (6)
== messages == [17/0ce5df] Submitted process > cor_utr (2)
== messages == [a3/2d4d29] Submitted process > cor_utr (1)
== messages == [ae/480a17] Submitted process > cor_utr (4)
...
== messages == [ab/976891] Submitted process > cor_utr (99)
== messages == [db/41a6be] Submitted process > cor_utr (95)
== messages == [b0/9ff97f] Submitted process > cor_utr (100)
== messages == Result saved to file: /llwang-works/installation/example_nextflow/utr.cor.result

pipeline overview

     ***************      *****************        ******************
     * utr channel *      * miRNA channel *        * lncRNA channel *
     ***************      *****************        ******************

        |                      |                         |
     validate               validate                  validate
        |                      |                         |
       \./                    \./                       \./
        .                      .                         .

     *****************    *****************        ******************
     * utr val chann *    * mir val chann *        * lnc val chann  *
     *****************    *****************        ******************

        |                    |           |                    |
       split                 |           |                  split
        |                    |           |                    |
       \./                  \./         \./                  \./
        .                    .           .                    .

     ****************************      ****************************
     * corelation channel       *      * corelation channel       *
     ****************************      ****************************
                 |                                 |    
              collect                           collect 
                 |                                 |    
                \./                               \./   
                 .                                 .        
              file_utr                          file_lnc

Map operator hangs the pipeline execution when applied to a dataflow variable

Process executing native code do not stage input files

In the following example, the print process can't access the x file.

process read_data {
  output:
  file "bla*" into read

  script:
  """
  echo 'Hello World' > bla.txt
  """
}


process print {
  input:
  file x from read

  exec:
  x.eachLine { str ->
    println str
  }
}

Add git commit hash number to version information

Modify the build script in such a way to include the git commit number in version information number

Scratch on Mac does not work

Using process.scratch=true locally on a Mac crashes the pipeline run

When running a "native" process using the GridGain executor an exception is raised

A NullPointerException is raised when a native process is executed using the gridgain executor

splitText confusing

there is 50 lines in my valid file .
why I got only one file in the validUtrChannelSplit Channel?
I am confused.

here is my script

params.mirfile = "/llwang-works/data/ceRNA/example_nextflow/data/mir.expr"
params.utrfile = "/llwang-works/data/ceRNA/example_nextflow/data/utr.expr"
params.chunkSize = 8

mirIo = file(params.mirfile)
utrIo = file(params.utrfile)

mirChannel = Channel.from(mirIo)
utrChannel = Channel.from(utrIo)

process validate_utr {
input:
file 'source' from utrChannel

    output:
    file 'valid' into validUtrChannel

    """
    perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
    """

}

process validate_mir {
input:
file 'source' from mirChannel

    output:
    file 'valid' into validMirChannel

    """
    perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
    """

}

validUtrChannelSplit = validUtrChannel.splitText(by: params.chunkSize)
process task_cat {
input:
file 'seq' from validUtrChannelSplit
file 'db' from validMirChannel
output:
file 'out' into corChannel

    """
    cat seq > out
    """

}

IncludeConfig does not support dot properties syntax

When a configuration file includes another config which declares config attributes using a dot property syntax an exception is raised.

For example having nextflow.config like this:

includeConfig 'foo.config'

and foo.config like the following:

params.bar = 1

The following error message is displayed:

ERROR ~ Not such variable: params

A process that define only an input of type 'each' never stops

For example:

process Hola {
  input:
  each x from 1,2

  output:
  set 'file_x', 'file_y' into results

  '''
  touch file_x
  touch file_y
  '''
}

Local Git support

I really like the integration of Github to version the pipeline files and hope nextflow to support local Git , not only Github. For example, we have local Gitlab for our internal code management. It could be better to support local Git repository. Thanks!

question for continuous checkpoints and pbs executor

Dear sir.
I encounter two question, as follow

_1_ . I don`t find the "Continuous checkpoints" function.
processes still Submitted after being sucessfully executed.

=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W  ~  version 0.12.0
=message= [warm up] executor > local
=message= [1b/f5ade3] Submitted process > validate_utr (1)
=message= [ed/0a1fa0] Submitted process > validate_mir (1)
=message= [3d/9b255f] Submitted process > cor_mir_utr (3)
=message= [fa/3a387a] Submitted process > cor_mir_utr (1)
=message= ...
=message= ...
=message= [88/8e6df7] Submitted process > cor_mir_utr (104)
=message= Result saved to file: /llwang-works/data/ceRNA/example_nextflow/result.txt
...
=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W  ~  version 0.12.0
=message= [warm up] executor > local
=message= [1b/f5ade3] Submitted process > validate_utr (1)
=message= [ed/0a1fa0] Submitted process > validate_mir (1)
=message= [3d/9b255f] Submitted process > cor_mir_utr (3)
=message= [fa/3a387a] Submitted process > cor_mir_utr (1)
=message= [e6/8b257c] Submitted process > cor_mir_utr (7)
=message= [95/dbd725] Submitted process > cor_mir_utr (5)

_2_ . using pbs executor , the processing stop at the last process...

=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W  ~  version 0.12.0
=message= [warm up] executor > pbs
=message= [ae/0543aa] Submitted process > validate_utr (1)
=message= [a7/4482ba] Submitted process > validate_mir (1)
=message= [51/abc5ae] Submitted process > cor_mir_utr (4)
=message= [b8/a1b37e] Submitted process > cor_mir_utr (3)
=message= [70/f6827c] Submitted process > cor_mir_utr (9)
=message= [16/af8b6e] Submitted process > cor_mir_utr (103)
=message= [b4/46e6e4] Submitted process > cor_mir_utr (104)

my script as follow

  1 // parameters
  2 params.mirfile     = "/llwang-works/data/ceRNA/example_nextflow/data/expr/mir.expr"
  3 params.utrfile     = "/llwang-works/data/ceRNA/example_nextflow/data/expr/utr.expr"
  4 params.chunkSize   = 200
  5
  6 params.out         = "/llwang-works/data/ceRNA/example_nextflow/result.txt"
  7
  8
  9 // Channels
 10 mirCh = Channel.fromPath(params.mirfile)
 11 utrCh = Channel.fromPath(params.utrfile)
 12
 13
 14 // validate utr sequence
 15 process validate_utr {
 16     input:
 17     file 'source' from utrCh
 18
 19     output:
 20     file 'valid' into utrValCh
 21
 22     """
 23     perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
 24     """
 25 }
 26
 27
 28 // validate microRNA sequence
 29 process validate_mir {
 30     input:
 31     file 'source' from mirCh
 32
 33     output:
 34     file 'valid' into mirValCh
 35
 36     """
 37     perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
 38     """
 39 }
 40
 41
 42 utrValSplitCh = utrValCh.splitText( by: params.chunkSize )
 43
 44
 45 process cor_mir_utr {
 46     input:
 47     file 'utr' from utrValSplitCh
 48     file 'mir'  from mirValCh.first()
 49
 50     output:
 51     file 'corel' into corCh
 52
 53     """
 54     Rscript /llwang-works/data/ceRNA/example_nextflow/scripts/cor.R utr mir 0.05 0.7 corel
 55     """
 56 }
 57
 58 corCh
 59   .collectFile(name: file(params.out))
 60   .subscribe {
 61      println "Result saved to file: $it"
 62    }
 63

NextFlow pipeline process hangs after silently dying in SGE

My cextflow.config is defined as follows:

process {
  executor='sge'
  queue='all.q'
}

Then I run the following:

nextflow run tutorial.nf
N E X T F L O W  ~  version 0.11.2
[warm up] executor > sge
[ad/b159ca] Submitted process > splitLetters (1)

At which point the job will appear in the SGE queue, hit an open slot, then disappear (not sure if it fails or not).. and the Submitted process > splitLetters (1) just hangs.

Broken test CacheHelperTest

Running gradle test with last version I get:

groovy.lang.MissingPropertyException at CacheHelperTest.groovy:38

ERROR:

groovy.lang.MissingMethodException: No signature of method: embed.com.google.common.hash.Murmur3_128HashFunction$Murmur3_128Hasher.putString() is applicable for argument types: (java.lang.String) values: [Hola]
Possible solutions: putString(java.lang.CharSequence, java.nio.charset.Charset), toString(), toString(), chopString(), putLong(long), chopString(java.util.Map)
at nextflow.util.CacheHelperTest.test hashCode(CacheHelperTest.groovy:38)

I can solve with

def aString = Hashing.murmur3_128().newHasher().putString('Hola', Charset.defaultCharset()).hash()

But I get more errors from the following tests

Best Regards

Add a method all() to params object

Add a method all() to params object that renders a string containing all command line parameters.

For example, having the following params map:

[a: 1, b: 2, c: 3]

It will return a string:

 --a 1 --b 2 --c 3

Add eventually some option to control the option prefix (- or --), the name-value separator (blank, or =) and to includes, excludes entries, quote values

String + in output declaration leads to "Not such variable."

This might not be an bug, but here's a simple nf script to demonstrate. Replacing val(fastq.baseName + '.out') with val('abcd' + '.out') works.

#!/usr/bin/env nextflow
seqs = file('seqs/a.fastq')

process proc1 {
  input:
  file fastq from seqs

  output:
  val(fastq.baseName + '.out') into proc1

  """
  echo 1
  """
}

N E X T F L O W  ~  version 0.13.4-SNAPSHOT
Launching strcat.nf
ERROR ~ Not such variable: fastq

 -- Check script 'strcat.nf' at line: 9 or see '.nextflow.log' file for more details

.nextflow.log:
https://gist.github.com/yesimon/fab422b9442ff83fd0c8

suggestion: run -dry

A dry run option would be great for validating a pipeline script without actually executing any processes.

nextflow run hello appears not to work

Calling nextflow run hello gets stuck with the message Pulling nextflow-io/hello ....

A common problem with my work environment is that I need to configure a proxy. Apparently, nextflow does not pick up the environment variables HTTP_PROXY and HTTPS_PROXY. How can I configure a proxy to use for the nextflow run hello command?

Implement clean up command

Implement a new command that allows the user to selectively clean the work/scratch directory.

Ideally the user should be able to select which runs have to discarded and keep the others. Currently to each pipeline execution is associated a UUID number. The history of all runs is stored in the file .nextflow.history (in the working directory). It can be viewed with the command nextflow history. It prints a report similar to the following (the oldest run is showed at the top, the latest at the bottom):

a77d4b73-8fdc-4111-bb19-31e30a7248d3    nextflow run piper 
aa4fd46b-fee7-40b4-bff6-e3acd5950254    nextflow run piper  --genome-file /path/to/file
49685297-1e52-41d5-9e0c-f00d09d1d17a    nextflow run grape --read1 '*_1.fa' --read2 '*_2.fa'
cbf5e933-f725-4a1f-a714-a0c8715bff74    nextflow run grape --annot /path/to/annotation.gff

It could be possibile to implement a new command clean command that delete all the files produced by a pipeline execution having a specified UUID number, or before that execution, or after that execution.

For example:

nextflow clean -before <UUID>
Delete all files produced before the pipeline run with the specified UUID

nextflow clean -after <UUID>
Delete all files produced after the pipeline run with the specified UUID

nextflow clean -run <UUID>
Delete all files produced by the pipeline run with the specified UUID

nextflow clean -but <UUID>
Delete all files expect the ones produced by the runs specified

nextflow clean
By default delete all files expect the ones produced by the last run.

Processes having identical commands and inputs are given the same hash key

When two or more processes have an identical command stript and get the identical inputs the and identical hash key is assigned to different process instances.

This result in multiple processes generating the command script in the same working directory and unpredictable content of script and wrapper files.

Using a variable declared in the global scope as parametric output name result in a exception

x = 1 

process foo {
  input:
  val y from 'a'

  output: 
  file "${x}_${y}" into q

  """
  echo Hello
  """
}

The following exception is thrown

Nov-11 15:40:53.701 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'foo (1)'

Caused by:
  x

groovy.lang.MissingPropertyException: x
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_51]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.script.BaseParam.resolveName(ParamsIn.groovy:205) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:368) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.script.FileOutParam$_getFilePatterns_closure1.doCall(ParamsOut.groovy:238) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.script.TokenGString$_resolve_closure1.doCall(ScriptTokens.groovy:130) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2348) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2318) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.dgm$63.invoke(Unknown Source) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.script.TokenGString.resolve(ScriptTokens.groovy:130) ~[nextflow-0.11.0.jar:na]
    at nextflow.script.TokenGString$resolve.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.script.FileOutParam.getFilePatterns(ParamsOut.groovy:238) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:166) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:68) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskRun$_memoizedMethodPriv$getOutputFilesNames_closure8.doCall(TaskRun.groovy:296) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1372) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.dgm$149.invoke(Unknown Source) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskRun.memoizedMethodPriv$getOutputFilesNames(TaskRun.groovy:295) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:368) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskRun$_closure1.doCall(TaskRun.groovy) ~[nextflow-0.11.0.jar:na]
    at nextflow.processor.TaskRun$_closure1.doCall(TaskRun.groovy) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:127) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:417) ~[groovy-2.3.7.jar:2.3.7]
    at java_util_concurrent_Callable$call.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskRun.getOutputFilesNames(TaskRun.groovy) ~[nextflow-0.11.0.jar:na]
    at nextflow.processor.TaskRun$getOutputFilesNames$4.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.executor.Executor.unstageOutputFilesScript(Executor.groovy:219) ~[nextflow-0.11.0.jar:na]
    at nextflow.executor.Executor.unstageOutputFilesScript(Executor.groovy) ~[nextflow-0.11.0.jar:na]
    at nextflow.executor.LocalExecutor.createTaskHandler(LocalExecutor.groovy:76) ~[nextflow-0.11.0.jar:na]
    at nextflow.executor.LocalExecutor$createTaskHandler.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskDispatcher.submit(TaskDispatcher.groovy:145) ~[nextflow-0.11.0.jar:na]
    at nextflow.processor.TaskDispatcher$submit$0.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskProcessor.submitTask(TaskProcessor.groovy:1313) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:510) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.7.jar:2.3.7]
    at nextflow.processor.ParallelTaskProcessor.invokeTask(ParallelTaskProcessor.groovy:364) ~[nextflow-0.11.0.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1074) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1253) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1209) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:807) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Script.invokeMethod(Script.java:78) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:414) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:353) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) ~[groovy-2.3.7.jar:2.3.7]
    at Script1$_run_closure1.doCall(Script1.groovy:1) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120) ~[gpars-1.2.1.jar:1.2.1]
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35) ~[gpars-1.2.1.jar:1.2.1]
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58) ~[gpars-1.2.1.jar:1.2.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]

Allow easier collection of stdout/stderr in log file

When putting echo true into my pipeline script, I get nice logging output that I can then redirect to a file. This would work even better for me if I could specify a file to direct/copy the stdout/stderr output to when using echo true.

This way, I would just see the (much smaller) list of the executed commands when executing nextflow run and still have the full log present if I (maybe days later) see that something went wrong with the pipeline run.

Installation, unable to run Java executable

I'd like to give this a try. I am trying to install on a cluster on which I do no have sudo. I use my own java, and the path to the executable is:

/home/stephen/bin/jre1.7.0/bin/java

Somewhere in lines 174-179 of the installer script it's trying to figure out where java lives, and it gets hung up at:

Nextflow cannot start -- unable to run Java executable: /home/stephen/bin/jre1.7.0/bin/java/bin/java

Apparently it's trying to add another /bin/java to the path. How can I fix this? Thanks.

Show a more detailed error message when hitting GitHub rate limits

GitHub public repositories have a very low rate limits. See https://developer.github.com/v3/#rate-limiting

When trying to download a repository for which the user reached the access rate limit, Nextflow reports that the repository does not exists.

Cannot find cbcrg/piper-nf pipeline -- Make sure exists a GitHub repository at https://github.com/cbcrg/piper-nf

This is confusing the user. It should provide a more detailed error message suggesting to enter its credentials.

HTTP/1.1 403 Forbidden
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Content-Encoding: gzip
Content-Security-Policy: default-src 'none'
Content-Type: application/json; charset=utf-8
Date: Tue, 09 Sep 2014 12:41:08 GMT
Server: GitHub.com
Status: 403 Forbidden
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-GitHub-Media-Type: github.v3
X-GitHub-Request-Id: 545842C2:592E:19837FF:540EF564
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1410267414
X-XSS-Protection: 1; mode=block

{
    "documentation_url": "https://developer.github.com/v3/#rate-limiting", 
    "message": "API rate limit exceeded for 84.88.66.194. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"
}

Error resuming process with output value and command defined on multiple lines

I have a test.nf file like the following:

process resume {

    output:
    val str into out

    script:

    str = 'test'

    command = "echo ${str}"
    command += " > test.out"

    return command
}

When I run it for the first time everything goes well:

$ nextflow run test.nf
N E X T F L O W  ~  version 0.9.0
[warm up] executor > local
[2b/aac531] Submitted process > resume (1)

When I run the pipeline with the -resume options I get an error:

$ nextflow run test.nf
N E X T F L O W  ~  version 0.9.0
[warm up] executor > local
ERROR: Error executing process > 'resume (1)'

Caused by:
  Class cannot be created (missing no-arg constructor): groovy.lang.GString$3

Tip: check the log file '.nextflow.log' for more details

Setting return command.toString() did not help. Here is the stacktrace:

Aug-27 15:01:03.702 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'resume (1)'

Caused by:
  Class cannot be created (missing no-arg constructor): groovy.lang.GString$3

Tip: check the log file '.nextflow.log' for more details
Aug-27 15:01:03.706 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Process resume raise the following exception:
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): groovy.lang.GString$3
    at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) ~[kryo-2.24.0.jar:na]
    at com.esotericsoftware.kryo.Kryo$readClassAndObject.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.6.jar:2.3.6]
    at nextflow.util.KryoHelper.deserialize(SerializationHelper.groovy:114) ~[main.jar:na]
    at nextflow.util.KryoHelper$deserialize.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.6.jar:2.3.6]
    at nextflow.processor.DelegateMap.read(DelegateMap.groovy:189) ~[main.jar:na]
    at nextflow.processor.DelegateMap$read$0.call(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) ~[groovy-2.3.6.jar:2.3.6]
    at nextflow.processor.TaskProcessor.checkCachedOutput(TaskProcessor.groovy:602) ~[main.jar:na]
    at nextflow.processor.TaskProcessor$checkCachedOutput$2.callCurrent(Unknown Source) ~[na:na]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.6.jar:2.3.6]
    at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:482) ~[main.jar:na]
    at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.6.jar:2.3.6]
    at nextflow.processor.ParallelTaskProcessor.invokeTask(ParallelTaskProcessor.groovy:338) ~[main.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1074) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1253) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1209) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:807) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.Script.invokeMethod(Script.java:78) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:414) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:353) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) ~[groovy-2.3.6.jar:2.3.6]
    at Script1$_run_closure1.doCall(Script1.groovy:1) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.6.jar:2.3.6]
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
    at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.6.jar:2.3.6]
    at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120) ~[gpars-1.2.1.jar:1.2.1]
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35) ~[gpars-1.2.1.jar:1.2.1]
    at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58) ~[gpars-1.2.1.jar:1.2.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
    at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Aug-27 15:01:03.707 [Actor Thread 3] DEBUG nextflow.Session - Session abort -- terminating all processors
Aug-27 15:01:03.708 [Actor Thread 1] DEBUG n.processor.ParallelTaskProcessor - <resume> After stop -- shareObjs [:]
Aug-27 15:01:03.709 [main] DEBUG nextflow.Session - <<< phaser deregister (session)

Any of the following actions solves the issue:

removing the output declaration block
putting the command on a single string like:

command = "echo ${str} > test.out"

defining the command as:

... 
command = "echo ${str}".toString()
command += " > test.out".toString()

return command
...

not using +=:

... 
command = "echo ${str}"
command = "${command} > test.out"

return command
...

NoClassDefFoundError: java/util/HashMap$Entry

./nextflow http://goo.gl/ngf8dL
N E X T F L O W  ~  version 0.8.1
ERROR: java.lang.NoClassDefFoundError: java/util/HashMap$Entry -- See the file '.nextflow.log' for more error details

Support login to a docker registry

It would be great to be able to log into a docker registry ("login" CLI command). This permits accessing private registries that authorization, as well as private repos on Docker Hub.

Channel type:dir not working

Looks like I won't have to wait long for the dir globbing -- I'm having problems getting it working though :

Here is my Channel declaration 👍

samps = Channel.path('/data/bifx/Sequence_data/TXOME/Project*/*', type:'dir')

and here's what happens when I try to run it:

[root@szdedicated /data/bifx/tlaurent/trinity_wd]# ./nextflow run_trinity.nf
N E X T F L O W  ~  version 0.7.3
ERROR: groovy.lang.MissingMethodException: No signature of method: static nextflow.Channel.path() is applicable for argument types: (java.util.LinkedHashMap, java.lang.String) values: [[type:dir], /data/bifx/Sequence_data/TXOME/Project*/*]
Possible solutions: path(java.lang.String), path(java.lang.Object), path(java.util.regex.Pattern), watch(java.lang.String), watch(java.lang.String, java.lang.String), watch(java.util.regex.Pattern, java.lang.String) -- See the file '.nextflow.log' for more error details

BTW I did pull the new package again:

Thanks,

The method toAbsolutePath resolve to an incoherent path when used in a process

For example

ch = file(params.in)

process foo {
  input:
  file x from ch 

  "echo ${x.toAbsolutePath()}" 

}

It is expected that it resolves the path against the process working path instead it is resolved agains the main thread execution path.

Race condition running multiple processes on file input.

I'm running an nf file with multiple processes that use storeDir and file inputs. Exception is non-deterministic and only occurs with over 5+ proc declarations.

race.nf:

#!/usr/bin/env nextflow
seqs = file('seqs/*.fastq')

process proc1 {
  storeDir 'results'

  input:
  file fastq from seqs

  output:
  set val(baseName), file(outName) into proc1

  script:
  baseName = fastq.baseName
  outName = baseName + '.proc1'
  """
  echo "blarg" > $outName
  """
}

process proc2 {
  storeDir 'results'

  input:
  file fastq from seqs

  output:
  set val(baseName), file(outName) into proc2

  script:
  baseName = fastq.baseName
  outName = baseName + '.proc2'
  """
  echo "blarg" > $outName
  """
}

... repeat as necessary

output:

N E X T F L O W  ~  version 0.13.3
Launching race.nf
[warm up] executor > local
[skipping] Stored process > proc2 (3)
[skipping] Stored process > proc3 (1)
[skipping] Stored process > proc3 (3)
[skipping] Stored process > proc2 (2)
[skipping] Stored process > proc2 (1)
[skipping] Stored process > proc1 (2)
[skipping] Stored process > proc5 (1)
[skipping] Stored process > proc5 (2)
[skipping] Stored process > proc5 (3)
[skipping] Stored process > proc1 (1)
[skipping] Stored process > proc1 (3)
[skipping] Stored process > proc3 (2)
[skipping] Stored process > proc4 (1)
[skipping] Stored process > proc4 (2)
[skipping] Stored process > proc4 (3)
ERROR ~ Error executing process > 'proc5 (1)'

Caused by:
  Cannot invoke method bindOutput() on null object


 -- Check '.nextflow.log' file for details
Uncaught exception occurred in gpars pool Actor Thread 14
java.util.concurrent.RejectedExecutionException: Task groovyx.gpars.actor.AbstractLoopingActor$1@2766ae7b rejected from java.util.concurrent.ThreadPoolExecutor@ad4cdb3[Shutting down, pool size =
 1, active threads = 1, queued tasks = 0, completed tasks = 49]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
        at groovyx.gpars.scheduler.DefaultPool.execute(DefaultPool.java:155)
        at groovyx.gpars.util.AsyncMessagingCore.store(AsyncMessagingCore.java:99)
        at groovyx.gpars.actor.AbstractLoopingActor.send(AbstractLoopingActor.java:241)
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.send(DataflowProcessorActor.java:85)
        at groovyx.gpars.actor.AbstractLoopingActor.send(AbstractLoopingActor.java:241)                                                                                                   [0/1821]
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.send(DataflowProcessorActor.java:85)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:215)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.terminate(DataflowProcessor.java:147)
        at nextflow.Session.abort(Session.groovy:375)
        at nextflow.Session$abort$7.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at nextflow.processor.TaskProcessor.resumeOrDie(TaskProcessor.groovy:756)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
        at nextflow.processor.TaskProcessor.handleException(TaskProcessor.groovy:694)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1206)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:80)
        at nextflow.processor.ParallelTaskProcessor.this$dist$invoke$2(ParallelTaskProcessor.groovy)
        at nextflow.processor.ParallelTaskProcessor$TaskProcessorInterceptor.methodMissing(ParallelTaskProcessor.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaClassImpl.invokeMissingMethod(MetaClassImpl.java:932)
        at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1255)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1208)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
        at nextflow.processor.ParallelTaskProcessor$TaskProcessorInterceptor.onException(ParallelTaskProcessor.groovy:296)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.fireOnException(DataflowProcessor.java:331)
        at groovyx.gpars.dataflow.operator.DataflowProcessor.reportError(DataflowProcessor.java:293)
        at groovyx.gpars.dataflow.operator.DataflowProcessorActor.reportException(DataflowProcessorActor.java:125)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:122)
        at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
        at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:744)

Channel question on result file in directory result of programe

I don`t know how do I put file into channel, the file lie in a directory.
while directory is a parameter rather than a file.

such as programe "tophat ".
the command line like following:
$ tophat -o tophat_out genome_index R1.fq R2.fq

the result like this

tophat_out/:
accepted_hits.bam deletions.bed junctions.bed prep_reads.info

align_summary.txt insertions.bed logs unmapped.bam

then I want pass the file "accepted_hits.bam" into a Channel, How should I do?

hearing from you!

Error: cannot find Java

I upgraded 0.12.2 to 0.13.1 and now see the error message "Error: cannot find Java or it's a wrong version". 0.12.2 worked and still works fine. 0.13.1 gives this error message.

~ ❯❯❯ nextflow -version
Error: cannot find Java or it's a wrong version -- please make sure that Java 7 or higher it's installed
Note: Nextflow is trying to use the Java VM defined by the following environment variables:
 JAVA_CMD: /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
 JAVA_HOME: 

~ ❯❯❯ java -version                                                           ⏎
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
~ ❯❯❯ brew switch nextflow 0.12.2
Cleaning /usr/local/Cellar/nextflow/0.12.2
Cleaning /usr/local/Cellar/nextflow/0.13.1
1 links created for /usr/local/Cellar/nextflow/0.12.2
~ ❯❯❯ nextflow -version

      N E X T F L O W
      Version 0.12.2 build 2674
      last modified 12-02-2015 11:17 UTC (12:17 CEST)
      http://nextflow.io

Add Support for Loading Files Dereferenced from Environmental Variable

That way this will work:

 file('env://MY_INPUT_FILE')

and will load whatever the value of MY_INPUT_FILE is set to in the environment.

MoveTo method should not throw a FileAlreadyExistsException when target file already exist

The documentation for moveTo method states that:

When a file with the same name as the target already exists, it will be replaced by the new one.

However currently a FileAlreadyExistsException is thrown when the a file already exist in the target directory.

Processes re-run unnecessarily

Processes are executed even when cached results of the same process with the same inputs are available.

Consider this minimal example (or git clone https://gist.github.com/b189ef40129bf751c8f8.git):

#!/usr/bin/env nextflow

params.query = './*.fasta'

fastaFiles = Channel.fromPath(params.query)

process getDefinitions {

        input:
        file 'fasta' from fastaFiles

        output:
        stdout defLines

        """
        sleep 10
        awk '/^>/ {print(substr(\$0,2))}' $fasta
        """
}

defLines.subscribe {
        println it
}

The nextflow documentation indicates that

(Any) attempt to execute the process along with the same inputs will cause the process execution to be skipped

I was expecting that if the program was re-run using the same inputs, the getDefinitions process would be skipped and the cached results used in place. This doesn't seem to happen:

$ rm -r work
$ time nextflow main.nf --query example.fasta
N E X T F L O W  ~  version 0.10.2
[warm up] executor > local
[f8/05f1ee] Submitted process > getDefinitions (1)
SEQUENCE_1 [Homo sapiens sapiens]
SEQUENCE_2 [Homo sapiens neanderthalensis]
SEQUENCE_3 [Mus musculus musculus]

real    0m12.070s
user    0m3.034s
sys 0m0.341s

$ time nextflow main.nf --query example.fasta 
N E X T F L O W  ~  version 0.10.2
[warm up] executor > local
[ad/28e06b] Submitted process > getDefinitions (1)
SEQUENCE_1 [Homo sapiens sapiens]
SEQUENCE_2 [Homo sapiens neanderthalensis]
SEQUENCE_3 [Mus musculus musculus]


real    0m12.041s
user    0m2.908s
sys 0m0.409s

I note that the hashLog values change for each indication. I had a look at the source, and it looks like when CacheHelper.hasher is given a file, it returns a Hasher that constructs a hash from:

the file path
the file size, and lostModified (if file.exists())

This is probably my inexperience with java showing, but I don't understand why the same file doesn't produce the same hash.

It is difficult to construct significant workflows when the whole thing needs to be re-run every time. Is this a bug, or just my mis-use?

Issues with wildcards in output section

I tried to resolve my problem described in #18 with the pattern proposed there. I'm opening another ticket here since I think this is a separate issue. The section below shows my script, the call and error message and the work directory.

Am I missing something or doing something wrong?

holtgrem@t-jbot01:~/Software/ngspipeline$ cat pipeline.nextflow 
#!/usr/bin/env nextflow

import ChannelUtil
import FastQC
import PathUtil
import PipelinePaths

// Uncomment for seeing the output of th eprocesses.
echo true

params.runID = 1234
params.baseDir = "$HOME/Projects/nextflow-test"

// Create helper for generating pipeline file paths. 
paths = new PipelinePaths(params.baseDir);

// Open channel for left and right files and merge it into triples, the
// first entry is the LCS of the file names that can be used as a read
// pair identifier.
readPairs = ChannelUtil.createFilePairChannel(
        Channel.fromPath(paths.fastqPattern("original", "R1")),
        Channel.fromPath(paths.fastqPattern("original", "R2"))
        )

// Genome file.
// TODO(holtgrew): Is this really required or could we as well use the path?
genomeFile = file(params.genome)

// Duplicate the read pairs into one queue for runFastQCOriginal
// and runTrimming.
readPairsFastQCOriginal = Channel.create()
readPairsRunTrimming = Channel.create()
readPairs.separate(readPairsFastQCOriginal, readPairsRunTrimming) { x -> [ x, x ] }

// --------------------------------------------------------------------------
// Run FastQC on the original reads.
// --------------------------------------------------------------------------

process runFastQCOriginal {
    module 'fastqc/0.11.2'

    input:
    set pairID, file(readL), file(readR) from readPairsFastQCOriginal

    output:
    file 'original/*.zip' into fastqcOutputOriginal

    script:
    """
    set -x
    mkdir -p original
    fastqc -o original ${readL} ${readR}
    """
}

holtgrem@t-jbot01:~/Software/ngspipeline$ nextflow run pipeline.nextflow 
N E X T F L O W  ~  version 0.11.3
[warm up] executor > local
[dc/2e80a7] Submitted process > runFastQCOriginal (1)
[83/572a74] Submitted process > runFastQCOriginal (2)
Error executing process > 'runFastQCOriginal (1)'

Caused by:
  Missing output file(s): 'original/*.zip' expected by process: runFastQCOriginal (1)

Command executed:

  set -x
  mkdir -p original
  fastqc -o original reads001_R1.fastq.gz reads002_R2.fastq.gz

Command exit status:
  0

Command output:
  + mkdir -p original
  + fastqc -o original reads001_R1.fastq.gz reads002_R2.fastq.gz
  Started analysis of reads001_R1.fastq.gz
  Analysis complete for reads001_R1.fastq.gz
  Started analysis of reads002_R2.fastq.gz
  Analysis complete for reads002_R2.fastq.gz

Work dir:
  /media/3dfdd68f-a74e-4fb8-8e7f-203343c39564/data6/home/holtgrem/Software/ngspipeline/work/dc/2e80a74320773446b671eec63f0083

Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'

holtgrem@t-jbot01:~/Software/ngspipeline$ tree /media/3dfdd68f-a74e-4fb8-8e7f-203343c39564/data6/home/holtgrem/Software/ngspipeline/work/dc/2e80a74320773446b671eec63f0083
/media/3dfdd68f-a74e-4fb8-8e7f-203343c39564/data6/home/holtgrem/Software/ngspipeline/work/dc/2e80a74320773446b671eec63f0083
|-- original
|   |-- reads001_R1_fastqc.html
|   |-- reads001_R1_fastqc.zip
|   |-- reads002_R2_fastqc.html
|   `-- reads002_R2_fastqc.zip
|-- reads001_R1.fastq.gz -> /home/holtgrem/Projects/nextflow-test/fastq/original/reads001_R1.fastq.gz
`-- reads002_R2.fastq.gz -> /home/holtgrem/Projects/nextflow-test/fastq/original/reads002_R2.fastq.gz

1 directory, 6 files

Per process containers not working with -with-docker option

Given the example pipeline and configuration below:

// test.nf
process test {
    """
    echo Test 
    """
}

// nextflow.config
process {
    echo = true

    $test.container = 'debian'
}

The following error message is displayed:

$ nextflow run test.nf -with-docker
N E X T F L O W  ~  version 0.12.5
You request to run with Docker but no image has been specified

nextflow-io / nextflow Goto Github PK

nextflow's Introduction

Quick start

Documentation

Community

Contributing

License

Citations

Credits

nextflow's People

Contributors

Stargazers

Watchers

Forkers

nextflow's Issues

pipeline overview

here is my script

the result like this

align_summary.txt insertions.bed logs unmapped.bam

Recommend Projects

Recommend Topics

Recommend Org