Thank you very much Felipe!

I'll try it next monday and tell you what happens.

Have a nice weekend!

JL

from mpgap.

Hello @jllavin77, thank you for trying this tool!
Actually, that error seems very strange (I have never seem it), but we'll make the best so that it is fixed!

As a first measure, could send me these:

1. The .nextflow.log file that has been generated
2. The version of the Docker image you're using (when was last pulled?)

Let's get it fixed 🙂

Sincerely,

Felipe.

from mpgap.

Hello @fmalmeida,

1, Find the file attached -> nextflow.log
2. I downloaded the docker this morning
(fmalmeida/mpgap latest 9ec556b0da9f 2 days ago 12.7GB)

Hope this helps

Best

JL

from mpgap.

hi @jllavin77, it actually helps! Taking a look at it, it actually seems more like problem with environment variables and nextflow than a problem with the pipeline.

As you can see, in the script of logging messages, I always try to display for users the variables: $HOME; $USER and $PWD.

If this is really the problem, is actually simple to solve. I've seen that you're running the pipeline in a Linux machine, which should have this variables per default.

Could you check if they are available for you?

• echo $USER
• echo $HOME
• echo $PWD

Let's see if this is the source of the problems, because in the log file, nextflow have not even entered the pipeline because Groovy could not find this env var.

😬

Ps. When I tried to reproduce with a small dataset of mine (nextflow run fmalmeida/mpgap --longreads dataset_1/ont/ont_15x.fq --lr_type nanopore --assembly_type longreads-only --outdir TESTE_ISSUE --threads 5 --try_flye --try_unicycler --genomeSize 3m), it actually works, thus I really believe on env var problem.

from mpgap.

Hello again fmalmeida,

I'm running the pipeline inside the own docker container, I mean,
I run the dockerfile (fmalmeida/mpgap latest 9ec556b0da9f)
docker run -it -v F:\:/storage 9ec556b0da9f /bin/bash
and then installed Nextflow and pulled the pipeline there.
The USER variable was empty in that environment, so I used export USER="JL" to set it.
Afterwards the pipeline run but other problems appeared...

Please find the log attached.

Best

JL
nextflow.log

from mpgap.

Hello @jllavin77, that solves the problem!

Actually, the pipeline is not to be run from inside the container. You must make the nextflow software and docker images available on your local (host) machine and execute the pipeline from there. From your machine, not from the container.

Then, when doing that, nextflow automatically loads the container for each step. As you can see (in the attached log) it tryed, but then failed because the docker command was not found inside the container where you executed the pipeline. Docker is available in our local machine, not on the container.

Is it clear? I don't know if I have made the explanation confuse. But essentially, the nextflow command must be executed on the local (host) machine, not inside the container.

What OS is your computer?

Best

Felipe

from mpgap.

Hello again @fmalmeida,

I was expecting that answer. I'm running it in a windows Workstation via docker.
So the solution will be to create a new docker container, install everything there and then run the pipeline inside the newly created container, right?

from mpgap.

Right! I'll set this information for Windows users in the Readme as well. Nextflow needs UNIX, see https://www.nextflow.io/docs/latest/getstarted.html

Two possible solutions are available:

1. Use the windows linux subsystem (Recommended. It is the easiest.). Source: https://docs.microsoft.com/pt-br/windows/wsl/install-win10
2. Run and install everything in a ubuntu container, making /var/run/docker.sock available inside docker.

Trying the hard one (all in docker container)

I'll give you some instructions that might help! As a workaround for Windows systems (since nextflow needs UNIX) is:

# First step -- get the latest available nextflow docker
docker pull nextflow/nextflow

# Then enter as this
docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v /YOURDATA:/YOURDATA nextflow/nextflow /bin/bash

# Already in docker container entered as shown above
# Remember to make $USER available!!
export USER=MY_USER
docker pull fmalmeida/mpgap 
nextflow run fmalmeida/mpgap [...]

Please take note that the -v /var/run/docker.sock:/var/run/docker.sock parameter is required so that nextflow properly accesses your docker installed in the local system.

Also, remember to set the locations of your data identically to the image, otherwise nextflow will fail, because it wont find the path locally. Saying something as mount denied! It may be trickier to make this on windows.

With that I believe it will now run smoothly. If something goes wrong please tell me, because it might be something with properly locating the docker.sock file in your system. But the /var/run/docker.sock location is docker's default!

Best.

Felipe

from mpgap.

Dear Felipe,

I have been trying to run the pipeline following your latest instructions but it doesn't seem to work either.

These are the errors I get (I cannot find the nexflow.log file in this new container, sorry).

Please tell me if you some kind of fix comes to your mind or If shloud give up trying to make this work

bash-4.4# nextflow run fmalmeida/mpgap --longreads '/storage/ONT_results_FLO-MIN106/pass/TEMP_2746.fastq' --lr_type 'nanopore' --assembly_type 'lon
greads-only' --try_canu --try_flye --try_unicycler --genomeSize '5m' --outdir '/storage/ONT_results_FLO-MIN106/assemblies' --threads 8
N E X T F L O W  ~  version 21.02.0-edge
Launching `fmalmeida/mpgap` [magical_majorana] - revision: 9860b84048 [master]

=================================================================
Docker-based, fmalmeida/mpgap, generic genome assembly pipeline

Output directory: /storage/ONT_results_FLO-MIN106/assemblies
Assembly method: longreads-only
Number of threads: 8
Longreads      : /storage/ONT_results_FLO-MIN106/pass/TEMP_2746.fastq
Longread technology: nanopore
Pipeline Release: master
Current home   : /root
Current user   : jllavin
Current path   : /
Command used   : nextflow run fmalmeida/mpgap --longreads /storage/ONT_results_FLO-MIN106/pass/TEMP_2746.fastq --lr_type nanopore --assembly_type longreads-only --try_canu --try_flye --try_unicycler --genomeSize 5m --outdir /storage/ONT_results_FLO-MIN106/assemblies --threads 8

=========================================
executor > local (2)
[76/1f72f9] process > lreadsonly_nf:canu_assembly (Performing a longreads only assembly with Canu) [ 0%] 0 of 1
[- ] process > lreadsonly_nf:quast_lreads_canu -
[06/62aec3] process > lreadsonly_nf:flye_assembly (Performing a longreads only assembly with Flye) [ 0%] 0 of 1
[- ] process > lreadsonly_nf:quast_lreads_flye -
[- ] process > lreadsonly_nf:unicycler_lreads [ 0%] 0 of 1
[- ] process > lreadsonly_nf:quast_lreads_unicycler -
Error executing process > 'lreadsonly_nf:flye_assembly (Performing a longreads only assembly with Flye)'
executor > local (2)
[- ] process > lreadsonly_nf:canu_assembly (Performing a longreads only assembly with Canu) -
[- ] process > lreadsonly_nf:quast_lreads_canu -
[06/62aec3] process > lreadsonly_nf:flye_assembly (Performing a longreads only assembly with Flye) [100%] 1 of 1, failed: 1 ✘
[- ] process > lreadsonly_nf:quast_lreads_flye -
[- ] process > lreadsonly_nf:unicycler_lreads [ 0%] 0 of 1
[- ] process > lreadsonly_nf:quast_lreads_unicycler -
Error executing process > 'lreadsonly_nf:flye_assembly (Performing a longreads only assembly with Flye)'

Caused by:
Process lreadsonly_nf:flye_assembly (Performing a longreads only assembly with Flye) terminated with an error exit status (1)

Command executed:

source activate flye ;
flye --nano-raw TEMP_2746.fastq --genome-size 5m --out-dir flye_TEMP_2746 --threads 8 &> flye.log ;
mv flye_TEMP_2746/assembly.fasta flye_TEMP_2746/assembly_flye.fasta

Command exit status:
1

Command output:
(empty)

Command error:
/bin/bash: .command.sh: No such file or directory

Work dir:
/work/06/62aec3a827f7c6f041b78b4e346ea5

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh
`

from mpgap.

Hi @jllavin77,

Probably the pipeline really won't work by running it from the container. Probably because nextflow has been written to automatically mount and work with Docker and not from it.

I cannot think of another alternative than installing the Linux subsystem for Windows and running it from there.

But keep in mind that any of these options given before are just suppositions, since nextflow has been designed to be run from UNIX systems, see: https://www.nextflow.io/docs/latest/getstarted.html

Guide for the Linux subsystem: https://docs.microsoft.com/pt-br/windows/wsl/install-win10

I am sorry but I cannot help more than that, because that is intrinsic to the software and not actually something with the pipeline.

Best.

Felipe.

from mpgap.

Hi @jllavin77,

Since no activity has been detected in the last three days I'll consider the issue closed. I hope you managed to execute the pipeline using the Linux subsystem for Windows. It is actually the easiest and simplest way of running linux software in Windows.

I have tested the pipeline with this subsystem and it worked. The Windows files are usually mounted in the /mnt folder inside the subsystem.

Best.

Felipe.

from mpgap.