Comments (11)
Now that the pipeline has been reconfigured to understand a YAML so it can analyse multiple samples at once. I will start to take a look on the possibility to set the modules to use bioconda channels and bioconda docker images so it has an easy integration for using conda, docker or singularity.
This will be tackled in branch https://github.com/fmalmeida/MpGAP/tree/issue-17-conda
from mpgap.
from mpgap.
I have re-checked the tools in the bioconda channel and it now seems that all the tools are available there. Thus, in the meantime, I will start re-shaping the pipeline so it can either run with Docker and conda as suggested by @AntonioBaeza.
Re: nf-core
Thank you again @mahesh-panchal for the information. I will bring the suggestion and discussion to our lab. group so we can then start discussions with nf-core community.
from mpgap.
In theory, both capabilities should be possible because it just depends on adjusting the nextflow config file to understand the profiles sge and singularity.
But definitely it would be easier if added to nf-core. The only problem is setting the processes to use conda and transforming the pipeline into the nf-core format.
I'll definitely need help on this.
I forgot to do it before, I will ask in the nf-core slack channel whether they're interested in adding it.
from mpgap.
I will just finish the implementation of the multi-sample workflow that is described in #15 before that.
This is very near to it's release and when done.
from mpgap.
Hello guys,
Just to update that I tried to address this issue in PR #25.
This is the first time that I try to implement my pipelines with either conda and singularity at once, moreover, some tools used in it are either only available for linux-64, or its conda packages are neither available nor up-to-date, causing incompatibility with other tools.
Thus, to solve the tools that had this problem, I have made pre-compiled linux-64 binaries of them available inside the main/root pipeline bin directory. Therefore, the -profile conda
will only work in linux-64 systems. Users in other operating systems must use docker or singularity.
Please, since it is the first time I did it, the conda profile
must be considered rather experimental and please, do not hesitate to contact me if a problem arise.
Read more at: https://github.com/fmalmeida/MpGAP#selecting-between-profiles
from mpgap.
Or better yet, would you consider adding/merging this to nf-core (https://nf-co.re/)?
All the modules would then support conda, docker, and singularity.
There's an old workflow that needs some modernization called hybrid-denovo where the assembly/annotation discussions are going on.
There is going to be an nf-core hackathon in October (https://nf-co.re/events/2021/hackathon-october-2021), where one could get help transforming your modules ready for an nf-core DSL2 workflow.
It would be really nice to see this integrated into nf-core.
from mpgap.
Hi @AntonioBaeza and @mahesh-panchal,
First of all, thanks for your interest in the pipeline.
about conda
Yes, updating the pipeline to also use conda environments instead of only Docker is definitely in my TODO/whish list.
However, I was not able to fully work on it yet because some of the tools are not yet available in conda. I am not very much of an expert in conda package development. I am still studying on how to help on that and create a bioconda package for the tools that yet don't have it. And a help on that would be deeply appreciated if any of you have some knowledge on it, or know someone that would like to collaborate.
about nf-core
About the nf-core consideration, I would gladly merge the pipeline into nf-core. However I have two considerations on it:
- nf-core says that new pipelines must not overlap already existent nf-core pipelines. Thus, it may be a problem for bringing the pipeline to nf-core since they already have some of their own.
- This pipeline "hybrid-denovo" seems to be focused on hybrid assemblies only, thus I don't know if it'd be of interest for them because MpGAP performs not only hybrid assemblies, but short reads only and long reads only too. Thus, we must properly discuss if it would be a possibility.
I will try to participate on the hackathon that you mentioned to see if the community together can help on solving this problems and work together on the best solution for that: either merging it in nf-core or developing new bioconda packages so that the pipeline can also run with conda-only.
In the mean time, do you know where I could starting discussing it with the nf-core community and developers?
Thanks again for your interests and ideas. Any help is immensely welcomed.
from mpgap.
Re: nf-core
Please join us on slack and discuss this in the #new-pipelines channel (https://nf-co.re/join/slack).
The primary issues I think are that all tools need to be in bioconda.
Although I mention 'hybrid-denovo', it is an old pipeline and I think one that needs to be replaced. I'm sure the community would benefit a workflow that could support as many types of input as possible, launching, short-read only, long-read only, or hybrid assemblies.
from mpgap.
Adding to nf-core would be really great! I was about to ask for the possibility to use the pipeline with sge and singularity. Both features come out of the box with nf-core.
from mpgap.
Made some tests with a simpler pipeline (https://github.com/fmalmeida/ngs-preprocess) and was able to make it run with docker, conda and singularity.
I will try then to implement this possibility in MpGAP using what I've learnt from the implementation in this other pipeline, focusing on using bioconda packages.
from mpgap.
Related Issues (20)
- Check new assemblers? HOT 4
- Add option hifi
- Add an option for multiple samples HOT 9
- Shovill with all assemblers? HOT 2
- add polypolish tool HOT 1
- update documentation about the configuration in either config or samplesheet HOT 1
- change to unicycler v0.5.0? HOT 1
- new directory called "final_output"
- add testing git actions for PRs HOT 1
- use nf-core framework for CLI help and log messages HOT 3
- No valid choice in the pipeline parameters for hybrid strategies and problem with -c config HOT 9
- quast generating empty files HOT 24
- mkdir: cannot create directory HOT 4
- add trycycler
- add 3 hybrid strategy HOT 1
- Requesting support with error "Explicit 'name separator' in class" HOT 4
- update tool versions
- add homopolish tool HOT 1
- Add skip parameter for sreads polishers and fix multiqc report names HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mpgap.