drjbarker / snakemake-gridengine Goto Github PK
View Code? Open in Web Editor NEWSnakemake profile for the (Sun) Grid Engine queueing system
Snakemake profile for the (Sun) Grid Engine queueing system
If a qsub fails, for example because a required resource (e.g. h_rt) is not found, then snakemake throws the exception from the subprocess that it returned with a non-zero return values.
We should see if there's an easy way to determine why qsub failed and print that for the end user.
Hi,
I am working on a POC which utilizes snakemake 7.30.1, I saw there is a commit specifically to support mem_mb disk_mb.
And apparently, in 7.30.1, there are mem_mib, disk_mib, which breaks the wrapper:
Traceback (most recent call last):
File "/home/derlin/snakie/snakemake-profile-demo/brenner/sge-submit.py", line 243, in <module>
update_double_dict(qsub_settings, parse_qsub_settings(job_properties.get("resources", {}), option_mapping={}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/derlin/snakie/snakemake-profile-demo/brenner/sge-submit.py", line 159, in parse_qsub_settings
raise KeyError(f"Unknown SGE option or resource: {skey}")
KeyError: 'Unknown SGE option or resource: mem_mib'
resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>
I am wondering if there is a workaround?
Thanks
Derrick
Hi,
I'm trying to run jobs using the SGE profile, but I run into issues when using 'threads':
Specifying 'threads' in the snakemake rule seems to be recognized by snakemake (i.e. snakemake output shows 'threads=8', for example), but when I look in the cluster log only 2 threads are 'propagated' to the cluster job (I assume this is a default value to be used in case nothing else is specified)
I thought maybe I need to adjust the profile for this, so I added pe: 'threaded {threads}'
under __options__
in the profile's cluster.yaml
, but to no avail.
My question: how do I correctly specify multiple threads using the SGE profile? Should I simply use the rule's 'thread' property?
Or is there a better way?
Let me know if you need any more information.
Many thanks in advance and best!
johann
At the moment we have to hard code custom resources into the sge-submit.py script. It would be convenient if we could read these in from a file so that the profile is easier to adapt for different clusters.
Hi,
after the patch, the mem_mib, disk_mib issue has gone away (thank you :)). But I think it breaks the rulename variable resolving.
All the logs files become like:
logs/sge/snakejob.snakejob.x.log
Cheers
Derrick
I wanted to ask a question about how to set different SGE options for each rule in my Snakefile.
For example, suppose I want each of my rules to have a different job name when submitted. If I were submitting these jobs directly, I would use the -N
option of the qsub
command.
With Snakemake I know that one way to do this is to specify it in the cluster.yaml
file. For instance, if the name of a rule in my Snakefile is run_test
and I would like the qsub
job to have the name testing
, I can add the following to cluster.yaml
after the __default__
settings:
"run_test":
name: "testing"
I was wondering if it would be possible to specify options like this in the Snakefile instead? I'm particularly interested in specifying different -pe
options for different rules directly in the Snakefile rather than through the cluster.yaml
file.
I have tried this:
rule run_test:
input: ...
output: ...
resources:
name='testing'
shell: ...
but get the following error: KeyError: 'Unknown SGE option or resource: name'
. I'm not sure what I'm doing wrong?
Thank you very much for your time and for making this code available!
I'm having an issue setting up a SGE profile with snakemake 8.10.0. I tried to use the executor flag but i couldn't set it to qsub or sge.
I get this error:
snakemake: error: unrecognized arguments: --cluster=qsub --sge-submit=sge/sge-submit.py --sge-status=sge/sge-status.py --sge-cancel=sge/sge-cancel.py
I also couldn't find official docs to set that up.
Thanks for your help!
Not strictly an issue on this repo but on https://github.com/Snakemake-Profiles/sge
When I try and install it, I get an error:
$ cookiecutter https://github.com/drjbarker/snakemake-gridengine.git
You've downloaded /home/ubuntu/.cookiecutters/snakemake-gridengine before. Is it okay to delete and re-download it? [yes]: yes
A valid repository for "https://github.com/drjbarker/snakemake-gridengine.git" could not be found in the following locations:
/home/ubuntu/.cookiecutters/snakemake-gridengine
I have no experience of cookiecutter so I am not even sure what it is trying to do....
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.