drjbarker / snakemake-gridengine Goto Github PK

I'm having an issue setting up a SGE profile with snakemake 8.10.0. I tried to use the executor flag but i couldn't set it to qsub or sge.

I get this error:
snakemake: error: unrecognized arguments: --cluster=qsub --sge-submit=sge/sge-submit.py --sge-status=sge/sge-status.py --sge-cancel=sge/sge-cancel.py

I also couldn't find official docs to set that up.

Thanks for your help!

Hi,

I am working on a POC which utilizes snakemake 7.30.1, I saw there is a commit specifically to support mem_mb disk_mb.

And apparently, in 7.30.1, there are mem_mib, disk_mib, which breaks the wrapper:

Traceback (most recent call last):
  File "/home/derlin/snakie/snakemake-profile-demo/brenner/sge-submit.py", line 243, in <module>
    update_double_dict(qsub_settings, parse_qsub_settings(job_properties.get("resources", {}), option_mapping={}))
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/derlin/snakie/snakemake-profile-demo/brenner/sge-submit.py", line 159, in parse_qsub_settings
    raise KeyError(f"Unknown SGE option or resource: {skey}")
KeyError: 'Unknown SGE option or resource: mem_mib'

resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>

I am wondering if there is a workaround?

Thanks
Derrick

I wanted to ask a question about how to set different SGE options for each rule in my Snakefile.

For example, suppose I want each of my rules to have a different job name when submitted. If I were submitting these jobs directly, I would use the -N option of the qsub command.

With Snakemake I know that one way to do this is to specify it in the cluster.yaml file. For instance, if the name of a rule in my Snakefile is run_test and I would like the qsub job to have the name testing, I can add the following to cluster.yaml after the __default__ settings:

"run_test":
  name: "testing"

I was wondering if it would be possible to specify options like this in the Snakefile instead? I'm particularly interested in specifying different -pe options for different rules directly in the Snakefile rather than through the cluster.yaml file.

I have tried this:

rule run_test:
    input: ...
    output: ...
    resources:
        name='testing'
    shell: ...

but get the following error: KeyError: 'Unknown SGE option or resource: name'. I'm not sure what I'm doing wrong?

Thank you very much for your time and for making this code available!

Hi,

after the patch, the mem_mib, disk_mib issue has gone away (thank you :)). But I think it breaks the rulename variable resolving.

All the logs files become like:

logs/sge/snakejob.snakejob.x.log

Cheers
Derrick

Not strictly an issue on this repo but on https://github.com/Snakemake-Profiles/sge

When I try and install it, I get an error:

$ cookiecutter https://github.com/drjbarker/snakemake-gridengine.git
You've downloaded /home/ubuntu/.cookiecutters/snakemake-gridengine before. Is it okay to delete and re-download it? [yes]: yes
A valid repository for "https://github.com/drjbarker/snakemake-gridengine.git" could not be found in the following locations:
/home/ubuntu/.cookiecutters/snakemake-gridengine

I have no experience of cookiecutter so I am not even sure what it is trying to do....

At the moment we have to hard code custom resources into the sge-submit.py script. It would be convenient if we could read these in from a file so that the profile is easier to adapt for different clusters.

Hi,

I'm trying to run jobs using the SGE profile, but I run into issues when using 'threads':

Specifying 'threads' in the snakemake rule seems to be recognized by snakemake (i.e. snakemake output shows 'threads=8', for example), but when I look in the cluster log only 2 threads are 'propagated' to the cluster job (I assume this is a default value to be used in case nothing else is specified)
I thought maybe I need to adjust the profile for this, so I added pe: 'threaded {threads}' under __options__ in the profile's cluster.yaml, but to no avail.

My question: how do I correctly specify multiple threads using the SGE profile? Should I simply use the rule's 'thread' property?
Or is there a better way?

Let me know if you need any more information.

Many thanks in advance and best!
johann

If a qsub fails, for example because a required resource (e.g. h_rt) is not found, then snakemake throws the exception from the subprocess that it returned with a non-zero return values.

We should see if there's an easy way to determine why qsub failed and print that for the end user.