Comments (27)
@alexxxxx very soon, hopefully! I've been chipping away at it. I'm working on processing sbatch directives inside the scripts. Once I have that completed, I'll add some tests and merge it into master.
I think currently the wrap and script s without sbatch directives do work.
from pyslurm.
@giovtorres Wondering how it goes with the job submission :)
from pyslurm.
Hi, sorry for the delay on this. I did manage to update the development code to 17.02. However, I haven't done much with the job submission portion just yet. I would have preferred to switch over to the newer API before and put efforts there when making newer changes, especially a big change such as job submission, in order to avoid duplication.
I'm currently setting up some automated testing, which should help move things along a bit.
from pyslurm.
If I find a solution, but for now I suppose I'll use subprocess with an sbatch command.
from pyslurm.
I would like to insist on this bug. It would be very interesting to have it. Obviously, drmaa (v1) is there and may be used. However, there is one drawback to drmaa : you can only run it as being the user using the program. What would be nice would be to have a launch(user="toto",...) function, where you could choose which user starts the job. It is a key function for any web interface (well there are ways to overcome that, but it will be not very smart).
from pyslurm.
Once all the new API code is in for 14.11 plus the additional contributions from others, test examples and any major outstanding issues that may affect 14.11 are resolved then this will shortcoming will be addressed. I hope to start coding up a basic submit class over this christmas break.
from pyslurm.
I second this feature request. After spending a while playing with the API to gather information about running jobs (I couldn't get information about the nodes yet, see #52) I was wondering about how to launch one. I was surprised not to find any reference to it. Is that a planned feature?
Best,
Aurélien
from pyslurm.
Yep, I know it's been open for a long time and I agree but I need to resolve the outstanding issues first.
from pyslurm.
Hi everyone,
I know it has been a while but I wanted to let you know that we are working on wrapping job submit functions and hope to have a decent amount of it wrapped by the time 17.02 is released. For example:
>>> import pyslurm
>>> pyslurm.job.allocate_resources_blocking()
python: job 27205708 queued and waiting for resources
python: job 27205708 has been allocated resources
[debug] job_id: 27205708
[debug] node list: cn0014
[debug] cancelled job id: 27205708
This is in the dev-16.05-props branch and will be eventually merged into master.
Giovanni
from pyslurm.
17.02 has been released by now. Is there any plan to go forward with merging the job submit wrapper functions in master? Thanks for your efforts in creating this software!
from pyslurm.
Hi @leezu, I've made some progress with this, but it's been a while. I need to create a branch based on 17.02. I'll try to get to that this week, then you will be able to test out the code. Thanks!
from pyslurm.
Thanks for the heads up @giovtorres. My university is using version 14.11.11, I'm not sure if your code relies on any APIs released in more recent versions. If not I'll be happy to help testing.
from pyslurm.
That version is a bit older. We've mostly been doing development on 16.05 and 17.02, focusing more on the newer versions of Slurm as they come out. Unfortunately, I don't think we will be backporting any new code into 14.11 versions.
from pyslurm.
Thanks, @giovtorres , just curious, have you got a chance to try to test this PR: #65?
Although it is based on slurm 2.6, it should be compatible (or minor modifications needed) with slurm 16.05.5 since no much code change on job submission between these two versions.
from pyslurm.
I did a while back. The C structs have changed some between that version and the current version. I would have to go through it more closely.
from pyslurm.
True and new options added I think. I am trying to make it work with 16.05 but I don't know much about C/Cython. So, don't know how it would go. Thanks for your reply and btw, really like your blog post of wrapping sdiag
with Cython. I just wish sbatch
would be less complicated 👍
from pyslurm.
Looks like someone created a project solely to do slurm job submission within Python: https://github.com/brentp/slurmpy
from pyslurm.
Thanks, though it is using subprocess to call sbatch
and not through the Slurm API.
from pyslurm.
I pull this #65 PR and make a few modifications for 16.05. It works for some simple scripts. Haven't tested on other options. But generally, that PR should work.
from pyslurm.
@0xaf1f: if all you need is job submission/control, I would rather use GC3Pie: https://gc3pie.readthedocs.io/en/master/
It doesn't interface the SLURM API though but just wraps the command line tools afaik.
from pyslurm.
@mstud, thanks. We had a user request installation of slurmpy when we already had pyslurm installed, and I found out from this issue that submission isn't handled by pyslurm. I thought I'd link slurmpy here since people commenting here have wanted this functionality for a few years. But GC3Pie does look better. Thanks again.
from pyslurm.
@giovtorres Just indicating another interested user in using PySlurm for Job Submission
from pyslurm.
Thanks @kmanalo. I hope to focus on this in the coming week.
from pyslurm.
Hello @giovtorres ,
Very interesting feature that I can't wait to test :) !!! I can see in the branch "job_submit" pretty recent commits. Is this branch currently in a "testable" state?
When do you except to have "beta" release for this feature?
Thanks in advance,
Alex
from pyslurm.
Hi @giovtorres. I greatly appreciate this work. I noticed commits as of January 2018. Is there any plan to merge this work soon and/or how usable do you estimate the branch to be at this point?
from pyslurm.
Is this issue still being worked on? I tried out "job_submit" and it ignores #SBATCH directives.
from pyslurm.
@pagrubel Not at the moment by me. Would you like to contribute?
from pyslurm.
Related Issues (20)
- pyslurm throw KeyError 5039 when statistics().get() called HOT 2
- symbol lookup error: undefined symbol: data_init HOT 7
- Installation fails for v22.5.x HOT 2
- Submittion of a batch job will be failed when argument "work_dir" contains a "_" HOT 1
- Unclear that `Jobs.load()` returns a dictionary. HOT 2
- Missing `Job` fields compared to the docs HOT 2
- slurmdbd: Implement Association API
- ValueError: invalid literal for int() with base 10: 'milton.m' HOT 1
- Copy/Paste error in Job class memory property HOT 1
- Cannot seem to get allocated gres info from Node HOT 4
- Node in Nodes collection do not report a MIXED state, only IDLE or ALLOCATED HOT 5
- A job is found with JobFilter if it is running on start_time HOT 6
- question: install on cluster without slurm-devel package and without root access HOT 2
- job mem returns as "None" HOT 3
- error: ‘KILL_JOB_ARRAY’ undeclared HOT 1
- Slurm 23.11 support HOT 3
- pyslurm is installation failed HOT 2
- Particular value of gres is causing cstr to crash HOT 1
- [guidance request] 'gres' : 'gpu' does not have the same behavior as when using bash HOT 2
- Problem in setup.py get_version() - packaging.version object is missing .major and .minor attributes in earlier module versions under Python 3.6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyslurm.