Giter Club home page Giter Club logo

Comments (6)

tazend avatar tazend commented on May 29, 2024

Hi @steenlysgaard

could you perhaps check how things are looking when you apply the same logic when using sacct? In sacct it would be CPUTimeRaw that corresponds to elapsed_cpu_time in pyslurm.

I'll also take your example and check out the issue on my Cluster to see whats going on.

from pyslurm.

steenlysgaard avatar steenlysgaard commented on May 29, 2024

Hi @tazend

Thanks, I can confirm that sacct behaves exactly like the new API:

> sacct --starttime=2023-08-31T13:01:03 --endtime=2023-08-31T13:01:07 -o JobID,Start,End,CPUTimeRAW
JobID                      Start                 End CPUTimeRAW 
------------ ------------------- ------------------- ---------- 
9            2023-08-31T13:01:05 2023-08-31T13:01:09          4 
9.batch      2023-08-31T13:01:05 2023-08-31T13:01:09          4 
> sacct --starttime=2023-08-31T13:01:07 --endtime=2023-08-31T13:01:11 -o JobID,Start,End,CPUTimeRAW
JobID                      Start                 End CPUTimeRAW 
------------ ------------------- ------------------- ---------- 
9            2023-08-31T13:01:05 2023-08-31T13:01:09          4 
9.batch      2023-08-31T13:01:05 2023-08-31T13:01:09          4 
> sacct --starttime=2023-08-31T13:01:11 --endtime=2023-08-31T13:01:15 -o JobID,Start,End,CPUTimeRAW
JobID                      Start                 End CPUTimeRAW 
------------ ------------------- ------------------- ----------

I noticed that in sacct --help it says:

-S, --starttime:                                                       
                   Select jobs eligible after this time. ...

By eligble, they apparently mean any job that is queued or running at that time.

from pyslurm.

tazend avatar tazend commented on May 29, 2024

Hi,

I see. I think what is needed is the -T / --truncate flag in sacct:

-T, --truncate
                 Truncate time.  So if a job started before --starttime the start time would be truncated to --starttime. 
                 The same for end time and --endtime.

The Job would still be found in both months, however the value for CPUTimeRaw is correctly adjusted for actual time-span the user requested, i.e:

> sacct -T -S 2023-09-1T12:51:30 -E 2023-09-1T12:55:13 -o JobID,Start,End,CPUTimeRaw,TotalCPU
JobID                      Start                 End CPUTimeRAW   TotalCPU 
------------ ------------------- ------------------- ---------- ---------- 
277383       2023-09-01T12:51:35 2023-09-01T12:55:13        218  10:18.002 
277383.batch 2023-09-01T12:51:35 2023-09-01T12:55:13        218  10:18.002 

> sacct -T -S 2023-09-1T12:51:30 -E 2023-09-1T12:54:13 -o JobID,Start,End,CPUTimeRaw,TotalCPU
JobID                      Start                 End CPUTimeRAW   TotalCPU 
------------ ------------------- ------------------- ---------- ---------- 
277383       2023-09-01T12:51:35 2023-09-01T12:54:13        158  10:18.002 
277383.batch 2023-09-01T12:51:35 2023-09-01T12:54:13        158  10:18.002

(The actual CPU efficiency, i.e. TotalCPU can't be truncated though to the time-interval)

Can you confirm that this is how you would like it to be?
Then I'd go ahead and implement this option

from pyslurm.

steenlysgaard avatar steenlysgaard commented on May 29, 2024

Yes, you are right the truncate option would work for my application.

Thanks!

from pyslurm.

tazend avatar tazend commented on May 29, 2024

Hi,

just added a truncate_time option on this branch (for slurm 23.02) in the pyslurm.db.JobFilter class:

import pyslurm
job_filter = pyslurm.db.JobFilter(truncate_time=True)
...

Additionally, I also added some new attributes to the pyslurm.db.Jobs class for convenience, so some statistics about jobs in that collection are automatically summed up after retrieval, for example:

import pyslurm
db_jobs = pyslurm.db.Jobs.load()

print(db_jobs.elapsed_cpu_time)
print(db_jobs.cpus)
print(db_jobs.memory)
...

Feel free to test out the branch, I'll push it to master soon after adding documentation etc.

from pyslurm.

steenlysgaard avatar steenlysgaard commented on May 29, 2024

I just tried out the branch and it works as expected. Furthermore, the new attributes make gathering the statistics a little simpler.

Thanks!

from pyslurm.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.