fasrc / slurmmon Goto Github PK

e.g. JobScript, JobName, etc.

Most (all?) of it currently is, but I think that's by virtue of the syntax highlighting code, and might get lost if that's configured off.

Pull request #2 has some nice PIL code for converting to an image instead, but the copy/paste and grep-ability of text is really helpful, and it would be nice to keep package dependencies to a minimum.

Yet another thing...
I like your plugin that's why I really want to use it... but some things remain not clear to me.
This is about the web page. I installed to the default path /var/www/html/slurmmon. The php's for the graphs are located at /var/www/html/ganglia/graph.d.
When I look on my ganglia web page I see the slurmmond_* reports. But when I open the slurmmon page I only see text and broken images although the graph.php calls look the same as on my ganglia page.
The whitespace report works at least for the first 3 columns. The rest is not shown (like CPU efficiency and so on). Unfortunately, it is not clear to me how to set up the whole thing that I could look at the graphs in ganglia and use the slurmmon web page. Or is it a ganglia OR slurmmon?
I'm thankful for any advices.

Hi again,
I'm looking through the code because I was wondering about the numbers on cores wasted in the whitespace report. In Node.py (l.89 I think) you define Cores_Wasted as CPUAlloc - CPULoad. As far as I could see CPULoad is the 1min load average from /proc/loadavg and CPUAlloc is the number of cores.
The naming "Cores wasted" sounds like wasted, unused, non-utilized cores to me, like, for example, somebody calling for a full node with N cores and just using one of the N cores and therefore wasting N-1 cores.
The more I think about it, I can see why you chose the quantity of CPUAlloc - CPULoad since the difference will approach zero when the cpus are "loaded" enough. Then still the name in the report is misleading somehow. I will add the definition of it to the whitespace report table for now. But I would be interested in your thoughts on this.
By the way, are GPUs supported already? I think I didn't see anything in the code about it.

Running /usr/sbin/slurmmon_whitespace_report shows:
Traceback (most recent call last):
File "/usr/sbin/slurmmon_whitespace_report", line 230, in
live_nodes_cpu=True,
File "/usr/sbin/slurmmon_whitespace_report", line 117, in write_report
for i, x in enumerate((j['User'], j['JobID'], int(round(j['CPU_Wasted']/(60_60_24))), '%d%%' % int(round(j['CPU_Efficiency']*100)), j['NCPUS'], config.syntax_highlight(j['JobScriptPreview']))):
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

Setting completed_jobs_cpu=False, I obtain:
*** ERROR *** unable to parse squeue job text ['']: need more than 0 values to unpack

Any advice? My current guess would be that you're using a different resource management in slurm. Maybe you could provide your settings in slurm.conf or add the configuration prerequisites for using slurmmon?

I'm seeing another issue, which may be just my own fault. When I install the rpms and start slurmmond afterwards, the daemon doesn't find the slurm commands that are located in /usr/local/bin. Also the PATH environment includes the /usr/local/bin. Here's a trace of the messages that I just took:

slurmmond[5341]: starting
slurmmond[5341]: started sdiag metrics process, pid [5342]
slurmmond[5341]: started jobcount metrics process, pid [5343]
slurmmond[5341]: started reserved cores metrics process, pid [5345]
slurmmond[5341]: started [probejob-compute,IB] metrics process, pid [5346]
slurmmond(sdiag)[5342]: metrics for [slurmmond(sdiag)] failed with message [[Errno 2] No such file or directory]
slurmmond(jobcount)[5343]: metrics for [slurmmond(jobcount)] failed with message [shell code ["squeue -h -o '%u' -t PD | wc -l"] failed with exit status [0], stderr is ['/bin/sh: squeue: command not found\n']]
slurmmond(probejob-compute,IB)[5346]: metrics for [slurmmond(probejob-compute,IB)] failed with message [job submission ["sbatch '-p' 'compute,IB' '-J' 'probejob' '-n' '1' '-t' '2' '--mem' '10' '-o' '/dev/null' '-e' '/dev/null' --wrap 'true'"] failed with non-zero returncode [127] and/or non-empty stderr ['/bin/sh: sbatch: command not found']]
slurmmond(reservations)[5345]: metrics for [slurmmond(reservations)] failed with message [[Errno 2] No such file or directory]

If I add /usr/local/bin/ to all of the command calls slurmmond works.
I also tried to add a sys.path.append but this didn't work. Am I doing somehting wrong?

any plans to migrate to Rhel7/centos7?
I got the rpm's rebuilt for RHEL7, rpm's installed and services running but ran into problems with the index.psp file in slurmmon-web. Seems mod_python is does not exist anymore and has moved to mod_wsgi. I'm not a python programmer for the web so I'm stuck here.

Hello,
I'm trying install slurmmon on Linux Mint 18.3
I sucssfully installed the RPMs but I'm getting an error when I try to start slurmmon service:

Job for slurmmond.service failed because the control process exited with error code. See "systemctl status slurmmond.service" and "journalctl -xe" for details.

And the "journalctl -xe output is:

-- Unit slurmmond.service has begun starting up.
Nov 29 09:42:18 master.lbn.com sudo[1745]:     root : TTY=unknown ; PWD=/ ; USER=slurmmon ; COMMAND=/usr/bin/python /
Nov 29 09:42:18 master.lbn.com sudo[1745]: pam_unix(sudo:session): session opened for user slurmmon by (uid=0)
Nov 29 09:42:18 master.lbn.com slurmmond[1732]: Starting slurmmond... Traceback (most recent call last):
Nov 29 09:42:18 master.lbn.com slurmmond[1732]:   File "/usr/sbin/slurmmond", line 142, in <module>
Nov 29 09:42:18 master.lbn.com slurmmond[1732]:     import slurmmon
Nov 29 09:42:18 master.lbn.com slurmmond[1732]: ImportError: No module named slurmmon
Nov 29 09:42:18 master.lbn.com sudo[1745]: pam_unix(sudo:session): session closed for user slurmmon
Nov 29 09:42:18 master.lbn.com slurmmond[1732]: *** FAILED ***
Nov 29 09:42:18 master.lbn.com systemd[1]: slurmmond.service: Control process exited, code=exited status=1
Nov 29 09:42:18 master.lbn.com systemd[1]: Failed to start SYSV: slurmmond - gather data about SLURM behavior.
-- Subject: Unit slurmmond.service has failed

Hoe to fix this ?
Thanks