dmwm / gwmsmon Goto Github PK

Currently CMSGWMS_Type represents 3 categories:

1. prodschedd
2. crabschedd
3. others
   3.1 mit schedd
   3.2 cmsconnect schedd

We will further divide prodschedd into prodschedd and tierschedd, so now all tier schedds will have CMSGWMS_Type=tier0schedd instead of prodschedd, nothing else will change apart from that.

totalview runtime is becoming long especially when schedds are overloaded, e.g. vocms0255/256... Get CPU Count part is running the most. In any case, prodview/analysisview/institutional view is already querying all schedds and can produce a file which could be re-used in totalview and speed it up

It will steal all memory and will raise kernel panic...

it would be extremely useful and insightful to have an extra layer of monitoring for

https://cms-gwmsmon.cern.ch/prodview/pdmvserv_task_SMP-RunIIFall17wmLHEGS-00165__v1_T_191004_060900_867

with a url like

https://cms-gwmsmon.cern.ch/prodview/pdmvserv_task_SMP-RunIIFall17wmLHEGS-00165__v1_T_191004_060900_867/SMP-RunIIFall17wmLHEGS-00165_0

that gives details about the task (sites, errors, ...)

Once things get stuck, this starts to send emails...

https://github.com/juztas/gwmsmon/blob/master/src/CondorMonitoring/JobCounter.sh#L18

Please replace this line to the appropriate one or just remove it.

I changed on HW not to receive these emails anymore

Here: https://github.com/dmwm/gwmsmon/blob/master/src/totalview-update#L837

Would be nice to scan also for downtimes inside the factory xmls, like: http://vocms0207.cern.ch/factory/monitor/schedd_status.xml

<entry name="CMSHTPC_T2_US_Caltech_cit">
<downtime status="True"/>
<total>
<ClientMonitor CoresIdle="2" CoresRunning="1039" CoresTotal="1056" GlideIdle="2" GlideRunning="282" GlideTotal="310" InfoAge="183" JobsIdle="267424" JobsRunHere="282" JobsRunning="49740"/>
<Requested Idle="0" IdleCores="0" MaxCores="0" MaxGlideins="0"/>
<Status Held="0" Idle="0" IdleOther="0" Pending="0" Running="22" RunningCores="704" StageIn="0" StageOut="0" Wait="0"/>
</total>
<frontends>
<frontend name="CMSG-v1_0_cmspilot">
<ClientMonitor CoresIdle="2" CoresRunning="1039" CoresTotal="1056" GlideIdle="2" GlideRunning="282" GlideTotal="310" InfoAge="733" JobsIdle="267424" JobsRunHere="282" JobsRunning="49740"/>
<Downtime status="True"/>
<Requested Idle="0" IdleCores="0" MaxCores="0" MaxGlideins="0">
<Parameters></Parameters>
</Requested>
<Status Held="0" Idle="0" IdleOther="0" Pending="0" Running="22" RunningCores="704" StageIn="0" StageOut="0" Wait="0"/>
<StatusEntries/>
</frontend>
</frontends>
</entry>

- means factory will not send new pilots.

As Unified is using the gwmsmon provided jsons, so can you please look at this issue we are facing from time to time while reading the json -
https://cms-gwmsmon.cern.ch/totalview//json/site_summary

As the command we use is :
https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L1914
it doesn't gives much details in error message except - No JSON object could be decoded.

I tried to use the same command on terminal and after a few retries got error message around 11am today morning, which tells that the certain column numbers in the JSON are not readable.
Error Message:
ValueError: Unterminated string starting at: line 1 column 434737 (char 434736)

The last time I see it from the unified logs is at Tue Feb 4 02:47:50 2020.
@z4027163 FYI

Please fix

this

https://cms-gwmsmon.cern.ch/prodview/json/historynew/memorycpu720/pdmvserv_task_BPH-RunIIFall18GS-00244__v1_T_200122_123649_5454/BPH-RunIIFall18GS-00244_0/success

is returning "No JSON object could be decoded".
that leaves unified unable to make any tuning of job requirements.

See "No JSON object could be decoded" from
https://cms-unified.web.cern.ch/cms-unified//logs/equalizor/last.log

I noticed that

https://cms-gwmsmon.cern.ch/poolview/graphs/cpus/cmsgwms-submit3-fnal-gov/daily

is flat to zero, while there are running jobs, and cpu

https://cms-gwmsmon.cern.ch/poolview/graphs/cmsgwms-submit3-fnal-gov/daily

looks like it only happens for fnal agents, but not all of them.
what can be going on?

https://ggus.eu/index.php?mode=ticket_info&ticket_id=126013
As far as I remember, only pilots from [FRONTENDS] can run CMS jobs from Global pool. Please let me know if there is anything wrong and other frontends should also be monitored.
Also just to add what improvements I can add to cms-gwmsmon and created github issues:
a) count number of CPUs for the pilots which running;
b) compare with pilots which are registered to the pool and provide a graph. Constant value should provide a diff of number unregistered to the pool;
c) There is already an API [5] (no GUI for each pilot stats, but can be added) for site admins, from which he get all pilots which are registered to the pool and also information about pilot, like name, what it is running, how cpu, mem is used. Addition to this would be also group by factory and entry name, so it could be debug deeper.
d) GUI for each pilot and mapping to task/user/specific job;

For documentation see here: https://sso-management.web.cern.ch/ and https://linux.web.cern.ch/linux/scientific6/docs/shibboleth.shtml

Currently it is under personal account, which is not good idea.

From @amaltaro here: #8
If we want to get back to the pledges metric, then we need to keep a cache of the pledges for periods of SSB instabilities (I'm short on time to follow that up).