dmwm / cms-htcondor-es Goto Github PK
View Code? Open in Web Editor NEWElasticSearch integration for CMS's HTCondor pool
ElasticSearch integration for CMS's HTCondor pool
(Starting to hand over, it would be good if someone else would start doing these things.)
Here's a short guideline:
message_type:validator_unknown_key
).JobMonitoring.json
file in /home/cmsjobmon/cms-htcondor-es/
(in a safe place) and add key/example value pairs.schemas/JobMonitoring.json
and jsonschemas/JobMonitoring.json
files. Create the latter using genson schemas/JobMonitoring.json > jsonschemas/JobMonitoring.schema
.Seems we sare still far from having full accounting of jobs in MONIT.
While investigating a problem with glideinWms
https://ggus.eu/index.php?mode=ticket_info&ticket_id=138535
I tried to use ES+Kibana etc. but found the very puzzling situation that for this CRAB Task [1]
I can not find in ES the jobs which completed at T2_TR_METU.
An example of such job (which completed successfully at that site a few hours ago) is e.g. [3]
So if the job ran, I am sure HTCondor knows about it, why it is not shown ?
Is me not looking in the right way, or something is wrong in the reporting ?
By the way way those jobs are in ES when running (although with incomplete
information, like missing CRAB_Id i.e. their ID inside the crab task), but somehow not as Completed [4]
[1]
Here are pointers in new and old dashboard.
https://monit-grafana.cern.ch/d/NvcuKTSiz/condor-jobs-test?orgId=6&var-task=181203_151212:areinsvo_crab_SingleMuon_Run2017F-31Mar2018-v1&var-user=areinsvo
http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=areinsvo&refresh=0&table=Mains&p=1&records=25&activemenu=2&pattern=181203_151212%3Aareinsvo_crab_SingleMuon_Run2017F-31Mar2018-v1&task=&from=&till=&timerange=lastWeek
Easiest solution is to move it from int_vals
to string_vals
:
https://github.com/bbockelm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L111
We'd then need to re-index the older ES indices to follow the change.
Currently all fields are converted and processed and then all but the ones in running_fields
are dropped. We could optimize by only processing those in the first place.
I found that repository does not contain shell scripts used in cronjobs on vocms0240. Why these scripts are not present in this repository? Where they are located?
I suggest to update repository with all shell script we need to use on production node.
For production jobs, we currently store the tarballs in CMS EOS (and they are readable by CMS members).
The URL can be constructed with the following recipe:
"https://eoscmsweb.cern.ch/eos/cms/store/logs/prod/recent/" + \
<task name> + "/" + \
<step name> + "/" + \
<schedd name> + <WMAgent job ID> + "-" + <retry #> + "-log.tar.gz"
For WMAgent jobs, can we add a new attribute (LogURL
?) containing a URL constructed as above?
For the tasks' index, we are only taking the last status for each job. However, the order in which the messages are consumed is not enough to ensure that the last seen message is the actual last update. We starting using the RecordTime
, and it worked when the status change, but for a completed job status updates always have the same RecordTime
making that updates over completed jobs didn't get overwritten (as they have the same version).
Currently, DataCollection
and RecordTime
use the CompletionDate
for Completed jobs. Could we use the _launch_time
for DataCollection (or a new field) and use it as a version hint?.
In the new ES schema, the retry number is part of the ID of the document, it is needed to avoid duplicates.
The QDate can be useful for the user in the tasks list for the running and pending tasks to order (and, when grafana enables us to do so, filter) the jobs.
Discussed with CERN/ES about this. They will start setting up a new endpoint "es-cms6" and mirror new data there. We can then observe possible issues and start fixing them.
The main thing is the removal of mapping types, but we only use a single type ("job"), so that should not be an issue for us.
with ref. to
https://github.com/bbockelm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L532
there are now more DAGs around in for CRAB due to automatic splitting which have task name other than ROOT yet do are not grid jobs.
We need a more solid way to tell things, and to communicate to developers what they can (not) do.
Using parameters to define the collectors or schedd factories (see #134) unify the different spiders.
I've notice there are some analysis jobs that are pushed as Type:production and trace this issue down to this part of the code:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L574
There are several analysis jobs, mostly jobUniverse 7 and 12, that do not have CRAB_Id neither define AccountingGroup and hence, they get marked as 'production', look at the condor query done on a crab schedd[1]
Wouldn't be better to use the CMSGWMS_Type attribute in the Schedd ClassAd to get the type of schedd (crabschedd, prodschedd, tier0schedd) and set the job Type respectively ?
[1]
[root@vocms0198 ~]# condor_q -const 'CRAB_Id=?=UNDEFINED' -af:h JobUniverse AccountingGroup | sort | uniq -c
240 12 undefined
10 5 production.cmsdataops
2 5 undefined
2 7 "highprio.spiga"
5242 7 undefined
We're currently maintaining a limited list of fields to be retained when processing the condor queues:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L423
And we're dropping any field not in that list when processing the queues:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L780-L781
(The behavior can be switched off by using the --keep_full_queue_data
option for spider_cms.py)
I had introduced this as a means of speeding up the uploading of documents to MONIT (it cuts about a factor of 5 in volume, iirc) as we were having trouble staying within the 12 min cycle. However, we were still uploading from a VM hosted at UNL at that time, and the faster upload time from the CERN hosted VM to MONIT might have made this unnecessary.
To check, we could run a few cycles without culling the fields and see how much the performance suffers.
Another aspect to keep in mind, however, is that this would significantly increase the data volume needed on the MONIT side, since it affects a large majority of the documents we send.
As suggested by @belforte in MM(Distrubuted Analysis channel), we are planning to put JobUniverse==5 constraint to our query scripts that query Condor Schedds, both queue and history.
As CMS Monitoring&Analytics team, we are just the operator of this Condor Job Monitoring. That's why, we would like to get confirmation from all stakeholders.
You can see Completed jobs with JobUniverse is different than 5 in [1] and see other jobs with JobUniverse is different than 5 in [2]
[1] https://es-cms.cern.ch/kibana/goto/9916fa1438a0d470cf620150e2ec8fb7
[2] https://monit-opensearch.cern.ch/dashboards/goto/9f7d3eaf122501164b28e0cb84b616b5?security_tenant=global
In order to avoid using javascript links to create the URLs, we'll send the URLs to ES.
Also, in order to use the value directly, we'll add the committed wall clock in seconds (to use the Grafana's duration formats).
We currently set the RecordTime
as the CompletionDate
from the ad, and then overwrite it with the current time in case the CompletionDate
is not set:
https://github.com/bbockelm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L536-L540
result['RecordTime'] = ad.get("CompletionDate", 0)
...
if not result['RecordTime']:
result['RecordTime'] = _launch_time
The problem is when jobs have a CompletionDate
, but have not actually terminated, e.g. because they are "Held". (I'm not sure actually there are any other examples.) So those docs will be submitted with the same RecordTime
over and over again, and pile up in the dashboard in one time bin.
Instead we could set RecordTime
to be the launch time for all jobs with statuses other than "Completed".
CRAB ID is being omitted for running and idle tasks and is needed for the task monitoring dashboard.
It seems that when a job gets removed, the monitoring system keeps adding duplicate records of it every 12 minutes. In this example:
https://monit-kibana.cern.ch/kibana/app/kibana::/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'2019-07-29T22:00:00.000Z',mode:absolute,to:'2019-08-07T21:59:59.999Z'))&_a=(columns:!(_source),index:'monit_prod_condor_raw_metric_v002-*',interval:auto,query:(query_string:(query:'data.CRAB_Retry:3+AND+data.CRAB_Id:11+AND+data.CRAB_Workflow:190702_095101*acarvalh_crab_2016v3_2019Jul02_ZZTo2L2Nu_13TeV_powheg_pythia8__RunIISummer16MiniAODv3-PUMoriond17_94X_mcotic_v3-v2')),sort:!(metadata.timestamp,desc))
you can see that all of the records point to the exact same job, same LastStartTime, same globalJobId, etc. One thing to notice is that the CpuEff does change. This could be the reason why the monitoring system thinks it is a different record. I think this is related to this bug in condor:https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=7083
The MachineAttrCMSSubSiteName0 was added at the condor level to identify opportunistic nodes on sites. It started appearing on CompletedJobs, now we need to add them to the Running Jobs list.
Hello,
While looking into the following issue:
dmwm/WMCore#11613
It was found out that some indexes like the one used for monit_prod_condor
seem to be missing some entries from this block:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L751-L757
Basically, if we have something like (as shown in the ES link above):
data.DESIRED_CMSDataset : /EGamma/Run2022C-v1/RAW
Then, we should see:
data.CMSPrimaryPrimaryDataset, data.CMSPrimaryDataTier, data.CMSPrimaryProcessedDataset
but only data.CMSPrimaryDataTier
is properly shown, with the other 2 set as Unknown
. I haven't see this in other indexes like cms-20*
, is this a bug?
Also, we had a few questions regarding the names:
CMSPrimaryPrimaryDataset: why Primary twice in the name?
CMSPrimaryDataTier, would it be better to rename it to DESIRED_CMSDatatier? In such case, we would like to do some renaming to make things clearer, like:
CMSPrimaryPrimaryDataset -> DESIRED_CMSPrimaryDataset
CMSPrimaryProcessedDataset -> DESIRED_CMSProcessedDataset
CMSPrimaryDataTier -> DESIRED_CMSDatatier
what do you think?
pinging @amaltaro
Given the discussion in dmwm/CRABServer#5860
we agree with @cronosnull to add a numerical field corresponding to the CRAB_Id
string and sort jobs accordingly.
At the moment, the possible values of CRAB_Id are of two types:
and the overall order should be 0-1, 0-2, 0-3, 0-..., 1, 2, 3, ..., 1-1, 1-2, 1-3, 1-..., 2-1, 2-2, 2-3, 2-..., 3-1, 3-2, 3-3, 3-...
This change has been already pushed to production using a temporary python2 branch (keeping the two collectors), but once the Python 3 version is ready we need to apply the change.
We use this strategy because the changes can be slightly different and because we want to keep the python3 branch-which is currently on review- able to automatically merge with the master)
As @belforte noticed from the task monitoring dashboard, the exit code is being sent as the mod 256 of the actual code (maybe it was cast as byte in some part of the process).
When looking into the following grafana dashboard:
https://monit-grafana.cern.ch/d/ifXAfjLVk/production-jobs-exit-code-monitoring?orgId=11
we realized we had some jobs that were "Removed" but the exit code was successful. When investigating this, we noticed the following line:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L1157
basically means when a job is removed and hence neither CRAB, WMCore or HTCondor gets to propagate a classad to report a exit code, it defaults to "0".
This shouldn't be the case, but we are wondering what the value should be instead. After discussing his, we came up with some options and we wanted to hear feedback from the monitoring team;
Options are:
-1
: Only issue with this is, exit codes are usually positiveDo you have any comments on what the best exit code to default should be (one that does have side effects with monitoring)
Related issues:
dmwm/WMCore#11614
Run py2to3
and see what happens.
In order to use the CRAB jobcount to infer the number of unsubmitted jobs we need this field to be present in the index for all the jobs.
Christian,
after review of cronAffiliation.sh
script in context of k8s migration I need two things:
spider_cms.py
This implies that you need to change AffiliationManager
class to accept input file which either will be created by this class or it will read from it.
I also suggest to get rid of cronAffiliation.sh
script and replace it with pure python version with proper arguments, similar as spider_cms.py
All the environment should be set externally such that scripts can be run from the shell.
This will allow to easily port the functionality of spider/affiliation python scripts to k8s.
At the end what we need is to call two files:
# to produce affiliation file
python affiliation.py --output=affiliation.txt
# to read affiliation
python affiliation.py --input=affiliation.txt
# to read affiliation in spider
python spider_cms.py --affiliation=affiliation.txt ...
Could you please adjust the code and make necessary changes.
As pointed out by @belforte , it would be good to have a single field to use for both Running/Idle and Completed jobs.
@bbockelm I'm assuming the logic is that RemoteHost
is the current host for a running job, and LastRemoteHost
is simply the last host that the job ran on?
If so, could we simply set RemoteHost
to LastRemoteHost
for completed jobs?
At present, a job campaign is defined in the following line:
https://github.com/dmwm/cms-htcondor-es/blob/vm-legacy/src/htcondor_es/convert_to_json.py#L699
We have recently implemented a mechanism to report the campaign name in the WM system
dmwm/WMCore#10914
The class ad name is: CMS_CampaignName
When this classad is defined, this information should be used and when not, the current logic in monit can be used instead.
Central services will be updated for next week and this feature will be present.
It seems that, for historical reasons, we are using CMS_JobType instead of CMS_TaskType to set TaskType for analysis jobs.
I don't fully trust how the code recovers history data after not finishing for a few hours (e.g. when the VM is down). For example, the VM feeding es-cms was down for a few hours after a reboot on Sunday afternoon (October 7th), and the script was restarted only on Monday afternoon. It recovers some, but not all of the data:
https://es-cms.cern.ch/kibana/goto/14b8189cfdd5119db8dc25405fa4a9f7
Looking at the code, I suspect this:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/history.py#L54
where we specify a limit of 10'000 jobs per query (per schedd). Depending on which 10'000 jobs this retrieves, the last_completion
time will be set such that older jobs are never recovered.
@bbockelm can you clarify which jobs are returned when a limit is passed to schedd.history
? Should we increase that number?
At present, physics task types are supported via:
https://github.com/dmwm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L682-L684
WMCore has recently added support for this. The classad is called: CMS_extendedJobType
and it shows the physics types when this is a production/processing job, or just the WMCOre job type (LogCollect, Merge, etc) when it is not.
We would need to replace the current CMS_TaskType
in the code lines above with this new classad CMS_extendedJobType
and still fallback to the guessTaskType function until this feature is fully propagated.
Do I understand correctly that CMS_TaskType
was a placeholder for a future WMCore native implementation of this by the way? I don't see this classad in the jobs in condor.
Documentation can be found below:
https://github.com/dmwm/WMCore/wiki/Job-task-type-characterization-based-on-cmsDriver-command-line-arguments
But it basically support the following physics types (and a combination of them)
Physics types:
There are still cases where spider_cms.py
runs out of time and fails to shut itself down properly.
Probably in the upload loop:
https://github.com/bbockelm/cms-htcondor-es/blob/master/spider_cms.py#L779-L810
Changes in e75c68b regarding the timezone (get_creation_time_from_taskname
) did not make any difference. We can remove the pytz dependency, and use:
import calendar
#...
calendar.timegm(_naive_date.timetuple())
I'll collect here some issues that appear.
text
to keyword
in the mappings for them to be used in aggregations. See herePer SI (Antonio) request, we want to have the job priority also for the not-completed jobs.
Rather than the existing format, that creates at least N(sitename) field names:
{
"ChirpCMSSW(.*?)IOSite_Site1_ReadBytes" : 123456,
"ChirpCMSSW(.*?)IOSite_Site1_ReadTimeMS" : 123456,
"ChirpCMSSW(.*?)IOSite_Site2_ReadBytes" : 234567,
"ChirpCMSSW(.*?)IOSite_Site2_ReadTimeMS" : 234567
}
we should store these in a single nested field:
{"ChirpCMSSW_IOSite" : [
{
"SiteName" : "Site1",
"ReadBytes" : 123456,
"ReadTimeMS" : 123456
},
{
"SiteName" : "Site2",
"ReadBytes" : 234567,
"ReadTimeMS" : 234567
}
]}
Aggregation by site name can still work when using a kibana plugin: https://ppadovani.github.io/knql_plugin/overview/
I will add this field, leaving the old fields in place for now. Then we can set up the plugin, test it, and decide how to proceed. @bbockelm what do you think? Do you know who's currently using this information?
It looks like in ES analysis jobs have their CMSSW version in the CRAB_JobSW attribute, but production jobs don't have anything like that. Would it be possible to add an attribute called JobSW for all jobs to
contain the CMSSW version used? The field should be defined for any job running CMSSW.
Because of the decision to not continue K8s migration development which is in master
branch, virtual machine based code which is in vm-legacy
will be forced pushed to master
branch. The fact that it is not done via Pull Request(master<-vm-legacy) is to not mix up K8s development code to production code(vm-legacy
).
It was not efficient to use master branch for K8s development but it was necessary for the quick development in this silent repo at that time.
Brian, is there any reason not to add HasBeenRouted_Overflow
to the data uploaded to es-cms instance? That would be quite helpful to investigate issues with "non-standard" jobs.
I was going to create a patch, but I saw this constraint
https://github.com/bbockelm/cms-htcondor-es/blob/master/src/htcondor_es/convert_to_json.py#L475
so I better ask before coding anything.
In order to filter the task in a more natural way for the user, we need the task creation date (it need to be stored as a timestamp, in order to be used in grafana as time field).
It can be generated from the CRAB_Workflow name, but maybe there is a more maintainable way to do this.
By Antonio's request, we need to have the GLIDECLIENT_Name value not only for the completed jobs.
In order to flag jobs intentionally killed by users, rather than by e.g. HTCondor PeriodicRemove because they exceeeded resource bounds. Since both result otherwise in jobs in status Removed with HTC history sayng: job removed by user.
see:
https://its.cern.ch/jira/browse/CMSMONIT-58
and
dmwm/CRABServer#5807
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.