Comments (12)
PreJob code seems OK
and not affected by recent changes
A problem is that prejob log is full of jobs because logging initialization is not working. But in there I find both these
CRAB_RequestedMemory = 2000;
RequestMemory = 4500;
Could it be a hint ? Or simply CRAB_RequestedMemory
is the initial value ?
hmm...
at same time as prejob_logs/prejob.6.7.txt
and prejob_logs/prejob.6.7.txt
were created, the input for PreJob was written in resubmit_info.job.6.txt
ls -l resubmit_info/job.6.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 resubmit_info/job.6.txt
that file is read in
CRABServer/src/python/TaskWorker/Actions/PreJob.py
Lines 181 to 188 in 3087e06
and content later used in
(but I do not know what that
inkey
outkey
stuff is)
but in the resubmit_info
file:
belforte@vocms0194/cluster108538413.proc0.subproc0> cat resubmit_info/job.6.txt |tr , '\n'|grep maxmem
'maxmemory': 2000
'maxmemory': 2000
'maxmemory': 2000
'maxmemory': 2000
'maxmemory': 2000
'maxmemory': 2000
'maxmemory': 2000
'maxmemory': 2000
belforte@vocms0194/cluster108538413.proc0.subproc0>
actually:
belforte@vocms0194/cluster108538413.proc0.subproc0> ls -l resubmit_info/
total 40
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.10.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.1.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 20:24 job.2.txt
-rw-r--r--. 1 crabtw zh 1324 Aug 23 20:24 job.3.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.4.txt
-rw-r--r--. 1 crabtw zh 757 Aug 22 21:32 job.5.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.6.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.7.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.8.txt
-rw-r--r--. 1 crabtw zh 1513 Aug 23 22:01 job.9.txt
belforte@vocms0194/cluster108538413.proc0.subproc0> cat resubmit_info/job.*.txt |tr , '\n'|grep maxmem|uniq
'maxmemory': 2000
belforte@vocms0194/cluster108538413.proc0.subproc0>
that value never changed, and IIUC it is what drives PreJob and Job.x.submit.
from crabserver.
the file in resubmi_info is created (and later read) by PreJob itself
CRABServer/src/python/TaskWorker/Actions/PreJob.py
Lines 554 to 558 in 3087e06
I do not find that directory referenced in any other place in the source base.
I am confused
from crabserver.
DagmanResubmitter has some helpful documentation
CRABServer/src/python/TaskWorker/Actions/DagmanResubmitter.py
Lines 97 to 111 in 3087e06
from crabserver.
hmmm... this should be the intial setting
CRABServer/src/python/TaskWorker/Actions/PreJob.py
Lines 292 to 293 in 3087e06
then DagmanResubmitter communicates via the simple
RequestMemory
CRABServer/src/python/TaskWorker/Actions/DagmanResubmitter.py
Lines 79 to 86 in 3087e06
and edits those in the target dagman job.
The Dagman job ads are then save to _CONDOR_JOB_AD in the SPOOL_DIR where indeed
belforte@vocms0194/cluster108538413.proc0.subproc0> grep RequestMemory _CONDOR_JOB_AD
RequestMemory = 4500
RequestMemory_RAW = 4500
and that file is read by PreJob
CRABServer/src/python/TaskWorker/Actions/PreJob.py
Lines 173 to 175 in 3087e06
from crabserver.
sounds like the only way is
- fix logging in PreJob --> done in #8668
- add zilions of logging
- try to resubmi in a test task
Maybe I can go through PreJob interactively, but I am not sure it will really reproduce the situation for resubmission https://twiki.cern.ch/twiki/bin/view/CMSPublic/Crab3OperatorDebugging#To_run_a_pre_job_on_the_schedd
from crabserver.
I should also test with an older tag. To sort of find when it broke. It will be a long work anyhow
from crabserver.
I finally managed to run PreJOb with pdb
and found that as feared I introduced this bug in f700045#diff-4523635b2647add8f64521c2cb1208629e53538fde1905e60c8a57b285ebee30
I have changed the ads which are used to communicate from TW to Dagman via prepending CRAB_
"for clarity", but somehow the change was not done consistently
from crabserver.
I guess the problem is here
CRABServer/src/python/TaskWorker/Actions/DagmanResubmitter.py
Lines 79 to 86 in 3087e06
The root case was the confusion between ads meant for the JDL of the intial dagman-startup script, and ads used to communicate to various scripts how to submit jobs to grid.
I tried to clean that up using CRAB_...
for the latter, but did not catch the resubmission use case.
from crabserver.
I had only taken care of setting CRAB_RequestedMemory
in DagmanSubmitter
from crabserver.
so..
Dagman(Re)Submitter should set values for CRAB_xxxs classAds, which are then used in PreJob to create the JDL for vanilla jobs submission
from crabserver.
I tested on one job in crab-dev-tw01 and it works.
I will add some documentation comments
from crabserver.
after review, it is fine as it is
from crabserver.
Related Issues (20)
- initial CRAB UI page is almost empty HOT 1
- SUSPEND a Rucio rule before deleting
- bump WMCore to 2.3.6rc1 HOT 2
- Canary deployments - REST, TW, Publisher HOT 4
- schedulers - implement automatic reboot HOT 3
- remove check for `username_CRAB` username for myproxy.
- #831: Test prod CRABClient using preprod REST instance and CMSSW_13_0_2 CMSSW release HOT 3
- test JobRouter on HTC 23.9 HOT 8
- fix ReportRecallQuota for new usage of Rucio account HOT 1
- add an API (URL) to retrieve list of SUBMITREFUSED in last N minutes
- Action Items from Udine Workshop
- add a GIT Etiquette section to crab-docs
- refactor crab-doc
- find a better way to prevent users from recalling PB's HOT 2
- use Stefano's OS token as S3 secrets in REST
- get rid of crab3/CRAB3ServerConfig
- Merge crab rest, log pipeline, and serviceaccount manifests together in single helm chart HOT 3
- Canary (QA) deployment for TW and Publishers
- CRAB monitoring diagram HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crabserver.