Comments (22)
spiga: In order to support a real analysis use case we need to be able to handle user libraries and modules etc.
All these stuff must be placed in the proper location on the runtime area in the WN. The piece of WMSpec handling this part need to be implemented.
from crabserver.
evansde: Can you elaborate a bit on what is needed here?
At some point this should reduce to a tar/untar of a scram project area containing libs, files etc.
How much more detail about it is needed? (Could you outline the current crab process for my enlightenment?)
from crabserver.
ewv: Dave, I think that is about it. In addition to libs, we tar up data/ I think which may contain data files from users. Maybe we want to change that to make them specify each file individually? Probably the first step of this for me is to actually look in detail at what CRAB actually does now.
from crabserver.
evansde: Is data under the scram project area as well?
Seems like if a user does:
scram project CMSSW CMSSW_X_Y_Z
cd CMSSW_X_Y_Z
#meddle with stuff...
Tarring up the CMSSW_X_Y_Z dir will get you every local change, FileInPath settings will work and you dont really need to care too much about whats in there at all? (I implemented an input sandbox for the PA/RelVal stuff back in the day before patch releases using this and it worked pretty well, do users stray outside this much?)
from crabserver.
ewv: The problem with that is that you pack up all their src/ code too which is worthless. And with CRAB running the way it does, you also pack up any old CRAB output they have, so tarballs grow exponentially if you do the naive thing. So we ignore src/ and bin/ for sure and encourage people to submit from those directories.
We should take this opportunity to re-evaluate of course, but we do have a formula that works pretty well and that users are used to.
Let me do a little forensics and I'll post here exactly what CRAB is currently doing.
from crabserver.
ewv: Ok, what crab is doing that may need to be done is:
- Check to see if the executable is part of the release. Usually this is cmsRun and is not put in the sandbox. For our initial stuff, this will DEFINITELY be cmsRun. I don't know if at some point we plan to support arbitrary scripts as jobs
- tar up the user's lib/ and module/ (CMSSW_3_6_3/lib and CMSSW_3_6_3/module)
- crawl through CMSSW_3_6_3/src/ looking for directories named data/. Tar if found
- tar the users CMSSW.py and CMSSW.pkl file
- We tar the crab.cfg file for some reason. To let the remote site try to understand what the user is doing?
I think 2 & 4 are the bare minimum. Something like 3 will be necessary.
CRAB2 also does things we definitely don't need to do:
a) Ship MonaLisa
b) Ship various ProdCommon, IMProv, and WMCore stuff
c) ship various CRAB utilities like writeCfg, cmscp, etc.
All this is taken care of by existing WMCore infrastructure.
from crabserver.
spiga: Replying to [comment:8 ewv]:
Ok, what crab is doing that may need to be done is:
- Check to see if the executable is part of the release. Usually this is cmsRun and is not put in the sandbox. For our initial stuff, this will DEFINITELY be cmsRun. I don't know if at some point we plan to support arbitrary scripts as jobs
- tar up the user's lib/ and module/ (CMSSW_3_6_3/lib and CMSSW_3_6_3/module)
- crawl through CMSSW_3_6_3/src/ looking for directories named data/. Tar if found
- tar the users CMSSW.py and CMSSW.pkl file
thanks Eric, In my original post I meant exactly your 2, 3, 4. This is what I would have in CRAB_3 soon. Not sure if Dave agree, for what I remember we were in the same page with this, some time ago.
- We tar the crab.cfg file for some reason. To let the remote site try to understand what the user is doing?
kind of. Don't remember exactly who asked to have it at the WN level.. If you want I can look for the related ticket/mail... personally I don't really see a use case.
I think 2 & 4 are the bare minimum. Something like 3 will be necessary.
CRAB2 also does things we definitely don't need to do:
a) Ship MonaLisa
b) Ship various ProdCommon, IMProv, and WMCore stuff
c) ship various CRAB utilities like writeCfg, cmscp, etc.All this is taken care of by existing WMCore infrastructure.
from crabserver.
ewv: Actually thinking about this, I think 4) is already taken care of by WMSpec, right. At least pulling the config out of a release is. It should be, at most, a minor change to use a user-supplied file.
from crabserver.
evansde: Couple of comments/questions:
Agree that 2,3,4 need done. 4. is already done by the config cache stuff & sandbox building.
- does lib & module get all the python stuff (configs etc?)
- If the analysis type is a CMSSW thing, then it will be cmsRun, if we want to support others, we add steps to support them explicitly which will then implement stuff to find the binary (Eg: FWLite, or a Script type, guessing the script stuff will cover madgraph etc)
- If you default to picking up the lib & module, you could probably just add a user settable list of extras to include in the sandbox (this could handle the data use case as well?)
The WMCore code will be packaged by the agent by default until we stabilise enough to build the runtime RPMs from it. Any extras that Crab needs can be included in that as well down the road.
from crabserver.
ewv: Replying to [comment:11 evansde]:
Couple of comments/questions:
Agree that 2,3,4 need done. 4. is already done by the config cache stuff & sandbox building.
- does lib & module get all the python stuff (configs etc?)
We don't need the python because we ship the pickled config file, so it has already grabbed all the python objects it needs out of either the user's area or the system area. I assumed WMAgent was doing the same, but maybe not? In any case, I think we want to continue to do this as we have users (encourage them actually) to have python programs as config files. By that, I mean their config can have if statements (on the simple side) to talking to web services (on the complicated end). I don't think we want any of that going on on the worker node.
- If the analysis type is a CMSSW thing, then it will be cmsRun, if we want to support others, we add steps to support them explicitly which will then implement stuff to find the binary (Eg: FWLite, or a Script type, guessing the script stuff will cover madgraph etc)
This is what I had in mind. We should probably eventually supply (or get users to supply) steps for all common actions. Maybe we supply a generic script step as well, but that should be extremely rare.
- If you default to picking up the lib & module, you could probably just add a user settable list of extras to include in the sandbox (this could handle the data use case as well?)
We could. I guess the question comes down to is the advantage of doing this (picking up un-needed data areas) worth re-training users? Now if they change a data/ file in their own area, the job uses it, just like if they change a src/ file and recompile.
Personally I lean towards just packing up the data area as we do now.
BTW, we do have a user-settable thing as well (I forgot to include that) for the odd file that they store that is not in a data area. We should keep this, whether it makes it into the first attempt or not.
The WMCore code will be packaged by the agent by default until we stabilise enough to build the runtime RPMs from it. Any extras that Crab needs can be included in that as well down the road.
Yup, agreed, which is why I put it in the "we don't need to do this category". Only there for completeness of what we do now. :-)
from crabserver.
evansde: Regarding config files:
ConfigCache can support python or pickled files, the interesting bit with the python code in the scram area is that you can decouple configs from a release and use the same one for several projects if you like. (Eg: the Conf/DataProcessing stuff makes the top level config look really lightweight for some of the standard things)
Eg: someone has a doReco.py config in configcache, but customises the event content in their scram area.
Or: Rerun myanalysis.py but with some cut changed in the scram stuff.
Probably not a first order kind of implementation, but may come in handy down the road...
The config cache stuff should make multicrab style operations on the server that bit easier, since you go same config, same sandbox over multiple datasets, and can add datasets after the initial submission if the interface supports it.
(Could also completely bypass the config cache and stuff the pickled config in the sandbox if needed, lots of freedom in the new system, probably need to pick a first pass and work from that)
from crabserver.
ewv: I'm making decent progress on this but before I get too far in, I want to make sure I'm not doing something too restrictive.
First, there is already a field in the Step for this. It's actually a list of sandboxes. Is there any reason it should be a list and not a single one? Does changing it mean a change to the underlying database?
Second, is there any reason that different steps in the job would need different sandboxes. Do we envision workflows like this?
from crabserver.
evansde: Replying to [comment:14 ewv]:
I'm making decent progress on this but before I get too far in, I want to make sure I'm not doing something too restrictive.
First, there is already a field in the Step for this. It's actually a list of sandboxes. Is there any reason it should be a list and not a single one? Does changing it mean a change to the underlying database?
Basically it can be a list of URLs or something like that. Could be reduced to a single file if needed.
No DB stores this based on just the spec information so its pretty free form.
Second, is there any reason that different steps in the job would need different sandboxes. Do we envision workflows like this?
No specific workflow in mind, just making it easy to be flexible. Eg: things like Madgraph etc seem to have a bunch of input sandboxes so it seemed prudent, but nothing concrete.
from crabserver.
ewv: Replying to [comment:15 evansde]:
Replying to [comment:14 ewv]:
I'm making decent progress on this but before I get too far in, I want to make sure I'm not doing something too restrictive.
First, there is already a field in the Step for this. It's actually a list of sandboxes. Is there any reason it should be a list and not a single one? Does changing it mean a change to the underlying database?
Basically it can be a list of URLs or something like that. Could be reduced to a single file if needed.
No DB stores this based on just the spec information so its pretty free form.
Ok, what I'll do for now is keep the structure as a list but keep the implementation as a single local file. Should we later need to expand this, it should be easier. If we're really going to do URLs (which makes sense for MadGraph probably since parts of it are centralized) then a untarUser.py script makes a lot more sense than my 5 lines of bash that I have now.
Second, is there any reason that different steps in the job would need different sandboxes. Do we envision workflows like this?
No specific workflow in mind, just making it easy to be flexible. Eg: things like Madgraph etc seem to have a bunch of input sandboxes so it seemed prudent, but nothing concrete.
OK. Different sand-boxes for different steps is also a minor perturbation for what I'm doing on the WN.
from crabserver.
ewv: Please review the patch. Multiple tarballs per step are not supported throughout, but some of the necessary code is in place.
from crabserver.
sfoulkes: I think it's better if the userSandbox parameter in StdBase defaulted to "None" so that we don't get a user sandbox for production jobs. Same goes in the Analysis Spec, if the user didn't pass in a sandbox we shouldn't be setting one for them.
I'm not sure that you're untarring the sandbox in the correct place. On the WN, CMSSW is run out of ./job/WMTaskSpace/stepName (I think), it looks like you're just untarring the sandbox in the base directory there.
from crabserver.
ewv: Yeah, definitely the default should be changed.
It is getting untarred in the right place. The job sandbox untars in job/ and the CWD at the time of the untar is CMSSW_x_y_z
from crabserver.
sfoulkes: Couple more issues:
- Fix the comment in WMCore/WMSpec/Steps/Templates/CMSSW.py for setUserSandbox()
- The code in WMCore/WMSpec/Steps/Executors/CMSSW.py in execute() will crash if a user sandbox isn't set.
You'll need to update your tree as well, Evans made some changes for the spec stuff to support multicore CMSSW and that caused your patch to not apply.
from crabserver.
ewv: Replying to [comment:21 sfoulkes]:
- The code in WMCore/WMSpec/Steps/Executors/CMSSW.py in execute() will crash if a user sandbox isn't set.
No, because the same code sets it as a default to [] and ','.join([]) is '' which is exactly what I want. And setUserSandbox(None) returns, doesn't actually replace the list with None
from crabserver.
ewv: Sorry formatting issue. Should read "is a blank string which is exactly..."
from crabserver.
sfoulkes: You're right, i'll apply the patch.
from crabserver.
ewv: BTW, if you want to enable this in the injectAnalysis script you can just
"userSandbox" : '/uscms/home/ewv/crab-test/CMSSW_3_6_1_patch7/default.tgz',
which is a CRAB-produced tarball. I didn't check in the injectAnalysis since I couldn't easily produce a clean patch with just that one change.
from crabserver.
Related Issues (20)
- #813: Test dev CRABClient using test2 REST instance and CMSSW_13_0_2 CMSSW release HOT 8
- checktaperecall - sometimes fails without clear cause HOT 4
- Improve PyPI images building process HOT 2
- #814: Test dev CRABClient using test2 REST instance and CMSSW_13_0_2 CMSSW release HOT 7
- RUCIO_Transfers should store also scope in filetransfersdb HOT 2
- rationalize configurations and common functions
- adapt Publisher_rucio to have rucio scope:name in tm_dbs_blockname HOT 2
- fix /opt/rucio/etc/rucio.cfg in pypi contaiiners
- [PyPI] Use local timezone for all images HOT 2
- avoid exiting container on command error HOT 4
- Change entrypoint of TW process to simple binary script
- keep tmp directory in TW only for 6 hours
- #815: Test prod CRABClient using test12 REST instance and CMSSW_13_0_2 CMSSW release HOT 7
- make text select via mouse for copy/paste work in pypi container
- stop extending TapeRecall rule when data is on disk HOT 3
- new DN for cmscrab service account
- FTS_Transfer does not handle external connection and bookkeeping properly, caused some files to be stale in the state "SUBMITTED" HOT 7
- incorrect handling of partial dataset
- Change entrypoint of Publisher process to simple binary script
- #816: Test prod CRABClient using preprod REST instance and CMSSW_13_0_2 CMSSW release HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crabserver.