Comments (9)
WMStats can't distinguish between different versions of the same workflow as long as the workflow is named exactly the same. Correct @amaltaro ?
But we have freedom to name workflows whatever we want in the Tier0. You could include a version number of string of sorts in the workflow name. Agent vobox hostname doesn't really work by itself, but how about vobox hostname plus processing version for the replay? We increment the latter anyways when we do new replays on the same agent/vobox. Should be easy to change the python in the config to force this to be consistent.
If you plan on running the same replay with the same processing version multiple times, you would have to invent some unique identifier/counter. But that still doesn't help you to keep track of things since you still need to be able which replay used which identifier. So might as well use the processing version, which has the useful side effect of also separating the output data of the replay.
One thing though, you likely want this to be optional since I don't think we want this for the production Tier0, correct?
from t0.
I initially thought about a random number, just to make it a unique workflow and able to properly use wmstats functionality.
However, as Dirk points out, we might want to have a logic where it's easy to identify the workflow names created. How about we make it:
normal workflow name + short hostname (not fqdn) + timestamp with date and time (same as we have for production)
? This should be enough.
Then as we discussed as well, it should be a configurable parameter for the Tier0Feeder which you only enable when you're running a replay or so.
from t0.
Alan, I would go with Dirk's suggestion to add the agent vobox hostname (or even the processing era, which is e.g. Tier0_REPLAY_vocms015) + processing version of the replay to the workflow names.
As I checked the Tier0Config.py and RunConfigAPI.py, an easy way could be adding an optional 'isReplay' parameter to ReplayOfflineConfiguration.py.
If it is defined, then the workflow name would be changed from
Express_Run326607_StreamHIExpressAlignment
to
Express_Run326607_StreamHIExpressAlignment_Tier0_REPLAY_vocms015_v212.
Dirk, Alan, would such naming make sense? We are identifying every replay with the processing version anyway. Or we could add just the processing version.
from t0.
If we create an isReplay parameter, would we want to use it to configure other settings too? There are a few parameters we just toggle between two settings, one for production and one for replays...
Otherwise I would just leave this at useUniqueWorkflowName or something like that, which tells you exactly what the parameter actually does.
from t0.
Thanks for your feedback Dirk. Indeed, since this parameter will only be used for naming workflows, then we would like to have it as specific as possible. Then I agree something like useUniqueWorkflowName will work.
I am going to create a patch for this and test it.
from t0.
So I added a new Global T0 configuration parameter UseUniqueWorkflowName.
The workflow name is formed in such manner:
Express_Run322057_StreamExpressAlignment_vocms047_v270
I implemented it following other global parameters setup and already ran a simple test replay on vocms047:
vytjan/T0@master...vytjan:versioning-workflows
Although, not sure whether the UseUniqueWorkflowName should be used to set the hostname of the VM or it should simply contain a boolean value in the replay configuration.
from t0.
Just an idea, how about setting that config somewhere here:
https://github.com/vytjan/T0/blob/ecf29162c776a4eecb5e87c94a0e2dcea70455ee/src/python/T0/RunConfig/Tier0Config.py#L247
and then evaluate it at Tier0Feeder before building the workflow name?
The hostname is defined in the WMAgent configuration file (or socket.gethostname() for instance).
In addition to that, it might be worth it adding the date, just in case someone run the two replays with the same settings, something like
currentTime = time.strftime('%y%m%d_%H%M%S', time.localtime(time.time()))
from t0.
I changed UseUniqueWorkflowName to EnableUniqueWorkflowName boolean parameter.
Also, added the acquisition era string to the workflow name as well as the timestamp. Thus, now the workflow name in replays is formed (excluding seconds in the timestamp on purpose):
PromptReco_Run322057_JetHT_Tier0_REPLAY_vocms047_v271_190211_1814.
vytjan/T0@master...vytjan:versioning-workflows#diff-798a0fd2fcb838e49ef7b0b833411e1aR558
Now it makes more sense and the workflow names are "enough" unique/separable. At the same time, the longest WF name is ~85 characters, so there is still enough until we reach 100:
Express_Run322057_StreamALCALUMIPIXELSEXPRESS_Tier0_REPLAY_vocms0313_v271_190211_1802
from t0.
from t0.
Related Issues (20)
- add more information in couchdb HOT 3
- prevent Repack/Express workflows from being cleaned up before SM notification HOT 1
- Cleanup script needs to check if run is active or not HOT 11
- change the wmbs_location_senames table HOT 2
- Report Repacked Files For Unclosed Runs HOT 1
- fix Express AlcaSkim task to use single core HOT 1
- Run tests in CI
- Where is the real Tier0 config? HOT 6
- Create the change for Tier0 on PNN, PSN mapping. HOT 2
- Consider throwing in case of duplicate data set definitions in Tier0 config
- Overriding SoftTimeout and GracePeriod ? HOT 25
- Fileset closing is not waiting for the data in the run_stream_fileset to be complete HOT 6
- Condition_t is not working properly HOT 1
- Tier0 and Rucio Subscriptions HOT 3
- SQL queries return unicode strings while CMSSW only accepts byte strings HOT 7
- Open files/urls with context manager HOT 3
- Update the T0 deployment script with the Rucio project metadata value HOT 1
- Replay test with 12_1_0_pre3 HOT 10
- Post description of Replay PR to HN
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from t0.