Comments (7)
T0 uses the same database access code as WMCore. With convert_unicode deprecated, how is this done now? Are we silently using unicode for any string return from a database call now and we only notice in the Tier0 since it passed some of these to CMSSW?
from t0.
As I understand, the convert_unicode was deprecated in the latest SQLAlchemy upgrade in April
cms-sw/cmsdist@2187529#diff-c359e72e06d238c2f7e059ec6f8c3c3cL1
Therefore, all old T0 releases used byte strings returned from database call and everything worked well (since the old SQLAlchemy release was used).
Only starting from the new Tier0 release (the one currently being tested) we are using unicode strings from any database call.
And yes, we only noticed that in a release test replay as some of these unicode strings got passed to CMSSW and caused all the jobs to fail.
Overall, I am wondering whether we should review these database calls more carefully or should we just go with the fix I am testing now, if it works without errors?
from t0.
I think converting every single DAO where we pull strings out of a database individually is the wrong approach. Maybe some central place would be better. I would coordinate this with @amaltaro , since this really is a more general WMCore problem.
from t0.
I think the generic fix is what DBFormatter format functions provide. Since you have to format every single DAO output, you just call one of those methods.
We might still have to improve something there - like casting the key as well, if needed - but I'd rather have this conversion only done where we need; I fear having that conversion for every single DAO in the fetch* method, for instance, might degrade the agent performance by a non-negligible factor.
from t0.
I agree with Alan that the DBFormatter is a more central/generic place for these checks.
I adjusted my changes accordingly, to use the formatDict method to convert results:
master...vytjan:deployment-updates
It seems the GetStreamDatasetTriggers DAO is the only one where we hit this unicode string incompatibility.
Maybe it could be enough to apply the patch I created and leave all the other DAOs as they are? @amaltaro Alan, what do you think?
In such case we wouldn't need to convert every single DAO result of unicode->byte string as Python handles them fine and there seems to be no other issues with T0 WMAgent of this kind.
from t0.
Your change looks fine to me, Vytas.
from t0.
Thanks Alan. Tested and merged #4503
from t0.
Related Issues (20)
- Cleanup script needs to check if run is active or not HOT 11
- change the wmbs_location_senames table HOT 2
- Report Repacked Files For Unclosed Runs HOT 1
- fix Express AlcaSkim task to use single core HOT 1
- Run tests in CI
- Where is the real Tier0 config? HOT 6
- Create the change for Tier0 on PNN, PSN mapping. HOT 2
- Consider throwing in case of duplicate data set definitions in Tier0 config
- Overriding SoftTimeout and GracePeriod ? HOT 25
- Fileset closing is not waiting for the data in the run_stream_fileset to be complete HOT 6
- Versioning T0 replay runs when the same run is used more than once HOT 9
- Condition_t is not working properly HOT 1
- Tier0 and Rucio Subscriptions HOT 3
- Open files/urls with context manager HOT 3
- Update the T0 deployment script with the Rucio project metadata value HOT 1
- Replay test with 12_1_0_pre3 HOT 10
- Post description of Replay PR to HN
- Clean up Heavy Ion Config files HOT 2
- version #100 is not clear HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from t0.