Comments (5)
vkuznet
commented on July 28, 2024
Andrea,
we should follow to whatever DBS schema is. And, what you found seems a typo, we
should have
StructField("processing_era_name", StringType(), True)
instead of
StructField("processing_version", StringType(), True)
Feel free to add PR to this issue.
V
…On 0, Andrea Sciaba ***@***.***> wrote:
In
https://github.com/dmwm/CMSSpark/blob/master/src/python/CMSSpark/schemas.py#L32
we have processing_version, but in
https://github.com/dmwm/CMSSpark/blob/7f72d51f1a9c58aa28483483cfbbd412264425e1/src/python/CMSSpark/schemas.py#L23
we have PROCESSING_ERA_NAME. It seems weird to me that the dataframe for the processing eras does not have the name, in fact. Is this just a typo?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#29
from cmsspark.
sciaba
commented on July 28, 2024
The only thing is that I checked and that field seems to have only numeric values (even if in principle they are strings), so it is more compatible with a version than with a name....
from cmsspark.
vkuznet
commented on July 28, 2024
I'm adding Yuyi to comment on this.
@yuyiguo , Yuyi could you please let us know what are the attributes of processing era table in DBS. Do we have PROCESSING_ERA_NAME in there or not and where processing version is stored.
from cmsspark.
yuyiguo
commented on July 28, 2024
@vkuznet @sciaba
Hi Valentin, Andrea,
Here is what in DBS schema:
PROCESSING_ERA_ID(NUMBER)
PROCESSING_VERSION(NUMBER)
CREATION_DATE(NUMBER)
**CREATE_BY(VARCHAR2)
DESCRIPTION(VARCHAR2)
from cmsspark.
vkuznet
commented on July 28, 2024
This is fixed in master head in this commit 4c7515b
Closing the ticket.
from cmsspark.
Related Issues (20)
- Last day of the month is not processed in bin/cron4dbs_condor HOT 4
- rucio_daily.py error due to wrong default date HOT 1
- Typo in date string HOT 1
- sparkexjar always empty HOT 23
- Evaluate to run yum update cern-hadoop-config in each Spark script
- Consider file pfn vs lfn in schemas
- --cvmfs option to run_spark de facto mandatory HOT 2
- Reading Avro files as Dataframes instead of RDD? HOT 5
- Make run_spark compatible with the ithdp-client cluster HOT 1
- Wrong type HOT 4
- Typo in variable name HOT 1
- dbs_events script missing header row in output HOT 2
- Corrupt timestamps in classad csvs HOT 1
- Authentication issues on it-hadoop-client HOT 8
- Simple setup for it-hadoop-client HOT 11
- Bug in run_spark? HOT 1
- Creating check functions for critic cron jobs HOT 1
- Main bash script for cron jobs and exposing cron metrics to push-gateway HOT 3
- Apply check util functions in cron jobs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmsspark.