agile-lab-dev / data-product-specification Goto Github PK

OpenMetadata has released a new version and it seems that they have introduced backward compatibility problems with the links used in our Readme.

The following link provided by the Readme does not work any more:
https://docs.open-metadata.org/openmetadata/schemas/entities/tagcategory
https://docs.open-metadata.org/openmetadata/schemas/entities/table#tabledata
https://docs.open-metadata.org/openmetadata/schemas/entities/table#tableprofile

Semantic Linking is a list of struct as per specification but in the example.yaml

Replace line 62:
semanticLinking: {}

with:
semanticLinking: []

May I suggest to maintain a json schema and possibly autogenerate the example.yaml (or simply check it during CI?) that would help users a lot!

if you agree I might work on it

At the moment, in the DP descriptor only the startDate of the dataset is available but not all dataset will have an infinite retention time: it's necessary to include a field that represents the time window data will be kept available in the dataset. The proposed solution is to add a RetentionTime field at StartDate level.

At the moment the section is representing more the swagger of the API itself, then the DP descriptor

Metadata introduced versioning to their documentation so the associated links on the README.md to the OpenMetadata docs are broken.

Semantic Linking is a metadata field of the Output Port that contains all the fields of the schema that can be joinable with other Output Ports' fields (even from other Data Products inside other domains).

Right now the Semantic Linking is a generic object, but it should be better defined with a basic structure.
The structure should define a way to reference external Output Ports' fields for some of the output port's fields.

output port section

As of 456b26d it is unclear to me what Storage Area > Type is for:

The description talks about workloads

• Type: [String] This is an enum [HouseKeeping|DataPipeline], Housekeeping is for all the workloads that are acting on internal data without any external dependency. DataPipeline instead is for workloads that are reading from outputport of other DP or external systems.

But a storage area is not a workload.

Can you please explain it to me @agile-lab

Define how the specification can be evolved and customized.
Set up a way to check compliance of the current version example with a structured definition.

In the data product specification the dataProductOwner field is set to be optional.
I expected it to be mandatory, since a data product must have an owner

Is there anything I am missing?

scenario summary

One of data as a product properties is bi-temporality in every output port

problem statement

being a property of every output port and considering it a foundational property that guarantees to consumers the possibility to time travel over the 2 temporal dimensions, this property should be explicitly managed as part of the output port’s data contract.

Also, it should enable automated checks as part of the computational federated governance.

proposed solution

Add in the data contract section 2 new String fields to be valorized with the output port’s corresponding technical and business temporal references, which should also be checked against the provided output port’s schema.

We should insert a field in the data sharing agreement that provides practical info to the consumer on how to consume the output port. For example:
"Download this JDBC driver here with version 10.0.x, and configure it with these parameters; you need to have a user and password as well, then you can verify your credentials into this URL, etc..."

The field could contain also links, so is possible to render them in a UI