View Code? Open in Web Editor NEW

A collection of tools for Data Mesh architectural approach. Data Contract validation against Open Data Contract Standards.

License: GNU General Public License v3.0

Shell 5.25% Python 94.75%

datamesh's Introduction

DataMesh

A collection of tools for Data Mesh architectural approach.

Data Contract Validation

The library can be used to validate your .yaml data contract against Open Data Contract Standard JSON schema from Bitol-io / open-data-contract-standard

Installation

pip install -U datamesh

Usage

data-contract-validation <path_to_contract_yaml_file> <optional_path_to_standards_json_schema>

or in Python

from datamesh.contract import Validator

validation_errors = (
    Validator(
        contract="examples/all/postgresql-adventureworks-contract.yaml",
        standard="schema/odcs-json-schema.json" # This is optional
    ) 
    .print_report() # Prints validation results and returns `self`
    .errors # List of errors 
)

🚩 Validation errors:
.
│
├──[]
│  └── 'datasetName' is a required property
│
├──[]
│  └── 'quantumName' is a required property
│
├──['kind']
│  └── 'managedDataset' is not one of ['DataContract']

can be used within your CI as

...
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install datamesh

      - name: Run Data Contract Validation on all examples
        run: |
          set -o pipefail
          failed_files=""
          for file in examples/all/*.yaml; do
            echo "Validating $file..."
            if ! data-contract-validation "$file"; then
              echo "Validation failed for $file"
              failed_files="$failed_files $file"
            fi
          done

          if [ -n "$failed_files" ]; then
            echo "Validation failed for the following files:$failed_files"
            exit 1
          else
            echo "All files validated successfully."
          fi
...