This repository is a mirror of the Datahub repo containing our modifications to support single-click data products in Superset. The following changes were made:

• An additional frontend componenent was created: datahub-web-react/src/app/entity/shared/containers/profile/sidebar/SidebarVisualizeWithSupersetSection.tsx
• The frontend component was added to datahub-web-react/src/app/entity/dataset/DatasetEntity.tsx
• An additional GraphQL mutation was added to datahub-graphql-core/src/main/resources/entity.graphql and datahub-web-react/src/graphql/mutations.graphql
• A resolver for the GraphQL mutation was added to datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/mutate/VisualizeWithSupersetResolver.java and registered in datahub-web-react/src/graphql/mutations.graphql

For this hackathon, we deployed services using Kubernetes. The following components should be present in order to run the hackathon submission:

• Kafka Zookeeper (we used 1)
• Kafka Brokers (we used 3)
• Confluent Schema Registry
• Elasticsearch
• Trino, with an additional catalog registered:

  additionalCatalogs:
    postgres: |
      connector.name=postgresql
      connection-url=jdbc:postgresql://postgres:5432/postgres
      connection-user=datahub
      connection-password=datahub
  

• Apache Superset
• Redis
• Postgres

The values.yaml from acryldata/datahub-helm was modified to add the following environment variables to datahub-gms:

    - name: SUPERSET_PASSWORD
      value: "admin"
    - name: SUPERSET_USERNAME 
      value: "admin"
    - name: SUPERSET_ENDPOINT
      value: "http://superset:8088"
    - name: SUPERSET_EXTERNAL_ENDPOINT
      value: "http://localhost:8088"
    - name: SUPERSET_PROVIDER
      value: "db"
    - name: DATASOURCE_PASSWORD
      value: ""
    - name: DATASOURCE_USERNAME 
      value: "trino"
    - name: DATASOURCE_HOST_AND_PORT
      value: "trino:8080"

1. Once the demo is deployed, download AIS_2020_01_01-first-10000.csv.
2. Copy the file into the Postgres Pod (or port-forward locally and run the following command using psql)
3. Create a file with the following SQL statements:

   CREATE TABLE ais_noaa_2020
   (mmsi varchar,
   basedatetime timestamp ,
   lat numeric(10, 6),
   lon numeric(10, 6),
   sog numeric(8, 2),
   cog numeric(8, 2),
   heading numeric(6, 0),
   vesselname varchar,
   imo varchar,
   callsign varchar,
   vesseltype numeric(6, 0),
   status numeric(6, 0),
   length numeric(6, 2),
   width numeric(6, 2),
   draft numeric(6, 2),
   cargo numeric(6, 0),
   transceiverclass varchar);
   
   copy ais_noaa_2020
   FROM '/tmp/AIS_2020_01_01-first-10000.csv'
   DELIMITER ','
   CSV HEADER;

4. On the pod or locally, run:

   psql -U postgres -d postgres -W -f <path-to-sql-create>
   

5. In Datahub, create the Trino ingestion source:

   source:
     type: trino
     config:
       host_port: 'trino:8080'
       username: trino
       database: postgres
       # Supported environment values: https://datahubproject.io/docs/graphql/enums/#fabrictype
       env: DEV # Change if using a different environment.  
       # Enables profiling of SQL tables (row count, column count, etc.)
       # More information about profile: https://datahubproject.io/docs/metadata-ingestion/docs/dev_guides/sql_profiles
       profiling:
         enabled: true
         include_field_median_value: false
   
   sink:
     type: datahub-rest
     config:
       server: 'http://datahub-gms:8080'

6. Run the ingestion.
7. Once it completes, port-forward Superset.
8. Navigate to the ais_noaa_2020 dataset in Datahub. Click the Visualize with Superset button. You should momentarily be redirected to Superset's SQL Editor page, with a pre-populated query to the dataset.

Quickstart | Documentation | Features | Roadmap | Adoption | Demo | Town Hall

📣 Next DataHub town hall meeting on Feb 25th, 9am-10am PDT (convert to your local time)

• Topic Proposals: submit here
• Signup to get a calendar invite: here
• Town-hall Zoom link: zoom.datahubproject.io
• Meeting details & past recordings

✨ Latest Update:

• Read our Monthly Project Updates here.
• Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data: Data Engineering Podcast
• Unleashing Excellent DataOps with LinkedIn DataHub: DataOps Unleashed Talk.
• Latest blog post DataHub: Popular Metadata Architectures Explained @ LinkedIn Engineering Blog.
• We're on Slack now! Ask questions and keep up with the latest announcements.

DataHub is an open-source metadata platform for the modern data stack. Read about the architectures of different metadata systems and why DataHub excels here. Also read our LinkedIn Engineering blog post, check out our Strata presentation and watch our Crunch Conference Talk. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented.

Please follow the DataHub Quickstart Guide to get a copy of DataHub up & running locally using Docker. As the guide assumes some basic knowledge of Docker, we'd recommend you to go through the "Hello World" example of A Docker Tutorial for Beginners if Docker is completely foreign to you.

If you're looking to build & modify datahub please take a look at our Development Guide.

There's a hosted demo environment where you can play around with DataHub before installing.

• datahub-project/datahub: This repository contains the complete source code for DataHub's metadata model, metadata services, integration connectors and the web application.

We have documentation available at https://datahubproject.io/docs/.

See Releases page for more details. We follow the SemVer Specification when versioning the releases and adopt the Keep a Changelog convention for the changelog format.

Check out DataHub's Features & Roadmap.

We welcome contributions from the community. Please refer to our Contributing Guidelines for more details. We also have a contrib directory for incubating experimental features.

If you need to understand how to extend our model with custom types, please see Extending the Metadata Model

Join our slack workspace for discussions and important announcements. You can also find out more about our upcoming town hall meetings and view past recordings.

Here are the companies that have officially adopted DataHub. Please feel free to add yours to the list if we missed it.

• Adevinta
• Banksalad
• Cabify
• DefinedCrowd
• DFDS
• Expedia Group
• Experius
• Geotab
• Grofers
• Haibo Technology
• hipages
• IOMED
• Klarna
• LinkedIn
• Moloco
• Peloton
• Saxo Bank
• Stash
• Shanghai HuaRui Bank
• ThoughtWorks
• TypeForm
• Uphold
• Viasat
• Wolt
• Zynga

• DataHub Blog
• DataHub YouTube Channel
• Saxo Bank: Enabling Data Discovery in Data Mesh
• Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data: Data Engineering Podcast
• DataHub: Popular Metadata Architectures Explained
• Driving DataOps Culture with LinkedIn DataHub @ DataOps Unleashed 2021
• The evolution of metadata: LinkedIn’s story @ Strata Data Conference 2019
• Journey of metadata at LinkedIn @ Crunch Data Conference 2019
• DataHub Journey with Expedia Group
• Data Discoverability at SpotHero
• Data Catalogue — Knowing your data
• DataHub: A Generalized Metadata Search & Discovery Tool
• Open sourcing DataHub: LinkedIn’s metadata search and discovery platform
• Emerging Architectures for Modern Data Infrastructure

See the full list here.

Apache License 2.0.