Giter Club home page Giter Club logo

marquezproject / marquez Goto Github PK

View Code? Open in Web Editor NEW
1.7K 47.0 302.0 46.3 MB

Collect, aggregate, and visualize a data ecosystem's metadata

Home Page: https://marquezproject.ai

License: Apache License 2.0

Java 75.97% Dockerfile 0.04% Shell 1.30% JavaScript 0.63% HTML 1.03% TypeScript 16.98% CSS 0.05% Python 3.42% Mustache 0.19% PLpgSQL 0.38%
data-lineage data-discovery data-governance data-provenance metadata-service data-dictionary marquez metadata data-ecosystem-metadata data-ops

marquez's Introduction

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez was released and open sourced by WeWork.

Badges

CircleCI codecov status Slack license Contributor Covenant maven docker Known Vulnerabilities CII Best Practices

Status

Marquez is an LF AI & Data Foundation Graduated project under active development, and we'd love your help!

Adopters

Want to be added? Send a pull request our way!

Try it!

Open in Gitpod

Quickstart

Marquez provides a simple way to collect and view dataset, job, and run metadata using OpenLineage. The easiest way to get up and running is with Docker. From the base of the Marquez repository, run:

MacOS and Linux users:

$ ./docker/up.sh

Windows users:

Before cloning Marquez, configure Git to check out files with Unix-style file endings:

$ git config --global core.autocrlf false

Verify that Bash and PostgreSQL have been installed and added to the PATH variable (Git Bash is recommended).

Start all services:

$ sh ./docker/up.sh

Tip: Use the --build flag to build images from source, and/or --seed to start Marquez with sample lineage metadata. For a more complete example using the sample metadata, please follow our quickstart guide.

Note: Port 5000 is now reserved for MacOS. If running locally on MacOS, you can run ./docker/up.sh --api-port 9000 to configure the API to listen on port 9000 instead. Keep in mind that you will need to update the URLs below with the appropriate port number.

WEB UI

You can open http://localhost:3000 to begin exploring the Marquez Web UI. The UI enables you to discover dependencies between jobs and the datasets they produce and consume via the lineage graph, view run metadata of current and previous job runs, and much more!

HTTP API

The Marquez HTTP API listens on port 5000 for all calls and port 5001 for the admin interface. The admin interface exposes helpful endpoints like /healthcheck and /metrics. To verify the HTTP API server is running and listening on localhost, browse to http://localhost:5001. To begin collecting lineage metadata as OpenLineage events, use the LineageAPI or an OpenLineage integration.

Note: By default, the HTTP API does not require any form of authentication or authorization.

GRAPHQL

To explore metadata via graphql, browse to http://localhost:5000/graphql-playground. The graphql endpoint is currently in beta and is located at http://localhost:5000/api/v1-beta/graphql.

Documentation

We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under docs/.

Note: To begin collecting metadata with Marquez, follow our quickstart guide. Below you will find the steps to get up and running from source.

Versions and OpenLineage Compatibility

Versions of Marquez are compatible with OpenLineage unless noted otherwise. We ensure backward compatibility with a newer version of Marquez by recording events with an older OpenLineage specification version. We strongly recommend understanding how the OpenLineage specification is versioned and published.

Marquez OpenLineage Status
UNRELEASED 2-0-2 CURRENT
0.49.0 2-0-2 RECOMMENDED
0.48.0 2-0-2 MAINTENANCE

Note: The openlineage-python and openlineage-java libraries will a higher version than the OpenLineage specification as they have different version requirements.

We currently maintain three categories of compatibility: CURRENT, RECOMMENDED, and MAINTENANCE. When a new version of Marquez is released, it's marked as RECOMMENDED, while the previous version enters MAINTENANCE mode (which gets bug fixes whenever possible). The unreleased version of Marquez is marked CURRENT and does not come with any guarantees, but is assumed to remain compatible with OpenLineage, although surprises happen and there maybe rare exceptions.

Modules

Marquez uses a multi-project structure and contains the following modules:

  • api: core API used to collect metadata
  • web: web UI used to view metadata
  • clients: clients that implement the HTTP API
  • chart: helm chart

Note: The integrations module was removed in 0.21.0, so please use an OpenLineage integration to collect lineage events easily.

Requirements

Note: To connect to your running PostgreSQL instance, you will need the standard psql tool.

Building

To build the entire project run:

./gradlew build

The executable can be found under api/build/libs/

Configuration

To run Marquez, you will have to define marquez.yml. The configuration file is passed to the application and used to specify your database connection. The configuration file creation steps are outlined below.

Step 1: Create Database

When creating your database using createdb, we recommend calling it marquez:

$ createdb marquez

Step 2: Create marquez.yml

With your database created, you can now copy marquez.example.yml:

$ cp marquez.example.yml marquez.yml

You will then need to set the following environment variables (we recommend adding them to your .bashrc): POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD. The environment variables override the equivalent option in the configuration file.

By default, Marquez uses the following ports:

  • TCP port 8080 is available for the HTTP API server.
  • TCP port 8081 is available for the admin interface.

Note: All of the configuration settings in marquez.yml can be specified either in the configuration file or in an environment variable.

Running the HTTP API Server

$ ./gradlew :api:runShadow

Marquez listens on port 8080 for all API calls and port 8081 for the admin interface. To verify the HTTP API server is running and listening on localhost, browse to http://localhost:8081. We encourage you to familiarize yourself with the data model and APIs of Marquez. To run the web UI, please follow the steps outlined here.

Note: By default, the HTTP API does not require any form of authentication or authorization.

Related Projects

  • OpenLineage: an open standard for metadata and lineage collection

Getting Involved

Contributing

See CONTRIBUTING.md for more details about how to contribute.

Reporting a Vulnerability

If you discover a vulnerability in the project, please open an issue and attach the "security" label.


SPDX-License-Identifier: Apache-2.0 Copyright 2018-2024 contributors to the Marquez project.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.