Mage is an open-source data pipeline tool for transforming and integrating data.

Here is a sample data pipeline defined across 3 files:

# data_loaders/load_data_from_file.py
@data_loader
def load_csv_from_file():
    return pd.read_csv('default_repo/titanic.csv')

# transformers/select_columns.py
@transformer
def select_columns_from_df(df, *args):
    return df[['Age', 'Fare', 'Survived']]

# data_exporters/export_to_file.py
@data_exporter
def export_titanic_data_to_disk(df) -> None:
    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

New? We recommend reading about blocks and learning from a hands-on tutorial.

1. Quick start
2. Demo
3. Tutorials
4. Features
5. Documentation
6. Core design principles
7. Core abstractions

Install Mage:

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \
  mageai/mageai mage start demo_project

_{Want to use Spark or other integrations? Read more about integrations.}

pip install mage-ai

or

conda install -c conda-forge mage-ai

_{For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.}

_{If you run into errors, please see Install errors.}

mage start demo_project

Open http://localhost:6789 in your browser and build a pipeline.

Try a hosted version of the tool here: http://demo.mage.ai.

WARNING

The live demo is public, please don’t save anything sensitive.

_{Click the image to play video}

• Train model on Titanic dataset
• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project

• Data pipeline management
• Notebook for building data pipelines

_{Read more here.}

Read more here.

Every user experience and technical design decision adheres to these principles.

1. Easy developer experience
2. Engineering best practices built-in
3. Data is a first-class citizen
4. Scaling made simple

_{Read more here.}

These are the fundamental concepts that Mage uses to operate.

• Project
• Pipeline
• Block
• Data product
• Trigger
• Run

_{Read more here.}

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.

Check out our FAQ page to find answers to some of our most asked questions.

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community:

For real-time news, fun memes, data engineering topics, and more, join us on:

• Twitter
• LinkedIn
• GitHub
• Slack

See the LICENSE file for licensing information.