Extract, transform, load using visual programming that can run Spark jobs on any environment

Create and debug from your browser and export into pure python code!

• Up and running as simple as docker pull

• Create complex pipelines and flows using drag and drop

• Debug and preview step by step

• Integrated dataview grid viewer for easier debugging

• Auto-generates comments so you don't have to

• Export to beautiful, pure python code

• Build artifacts for AWS Glue deployment (Work in progress)

$ docker pull zalmane/basin:latest

$ mkdir data This is the folder that will hold all input and output files

Run image mapping data directory to your local environment. This is where input/output goes (extract and load)

docker run --rm -d -v $PWD/data:/opt/basin/data --name basin_server -p 3000:3000 zalmane/basin:latest

That's it. Point your browser to http://localhost:3000 and you're done!

Notes:

• Metadata is stored in the browser's indexeddb.

docker-compose up

This will set up 2 containers: basin-client and basin-server

That's it. Point your browser to http://localhost:8860 and you're done!

To run npm commands in the basin-client container use:

docker exec basin-client npm <command>

To update changes in py files (block templates, lib), use:

docker exec basin-client npm run build-py

A source defines the information needed to parse and import a dataset. Sources are referenced when using an Extract block. The source defines the following information:

• type of file (delimited, fixed width, json, parquet)
• regular expression to match when identifying the file. This will match against the file name
• information about headers and footers
• specific metadata based on type of file (for csv includes the delimiter etc)

Each block type consists of:

• Descriptor json
• code template
• optional code library template
• Properties panel

This program is free software: you can redistribute it and/or modify it under the terms of the Server Side Public License, version 1, as published by MongoDB, Inc. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Server Side Public License for more details. You should have received a copy of the Server Side Public License along with this program. If not, see http://www.mongodb.com/licensing/server-side-public-license

Copyright © 2018-2020 G.M.M Ltd.