Giter Club home page Giter Club logo

m2p-etl's Introduction

M2P-ETL

Mongo to Postgres ETL

Development workflow

Getting started

Prerequisites

  • Docker version 19.03.5, build 633a0ea
  • Python 3.8.2

Start mongo and postgres

Docker-Compose

docker-compose -f ./provisioning/docker-compose.yml up -d

Docker command

Mongo Database

docker pull mongo:4.2.2
docker run -d -p 27017:27017 -v /opt/mongo/data/configdb:/data/configdb -v /opt/mongo/data/db:/data/db --name mongodb mongo:4.2.2

Postgres Database

docker pull postgres:12.2
docker run -d -p 5432:5432 -v /opt/postgresqk/var/lib/postgresql/data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=VMware1! --name postgresql postgres:12.2

Pylint

  • source-code, bug and quality checker for the Python programming language
python lint.py --path ../M2P-ETL --threshold 9.5

Docker build

docker build -t [name]:[version-number] .

Requirements

Pylint

  • astroid==2.3.3
  • isort==4.3.21
  • lazy-object-proxy==1.4.3
  • mccabe==0.6.1
  • pylint==2.4.4
  • six==1.14.0
  • wrapt==1.11.2

M2P-ETL

  • numpy==1.18.1
  • pandas==1.0.1
  • psycopg2-binary==2.8.4
  • pymongo==3.10.1
  • python-dateutil==2.8.1
  • pytz==2019.3
  • six==1.14.0
  • SQLAlchemy==1.3.14

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.