bereketkibru / data_engineering_sensor_data Goto Github PK

A fully dockerized ELT pipeline project, using MYSQL, dbt, Apache Airflow, and Redash.
Explore the docs »

View Demo · Report Bug · Request Feature

Using a docker-compose file, developed a completely dockerized ELT pipeline with MySQL for data storage, Airflow for automation and orchestration, DBT for data transformation, and a Redash dashboard connected to the MySQL database.

Tech Stack used in this project

• MYSQL
• Apache Airflow
• dbt
• Redash

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

1. Clone the repo

   git clone https://github.com/bereketkibru/Data_engineering_sensor_data

2. Run

    docker-compose build
    docker-compose up

3. Open Airflow web browser

   Navigate to `http://localhost:8000/` on the browser
   activate and trigger dbt_load_dag
   activate and trigger dbt_dbt_dag

4. Access redash dashboard

   Navigate to `http://localhost:5000/` on the browser

5. Access your mysql database using adminar

   Navigate to `http://localhost:8080/` on the browser
   choose mysql databse
   use `root` for username
   use `pssd` for password

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Distributed under the MIT License. See LICENSE for more information.

Bereket Kibru - @email - [email protected]

• 10 Academy