The goal of this project is to automate the flow of data from a data lake into a data warehouse to power a dashboard which enables users to extract insights on the Green taxi trips made in NYC in 2022.
The data was downloaded from https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page in parquet format and loaded into Google Cloud Storage for access by the data pipeline.
The taxi trips table is about the Green taxi trips made in NYC and is enriched with generated data about taxi zones, payment types, rate codes and trip types from the provided data dictionary and dates by dbt.
- Terraform
- Docker
The following steps will help you spin up the required cloud infrastructure:
- Move into the Terraform directory
cd terraform
- Initiate Terraform and download the required dependencies
terraform init
- View the Terraform plan
terraform plan
- Apply the Terraform plan
terraform apply
- Move into Airflow directory
cd airflow
- Run the Docker container
docker compose up
- Inlclude CI/CD pipelines
- Include more data quality checks
- Add data visualizations to present insights from the data