This project aims to analyze video game data to identify trends and insights. The data is extracted from RAWG using Python and stored in Google Cloud Storage (GCP). The data is then transformed using dbt Cloud and Mage. The entire process is orchestrated using Mage.

Data is extracted from the RAWG Video Games Database API. The API provides information on video games, including game title, genre, platform, release date, and more. The API documentation can be found here.

For the data dictionary, refer to the dbt documentation (Click Sources) here

1. Python Script using Mage

2. Google Cloud Storage (GCP)

1. dbt Core

1. Google BigQuery

1. Google Looker Studio

1. Mage

1. Terraform

1. GitHub Actions - to generate and host dbt documentation

1. End to end pipeline to extract, transform, and load video game data

   

2. dbt docs to view the data model and documentation

3. Dashboard using Looker Studio

1. Fork this repository
2. Create a codespace using the forked repository

3. Create a Google Cloud account and project.
4. Install Terraform
5. In Google Cloud Storage, create 2 buckets (one for historical data and one for the latest data).
6. Create a service account. Required access:
   1. Storage Admin
   2. BigQuery Admin
7. Download the service account key and save it as ./keys/gcp-creds.json

   [!IMPORTANT] Do not commit the service account key to the repository.

8. Install Terraform (follow the instructions here
9. Run the following commands to set up the GCP environment:

terraform init
terraform plan
terraform apply

10. Prepare config files and create directories:

bash script/00_repo_initial_setup.sh

11. Prepare the files .env

12. Start the docker containers:

docker-compose up -d

13. Open the mage application

14. Run the pipeline end_to_end_pipeline

15. Check the data in BigQuery

16. Clone the dashboard in Looker Studio and connect the data source to BigQuery