anudwigna / spark_project Goto Github PK

Submission By: Abhinandan Aryal & Kassem Bou Zeid

The assignment was to perform several analysis on NYC Taxi dataset using spark. For this purpose, GCP was used and a cluster was made to run the analysis.

The project can be cloned using the link provided in the clone option. After that following steps need to be taken to run the project for the linux environment as our environment was LINUX.

1. Download the CLI

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-441.0.0-linux-x86_64.tar.gz

2. Unzipping the File

tar -xf google-cloud-cli-441.0.0-linux-x86_64.tar.gz

3. Installing the CLI

./google-cloud-sdk/install.sh

4. Initialize the CLI

./google-cloud-sdk/bin/gcloud init

A cluster was created in GCP to run the analysis using the following command:

make create_cluster

The jobs are organized inside the src/jobs folder. Each job has its own folder. For running each job, we changed the job name in the .env file and used make to submit the job to GCP using the following command.

make submit_job

After each job is run, the results are stored in the bucket pyspark-tutorial-abhinandan-aryal. The results are stored as per the task number. The number in front of the results refer to the task number in the following order.

1. Trip Analysis
2. Tip Analysis
3. Fare Analysis
4. Traffic Analysis
5. Prediction Job

Multiple files are stored as results for each job to ensure the intermediate results as per necessity.