This BigData study intends to identify the most revenue-generating Taxi zones in New York City for the year 2019. Three MapReduce algorithms were developed and their performance was analyzed on different size of input datasets and different size clusters in EMR.

The zip file containing Java source code package is located at ~~\Package - s3400652\Java Source Code\Assignment.zip

Load this zip file to eclipse workspace.

• The dependancies (.pom) have correclty been set in the provided Java application package
• Follow the standard procedure to build standalone .jar file in eclipse-workspace through Maven build

[** - .jar files have been provided in the "Java Souce File" folder]

Drag and drop the .jar file to HDFS (location - Home/user/s3400652) through HUE web interface.

[** - destination location has to be correctly set]

hadoop fs -copyToLocal /user/s3400652/Assignment-0.0.1-SNAPSHOT_yellow.jar /home/hadoop/

hadoop fs -copyToLocal /user/s3400652/Assignment-0.0.1-SNAPSHOT_yellow_combiner.jar /home/hadoop/

hadoop fs -copyToLocal /user/s3400652/Assignment-0.0.1-SNAPSHOT_yellow_IMC.jar /home/hadoop/

[** - source location (/user/s3400652) has to be correctly set]

Experiment were conducted on NYC TLC Yellow Taxi Trip Data .csv for year 2019. [make sure the .csv column format is the same if older year files are used in for analysis]

hadoop distcp s3a://nyc-tlc/"trip data"/yellow_tripdata_2019-01.csv /user/s3400652/

hadoop distcp s3a://nyc-tlc/"trip data"/yellow_tripdata_2019-02.csv /user/s3400652/

hadoop distcp s3a://nyc-tlc/"trip data"/yellow_tripdata_2019-03.csv /user/s3400652/

hadoop distcp s3a://nyc-tlc/"trip data"/yellow_tripdata_2019-04.csv /user/s3400652/

hadoop distcp s3a://nyc-tlc/"trip data"/yellow_tripdata_2019-05.csv /user/s3400652/

hadoop distcp s3a://nyc-tlc/"trip data"/yellow_tripdata_2019-06.csv /user/s3400652/

[** - make sure destination "/user/s3400652/" is correctly set accordingly]

hadoop jar Assignment-0.0.1-SNAPSHOT_yellow.jar /user/s3400652/*.csv /user/s3400652/outputNYC_sixfile

hadoop jar Assignment-0.0.1-SNAPSHOT_yellow_combiner.jar /user/s3400652/*.csv /user/s3400652/outputNYC_sixfile_combiner

hadoop jar Assignment-0.0.1-SNAPSHOT_yellow_IMC.jar /user/s3400652/*.csv /user/s3400652/outputNYC_sixfile_IMC

[** - all the csv files in HDFC location "/user/s3400652/" have been taken as inputs; output files are saved at "/user/s3400652" under provided output folder name]

sh boost_cluster.sh

[** - this will increase the number of nodes in the EMR cluster by 2]

hadoop fs -cat /user/s3400652/outputNYC_sixfile_combiner/part-r-0000* | sort -g -k2 -r | head -n10

[** - output file path has to be correcty set; in above example the output files path is "/user/s3400652/outputNYC_sixfile_combiner"]