nikhil-saxena / tweets_analysis_in_hadoop_using_flume Goto Github PK

The scripts will work fine given json text as input, they have been tested on the linux system, Apache Hadoop in pseudo-distributed mode. But they fail on avro formatted input (the format which Apache Flume uses to store data in HDFS). I figured out the solution which is provided by hadoop itself, i.e., to append "-inputformat org.apache.avro.mapred.AvroAsTextInputFormat" to the hadoop streaming statement and incude the various jar file. This would convert avro to json as required and feed the same to the mapper. But it didn't workout.

I was forced to enter the stopword list manually into the mapper, after numerous attempts of including the NLTK corpus into the Hadoop system had failed.

Currently, hand classified tweets are used. But the occurrence of the tweets with intent was observed to be very low. The keyword used to fetch the tweets were, ['where can','what can','what should','where should','go','i go','we go','buy', 'sell', 'rent', 'purchase', 'recommend', 'new', 'buying','selling', 'renting','want', 'wanted', 'need', 'needed'].