In this repository, I will show how to analyze the geo-coded social media data posted in Hong Kong. The general procedure is the following:

1. Tweet filtering. For more information, please check the following two Jupyter notebooks:

   • Tweet filtering without visitors
   • Tweet filtering process for tweets posted by visitors

   The final dataset we use is the combined datafile derived from the above two Jupyter notebooks

2. Tweet text preprocessing

   • Please check the clean the text sample notebook

3. Generate tweet representation using FastText word embedding based on sentiment140

4. Manually label 5000 tweets randomly sampled from our tweet dataset

5. Sentiment analysis classifiers cross validation and sentiment prediction

6. Cross sectional analysis and longitudinal analysis

7. Result visualization(word cloud, topic modelling, etc)

In this project, I am using Python 3.5 to analyze the tweets. You could install all relevant packages by running the following code in the command line:

pip install -r requirements.txt

However, in the transit_non_transit_comparison folder, you need the ArcPy package to do the geographical analysis. This package is only supported in Python 2+ and could only be imported after downloading the ArcGIS.

To be continued.....

MIT