In this repository, I will show how to analyze the geo-coded social media data posted in Hong Kong. The general procedure is the following:
-
Tweet filtering. For more information, please check the following two Jupyter notebooks:
The final dataset we use is the combined datafile derived from the above two Jupyter notebooks
-
Tweet text preprocessing
- Please check the clean the text sample notebook
-
Generate tweet representation using FastText word embedding based on sentiment140
-
Manually label 5000 tweets randomly sampled from our tweet dataset
-
Sentiment analysis classifiers cross validation and sentiment prediction
-
Cross sectional analysis and longitudinal analysis
-
Result visualization(word cloud, topic modelling, etc)
In this project, I am using Python 3.5 to analyze the tweets. You could install all relevant packages by running the following code in the command line:
pip install -r requirements.txt
However, in the transit_non_transit_comparison folder, you need the ArcPy package to do the geographical analysis. This package is only supported in Python 2+ and could only be imported after downloading the ArcGIS.
To be continued.....