jinmanx / data-fair-g18 Goto Github PK

How do you think we have a catalog before all the contents?
Any suggestions on our web ui please tell me.

View this to get an idea of how we can use Mapbox and Plotly to display our results: https://towardsdatascience.com/how-to-create-an-attractive-bubble-map-5cf452c244e9

The two most common methods of topic analysis are topic modeling and topic classification. Topic modeling a text analysis technique that uses unsupervised machine learning to process text by topic or subject and cluster and categorize similar words and phrases, without the need to create defined tags (categories) ahead of time. With topic classification or topic extraction, on the other hand, you must pre-define the tags, use them to train a classification model, and the model will then categorize your texts (tweets) into the tags you set up.

There is a small change so we can use +GitRepo on Edina. (We cannot upload main branch)
Github changed the default branch name recently to use neutral term.

https://www.zdnet.com/article/github-to-replace-master-with-main-starting-next-month/

• Can we use Mapbox, OpenStreetMap or Google on our website? Not sure yet, Google API should be paid. I notice some example map can be interactive which is a good sign.
• Since we don't have accurate location (During my own data mining, I notice the locations are from the user account page which normally are the home address or fake address of users.), some boundary map is not so accurate. Maybe we can just use color to show difference with your recommended classification as legends. So I mean for location, we can focus on tweets and hashtags.
• I'm not so sure should we use other visulisations besides geomap, our data holder didn't tell so clearly about that.

Here is my suggestion for our next steps:

Based on the conflictlocation file from 23nd October (to start with):

Additional cleaning steps:

• use separator package to divide hashtags or words (e.g. #ArmeniaStopWar)
• Filter for 'news', 'media' , 'global', 'UK', 'US', 'politics', 'info', 'information' OR users more than 2500 followers
• Drop all rows with irrelevant hashtags - not case sensitive (top 10 hashtags such as work / management / fitness / Manage / Leadership / coaching / workplace /conflictmanagement / employees / unproductive / companyif applicable)
• Drop all rows with only one hashtag (conflict) --> wont give as any additional value/information

1. Create DataFrame with lat and on (latitude and longitude) values. Drop all rows without location. This can be achieved with Geocoding. For example, we can use GeoPy (check this out: https://towardsdatascience.com/geocode-with-python-161ec1e62b89 or https://geopy.readthedocs.io/en/latest/#nominatim)

2. Add sentiment value, hashtags, username and time for each row

3. Identify top tokenised words AND top hashtags used. Based on that decide which major event to concentrate on (lets say Armenia, Azerbaijan, Covid). Maybe: Segregate these in different DataFrames

4. After geocoding and cleaning, perform mapping with plotly:
   https://medium.com/analytics-vidhya/plotly-for-geomaps-bb75d1de189f

5. If we want to be really funky (and have time) then we can also do a real-time stream of #conflict tweets VS #peace tweets:
   https://medium.com/dev-genius/twitter-api-data-visualizations-with-pandas-and-plotly-1e9d0c00ab26

DOCUMENTATION FOR GEOPY: https://geopy.readthedocs.io/en/stable/#nominatim