We are students from Singapore Institute of Technology and this is our project for ICT 1002, where we were tasked to scrape data from LinkedIn and conduct job sentiment analysis.
The objective of this project is to identify competencies that employers are looking out for. We scraped data from LinkedIn - one of the leading international recruiting platforms - and proceeded to process and analyze our data. The countries analyzed were Singapore, USA, China and Russia.
| Library | Function |
|---|---|
| Selenium | Data crawling from LinkedIn, populate dataset |
| Pandas | Data manipulation and analysis, manipulate numerical tables |
| NLTK | Natural language processing, working with human language data |
| Langdetect | Detecting language of data |
| PandasGUI | Viewing, plotting and analyzing Pandas DataFrames |
| Tkinter | Graphical user interface |
| Seaborn | High-level interface for drawing statistical graphs |
| Matplotlib | Utility library to assist Seaborn in creating graphs |
| Wordcloud | Creating visual representations of keywords |
| Pillow | Image processing |
View the detailed user guide here.
Users are able to crawl data, clean data, as well as open an interactive excel reader using the GUI.
- To crawl data, input the job, country, seniority level and number of data to be crawled. Do note that the maximum data to crawl is 1000.
- To view cleaned data, you can use the interactive excel reader, and import the excel data file.
- Here are the list of keywords our team has been provided to analyze in job descriptions.
# Dependent Competencies
['customer', 'team', 'partner', 'people', 'relationship', 'communication', 'support', 'contact', 'understanding', 'responsibility', 'care', 'group', 'communicate', 'staff', 'manner', 'help', 'follow', 'share', 'partner', 'support', 'home', 'assist', 'family', 'serve', 'consultant']
# Independent Competencies
['job', 'solution', 'operate', 'knowledge', 'comply', 'degree', 'legislation', 'technology', 'write', 'deliver', 'sell', 'learn', 'software', 'performance', 'project', 'service', 'healthcare', 'perform', 'compliance', 'emergency', 'risk', 'bachelor', 'issue', 'retail', 'conflict', 'accounting', 'forecast', 'negotiation', 'achieve', 'jurisdiction', 'quality', 'information', 'territory', 'training', 'report', 'tool', 'presentation', 'problem', 'success', 'implement', 'individual', 'engineering', 'order', 'result', 'negotiate', 'specialist', 'deal', 'operation', 'promote', 'study', 'qualification', 'program', 'self', 'execute', 'initiative', 'task', 'win', 'wealth', 'education']
- Detailed analysis can be found here
The installation guide can be found here.
- Completed!!!!
- Components:
Data Crawling - 27 SeptSelenium
Data Cleaning - 6 OctLang DetectNLTK
Data Analysis - 14 OctSeabornMatplotlib
GUI - 20 Oct
Files are split into 3 folders, src. images and venv. DO NOT ADD OR REMOVE FILES IN THE VENV FOLDER! Venv is installed by the user.
venv Folder (activate when coding)
|_ ...
images Folder (images)
|_ ...
src Folder
|_ controller Folder (all logic related code)
|_ __init__.py
|_ augmentor.py
|_ cleaner.py
|_ counter.py
|_ crawler.py
|_ merger.py
|_ processor.py
|_ models Folder (data objects)
|_ __init__.py
|_ jobs.py
|_ keywords.py
|_ keywordsLook.py
|_ data Folder
|_ augmentedData Folder
|_ (augmented data files...)
|_ mergedData Folder
|_ (merged data files...)
|_ cleanedData Folder
|_ (cleaned data files...)
|_ rawData Folder
|_ (raw data files...)
|_ keywords Folder
|_ (keywords data files...)
|_ views Folder (analysis and GUI)
|_ __init__.py
|_ graphicUI.py
|_ analysisPlots Folder
|_ (plots....)
|_ analysisWordCloud Folder
|_ word_cloud.py
|_ misc Folder (misc code, installer)
README.md
Installation.md
Analysis.md
requirements.txt (used to install modules in venv)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent