A Data Engineering project for visualizing the top skills demanded by roles based on the batch parsing of scraped LinkedIn job postings.

Creation of a fully orchestrated batch data pipeline and a dashboard app for interacting with the raw scraped LinkedIn job data in order to gain insights regarding in demand skills for jobs.

I am utilizing the following '1.3M Linkedin Jobs & Skills (2024)' dataset from Kaggle. https://www.kaggle.com/datasets/asaniczka/1-3m-linkedin-jobs-and-skills-2024/

The dataset is comprised of 3 csvs containing 1.3 million rows of scraped job data from LinkedIn from January 2024:

• linkedin_job_postings.csv
• job_skills.csv
• job_summary.csv

• Cloud - Google Cloud Platform
• Infrastructure as Code software - Terraform
• Containerization - Docker, Docker Compose
• Orchestration - Apache Airflow
• Transformation and Batch Processing - Apache Spark
• Data Lake - Google Cloud Storage
• Data Warehouse - BigQuery
• Data Visualization - Streamlit
• Dashboard App Deploymment - Cloud Run
• Language - Python

A live version of the dashboard app can be accessed at: https://streamlit-dashboard-ucsfwi2asq-uc.a.run.app/

See deploy.md

This has been a capstone project for the 2024 cohort of DataTalks.Club Data Engineering Zoomcamp. I am deeply grateful to Alexey Grigorev and the team for making this quality course available completely free and painstaklingly going through the effort of making it all possible.

Since the 2024 cohort of the Zoomcamp primarily utilized Mage for orchestration and I chose to go with Airflow for this project, I benefited a lot from these two projects for my Airflow deployment. So a thank you to Ankur Chavda and Yusuf Ganiyu

• https://github.com/ankurchavda/streamify
• https://github.com/airscholar/SparkingFlow

I would also like to thank Asaniczka of course for the Dataset that made this project possible.