🌟 Welcome to "Skills-Compass"

Our mission is to guide you on the most in-demand technologies in Israel's tech job market. We help you stay relevant by providing the latest information on the skills you need for different tech roles.

🌍 What Makes Us Unique:

Every day, our system scans the latest job postings online, ensuring that our data is the freshest and most accurate, especially for jobs in Israel. This way, you get real-time insights into what employers are looking for.

🔮 What's Coming Next:

🚀 Exciting things are on the horizon! We're planning to add more job roles, highlight emerging and declining tech trends, and provide even more data to help you navigate the job market with confidence.

Website: Skills-Compass

Features
Technologies Used
Project Structure
How It Works?
Screenshots
Contact Information
License

🔰 Features

Automated daily scanning of job postings on the internet.
Text analysis and extraction of tech keywords from job descriptions.
Aggregation and categorization of data.
Visualization of in-demand skills categorized by tech roles.

👨‍💻 Technologies Used

Server: Django (hosted on Render)
Database: PostgreSQL (hosted on Neon)
Frontend: React Vite (hosted on Netlify)
Programming Language: Python 3.12.3
Packages Management: Poetry

🏗️ Project Structure

Django Apps:
- /core: Base models for the project database.
- /usage_stats: Advanced models for database usage statistics.
/init_db: Scripts for database initialization.
/logic: Business logic modules.
- /web_scraping: Logic and modules for data scraping.
- /text_analysis: Modules for analyzing job description texts.
- /data_processing: Logic for processing keywords and grouping them.
/templates: HTML templates used in the project.
/tmp/django_cache: Server cache storage.
/utils: Global configurations and settings.
- /mail_module: SMTP configuration for email.

❓ How It Works?

✅ Data Reliability

How do you avoid collecting data from the same listings on different sites?

We gather job listings from aggregator sites that collect postings from various sources. To ensure comprehensive coverage, we also check individual job sites directly. If a listing is already collected from another source, we skip it to avoid duplicates.

How do you avoid collecting the same listing every day?

Our system scans job listings every 24 hours, focusing on posts from the past day. This method ensures we don't collect the same listing on different days.

How do you handle duplication of listings within the same list?

We compare company names, locations, and job titles to identify and skip duplicate listings during our scanning process.

How do you filter out non-relevant listings?

We analyze job titles to ensure they match the desired role. For example, we filter out listings like "Full-stack Developer" in a search for "Backend Developer."

🔎 Text Analysis and Data Aggregation

How do you extract relevant words from a text?

We use a dataset of technological keywords to identify and extract relevant terms from job descriptions.

How do you handle different spellings and synonyms of the same technology?

We group synonyms and different spellings, like [node.js, node, nodeJs], into a single category during our data aggregation process.

How do you count the appearance of tech keywords in a single job description?

We use a Set data structure to count each technology's appearance uniquely, regardless of how many times it appears in the text.

How do you count the same keyword for different roles?

We maintain separate counts of each technology for each role. When scanning job listings, we specify the role related to those postings, ensuring the counts are associated with the correct role.

📸 Screenshots

Website: https://skills-compass.netlify.app/

Contact Information

LinkedIn:
GitHub:

📜 License

This project is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) License.

You are free to:

Share — copy and redistribute the material in any medium or format.

The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.
No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Notices:

You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.
No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

For more details, see the LICENSE file.

korenkaplan / dev-skill-compass-server Goto Github PK

dev-skill-compass-server's Introduction