scrapping job information in stackoverflow and indeed
-
run main.py
-
input what you want to query and where you want to work.
-
open jobs.csv and utilize scrapped information as you want.
-
get_full_URL(URL): combine base URL with query, location and option(limit in indeed, sort in SO) into one full URL.
-
get_page(URL): parse HTML to text.
-
get_last_page(page, website): scrap pagination and return the last page.
-
extract_jobs(URL, last_page, website): scrape jobcards.
-
extract_job(job_card, website): scrape title, company, location, link.
save_to_file(jobs): convert jobs into csv(comma seperated values) file