Jupyter Notebook file from the Medium post: Selenium Tutorial: Scraping Glassdoor.com
arapfaik / scraping-glassdoor-selenium Goto Github PK
View Code? Open in Web Editor NEWJupyter Notebook from Selenium Tutorial: Scraping Glassdoor.com"
Jupyter Notebook from Selenium Tutorial: Scraping Glassdoor.com"
Jupyter Notebook file from the Medium post: Selenium Tutorial: Scraping Glassdoor.com
The code given in this repository to close the sign window seems to expire,since changes are made in html script . S, Can you provide code to close the pop- up sign up window automatically.
Hi, Thanks.
Im learning data science, I needed to scrape the glass door website but seems to fail when the cookies (Accept Cookies) pop up.
Please help if possible.
Hi,
It seems that for your code, when selenium clicks the next page button. It just re scrape everything from the beginning, and did not collect the new jobs from the new page.
The error comes from the line :
driver.execute_script("arguments[0].click();", job_button)
Note : I used this instead of your original code : #job_button.click() #You might
But now as I iterate over 500 or more , I am getting StaleElementReferenceException error,
Traceback (most recent call last):
File "C:\Users\suy\Documents\Github Backup\Glassdoor_Salary_Project\data_collection.py", line 15, in
dataframe = sc.get_jobs("data scientist",frame,False,path,2)
File "C:\Users\suy\Documents\Github Backup\Glassdoor_Salary_Project\glassdoor_scrapper.py", line 63, in get_jobs
driver.execute_script("arguments[0].click();", job_button)
File "C:\Users\suy.conda\envs\nlp\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 636, in execute_script
'args': converted_args})['value']
File "C:\Users\suy.conda\envs\nlp\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\suy.conda\envs\nlp\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
StaleElementReferenceException: stale element reference: element is not attached to the page document
(Session info: chrome=83.0.4103.116)
Please let me know if you know this error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.