Failed to click recent button about googlemaps-scraper HOT 22 CLOSED

Hi @kmangalorekar ,i am getting the same issue while executing the script,have you get any solution

from googlemaps-scraper.

Hey @Azim-pix You could try using my fork until this issue is fixed.

from googlemaps-scraper.

hi @venomouscyanide I have tried your fork, but after the chrome window appears, the link that has been inputted on url.txt cannot be opened. on chrome it says no internet even though there are no problems on my internet

from googlemaps-scraper.

@ryanramadhanii The main motivation for me to fork this was to add a proxy module. The error you are getting is because of the lack of proxy credentials you have provided in proxy_setup.py

In case you want to use the scraper without proxy please use the branch remove_proxy and run the program as is.

from googlemaps-scraper.

after i use your branch remove_proxy @venomouscyanide , the log shows like this

2020-02-29 12:59:15,419 - INFO - Closing chromedriver...
2020-02-29 12:59:15,419 - INFO - Closing chromedriver...
2020-02-29 12:59:15,419 - INFO - Closing chromedriver...
2020-02-29 12:59:15,419 - INFO - Closing chromedriver...
2020-02-29 12:59:15,419 - INFO - Closing chromedriver...
2020-02-29 12:59:15,419 - INFO - Closing chromedriver...

can you help me explain why it is like this?

from googlemaps-scraper.

Hi @venomouscyanide i am getting the same in log after using remove_proxy

here is the exception and log

Traceback (most recent call last):
File "scraper.py", line 26, in
scraper.get_reviews(url, index)
File "/var/www/html/google_azim/googlemaps-scraper/googlemaps.py", line 66, in get_reviews
menu_bt = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@Data-Value='Sort']')))
File "/home/azim/.local/lib/python3.6/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

2020-02-28 23:28:31,939 - INFO - Closing chromedriver...
2020-02-28 23:31:11,910 - INFO - Closing chromedriver...
2020-02-28 23:32:11,452 - INFO - Closing chromedriver...
2020-02-28 23:34:08,500 - INFO - Closing chromedriver...
2020-02-28 23:37:31,977 - INFO - Closing chromedriver...
2020-02-29 11:47:15,114 - INFO - Closing chromedriver...

from googlemaps-scraper.

@Azim-pix Can you give me an example of the URL you are trying to run?
Because for the included URL's in urls.txt the scraper is working. Or are you having issues with the included URL's as well?

from googlemaps-scraper.

@venomouscyanide in your remove_proxy branch proxy code is not commented ,i commented it and ran successfully
Thanks very much

from googlemaps-scraper.

Closing this issue, please use @venomouscyanide 's remove proxy branch, it works successfully with any URLs :

https://github.com/venomouscyanide/googlemaps-scraper/tree/remove_proxy

Thanks!

from googlemaps-scraper.

i've used @venomouscyanide remove proxy branch, but still does not work well, the log still shows as below (use included URL's in urls.txt)

2020-02-29 16:31:34,501 - INFO - Closing chromedriver...
2020-02-29 16:31:34,501 - INFO - Closing chromedriver...
2020-02-29 16:31:34,501 - INFO - Closing chromedriver...
2020-02-29 16:31:34,501 - INFO - Closing chromedriver...

can anyone help me? the chrome I use is version 80, does this matter?

from googlemaps-scraper.

Hi,
sorry for the delay in the answer.
@kmangalorekar I noticed that the click to get the recent button sometimes failed for a reason still unknown to me, so I added the control you highlighted. If the click always fails, it is probably due to a change in the name of the button, which is never found. Anyway, I tested it locally using the master branch and it works.

@ryanramadhanii I also updated my chromedriver version to 80 and it works: have a try and let me know.

from googlemaps-scraper.

hi @gaspa93 and @venomouscyanide I downloaded the @venomouscyanide repository and updated my chromedriver to version 80 but I still get the error:
2020-03-20 16:27:19,162 - INFO - Closing chromedriver...

in the console I get the following:

DevTools listening on ws://127.0.0.1:52445/devtools/browser/0846cd9d-f272-4b2b-ad25-6bbe8e4a8db5
Traceback (most recent call last):
File "C:\Users\Alberto\AppData\Local\Programs\Python\Python37\Scripts\scraper.py", line 26, in
scraper.get_reviews(url, index)
File "C:\Users\Alberto\AppData\Local\Programs\Python\Python37\Scripts\googlemaps.py", line 66, in get_reviews
menu_bt = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@Data-Value='Sort']')))
File "C:\Users\Alberto\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Any solution?

Greetings.

from googlemaps-scraper.

Hi @tobeal ,

this error means that the scraper cannot find the element specified by XPath expression //button[@Data-Value='Sort']. If you run the driver without graphical interface (headless, which is the default behaviour), the elements names may change. The line of code should be substituted by:
menu_bt = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'div.cYrDcjyGO77__container')))

If you check master code, it should be already like that, while the other expression is commented.

Let me know if changing the line of code solves the problem.

from googlemaps-scraper.

Just for completeness, if you run the same code but without the --headless flag, you should not get the error either.

It is an improvement that I still have to implement: automatically define different expressions if the driver is headless or not.

Cheers,

Mattia

from googlemaps-scraper.

Hi @gaspa93 I´ve tried to change the code but the same mistake appears:
DevTools listening on ws://127.0.0.1:64696/devtools/browser/cf8f7722-420f-4d60-8ff5-9e469bc595a6
Traceback (most recent call last):
File "C:\Users\Alberto\AppData\Local\Programs\Python\Python37\Scripts\scraper.py", line 26, in
scraper.get_reviews(url, index)
File "C:\Users\Alberto\AppData\Local\Programs\Python\Python37\Scripts\googlemaps.py", line 66, in get_reviews
menu_bt = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'div.cYrDcjyGO77__container')))
File "C:\Users\Alberto\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Respect your other solution i don´t know what you mean with headless flag, I am a begginer on this sorry.
If you can give me a solution i will aprecciate it so much.
Cheers,
Alberto

from googlemaps-scraper.

Ok, I am not sure about what version of the software you are using: could you please try to download the latest version of my repository?
And, if it does not work, then can you describe your configuration (OS, python version, Chromedriver version,...)?

Thank you,

Mattia

from googlemaps-scraper.

I am trying to use the next url ,with the next command 'scraper.py --N 50':
https://www.google.es/maps/place/Alcazaba+de+Badajoz/@38.8838882,-6.9704566,17z/data=!4m7!3m6!1s0xd16e435eeb32fd5:0x9ee731b4e09cdf91!8m2!3d38.8838882!4d-6.9682679!9m1!1b1

Version:
Google chrome: 80.0.3987.149
Python: 3.7.7
ChromeDriver: 80.0.3987.106
Console:
[0320/233845.705:INFO:CONSOLE(0)] "A cookie associated with a cross-site resource at http://google.com/ was set without the SameSite attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with SameSite=None and Secure. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.", source: https://www.google.es/maps/place/Alcazaba+de+Badajoz/@38.8838882,-6.9704566,17z/data=!4m7!3m6!1s0xd16e435eeb32fd5:0x9ee731b4e09cdf91!8m2!3d38.8838882!4d-6.9682679!9m1!1b1 (0)

GM-Scraper.txt:
2020-03-20 23:40:25,638 - WARNING - Failed to click recent button

from googlemaps-scraper.

Ok, I tested your link with my latest version and it is working right now.
I just noticed that you said you used the @venomouscyanide repository: I don't know what could be the problem there, try with my repo latest update and let me know.

from googlemaps-scraper.

Hi Mattia ,I am working on your last version.
I tried to use my other computer with the same software versions I mentioned but I get the same error in gm-scraper.txt:
2020-03-21 14:17:48,786 - WARNING - Failed to click recent button

Nothing is being written on the csv when you run it, it keeps the information you wrote yourselfCould you please tell me which versions you work on?
I hope we can solve it.
Thank you soo much for the help.

from googlemaps-scraper.

After 1 hour of testing and reviewing I found the error, when I run in debug mode Google was putting the page in Spanish so the 'sort' button in your code needed to be translated, once changed everything worked perfectly.
Thank you very much for your attention Mattia, sorry to be so clumsy hahaha.
Now my final project is on track!

from googlemaps-scraper.

@tobeal now it is clear: my machine was running in either English or Italian, so I would never detect such error.
But this feedback let me think how to generalise: I set a flag at the beginning to force English for chromedriver, but probably the browser page is "overwritten" by the .es at the end of url. I will have some test before and update the code before closing the issue.

I am glad that now you can work on your project: let me know if you find some other problems.

Thanks,

Mattia

from googlemaps-scraper.

latest commit 5b0922d on master branch should fix the problem: forced en_GB as only accepted language

from googlemaps-scraper.