Scraper hangs if page takes too long to render about images-scraper HOT 7 CLOSED

The other possibility is that ksb._kvc never appears on the page sometimes.
Definitely not this. I also had to bump up my time to 2 seconds, which seems to be working much better.

from images-scraper.

Thanks for finding this issue @ururk

I think it would be better to wait for page load. That can be achieved by doing this as part of the goto.

await page.goto(query.replace('%', encodeURIComponent(self.keyword)), {
  waitUntil: 'networkidle0'
});

from images-scraper.

Great! I'm still getting the same issue though - page loads but does not find any results. What I've found, is every time this happens, the meta JSON dos not load on the page (no .rg_meta element) nor are there any .rg_l elements. I see some errors in the web browser console:

Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'report-sample' 'nonce-5fBQzWdGZVpdbsJiddVttw' 'unsafe-inline'". Note that 'unsafe-inline' is ignored if either a hash or nonce value is present in the source list.

addScriptContent @ VM52 __puppeteer_evaluation_script__:7

No network errors, though, so I'm at a slight loss as to why this is happening. I even tried updating to the latest puppeteer (which includes a newer Chromium) and am still having problems.

from images-scraper.

Thanks for investigating @ururk . I can't seem to reproduce it

The error you see might be resolved by doing this: https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pagesetbypasscspenabled

Can you please try? Thanks, I'm reopening this issue.

from images-scraper.

I managed to reproduce it @ururk . I think this fixes it. I'm pushing it to a new version shortly.

from images-scraper.

This appears to fix it - I'll run some test searches and let you know how it goes.

from images-scraper.

I've done about 50 searches - this fixes the problem! I still have random Chromium crashes, but I don't think it's related to this node module's code.

from images-scraper.