http headers about livescrape HOT 3 CLOSED

I suspect you're describing a pretty common use case, I'll build support into livescrape for properly emulating a browser (user agent and cookie persistence).

In the mean time, this is the type of thing I'd implement by overriding scrape_fetch. The next version will also contain some example recipes on how one would do that.

from livescrape.

I think that v0.9.4 will ease this usecase. I've uploaded it to pypi and readthedocs, although the latter is still building.

With v0.9.4, just add scrape_headers = {'User-Agent': 'Mozilla/0.2'} to your scraper definition.

from livescrape.

Excellent - your previous suggestion rather challenged my python skills.

Many thanks,

Mat
On 14 Feb 2016 13:10, "Koert van der Veer" [email protected] wrote:

I think that v0.9.4 will ease this usecase. I've uploaded it to pypi and
readthedocs, although the latter is still building.

With v0.9.4, just add scrape_headers = {'User-Agent': 'Mozilla/0.2'} to
your scraper definition.

—
Reply to this email directly or view it on GitHub
#3 (comment)
.

from livescrape.