Long-lived cookies might have unintended consequences on a crawling session about heritrix3 HOT 4 OPEN

Has there been a good analysis of the pros/cons of crawling with or without cookies being enabled? Maybe I should turn them off for most crawls? /cc @kris-sigur @nlevitt

from heritrix3.

I am not aware of a study of this particular issue, but from my web browsing and cURLing experience I can say that some sites might trap the crawler into an infinite redirect (which should be caught by the max redirect limit) and some sites might not return the useful page content in the first fetch without any cookies being present.

from heritrix3.

I don't know of any such analysis. It would be interesting. Archive-it crawlers log in with configured credentials in some cases, so for that to work we'd definitely need the cookies, but other than that it's hard to say.

from heritrix3.

Here goes another analysis that uncovers another issue of long-lasting cookies, which cause phrases of multiple languages show up on a Twitter timeline page.

https://ws-dl.blogspot.com/2019/03/2019-03-18-cookie-violations-cause.html

from heritrix3.