Consider using selenium instead of requests + regex about anubis HOT 3 CLOSED

Using a headless browser would add quite a bit of overhead (and make it considerably slower), and would complicate the install process. Provided the pages Anubis is scraping don't use js to fetch data or use a framework, the results should be identical.

Why do you think I should use selenium (genuinely asking)?

from anubis.

Because if you mimic browser you are trying to be stealthy. If you wanna be stealthy, there is no better mimic of a browser than actula browser.

And selenium is not headless. It's just a library implementing the api side of the webdriver standard. On the other side is usual desktop web browser.

from anubis.

I don't think the overhead is worth it. It's possible to revisit this in the future, but for now requests + regex is good enough.

from anubis.