benchmarks? about portia HOT 4 CLOSED

No, it doesn't. You can check that with the CPU usage in your machine while you crawl.

Crawling speed depends on many factors, not only your machine capabilities. It depends also on the settings you are using for the job, and on the target server.

from portia.

i would rephrase my question, what difference would i get if i do the same crawling with pure scrapy and scrapy using portia

from portia.

There can be of course a small CPU usage cost, but that is not the usual case (unless the target site has very very big pages, and even on that case probably the cost is not important). As I said, you can check CPU usage by yourself. Also, we crawl at much higher speeds than you get, using scrapely/slybot (portia).

Again, crawling speed depends on many other factors, which does not came from a difference between pure scrapy and portia. I would start by checking settings/middleware differences between your runs with scrapy and with portia.

from portia.

There are some benchmarks in slybot that compare parsel extraction speed to scrapely extraction speed:

test_extraction_speed.TestExtractionSpeed.test_slybot_parse_and_extract: 20.4085s
test_extraction_speed.TestExtractionSpeed.test_parsel_parse_and_extract: 16.3027s
test_extraction_speed.TestExtractionSpeed.test_parsel_extract: 12.9132s
test_extraction_speed.TestExtractionSpeed.test_slybot_extract: 10.6147s

These tests are extracting from 300 pages extracting 10-15 items from each page. As you can see the part that slows down scrapely compared to parsel is the parsing which is written in python compared to lxml which is written in C. There was an attempt made to convert the scrapely parser from python to cython but unfortunately this didn't work on all pages so it was disabled. With the cython parser scrapely was faster than parsel so we need to look at fixing the bugs in cython parser and re-enabling it.

from portia.