AbotX not respecting CrawlConfigurationX MaxPagesToCrawlPerDomain and MaxCrawlDepth about abotx HOT 5 CLOSED

Can you add this line...

impls.WebCrawlerFactory = new WebCrawlerFactory(config); //!!!!!!!!!!!!!!!!!This is new!!!!!

before this one...
ParallelCrawlerEngine crawlEngine = new ParallelCrawlerEngine(config, impls);

Let me know if that solves your problem.

from abotx.

Thank you, but it is still not working properly. I added the line of code, recompiled, and ran a test with 8 Domains and 1000 MaxPagesToCrawlPerDomain. It is currently surpassing 15,000 pages crawled. It shouldn't have gone past 8000 pages.

My counter is incremented each time the crawler fires the PageCrawlCompletedAsync() event.

from abotx.

Also, my project is running Version 1.3.81 of AbotX and 1.6.0.5 of Abot

from abotx.

I was able to get it working by using the SiteToCrawl.CrawlConfiguration property instead of a global crawl configuration for the crawl.

from abotx.

I've been unable to reproduce this issue. If this springs up again in version 2.0+ feel free to reopen

from abotx.