How would I crawl a single site with multiple pages in parallel? about abotx HOT 2 CLOSED

Abot's PoliteWebCrawler alone can crawl multiple PAGES of a single site concurrently. AbotX's ParallelCrawlerEngine is to manage multiple instances of Abot's PoliteWebCrawler instances, effectively allowing you to crawl multiple SITES concurrently.

Example shows how to get the content of a crawled page.
var crawler = new PoliteWebCrawler(config);
crawler.PageCrawlCompleted += PageCrawlCompleted;//Several events available...
var crawlResult = await crawler.CrawlAsync(new Uri("http://!!!!!!!!YOURSITEHERE!!!!!!!!!.com"));

    private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
    {
        var httpStatus = e.CrawledPage.HttpResponseMessage.StatusCode;
        var rawPageText = e.CrawledPage.Content.Text;
    }

from abotx.

Thank-you for the info. Very helpful

One last clarification. If I have a website and I want it to crawl specific pages, not the whole site, do I have to make it crawl the entire site?

How do I direct it to crawl, say paged content. For example:

www.mysite.com/puppies?page=1
www.mysite.com/puppies?page=2
www.mysite.com/puppies?page=3
www.mysite.com/puppies?page=4
www.mysite.com/puppies?page=5

...and I do not want it to crawl

www.mysite.com/contact
www.mysite.com/puppies/blog
www.mysite.com/services

Thank-you for your patience!

from abotx.