Comments (5)
Can you add this line...
impls.WebCrawlerFactory = new WebCrawlerFactory(config); //!!!!!!!!!!!!!!!!!This is new!!!!!
before this one...
ParallelCrawlerEngine crawlEngine = new ParallelCrawlerEngine(config, impls);
Let me know if that solves your problem.
from abotx.
Thank you, but it is still not working properly. I added the line of code, recompiled, and ran a test with 8 Domains and 1000 MaxPagesToCrawlPerDomain. It is currently surpassing 15,000 pages crawled. It shouldn't have gone past 8000 pages.
My counter is incremented each time the crawler fires the PageCrawlCompletedAsync() event.
from abotx.
Also, my project is running Version 1.3.81 of AbotX and 1.6.0.5 of Abot
from abotx.
I was able to get it working by using the SiteToCrawl.CrawlConfiguration property instead of a global crawl configuration for the crawl.
from abotx.
I've been unable to reproduce this issue. If this springs up again in version 2.0+ feel free to reopen
from abotx.
Related Issues (20)
- Javascript rendering even if ShouldRenderJavascript returns false HOT 1
- User agent HOT 5
- Robots.txt is not reloaded when uri scheme is changed (http/https) HOT 1
- custom implementations for IThreadManager doesn't work HOT 2
- Problems with Javascript rendering (phantomjs.exe not found)
- What happened to some of the properties that were in v1? HOT 1
- Parallel engine not working HOT 8
- If paid feature is enabled, Abotx doesn't do anything and no warning or exception is present HOT 1
- How would I crawl a single site with multiple pages in parallel? HOT 2
- crawledPage.HttpRequestException. on https://aanhangwagenspattyn.be/ while ok in browser or Postman HOT 1
- Incompatibility with net5.0 HOT 5
- abotx.org SSL certificate failing HOT 2
- Javascript rendering - detecting window.location changes HOT 3
- Crawl through a proxy server HOT 2
- AbotX produces huge amount of warnings on Linux HOT 1
- crash when stop
- Current AbotX license does not include rendering of javascript. HOT 1
- Add Elapsed property on the AllCrawlsCompleted event
- Constructing wrong URLs to crawl from anchor tags without scheme
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abotx.