Comments (5)
Not enough information to keep this bug open. Need full example and configuration.
from abotx.
The problem is when you pass to crawler the Impls
var crawler = new CrawlerX(crawlConfig, new ImplementationContainer() {
PageRequester = new PageRequester(crawlConfig, proxy)
});
Changing this to:
var crawler = new CrawlerX(crawlConfig);
crawler.Impls.PageRequester = new PageRequester(crawlConfig, proxy);
Solves the problem. But this issue is still valid. Maybe ImplementationContainer should be abstract and CrawlerX could throw a meaningful exception.
from abotx.
If you pass in an ImplementaionContainer it is expecting that all properties must set. If you use the ImplementationOverride class, then you can just set the impls that you care about and the defaults will be used for all other impls. See an example here.
I understand this is confusing and will consider making the ImplementationOverride an abstract class or by just requiring the ImplementationOverride explicitly in the constructor.
from abotx.
Todo:
1: Either make ImplementationContainer an abstract class or require ImplementationOverride in the constructor.
2: Do the same for parallelCrawler
3: Update parallelCrawler docs page
from abotx.
Now Both CrawlerX and ParallelCrawlerEngine REQUIRE an ImplementationOverride instead of ImplementationContainer. This has been fixed with nuget version 1.2.126
from abotx.
Related Issues (20)
- Javascript rendering even if ShouldRenderJavascript returns false HOT 1
- User agent HOT 5
- Robots.txt is not reloaded when uri scheme is changed (http/https) HOT 1
- custom implementations for IThreadManager doesn't work HOT 2
- Problems with Javascript rendering (phantomjs.exe not found)
- AbotX not respecting CrawlConfigurationX MaxPagesToCrawlPerDomain and MaxCrawlDepth HOT 5
- What happened to some of the properties that were in v1? HOT 1
- Parallel engine not working HOT 8
- If paid feature is enabled, Abotx doesn't do anything and no warning or exception is present HOT 1
- How would I crawl a single site with multiple pages in parallel? HOT 2
- crawledPage.HttpRequestException. on https://aanhangwagenspattyn.be/ while ok in browser or Postman HOT 1
- Incompatibility with net5.0 HOT 5
- abotx.org SSL certificate failing HOT 2
- Javascript rendering - detecting window.location changes HOT 3
- Crawl through a proxy server HOT 2
- AbotX produces huge amount of warnings on Linux HOT 1
- crash when stop
- Current AbotX license does not include rendering of javascript.
- Add Elapsed property on the AllCrawlsCompleted event
- Constructing wrong URLs to crawl from anchor tags without scheme
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abotx.