Comments (5)
What I interpret this as (and @sian1468, please correct me if wrong) is an option to only follow links if the links are in the same domain.
Normally I would think this is covered by domain restriction but I think what is being suggested is to ignore the "hops" restriction and keep following any links that are in the same domain. This is somewhat similar to "archive the whole site" provided the entire site is inter-linked from the starting seed.
You correct @machawk1
I think Squidwarc can do more than capture from depth setting by capture whole site with single/by depth page's offsite links or without offsite links setting.
from squidwarc.
What I interpret this as (and @sian1468, please correct me if wrong) is an option to only follow links if the links are in the same domain.
Normally I would think this is covered by domain restriction but I think what is being suggested is to ignore the "hops" restriction and keep following any links that are in the same domain. This is somewhat similar to "archive the whole site" provided the entire site is inter-linked from the starting seed.
from squidwarc.
@sian1468 thanks you for suggesting this and I believe your suggestion would be an excellent feature for Squidwarc.
I will be putting some thought into how to accomplish this nicely alongside the existing crawl modes.
Do you have any suggestions as to how you would like to be able to specify this crawl mode?
from squidwarc.
@sian1468 thanks you for suggesting this and I believe your suggestion would be an excellent feature for Squidwarc.
I will be putting some thought into how to accomplish this nicely alongside the existing crawl modes.
Do you have any suggestions as to how you would like to be able to specify this crawl mode?
Recursive crawl mode
I got inspiration from other archiving tools & software eg. Wpulll, grab-site and crocoite
from squidwarc.
implemented and merged into master PR #47
from squidwarc.
Related Issues (20)
- TypeError: input.on is not a function HOT 6
- Add docker file
- Allow crawl config to contain user scripts for a page HOT 16
- Instagram pages not recorded properly? HOT 3
- Squidwarc on Windows 10 (using npm) HOT 1
- Generated WARC files fail indexing with OpenWayback cdx-indexer HOT 11
- Exception immediately thrown on first-run HOT 5
- Make option for gzipped WARC:s? HOT 1
- Feature request: set browser accept language HOT 6
- Describe requirements in the README HOT 1
- Unable to reuse local Chrome user dir/cookies HOT 7
- Append config seem working not correctly. HOT 3
- Feature request: ignore sets
- Error: options.stripFragment is renamed to options.stripHash HOT 1
- Installation fails with missing node-warc submodule HOT 3
- TypeError in node-warc causes Squidwarc to crash HOT 2
- Exception with defaults using Docker in Puppeteer config HOT 2
- Setting mode in conf.json is ignored HOT 1
- Fatal error when starting crawl HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from squidwarc.