This is a small utility to scrape articles from Haveeru.
Make sure you have the latest version node.js installed
- Clone from github
git clone https://github.com/iulogy/haveeru-scraper
cd haveeru-scraper
- Install dependencies
npm install
- Scrape first 20 articles
node scrape
- Scrape articles from 100 to 500
node scrape --start 100 --end 500
- Scrape articles from 100 to 70000 with 20 concurrent requests and save full page
node scrape --start 100 --end 70000 --limit 20 --save-full-page