This is a firmware scraper that aims to download firmware images and associated metadata from supported device vendor websites.
Configure the scrapy/settings.py
file, including whether metadata about
downloaded firmware should be inserted into a SQL server.
To run a specific scraper, e.g. dlink
:
scrapy crawl dlink
To run all scrapers with maximum 4 in parallel, using GNU Parallel:
parallel -j 4 scrapy crawl ::: `for i in ./scraper/spiders/*.py; do basename ${i%.*}; done`