darthbear / scrapy-redis Goto Github PK
View Code? Open in Web Editor NEWThis project forked from rmax/scrapy-redis
Redis-based components for scrapy that allows distributed crawling. Small update to make it work for Scrapy 0.16+ and added QUEUE_TYPE and DUPE_FILTER options
License: BSD 3-Clause "New" or "Revised" License