Use scrapy with a list of proxies generated from proxynova.com

The first run will generate the list of proxies from http://proxynova.com and store it in the cache.

It will individually check each proxy to see if they work and remove the ones that timed out or cannot connect to.

Example:

./run_example.sh

To regenerate the proxy list, run: python proxies.py

In settings.py add the following line: DOWNLOADER_MIDDLEWARES = { 'scrapy_proxynova.middleware.HttpProxyMiddleware': 543 }

Set these options in the settings.py.

• PROXY_SERVER_LIST_CACHE_FILE — a file to store proxies list. Default: proxies.txt.
• PROXY_BYPASS_PERCENT — probability for a connection to use a direct connection and not use a proxy