vincetran96 / scrape-finance-data-v2 Goto Github PK

2021-11-02 15:28:11 [scrapy.extensions.telnet] INFO: Telnet Password: 60c4207748982772
2021-11-02 15:28:11 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle']
2021-11-02 15:28:11 [financeInfo] INFO: Reading start URLs from redis key 'financeInfo:start_urls' (batch size: 16, encoding: utf-8
2021-11-02 15:28:11 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'rotating_proxies.middlewares.BanDetectionMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-11-02 15:28:11 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-11-02 15:28:11 [scrapy.middleware] INFO: Enabled item pipelines:
['scrapy_redis.pipelines.RedisPipeline']
2021-11-02 15:28:11 [scrapy.core.engine] INFO: Spider opened
2021-11-02 15:28:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-11-02 15:28:11 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
2021-11-02 15:28:11 [celery.app.trace] INFO: Task celery_tasks.finance_task[cd208afb-3cd5-40f7-a938-248c7d98603c] succeeded in 0.11887021500001538s: None
2021-11-02 15:28:12 [financeInfo] INFO: === IDLING... ===
2021-11-02 15:28:16 [financeInfo] INFO: === IDLING... ===
2021-11-02 15:28:21 [financeInfo] INFO: === IDLING... ===
2021-11-02 15:28:26 [financeInfo] INFO: === IDLING... ===
2021-11-02 15:28:31 [financeInfo] INFO: === IDLING... ===
2021-11-02 15:28:31 [financeInfo] INFO: corpAZ closed key: 1
2021-11-02 15:28:31 [financeInfo] INFO: corpAZ key financeInfo:corpAZtickers contains: set()
2021-11-02 15:28:31 [financeInfo] INFO: set()
2021-11-02 15:28:31 [financeInfo] INFO: Deleted status file at run/scrapy/financeInfo.scrapy
2021-11-02 15:28:31 [scrapy.core.engine] INFO: Closing spider (CorpAZ is closed; CorpAZ queue is empty; Spider is idling)
2021-11-02 15:28:31 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'elapsed_time_seconds': 20.031222,
 'finish_reason': 'CorpAZ is closed; CorpAZ queue is empty; Spider is idling',
 'finish_time': datetime.datetime(2021, 11, 2, 15, 28, 31, 938732),
 'log_count/INFO': 21,
 'log_count/WARNING': 1,
 'memusage/max': 65544192,
 'memusage/startup': 65331200,
 'start_time': datetime.datetime(2021, 11, 2, 15, 28, 11, 907510)}
2021-11-02 15:28:31 [scrapy.core.engine] INFO: Spider closed (CorpAZ is closed; CorpAZ queue is empty; Spider is idling)

Scrape Finance Data - version 2

Do you wish to mass scrape? [y/n] y
Do you wish clear ALL scraped files and kill ALL running Celery workers? [y/n] y
Clearing scraped files and all running workers, please wait...

OK
rm: cannot remove './run/celery/*': No such file or directory
removed './run/scrapy/financeInfo.scrapy' removed './logs/corporateAZExpress_log_verbose.log' removed './logs/corporateAZOverview_log_verbose.log' removed './logs/financeInfo_log_verbose.log'
Do you wish to start mass scraping now? Process will automatically exit when finished. [y] y
Creating Celery workers...
Running Celery tasks for mass scrape...
Scrapy is still running…
Scrapy is still running...
Scrapy is still running...
…
Scrapy is still running...
Scrapy has finished
Killing Celery workers, flushing Redis queues, deleting Celery run files...
OK
removed './run/celery/workercorpAZ.pid'
removed './run/celery/workerfinance.pid'
Exiting...

2021-09-24 20:21:29 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'bans/status/403': 1,
 'elapsed_time_seconds': 1402.909312,
 'finish_reason': 'CorpAZ is closed; CorpAZ queue is empty; Spider is idling',
 'finish_time': datetime.datetime(2021, 9, 25, 1, 21, 29, 850110),
 'httpcompression/response_bytes': 162246573,
 'httpcompression/response_count': 2135,
 'log_count/INFO': 2910,
 'memusage/max': 61169664,
 'memusage/startup': 55721984,
 'response_received_count': 2136,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/403': 1,
 'scheduler/dequeued/redis': 2135,
 'scheduler/enqueued/redis': 2135,
 'start_time': datetime.datetime(2021, 9, 25, 0, 58, 6, 940798)}