topiccrawler / jkcrawler Goto Github PK
View Code? Open in Web Editor NEW使用 Scrapy 写成的 JK 爬虫,图片源自哔哩哔哩、Tumblr、Instagram,以及微博、Twitter
License: GNU General Public License v3.0
使用 Scrapy 写成的 JK 爬虫,图片源自哔哩哔哩、Tumblr、Instagram,以及微博、Twitter
License: GNU General Public License v3.0
在爬取微博话题 的时候遇到问题,不知道怎么去修改爬取的话题,还有保存的格式
你好,我想爬取微博其他话题下的图片,m_weibo.py中的start_urls值是怎么获取的
首先呢,头像周杰伦好评 周杰伦10多年老粉了
然后,readme可不可以写的具体一些,不知道怎么把程序跑起来?
python3.9 -m scrapy crawl instagram -o data/instagram.jsonlines
2021-01-28 15:33:14 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: jkcrawler)
2021-01-28 15:33:14 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.9.1 (default, Jan 8 2021, 17:17:17) - [Clang 12.0.0 (clang-1200.0.32.28)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1i 8 Dec 2020), cryptography 3.3.1, Platform macOS-10.15.7-x86_64-i386-64bit
2021-01-28 15:33:14 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-01-28 15:33:14 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'jkcrawler',
'FEED_EXPORT_ENCODING': 'utf-8',
'NEWSPIDER_MODULE': 'jkcrawler.spiders',
'SPIDER_MODULES': ['jkcrawler.spiders']}
2021-01-28 15:33:14 [scrapy.extensions.telnet] INFO: Telnet Password: 9611507f386f33a4
2021-01-28 15:33:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2021-01-28 15:33:14 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-01-28 15:33:14 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-01-28 15:33:14 [scrapy.middleware] INFO: Enabled item pipelines:
['scrapy.pipelines.images.ImagesPipeline']
2021-01-28 15:33:14 [scrapy.core.engine] INFO: Spider opened
2021-01-28 15:33:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-01-28 15:33:14 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-01-28 15:33:16 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://instagram.com/explore/tags/jk%E5%88%B6%E6%9C%8D/?__a=1> from <GET http://instagram.com/explore/tags/jk%E5%88%B6%E6%9C%8D/?__a=1>
2021-01-28 15:33:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.instagram.com/explore/tags/jk%E5%88%B6%E6%9C%8D/?__a=1> from <GET https://instagram.com/explore/tags/jk%E5%88%B6%E6%9C%8D/?__a=1>
2021-01-28 15:33:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.instagram.com/accounts/login/?next=/explore/tags/jk%25E5%2588%25B6%25E6%259C%258D/%3F__a%3D1> from <GET https://www.instagram.com/explore/tags/jk%E5%88%B6%E6%9C%8D/?__a=1>
2021-01-28 15:33:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.instagram.com/accounts/login/?next=/explore/tags/jk%25E5%2588%25B6%25E6%259C%258D/%3F__a%3D1> (referer: None)
2021-01-28 15:33:17 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.instagram.com/accounts/login/?next=/explore/tags/jk%25E5%2588%25B6%25E6%259C%258D/%3F__a%3D1> (referer: None)
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/scrapy/utils/defer.py", line 120, in iter_errback
yield next(it)
File "/usr/local/lib/python3.9/site-packages/scrapy/utils/python.py", line 353, in next
return next(self.data)
File "/usr/local/lib/python3.9/site-packages/scrapy/utils/python.py", line 353, in next
return next(self.data)
File "/usr/local/lib/python3.9/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/local/lib/python3.9/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/usr/local/lib/python3.9/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/local/lib/python3.9/site-packages/scrapy/spidermiddlewares/referer.py", line 340, in
return (_set_referer(r) for r in result or ())
File "/usr/local/lib/python3.9/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/local/lib/python3.9/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in
return (r for r in result or () if _filter(r))
File "/usr/local/lib/python3.9/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/usr/local/lib/python3.9/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in
return (r for r in result or () if _filter(r))
File "/usr/local/lib/python3.9/site-packages/scrapy/core/spidermw.py", line 62, in _evaluate_iterable
for r in iterable:
File "/Volumes/Untitled 2/jkcrawler/jkcrawler/spiders/instagram.py", line 13, in parse
graphql = json.loads(response.text)['graphql']
File "/usr/local/Cellar/[email protected]/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/[email protected]/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/[email protected]/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2021-01-28 15:33:17 [scrapy.core.engine] INFO: Closing spider (finished)
2021-01-28 15:33:17 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1294,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 4,
'downloader/response_bytes': 22360,
'downloader/response_count': 4,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/301': 2,
'downloader/response_status_count/302': 1,
'elapsed_time_seconds': 3.213926,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 1, 28, 7, 33, 17, 915022),
'log_count/DEBUG': 4,
'log_count/ERROR': 1,
'log_count/INFO': 10,
'memusage/max': 54632448,
'memusage/startup': 54628352,
'response_received_count': 1,
'scheduler/dequeued': 4,
'scheduler/dequeued/memory': 4,
'scheduler/enqueued': 4,
'scheduler/enqueued/memory': 4,
'spider_exceptions/JSONDecodeError': 1,
'start_time': datetime.datetime(2021, 1, 28, 7, 33, 14, 701096)}
2021-01-28 15:33:17 [scrapy.core.engine] INFO: Spider closed (finished)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.