scrapy-itzhaopin's Introduction
scrapy-itzhaopin's People
Contributors
Stargazers
Watchers
Forkers
yolanda1989 newlionwang xiaoyebupt charmlynn zuiwufenghua beiyexertz lapuda xiaodc aliqiaoqiao fyhsky luis-wang jackwang429 veinyy helloliubai point-line-surface-body jguoguo sayounara richardleeh hongxs rockflying aylhex lee-eve waha590927 qgf haozhuoran1991 tensorspace guiying212 kimisshao smithzhang010 duguruiyuan jhangli cousepig py-web wangshen2014 gischen agz1990 beibeirory fsfree satiago yiyoxy sinbh ccavxx liangmingjie jiezhu2007 wwdgit xukui44 chijiaodaxie kingstorm ver007 fvames daize1994 augustlong yiyihome shliujing fanyingying batmancn zsmj513 nanjinghhu stupidboy2015 baitongda supereagleflying fengchunsgit xiaosimao wodear gaosongyue xianwenye bertomartin leelong80 xiaoabiang piaosanlang swordlidev gallantzhangyu five3 annaye skyeyeslive bigtiger079 fds whpiano ylcolala xiaohhhh cherish24 lanlingsheng zhugeburu paulmrzhang torans ziyubiti frcmail xiaoliandanzi bygonetime woden234 hanmichael nuadaandre memego hunterchao kingleoric2010 861 xiju2003 hack-cc always20 caolusgscrapy-itzhaopin's Issues
运行成功后没有生成tencent.json文件
运行成功后没有生成tencent.json文件
第一页的数据没有爬下来，探讨解决
最近在学scrapy框架，觉得你写的这个实例不错，然后也按照最简单多方法写了一个爬虫同样是爬腾讯招聘，但是我发现虽然爬虫运行良好，但是始终爬不到第一页的数据，然后clone里你多程序试一试，发现你的程序同样有这个问题，所以想问问是哪里出了问题，我们一起进步一下。
这里是主要部分的代码，运行后能同样爬出2000+的数据，但是就是没有第一页：
class TencentSpider(CrawlSpider):
name = "tencenthr"
# download_delay = 1
allowed_domains = ["tencent.com"]
start_urls = ["http://hr.tencent.com/position.php"]
rules = [
Rule(LinkExtractor(allow = ('/position.php\?&start=\d*#a',),restrict_xpaths=('//*[@id="next"]')), follow=True, callback='parse_item')
]
def parse_item(self, response):
self.logger.info('Now is spidering in this page: %s', response.url)
base = response.xpath('//div[@id="position"]/div[1]/table/tr[@class="even" or @class="odd"]')
pages = response.xpath('//a[@class="active"]/text()').extract()
for sel in base:
item = TencenthrItem()
item['work'] = sel.xpath('td[1]/a/text()').extract()
item['worktype'] = sel.xpath('td[2]/text()').extract()
item['number'] = sel.xpath('td[3]/text()').extract()
item['location'] = sel.xpath('td[4]/text()').extract()
item['date'] = sel.xpath('td[5]/text()').extract()
item['page'] = pages
yield item
第一页的数据没有爬下来，探讨解决
最近在学scrapy框架，觉得你写的这个实例不错，然后也按照最简单多方法写了一个爬虫同样是爬腾讯招聘，但是我发现虽然爬虫运行良好，但是始终爬不到第一页的数据，然后clone里你多程序试一试，发现你的程序同样有这个问题，所以想问问是哪里出了问题，我们一起进步一下。
这里是主要部分的代码，运行后能同样爬出2000+的数据，但是就是没有第一页：
class TencentSpider(CrawlSpider):
name = "tencenthr"
# download_delay = 1
allowed_domains = ["tencent.com"]
start_urls = ["http://hr.tencent.com/position.php"]
rules = [
Rule(LinkExtractor(allow = ('/position.php\?&start=\d*#a',),restrict_xpaths=('//*[@id="next"]')), follow=True, callback='parse_item')
]
def parse_item(self, response):
self.logger.info('Now is spidering in this page: %s', response.url)
base = response.xpath('//div[@id="position"]/div[1]/table/tr[@class="even" or @class="odd"]')
pages = response.xpath('//a[@class="active"]/text()').extract()
# items = []
for sel in base:
item = TencenthrItem()
# item['company'] = sel.xpath('div[@class="jobnote-r"]/a/@href').extract()
item['work'] = sel.xpath('td[1]/a/text()').extract()
item['worktype'] = sel.xpath('td[2]/text()').extract()
item['number'] = sel.xpath('td[3]/text()').extract()
item['location'] = sel.xpath('td[4]/text()').extract()
item['date'] = sel.xpath('td[5]/text()').extract()
item['page'] = pages
yield item
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.