lanbing510 / lianjiaspider Goto Github PK

Traceback (most recent call last):
File "LianJiaSpider.py", line 12, in
from bs4 import BeautifulSoup
ImportError: No module named bs4

您好，求问您还留着之前的爬取结果嘛？马上要写毕业论文了，救救孩子吧…愿付费的
@lanbing510

登陆不成功。

登入也成功，爬的是北京的，没有报错，就是db文件是空的，请问哪里出错了

我把所有的
bj.lianjia.com
改成了
qd.lianjia.com
现在有报错，如下
`/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/mars/git/LianJiaSpider/LianJiaSpider.py
<urllib2.Request instance at 0x102cf1f38>
404
Not Found
http://qd.lianjia.com/?ticket=ST-7736156-tIILbXMnsuLyfc4PUyfx-www.lianjia.com
Date: Sun, 27 Nov 2016 10:08:06 GMT
LB_HEADER: aws-1a-lj-nginx00.ares.lianjia.com
Server: Tengine/2.2.0
Content-Length: 0
Connection: Close

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bs4/init.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 371 of the file /Users/mars/git/LianJiaSpider/LianJiaSpider.py. To get rid of this warning, change code that looks like this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "html.parser")

markup_type=markup_type))
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bs4/init.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 774 of the file /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py. To get rid of this warning, change code that looks like this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "html.parser")

markup_type=markup_type))
timed out
爬下了 市南 区全部的小区信息
timed out
爬下了 市北 区全部的小区信息
爬下了 李沧 区全部的小区信息
爬下了 崂山 区全部的小区信息
Traceback (most recent call last):
爬下了 四方 区全部的小区信息
File "/Users/mars/git/LianJiaSpider/LianJiaSpider.py", line 377, in
done
exception_spider(db_cj)
File "/Users/mars/git/LianJiaSpider/LianJiaSpider.py", line 341, in exception_spider
excep_list=exception_read()
File "/Users/mars/git/LianJiaSpider/LianJiaSpider.py", line 326, in exception_read
f=open('log.txt','r')
IOError: [Errno 2] No such file or directory: 'log.txt'

Process finished with exit code 1`

爬了两个城区后出现下列错误。同时网页打开后提示流量异常，要我做图片验证码，做完后网页浏览正常，但是爬取仍然如此。
此外，拿到的db是12K，是不是也是只有空文件？ 另外这个文件使用sqlite查看吗？

/Library/Python/2.7/site-packages/beautifulsoup4-4.5.3-py2.7.egg/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 371 of the file LianJiaSpider.py. To get rid of this warning, change code that looks like this:

 BeautifulSoup([your markup])

to this:

 BeautifulSoup([your markup], "html.parser")

Traceback (most recent call last):
  File "LianJiaSpider.py", line 371, in <module>
    do_xiaoqu_spider(db_xq,region)
  File "LianJiaSpider.py", line 187, in do_xiaoqu_spider
    d="d="+soup.find('div',{'class':'page-box house-lst-page-box'}).get('page-data')
AttributeError: 'NoneType' object has no attribute 'get'

现在链家的所有网页，查看source的话都放在这里了，抓不到了。

<link rel="stylesheet" href="http://static1.ljcdn.com/pc/asset/ershoufang/xiaoquDetail/index.css?_v=20160721182616">
<link rel="stylesheet" href="http://static1.ljcdn.com/dep/ueditor/themes/iframe.css?_v=20160721182616">