kkys / zhihuspider Goto Github PK

View Code? Open in Web Editor NEW

390.0 390.0 208.0 2.96 MB

Java无框架实现爬取知乎用户信息、图片和知乎推荐内容并下载到本地或数据库中

Java 100.00%

zhihuspider's People

Contributors

Stargazers

Watchers

Forkers

cludex zy0001 xjchuan kfengc27 kdd-xiaoxianrou 850075610 lurunhong chenglei1986 arvinshu mo-di biggbuddy matwu12 simplesuhao newdream8848 doujianghuijia dadaobixing newdimitri razrjay c54s888 heguanyuan doctli fangchenhui3 jounywang pennycoding tcf245 qinyunsen hua25 guoyu07 xzhiqin i-12-14 duaff sjxinrui andadepiaotaidong mayi203 ld1987 chisijun angelen10 zarelone helw kermit5 gcjyzdd wangshaoboxy foolself qulongjun xblydxj whitesnowchen gongqunhao uoor vigarbuaa zizheng1989 firetercel die-deutsche-orthopaedie fdzcz carlos-chu proaway topbean huty tonytonycho gordenchoy bigcapitalist haozibe lijian8 cikai edtenz thinerzq huiseee zalej chenshashafly lovelife1994 wu-sicheng androidzhaoxiaogang lichong yzmin1106 jasoncai0 alongkin jiage8866 tranbean swallown1 robin2017 flywencww aabb667 nblookup supass xjl219 goto9ccc chinazhouwy xiaohao204 zhongjie-chen zhan0356 benzhenzhang nfhy planabcd louishua1991 sunyongle swifth liuxiaoqiang perryhau fengwen bransummer dustinli

zhihuspider's Issues

正在抓取知乎链接：https://www.zhihu.com/question/28594126
发送GET请求出现异常！java.io.IOException: HTTPS hostname wrong: should be <www.zhihu.com>
java.io.IOException: HTTPS hostname wrong: should be <www.zhihu.com>
at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
at mothed.Util.SendGet(Util.java:26)
at bean.ZhiHuPicBean.(ZhiHuPicBean.java:47)
at mothed.Spider.downloadPic(Spider.java:21)
at test.ZhiHuMain.main(ZhiHuMain.java:29)

标题：null

即将开始下载图片到D:/知乎爬虫/null

开始下载................

图片下载完毕，请到D:/知乎爬虫/null里去看看吧！！！

我把里面的url改成http://www.zhihu.com 没有发生异常，但是爬取得内容为null,没有爬到内容

请问你是怎么解决知乎反爬虫问题的？

我在抓取回答的时候设置了sleep依然被限制访问了有什么办法吗？

一个新的ip第一次爬京东商品，就被重定向了？

<title>302 Found</title>

302 Found

JDWS 大神遇到过吗？你是怎么解决的啊。

Echarts展示网页打不开

导入工程后报错

1,'<>' operator is not allowed for source level below 1.7 HttpUtils.java 一般这种问题是修改java compiler
2,BasicHeader cannot be resolved to a type 一般这种问题是修改java jdk版本
3.can't find mysql-connector-java-5.1.7.jar 一般是导入jar包

为什么我这些都改了还是一堆报错,我表示很苦恼.
百度已经解决不了我的哀愁了.请解答一下.

现在精华问题翻页成js了 ..

打出jar包后，包内那么多public static void main函数，怎么知道从哪个main开始的呢？

抓取的图片存在哪里？

新手求教

希望添加一下对知乎专栏的爬取

今天准备写一个爬虫，结果发现知乎专栏是Angular渲染的。。爬不到

能否提供一下创建数据库的sql？谢谢

爬取知乎回答数返回多余字符

正在抓取用户链接：https://www.zhihu.com/people/piaoyegege
Exception in thread "main" java.lang.NumberFormatException: For input string: "回答86"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at ZhiHuSpider.bean.ZhiHuUserBean.(ZhiHuUserBean.java:109)
at ZhiHuSpider.mothed.Spider.getTopUserInfo(Spider.java:89)
at ZhiHuSpider.test.ZhiHuMain.main(ZhiHuMain.java:40)

无法创建文件路径

C:/..../ZhihuSpider/知乎爬虫/<span class="zm-editable-content">####</span>
问题出在ZhihuPicBean获取问题标题的pattern
加粗部分并不能作为文件夹名称使用，需要改进pattern的表达式。

临时解决办法：
如遇到同样问题, 可修改FileReaderWriter,line 81: String dir = filePath + zhiHuPicBean.getQuestion() 加粗部分为任意你想要的String 如我的：String dir = filePath + "test";

thanks

貌似不能抓取“更多”按钮之后的图片？

只能抓取第一页的图片，更多之后的图片貌似抓取不到？

抓取图片报错

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching www.zhihu.com found.
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1747)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1209)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:135)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:943)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:133)
at mothed.Spider.SendGet(Spider.java:25)
at bean.ZhiHuPicBean.(ZhiHuPicBean.java:45)
at test.ZhiHuMain.main(ZhiHuMain.java:22)