dongweiming / weapp-zhihulive Goto Github PK

其实本来也不想麻烦您，但是实在是最近被雅思压得厉害，不知道我是否能有幸请到您来帮忙，求加扣扣306732350.感谢感谢

无法使用 await 方法

有简单的类图分析吗？感谢。讲解说几句也行，就是实体类，界面类，控制类什么的，还有之间的简单的关系

嗨，主人。可以把预览图去掉，或者缩小一些吗。
clone的时候实在太痛苦了

代码链接，因为api登录大部分时候self.need_captcha()不需要验证码会返回False，所以可能没遇到，ZhihuClient类中并没有给出self.get_captcha()的实现以及设置CAPTCHA_FILE。

`python crawl.py
PUT http://localhost:9200/live130 [status:400 request:0.008s]
Traceback (most recent call last):
File "crawl.py", line 268, in
loop.run_until_complete(live_init())
File "./Server/models/live.py", line 171, in init
await Live.init()

raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)

elasticsearch.exceptions.RequestError: TransportError(400, 'illegal_argument_exception', 'Custom Analyzer [ik_analyzer] failed to find tokenizer under name [ik_max_word]')
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x1047e4128>`

环境： Mac, Python3.6.5, es三个版本2.4.6/ 5.6.2 / 6.2.2 都试过，
脚本运行都出错，创建index失败

本地搭建的es是可以运行的，head插件可以正常访问es.

代码链接

    async def fetch(self, url, max_redirect):
        ...
        try:
            ...
            if next_url is not None:
                self.add_url(next_url, max_redirect)
            else:
                for type in self.__stopped:
                    if type in url:
                        self.__stopped[type] = True
        finally:
            response.release()

知乎api的设计中，next_url永远都不会返回None，可以试一下https://api.zhihu.com/lives/ended?purchasable=0&limit=10&offset=10000000000

这样会导致self.__stopped[type]永远都是False，然后在work中会源源不断的往queue添加url，永远不会退出，crawl()会阻塞在await self.q.join()，work不会cancel：

    async def work(self):
        try:
            while 1:
                url, max_redirect = await self.q.get()
                if url in self.seen_urls:
                    type = url.split('/')[-1].split('?')[0]
                    if not self.__stopped[type]:
                        self.add_url(get_next_url(url), max_redirect)
                await self.fetch(url, max_redirect)
                self.q.task_done()
                asyncio.sleep(1)
        except asyncio.CancelledError:
            pass

而且self.__stopped = {}.fromkeys(['ended', 'ongoing', 'posts'], False)，好像api只有ended和outgoing两种类型？

Password:
Traceback (most recent call last):
File "crawl.py", line 267, in
crawler = Crawler()
File "crawl.py", line 66, in init
self.client = ZhihuClient()
File "/code/weapp-zhihulive/Server/client.py", line 98, in init
self.login(username, password)
File "code/weapp-zhihulive/Server/client.py", line 141, in login
self.save_token(self._login_auth, data)
File "code/weapp-zhihulive/Server/client.py", line 106, in save_token
raise LoginException(json_dict['error']['message'])
exception.LoginException: Login Fail: 为了您的帐号安全，请使用短信验证码登录
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f7498e5b6d0>

请问您做后台了吗？数据库什么的。小白一个，还没入门，请多包涵