hezhang1994 / weibo-crawler Goto Github PK

It seems like this code cannot crawl video from re-tweeted post.

换了不同人的账号，一到live photo就会报错
FileNotFoundError: [WinError 2] 系统找不到指定的文件
谢谢

see "bmiddle_pic" under "retweeted_status" and modify the code.

It happens at the following code:
if card['mblog']['isLongText'] == 'False': ... else: ...

In details, when card['mblog']['isLongText'] = False, the code still excutes the else content.

1. If the cookie is expired, the crawling process will stop.

   • How to continue to crawl data from the stopped post?
   • How to automatic crawl data?

2. How to crawl video?

Currently, this implementation can not crawl images posted in transferred text.

1. If the cookie is expired, the crawling process will stop.

   • How to continue to crawl data from the stopped post?
   • How to automatic crawl data?

2. How to crawl videos?

3. How to crawl GIFs?

Sometimes, the value of time_list could be ['N/A\r'] (def time_str2float). It may mean the video missed. Then, it would break.
My temporary solution is that set 'time_num = 0', if the 'N/A\r' appears. However, I do not know if it would cause unknown flaws.

因为某些微博表情无法解码保存。

Error message:
Traceback (most recent call last):
File "H:/prbkp/weibo-crawler/run_WeiboCrawler.py", line 245, in
photo_str = card['mblog']['pic_video']
KeyError: 'pic_video'

I do not know why it appeared. I just use:
if card_info:
if 'pic_video' in card_info:
try:
photo_str = card['mblog']['pic_video']
photo_list = re.split('[,]', photo_str)
for photo in photo_list:
# E.g., 'photo' = '0:000voDMsjx07t57qM583010f0100alDF0k01'.
photo_code = re.split('[:]', photo)[1]
photo_url = photo_url_start + photo_code + photo_url_end
photo_urls.append(photo_url)
except:
pass

to skip the part of code to prevent sudden interruption.

Hope it can help for further improvement.

It seems I can only find "getdashinfoxxx" and "proxy?xxx" on my XHR tag. Is there other ways to find the containder id? Thankss