Giter Club home page Giter Club logo

chat-downloader's Introduction

Chat Downloader

Python

PyPI version

Downloads

License

Coverage

Chat Downloader is a simple tool used to retrieve chat messages from livestreams, videos, clips and past broadcasts. No authentication needed!

Installation

This tool is distributed on PyPI and can be installed with pip:

$ pip install chat-downloader

To update to the latest version, add the --upgrade flag to the above command.

Alternatively, the tool can be installed with git:

$ git clone https://github.com/xenova/chat-downloader.git
$ cd chat-downloader
$ python setup.py install

Usage

Command line

usage: chat_downloader [-h] [--version] [--start_time START_TIME]
                       [--end_time END_TIME]
                       [--message_types MESSAGE_TYPES | --message_groups MESSAGE_GROUPS]
                       [--max_attempts MAX_ATTEMPTS]
                       [--retry_timeout RETRY_TIMEOUT]
                       [--interruptible_retry [INTERRUPTIBLE_RETRY]]
                       [--max_messages MAX_MESSAGES]
                       [--inactivity_timeout INACTIVITY_TIMEOUT]
                       [--timeout TIMEOUT] [--format FORMAT]
                       [--format_file FORMAT_FILE] [--chat_type {live,top}]
                       [--ignore IGNORE]
                       [--message_receive_timeout MESSAGE_RECEIVE_TIMEOUT]
                       [--buffer_size BUFFER_SIZE] [--output OUTPUT]
                       [--overwrite [OVERWRITE]] [--sort_keys [SORT_KEYS]]
                       [--indent INDENT] [--pause_on_debug | --exit_on_debug]
                       [--logging {none,debug,info,warning,error,critical} | --testing | --verbose | --quiet]
                       [--cookies COOKIES] [--proxy PROXY]
                       url

For example, to save messages from a livestream to a JSON file, you can use:

$ chat_downloader https://www.youtube.com/watch?v=jfKfPfyJRdk --output chat.json

For a description of these options, as well as advanced command line use-cases and examples, consult the Command Line Usage page.

Python

from chat_downloader import ChatDownloader

url = 'https://www.youtube.com/watch?v=jfKfPfyJRdk'
chat = ChatDownloader().get_chat(url)       # create a generator
for message in chat:                        # iterate over messages
    chat.print_formatted(message)           # print the formatted message

For advanced python use-cases and examples, consult the Python Documentation.

Chat Items

Chat items/messages are parsed into JSON objects (a.k.a. dictionaries) and should follow a format similar to this:

{
    ...
    "message_id": "xxxxxxxxxx",
    "message": "actual message goes here",
    "message_type": "text_message",
    "timestamp": 1613761152565924,
    "time_in_seconds": 1234.56,
    "time_text": "20:34",
    "author": {
        "id": "UCxxxxxxxxxxxxxxxxxxxxxxx",
        "name": "username_of_sender",
        "images": [
            ...
        ],
        "badges": [
            ...
        ]
    },
    ...
}

For an extensive, documented list of included fields, consult the Chat Item Fields page.

Frequently Asked Questions

Coming soon

Issues

Found a bug or have a suggestion? File an issue here. To assist the developers in fixing the issue, please follow the issue template as closely as possible.

Contributing

If you would like to help improve the tool, you'll find more information on contributing in our Contributing Guide.

Supported sites:

  • YouTube.com - Livestreams, past broadcasts and premieres.
  • Twitch.tv - Livestreams, past broadcasts and clips.
  • Zoom.us - Past broadcasts
  • Facebook.com (currently in development) - Livestreams and past broadcasts.

chat-downloader's People

Contributors

9001 avatar felk avatar herowinb avatar jc-chung avatar joshualochner avatar krichbanana avatar leodmanx2 avatar marsam avatar mikoim avatar naokiy avatar pboettcher avatar quagsirus avatar taizan-hokuto avatar xenova avatar yay295 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

chat-downloader's Issues

Retrieve Banned / Redacted Messages [FEATURE]

I am unable to see banned or redacted messages at the output, i wonder if there is a way to do this through this package, if theres not, then will it be implemented?

[Edit: Sorry, after reading through some closed issues, i found theres an implementation, so im closing this issue]

Retracted messages

Sorry, this isn't much of an issue, but do you think it's possible to get retracted / unsent messages with the script?

[FEATURE] Get live chat when user goes live

Is your feature request related to a problem? Please describe.

So, some streamers have stream rooms up before they stream, right?
And some of them even disable the live chat before the stream actually begins, like an hour or less before it actually starts.

Describe the solution you'd like

So, rather than manually trying to re-enter the command line or refreshing sites like Youtube to see if the live chat's available to public, I'm suggesting some kind of retry system to keep the command running and grab the live chats when it starts.

Additional context

Sorry if what I'm requesting is already a thing with python scripts, like below? gist.github.com/plieningerweb/39e47584337a516f56da105365a2e4c6

I'll try the link above first to run the feature that I want, although I can only copy paste stuffs since I don't understand much other than texts. So, I'm hoping this feature could be added if I fail to run it. Also, I'm going to sleep so yeah

Script error for message types presenting that a user purchased a product linked to the YouTube channel merchandising

python chat_replay_downloader.py https://youtu.be/y5ih7nqEoc4 -message_type all -output y5ih7nqEoc4

Just after reachin 1:52:29 into parsing I see:

Traceback (most recent call last):
  File "chat_replay_downloader.py", line 455, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "chat_replay_downloader.py", line 409, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type)
  File "chat_replay_downloader.py", line 333, in get_youtube_messages
    data = self.__parse_item(item)
  File "chat_replay_downloader.py", line 235, in __parse_item
    self.__time_to_seconds(data['time_text']))
KeyError: 'time_text'

I think this is because the last data variable values are:

{'message': 'NEVER EVER GIVE IN', 'author': 'Stephen Kelly', 'timestamp': 1603158759859453, 'time_text': '1:52:29'}

{'timestamp': 1603158761042020, 'message': None}

Watching the video and chat replay on YouTube during that time I see...aha! I see now!

Someone just bought
The Death of Censorship
https://teespring.com/en/silence-victory-accelerates

Therefore, it seems like event handling for this other message type is what the problem is

Specifically liveChatPurchasedProductMessageRenderer

Adding it as line 45 fixes it:

        'ignore': [
            # message saying Live Chat replay is on
            'liveChatViewerEngagementMessageRenderer',
            'liveChatPurchasedProductMessageRenderer',
            'liveChatPlaceholderItemRenderer',  # placeholder
            'liveChatModeChangeMessageRenderer'  # e.g. slow mode enabled
        ],

I'm surprised that I can't find any API documentation info about it at https://developers.google.com/youtube/v3/live/docs/liveChatMessages

I thought Google used to be a company that excelled at being on top of its game providing #1 best excellent resource of information and documentation and analysis, but I think that is no longer the case. Bummer!

Comments from first minutes aren't downloaded

Today I tried download two different live chats, and on each one first couple of minutes of comments aren't downloaded. First comment downloading is like 3 or 5 minutes after truely first comment.

Donation link

Hey Xenova, got any links to donate a lil something to you? You saved something very special to me. :)

Error on 2 YouTube Video IDs

Upon trying to download DzEbfQI4TPQ, the vast majority of the chat is captured, and then this happens:

[3:34:24] marshallbenn33: eft lets goooo
[3:34:25] BrokeProphet: Need a new team
Traceback (most recent call last):
  File "/home/xnaas/youtube2/chat-replay-downloader/chat_replay_downloader.py", line 449, in <module>
    args.url, start_time=args.start_time, end_time=args.end_time, message_type=args.message_type)
  File "/home/xnaas/youtube2/chat-replay-downloader/chat_replay_downloader.py", line 403, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type)
  File "/home/xnaas/youtube2/chat-replay-downloader/chat_replay_downloader.py", line 331, in get_youtube_messages
    data = self.__parse_item(item)
  File "/home/xnaas/youtube2/chat-replay-downloader/chat_replay_downloader.py", line 234, in __parse_item
    self.__time_to_seconds(data['time_text']))
  File "/home/xnaas/youtube2/chat-replay-downloader/chat_replay_downloader.py", line 95, in __time_to_seconds
    return sum(abs(int(x)) * 60 ** i for i, x in enumerate(reversed(time.split(':')))) * (-1 if time[0] == '-' else 1)
  File "/home/xnaas/youtube2/chat-replay-downloader/chat_replay_downloader.py", line 95, in <genexpr>
    return sum(abs(int(x)) * 60 ** i for i, x in enumerate(reversed(time.split(':')))) * (-1 if time[0] == '-' else 1)
ValueError: invalid literal for int() with base 10: '443,856'

Unfortunately, no file is output when this happens.

No problems, just thanks

I'm bad at github and couldn't see any other way to send this.

Just wanted to say how much I appreciate this. I found this script via google leading me to a random stackexchange answer, and it was just what I needed when I needed it most. Helped me save some things in the eleventh hour that had a lot of sentimental value and would have otherwise been totally lost.

Thanks so much for making this, honestly.

Message types

Not really an issue, but I've realized that the script isn't grabbing superchats. I feel like it's --message_types superchat_message but the script isn't working after being run. But then again, can I know what to use for grabbing both text and superchats? Since all_message is definitely wrong.

Receving duplicated superchats

As title, superchats received are duplicated (Some superchats are received twice).

May try this and search for a few superchat authors or text contents.

about SQLite3 support to which branch

I really appreciate your work, and I find a small problem that sometimes the script break down (due to google) and all the data got lost before write to a file. So I'm going to add support for SQLite to write data into a file, like chat_data.db so that nothing will get lost. I just wondering which branch should I address on, the master branch that have one py file or the
new package branch, or both?

[YouTube] Add exception for privated/deleted video.

I have a list of videos that I'm trying to get chat replay for, and I've discovered that if the video is privated, the script will just crash like this:

H:\YouTubeArchive\Test>python chat_replay_downloader.py https://www.youtube.com/watch?v=ijFMXqa-N0c -message_type all -output ijFMXqa-N0c.chat.json
Traceback (most recent call last):
  File "chat_replay_downloader.py", line 449, in <module>
    args.url, start_time=args.start_time, end_time=args.end_time, message_type=args.message_type)
  File "chat_replay_downloader.py", line 403, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type)
  File "chat_replay_downloader.py", line 253, in get_youtube_messages
    continuation_by_title_map = self.__get_initial_youtube_info(video_id)
  File "chat_replay_downloader.py", line 161, in __get_initial_youtube_info
    columns = ytInitialData['contents']['twoColumnWatchNextResults']
KeyError: 'contents'

Would it be possible to add an exception to the script so that it would not crash? I'm using a script to basically step through a list of stream IDs and download chat for them one by one, and it would be nice to be able to skip past this without crashing.

[BUG] ParsingError: Unable to parse video data.

Basic information

  • Python version: 3.9.0

Describe the bug

Tracebacks happening with the usual command I've used.

If running from the command line, provide the following:

  1. The command used (including the verbose tag, -v):
    python -m chat_replay_downloader --message_types "text_message paid_message membership_item" https://www.youtube.com/watch?v=aqXFQuxgo5c -o aqXFQuxgo5c1.txt

Otherwise, if using the python module, provide the following:

  1. A minimal reproducible example:
 -m chat_replay_downloader --message_types "text_message paid_message membership_item" https://www.youtube.com/watch?v=aqXFQuxgo5c -o aqXFQuxgo5c1.txt
  1. Output, traceback or other information relating to the bug:
  File "C:\Users\arthur\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\arthur\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\arthur\AppData\Local\Programs\Python\Python39\lib\site-packages\chat_replay_downloader\__main__.py", line 4, in <module>
    chat_replay_downloader.main()
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\chat_replay_downloader\cli.py", line 217, in main
    chat = downloader.get_chat(**chat_params)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\chat_replay_downloader\chat_replay_downloader.py", line 128, in get_chat
    info = correct_site.get_chat(**params)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\chat_replay_downloader\sites\youtube.py", line 1307, in get_chat
    return self.get_chat_by_video_id(match.group('id'), kwargs)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\chat_replay_downloader\sites\youtube.py", line 1273, in get_chat_by_video_id
    initial_info = self._get_initial_video_info(video_id)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\chat_replay_downloader\sites\youtube.py", line 782, in _get_initial_video_info
    raise ParsingError(
chat_replay_downloader.errors.ParsingError: Unable to parse video data. Please try again.

Expected behavior

Retrieving chat for "title"..
[00:01:04]Username: Message types
[00:01:05]Username: Message types

continuation_by_title_map['Live chat replay']

This might not work returning a KeyError depending on the person's location (This phrase varies for each localization of YT). I think forcing the HTTP header language to English should do it.

Active YouTube livestreams result in error (sometimes)

"Video does not have a chat replay."

I noticed that that appears after a livestream is finished but before the live chat is available.

However, for active livestreams that in progress, as I test this more than the first occurrence I experienced, for a 2nd different active livestream, I wasn't able to reproduce it because the script output the same as above, however, trying a few livestreams at https://www.youtube.com/results?search_query=live&sp=EgJAAQ%253D%253D to reproduce the error, I can find other livestreams that result in the same error. Due to the nature of time-sensitivity, I can't provide any specific examples that will work in future, but I'm sure you can find one to reproduce the same error experience where the script fails with:

Traceback (most recent call last):
  File "chat_replay_downloader.py", line 449, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "chat_replay_downloader.py", line 404, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type)
  File "chat_replay_downloader.py", line 332, in get_youtube_messages
    data = self.__parse_item(item)
  File "chat_replay_downloader.py", line 235, in __parse_item
    self.__time_to_seconds(data['time_text']))
KeyError: 'time_text'

Note: The line numbers might be off by 1 because I edited the script per my last issue report

The data['time_text'] doesn't exist, but I see that some of the data is available. For example this was the value of data:

{'message': 'Oh ty', 'author': '• Lacy •', 'timestamp': '1603504526543501'}

Perhaps for active livestreams that are in progress, the python script can, instead of using already post-processed calculated time values, the script can generate these calculations or otherwise remove them?

I can't find a way to get the unix epoch timestamp for when a livestream started, so that it can be used to calculate time elapsed for each message in data that doesn't contain 'time_text'

Add Support for YouTube Streams that Aren't Finished

Just a thought.

The __YOUTUBE_API_TEMPLATE would be https://www.youtube.com/live_chat/get_live_chat?continuation={}&hidden=false&pbj=1, and looking at the JSON a lot of the keys are different (ex. Live chat replay -> Live chat).

Automatically restart on network error

Right now when there is a network error, it is stopped. Is there a way to automatically restart without clearing the output file and have no duplicate messages?

Super chat messages are skipped during live stream

My command:
python chat_replay_downloader.py https://www.youtube.com/watch?v=* -message_type all -output *.csv

Used parameter -message_type all, but paid super chat messages are skipped during live stream. They are missing in the file and in the console output. Several of regular messages following the super chat are missing too.
Super chat from completed streams are saved correctly. But no need to write colors and duration of pinned messages, as useful information is only sum of donation. it would be great if it was possible to customize output.

callback function on new API

I'm using the latest commit in master branch.

Before this update, I used to do:

ChatReplayDownloader().get_youtube_messages(video_id, callback=my_callback)

But now I don't see a callback parameter provided, is there any fix? Thanks!

Unable to parse video data

I've tried the usual script on like 3 different streams now.
The response is always "Unable to parse video data."
I know it's not my connection since I've tried it with and without VPN numerous times. Any idea why this happens?

Save new messages to file as they come in

Is there a way to append new messages to the end of a file for a live video as they come in? Right now it only saves when you exit. Also, trying to send the output to a file using > results in an empty file.

Add support for YouTube member only live stream.

I noticed when trying to download chat replay from a YouTube "member only" live stream, the program returns Video does not have a chat replay., which is not true. Is it possible to add support so that chat replay can be downloaded from these streams by passing in a suitable cookie?

[BUG] Youtube Membership welcome messages produce JSON object in place of message text

Basic information

  • Program version: Current pip version
  • Python version: 3.8
  • Operating system: Linux/Fedora

Describe the bug

Instead of a simple text string the "message" field for membership welcome messages is a nested object. This seems to be a regression with the newest version.

Command/Code used

If running from the command line, provide the following:

  1. The command used (including the verbose tag, -v):
chat_downloader "https://www.youtube.com/watch\?v\=ie5oTaH3nAk" --message_groups all --output "output.json" -v

One particular example message from that chat replay, though this applies to both add_live_chat_ticker_item and add_chat_item.

        "message": {
            "message": "Welcome to \u6850\u751f\u4f1a!"
        },
        "message_id": "ChwKGkNLRDV0YV9tMXU0Q0ZZMm13UW9kVlBnQjhR",

Expected behavior

All "message" fields should be simple strings

        "message": "Welcome to \u6850\u751f\u4f1a!"
        "message_id": "ChwKGkNLRDV0YV9tMXU0Q0ZZMm13UW9kVlBnQjhR",

Duplicate super chat, membership purchase; Print issue; FileNotFoundError

  • superchat_ticker is not ignored when output format is json.
  • Superchat without message display as 'None' after print. (like this: [01:23] *$5.00* authorZ: None)
  • liveChatMembershipItemRenderer is an auto-message but after print is look like normal message.
  • FileNotFoundError when parent folder of output filepath not exist.

(Sorry for my bad english.)

Error caused by empty author name during youtube live stream

After an error, the script stops and the result is not saved to file.
Used .csv output.
Error caused by empty author name.
It would be better if the file was written by adding new lines continuously then the result would not be lost.

Traceback (most recent call last):

  File "*\chat_replay_downloader.py", line 638, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "*\chat_replay_downloader.py", line 562, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type, chat_type, callback)
  File "*\chat_replay_downloader.py", line 473, in get_youtube_messages
    callback(data)
  File "*\chat_replay_downloader.py", line 615, in print_item
    chat_downloader.print_item(item)
  File "*\chat_replay_downloader.py", line 208, in print_item
    message = emoji.demojize(self.message_to_string(item))
  File "*\chat_replay_downloader.py", line 199, in message_to_string
    item['author'],
KeyError: 'author'

Suggestion: add option to print youtube author id along with author name

Youtube doesn't allow you to view channel of user in chat replay, but i really needed to find one user's channel, so i looked into code of thins tool, and after some debugging found out that this tool have access to message author id, it's stored in item_info object under key authorExternalChannelId in method __parse_item, so i was able to find channel of needed user. After all of that, i think, there should be option to display message author id along with author name for easier access.

ModuleNotFoundError: No module named 'emoji'

Sorry if this is dumb, but I unzipped and did the pip requirements install, but every time I run the script I get

Traceback (most recent call last):
  File "run_tests.py", line 1, in <module>
    from chat_replay_downloader import *
  File "~/chat-replay-downloader-master/chat_replay_downloader.py", line 9, in <module>
    import emoji
ModuleNotFoundError: No module named 'emoji'

No experience with python so I have no clue what it means or how to fix it

support youtube top char

Youtube live normal normal message is two types of messages

Top chat: This view filters messages such as potential spam to help make chat easier to read and more useful.
Live chat: This view is not filtered. It shows all chat messages as they come in.

I noticed that the message obtained by this project is all the messages, that is, Live chat, which contains a lot of spam.
Is there any way to get only Top chat, or separate Top chat

Receiving Error after execution

After running the program I receive this error at the end
'list' is not recognized as an internal or external command, operable program or batch file.

Any idea how to fix this?

Negative YouTube Start Times Don't Work

It's possible to comment on YouTube streams before they go live, and the comments get a negative timestamp. If you use a negative value with this though it crashes either after the first batch of comments, or after it reaches positive comments. I'm not sure which; I don't have a video with enough pre-comments for it to have more than one batch.

Superchats

I'm so sorry but the new version isn't saving superchats..

Changing the time display in txt files

Hey, I really love the work you put into this tool!
I wanted to use it and realized that the time of messages doesn't have a continues format.
What I mean is, that time gets displayed with only one digit if it is the left most time format thingy (hour, minute, second).
(ex 1:00 instead of 01:00) (ex. 2:05:00 instead of 02:05:00)

Could you maybe change that in a following release or is this intentional?

Some new features

Hey there, and thanks for making this! I've been adding some minor features to cover my own usecases, and was wondering if you'd like to have some of this merged in? I'd be happy to create some PR's all depending on how you'd like it wrapped :>

with all included, it currently looks like this: https://ocv.me/dev/?chat_replay_downloader.py

the following changes are fairly clean and could mostly be copied over as-is:

  • use videoOffsetTimeMsec rather than timestampText when available, for subsecond precision
  • in json output, include authorExternalChannelId as author_id to separate users with the same display name
  • in json output, include user attributes (moderator flag etc, taken from authorBadges)
  • in __print_item, have an additional try/catch around print(safe_string) in case the stdout codec is not unicode

now these i'm a bit uncertain about... maybe i'm just fixing symptoms rather than the root cause:

  • use item.get('time_text', None) rather than item['time_text'] in message_to_string as some messages don't appear to have this key at all
  • another conditional for the presence of time_text in __parse_item

additionally, i added a crude hack to persist the original json replies from youtube in a logfile (in addition to the generated json output) so I didn't have to call youtube's servers all the time when testing changes... was super useful, but would need a total rewrite if it's a feature to keep :p

most of that is the logfile stuff in get_youtube_messages and __main__, plus the new get_chat_replay_from_json() which replaces the __get_live_info and __get_replay_info functions to grab responses from the logfile rather than youtube, so it's by all means not pretty

and finally, the reason i wanted subsecond precision in the first place :p https://twitter.com/tripflag/status/1317862794725855235

Live Chat Replay

How can I be sure youtube live chat replay is returning the all the comments and not just the top chat?

It definitely is excluding a lot of them.

[YouTube] Support super chats

Super chats are donations to a partnered YouTube streamer.

  • have messages if the donation amount is enough (for USD, this is $2 or greater)
  • pin themselves to the top of the chat window (up to 5 hours for the max donation. For USD, this is $500)
  • are colored based on the amount donated
  • can be made in currencies other than USD

Currently no super chat data is saved. Ideally the output would have several additional fields: color, pin duration, amount, and the original currency code. And the presence of amount would differentiate super chats from regular chat.

Example video with a large number of super chats: https://www.youtube.com/watch?v=UlemRwXYWHg

traceback

Is there any reason for the script to stop in the middle of a stream? I've been getting lots of these lately. My internet's pretty okay with 10 MBps download speed..
python tester.py -message_type all https://www.youtube.com/watch?v=Ei8iGpELhUc -output Ei8iGpELhUc1.txt

Running the script after the stream ends always works fine of course, it's just that some messages are still being sent out after the stream ends, so I'd like to get them as well. Also, I've been trying to find a way to grab the ones that aren't even showing in the live chat replay although that's not the case here.

Traceback (most recent call last):
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 639, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 563, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type, chat_type, callback)
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 474, in get_youtube_messages
    callback(data)
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 625, in write_to_file
    print_item(item)
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 616, in print_item
    chat_downloader.print_item(item)
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 208, in print_item
    message = emoji.demojize(self.message_to_string(item))
  File "C:\Users\x\Downloads\Compressed\New folder\tester.py", line 199, in message_to_string
    item['author'],
KeyError: 'author'

Youtube chat issues

Sorry, I have no idea how to code, but it seems that there's an error for retrieving Youtube live chats. Here's the traceback:

  File "C:\Windows\System32\chat-replay-downloader-master\test.py", line 477, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "C:\Windows\System32\chat-replay-downloader-master\test.py", line 429, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type, chat_type)
  File "C:\Windows\System32\chat-replay-downloader-master\test.py", line 271, in get_youtube_messages
    continuation_by_title_map = self.__get_initial_youtube_info(video_id)
  File "C:\Windows\System32\chat-replay-downloader-master\test.py", line 168, in __get_initial_youtube_info
    ytInitialData_script = next(script.string for script in soup.find_all(
StopIteration

Tracebacks

Traceback (most recent call last):
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 641, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 565, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type, chat_type, callback)
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 408, in get_youtube_messages
    info = self.__get_live_info(continuation)
  File "C:\Users\r\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 303, in __get_live_info
    return(self.__get_continuation_info(url))
  File "C:\Usersr\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 307, in __get_continuation_info
    info = self.__session_get_json(url)
  File "C:\Users\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 145, in __session_get_json
    return self.__session_get(url).json()
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2>python tester.py -message_type all https://www.youtube.com/watch?v=jJ8EkajpnAo -output jJ8EkajpnAo3.txt
Traceback (most recent call last):
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 641, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 565, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type, chat_type, callback)
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 408, in get_youtube_messages
    info = self.__get_live_info(continuation)
  File "C:\UsersDownloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 303, in __get_live_info
    return(self.__get_continuation_info(url))
  File "C:\Users\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 307, in __get_continuation_info
    info = self.__session_get_json(url)
  File "C:\Users\\Downloads\Compressed\chat-replay-downloader-1.0.2\tester.py", line 145, in __session_get_json
    return self.__session_get(url).json()
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Tracebacks

Traceback (most recent call last):
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\response.py", line 438, in _error_catcher
    yield
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\response.py", line 767, in read_chunked
    chunk = self._handle_chunk(amt)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\response.py", line 711, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 612, in _safe_read
    data = self.fp.read(amt)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\response.py", line 572, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\response.py", line 793, in read_chunked
    self._original_response.close()
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 643, in <module>
    chat_messages = chat_downloader.get_chat_replay(
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 567, in get_chat_replay
    return self.get_youtube_messages(match.group(1), start_time, end_time, message_type, chat_type, callback)
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 410, in get_youtube_messages
    info = self.__get_live_info(continuation)
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 305, in __get_live_info
    return(self.__get_continuation_info(url))
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 309, in __get_continuation_info
    info = self.__session_get_json(url)
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 145, in __session_get_json
    return self.__session_get(url).json()
  File "C:\Users\x\Downloads\Compressed\chat_replay_downloader_test\tester.py", line 141, in __session_get
    return self.session.get(url)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 697, in send
    r.content
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 831, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "C:\Users\x\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 756, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

I'm sure 50% of it's because of my slow internet and 50% because of the script. Can you upload a separate version, especially the last one, before "print(html.text)" had to be added? Please.. I'm so done with reporting issues back to back.. I'm trying to save live chat logs everyday and the last version was perfect since there wasn't any tracebacks at all, no matter how bad my internet was.

I really hope that you could do that since this script is always updated via internet, if my assumption isn't wrong. I just don't want anymore issues in the future..

zsh: command not found: chat_replay_downloader

  • chat-replay-downloader version: v0.0.8
  • Python version: Python 2.7
  • Operating System: MacOS

Description

Hey ya'll,
First off, thanks for creating such a handy tool, and sorry if this is a newbie question. I recently got a Mac, so I still haven't gotten used it.

I tried the tool on Windows and it works perfectly, without any issues.
However, on my Mac, I just can't get it to work.
I've installed the package using pip install.
I've also cloned the git repository.

But when I use the 'chat_replay_downloader' command in terminal and VS code, I get a zsh: command not found: chat_replay_downloader error.

I'm sure there's an easy way to fix this, but wanted to check with ya.

On Windows, I simply used the pip command, installed the tool and ran the chat_replay_downloader command and it worked without any issues.

Any Ideas how to fix this?

Cheers,

What I Did

amir@Amirs-MBP chat-replay-downloader % chat_replay_downloader 
zsh: command not found: chat_replay_downloader
amir@Amirs-MBP chat-replay-downloader % chat_replay_downloader -h
zsh: command not found: chat_replay_downloader
amir@Amirs-MBP ~ % python chat_replay_downloader.py https://www.youtube.com/watch?v=xxxxx -start_time 14400 -end_time 15000 -output example.csv
zsh: no matches found: https://www.youtube.com/watch?v=xxxxx
amir@Amirs-MBP ~ % python chat_replay_downloader.py https://www.youtube.com/watch?v=xxxxx -output chat.csv   
zsh: no matches found: https://www.youtube.com/watch?v=xxxxx```

Comments before stream starts

I was puzzled as to why comments were not grabbing for ones written before stream starts, it then dawned on I needed to set a minus number start time to grab them all.

Could this be added to the help.

Cheers

Unable to parse video data

Suddenly not working, for the last 2 hours ago, the script was still working perfectly.
Script was something like this:
python tester.py -message_type all https://www.youtube.com/watch?v=EVLjth4lJXY -output EVLjth4lJXY1.txt

Awaiting your reply.

Help much appreciated

Really sorry to have to do this but I have no knowledge of Python and this is my last hope. I live streamed a wedding and trimmed it not realising YouTube removed live chat replay if you trim. The couple had lots of lovely messages from family there and I really need to get them back. If I share the video URL, can anybody help me with this? Thank you so much.

Bad python packaging and dependency management

it needs chmod 755 ./chat_replay_downloader.py on macos or gives zsh: permission denied: Installation is easier if you package it as a console script.

  1. Add setup.py
  2. make console script
  3. Write the version numbers into setup.py and requirements.txt

Also, long argument names start with double hyphens. Other argument names are not similar with -h --help.

Failed, Didn't work (SyntaxError: invalid syntax)

I followed the instructions (installed all the requirements except (csv) (didn't install), and when I tried to execute the command to download (all the chat), I got an error message, here is my example:

  • the command:
    Debian@Linux$ python chat_replay_downloader.py https://www.youtube.com/watch?v=xEEdiQ_B62I -output NewFile.txt

  • and I got this error message in the terminal:

  File "chat_replay_downloader.py", line 252
    print('['+message['time_text']+']',message['author']+':',message['message'],file=f)
                                                                                    ^
SyntaxError: invalid syntax

I'm not sure if it is a bug or do I miss something?
thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.