brisppy / twitch-archiver Goto Github PK
View Code? Open in Web Editor NEWA simple, fast, platform-independent tool for downloading Twitch streams, videos, and chat logs.
License: GNU Affero General Public License v3.0
twitch-archiver's People
Contributors
Stargazers
Watchers
twitch-archiver's Issues
Use of Real Time Archiver always drops "No new VODs are available in the requested formats for thechannel"
Hi. I tried to use the Real Time Archiver mode with these arguments
twitch-archiver -l -R -c TheChannel -d "C:\thedirectory"
But always says:
2024-01-27 16:23:30 [ INFO] Twitch Archiver v4.0.4 - https://github.com/Brisppy/twitch-archiver
2024-01-27 16:23:31 [ INFO] Now archiving channel 'thechannel'.
2024-01-27 16:23:32 [ INFO] No new VODs are available in the requested formats for thechannel.
And the log file drops this:
2024-01-27 16:16:53 [processing.py:68 - get_channel()] Now archiving channel 'thechannel'.
2024-01-27 16:16:55 [processing.py:169 - get_channel()] No new VODs are available in the requested formats for thechannel.
Is it because I can't start the Real Time Archiver mode while the stream is live? Or do I have to use other arguments? Thanks.
If a title contains multibyte characters, they are replaced with "_"
Describe the bug
twitch-archiver --archive-only --chat -v 1959912263 -d /content/chats2
2023-10-27 19:14:21 [ WARNING] New version of Twitch-Archiver available - Version 3.0.7:
https://github.com/Brisppy/twitch-archiver/releases/latest
Release notes:
**(2023-08-31) Version 3.0.7**
**Changes and Fixes:**
* Increased the start delay for newly live channels.
2023-10-27 19:14:21 [ INFO] OAuth token verified successfully. Expiring in 887521
2023-10-27 19:14:21 [ INFO] Now processing VOD: 1959912263
2023-10-27 19:14:22 [ INFO] VOD offline.
2023-10-27 19:14:22 [ INFO] Grabbing chat logs...
2023-10-27 19:14:23 [ INFO] Found 199 messages. 28716 - ETA: 00:00:01
cd C:\content\chats2
(base) PS C:\content\chats2> ls
: C:\content\chats2
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2023/10/27 19:14 2023-10-25_12-36-35 - lol____ - 1959912263
Operating system
Your operating system (Windows, Linux, Mac)
Windows10 and Ubuntu(WSL2)
This program is very useful for archiving, thank you for making this!
Error occurs when the stream title is long
Describe the bug
Error occurs when the stream title is long
Supplied arguments
Arguments supplied to twitch-archiver which produced the error.
!twitch-archiver --archive-only --chat -c ta1yo_tv
Error log
2024-01-17 09:47:37 [ INFO] Twitch Archiver v4.0.2 - https://github.com/Brisppy/twitch-archiver
2024-01-17 10:29:23 [ INFO] Now archiving channel 'ta1yo_tv'.
2024-01-17 10:29:24 [ INFO] 42 VOD(s) in download queue.
Traceback (most recent call last):
File "/usr/local/bin/twitch-archiver", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/__init__.py", line 276, in main
process.get_channel(channels)
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/processing.py", line 171, in get_channel
self.vod_downloader(download_queue)
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/processing.py", line 235, in vod_downloader
_chat_download_queue.append(Chat(_vod, self.output_dir, self.quiet))
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/downloaders/chat.py", line 67, in __init__
self.load_from_file()
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/downloaders/chat.py", line 77, in load_from_file
with open(
OSError: [Errno 36] File name too long: '/content/chats2/ta1yo_tv/2023-12-21_08-38-59 - マウガ実装！！！TOP500実装！！！マウガ実装！！！TOP500実装！！！マウガ実装！！！TOP500実装！！！マウガ実装！！！TOP500実装！！！マウガ実装！！！TOP500実装！！！ - 2009919164/verbose_chat.json'
[OPTIONAL] Applicable files
If your issue is related to a VOD video or other file, upload it somewhere and place the link here.
VOD ID : 2009919164
Operating system
Colab / Python3.10
Additional context
this program is very useful for archiving, thank you for making it
I especially like the feature to avoid duplicates!
Streams with corruption not saved
Describe the bug
You have included a check to see if the TS files were downloaded completely and everything can be merged without corrupting. Unfortunately, if the streamer had problems with his internet during the stream, it can happen that these "corruptions" are already streamed to twitch and thus the VoD is never completely downloaded.
Supplied arguments
twitch-archiver.py -d /data/twitch -c kimtnt -N -D
Error log
2023-01-30 16:19:45 [ INFO] Merging VOD parts. This may take a while.
2023-01-30 16:21:39 [ INFO] Converting VOD to mp4. This may take a while.
2023-01-30 16:24:07 [ DEBUG] Verifying length of VOD file. ETA: 00:00:00
2023-01-30 16:24:07 [ DEBUG] Downloaded VOD length is 14956. Expected length is 14956.
2023-01-30 16:24:07 [ DEBUG] VOD passed length verification.
2023-01-30 16:24:07 [ DEBUG] Generating readable chat log and saving to disk...
2023-01-30 16:24:07 [ DEBUG] Downloading VOD thumbnail.
2023-01-30 16:24:09 [ DEBUG] Cleaning up temporary files...
2023-01-30 16:24:12 [ DEBUG] Adding VOD info to database.
2023-01-30 16:24:12 [ DEBUG] Database path: /home/downloader/.config/twitch-archiver/vods.db
2023-01-30 16:24:12 [ DEBUG] Connection to SQLite DB successful.
2023-01-30 16:24:12 [ DEBUG] Executing SQL statement: SELECT stream_id, video_archived, chat_archived FROM vods WHERE stream_id IS ?
2023-01-30 16:24:12 [ DEBUG] Values: {'stream_id': '40411186056'}
2023-01-30 16:24:12 [ DEBUG] Executing SQL statement:
INSERT INTO
vods (stream_id, user_id, user_login, user_name, title, description, created_at, published_at, url, thumbnail_url,
viewable, view_count, language, type, duration, muted_segments, vod_id, store_directory, video_archived,
chat_archived)
VALUES
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
2023-01-30 16:24:12 [ DEBUG] Values: {'stream_id': '40411186056', 'user_id': '81127376', 'user_login': 'kimtnt', 'user_name': 'KimTNT', 'title': 'Immer der Stra�e nach!', 'description': None, 'created_at': '2023-01-30T11:00:30Z', 'published_at': '2023-01-30T11:00:30Z', 'url': 'https://www.twitch.tv/videos/1722915620', 'thumbnail_url': 'https://static-cdn.jtvnw.net/cf_vods/dgeft87wbj63p/ecc1bb460b9554f8b9e8_kimtnt_40411186056_1675076425//thumb/thumb0-%{width}x%{height}.jpg', 'viewable': 'public', 'view_count': 0, 'language': 'de', 'type': 'archive', 'duration': 14956, 'muted_segments': None, 'vod_id': '1722915620', 'store_directory': '/data/twitch/KimTNT/2023-01-30_11-00-30 - Immer der Stra_e nach_ - 1722915620', 'video_archived': True, 'chat_archived': True}
2023-01-30 16:24:12 [ DEBUG] Removing lock file.
2023-01-30 16:24:12 [ DEBUG] Processing VOD 1712901026 by KimTNT
2023-01-30 16:24:12 [ DEBUG] Creating lock file for VOD.
2023-01-30 16:24:12 [ DEBUG] Database path: /home/downloader/.config/twitch-archiver/vods.db
2023-01-30 16:24:12 [ DEBUG] Connection to SQLite DB successful.
2023-01-30 16:24:12 [ DEBUG] Executing SQL statement: SELECT vod_id, video_archived, chat_archived FROM vods WHERE stream_id IS ?
2023-01-30 16:24:12 [ DEBUG] Values: {'stream_id': 40360528968}
2023-01-30 16:24:12 [ INFO] Now processing VOD: 1712901026
2023-01-30 16:24:13 [ INFO] VOD offline.
2023-01-30 16:24:13 [ INFO] Grabbing video...
2023-01-30 16:24:15 [ INFO] Grabbing chat logs...
2023-01-30 16:24:15 [ DEBUG] Grabbing chat logs from offset: 33538
2023-01-30 16:24:16 [ INFO] Found 15 messages.
2023-01-30 16:24:16 [ INFO] Merging VOD parts. This may take a while.
2023-01-30 16:28:35 [ INFO] Converting VOD to mp4. This may take a while.
...
2023-01-30 16:33:53 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3225
2023-01-30 16:33:53 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3226
2023-01-30 16:33:53 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3227
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3228
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3229
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3230
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3231
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3232
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3233
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3234
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3235
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3236
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3237
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3238
2023-01-30 16:33:54 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3239
2023-01-30 16:33:55 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3240
2023-01-30 16:33:55 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3241
2023-01-30 16:33:55 [ DEBUG] Ignoring corrupt packet as part in whitelist. Part: 3242
2023-01-30 16:33:55 [ ERROR] Corrupt packet encountered. Part: 3243
2023-01-30 16:33:55 [ ERROR] Corrupt packet encountered. Part: 3244
2023-01-30 16:33:55 [ ERROR] Corrupt packet encountered. Part: 3245
2023-01-30 16:33:55 [ ERROR] Corrupt packet encountered. Part: 3246
2023-01-30 16:33:55 [ ERROR] Corrupt packet encountered. Part: 324700:11
2023-01-30 16:33:55 [ ERROR] Corrupt packet encountered. Part: 3248
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3249
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 325000:11
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3251
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3252
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3253
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3254
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3255
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 325600:10
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3257
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3258
2023-01-30 16:33:56 [ ERROR] Corrupt packet encountered. Part: 3259
2023-01-30 16:33:57 [ ERROR] Corrupt packet encountered. Part: 3260
2023-01-30 16:34:08 [ ERROR] Corrupt segments found while converting VOD. Attempting to retry parts:
3243, 3244, 3245, 3246, 3247, 3248, 3249, 3250, 3251, 3252, 3253, 3254, 3255, 3256, 3257, 3258, 3259, 3260
...
2023-01-30 16:44:59 [ ERROR] Error downloading VOD 1712901026.: 00:00:00
Traceback (most recent call last):
File "/home/downloader/twitch-archiver/src/processing.py", line 428, in get_vod_connector
Utils.convert_vod(vod_json, muted_segments, print_progress=False if self.quiet else True)
File "/home/downloader/twitch-archiver/src/utils.py", line 274, in convert_vod
raise CorruptPartError(corrupt_parts, formatted_ranges)
src.exceptions.CorruptPartError: Corrupt parts found when converting VOD file. Parts: ['3243-3260.ts']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/downloader/twitch-archiver/src/processing.py", line 454, in get_vod_connector
Utils.convert_vod(vod_json, muted_segments, print_progress=False if self.quiet else True)
File "/home/downloader/twitch-archiver/src/utils.py", line 274, in convert_vod
raise CorruptPartError(corrupt_parts, formatted_ranges)
src.exceptions.CorruptPartError: Corrupt parts found when converting VOD file. Parts: ['3243-3260.ts']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/downloader/twitch-archiver/src/processing.py", line 457, in get_vod_connector
raise VodDownloadError(
src.exceptions.VodDownloadError: Video download failed. Error: Corrupt part(s) still present after retrying VOD download. Ensure VOD is still available and either delete the listed #####.ts part(s) from 'parts' folder or entire 'parts' folder if issue persists.
Operating system
Linux
Additional context
Seems to be a problem regarding this VoD. already tried to download the VoD multiple times and only this one has so many errors. Maybe add a switch to ignore coorupted files?
Question about playing downloaded content
Hi. This tool is really amazing. But how do I play it like on twitch? I mean, how do i display the video and also have the chat like in the twitch player? Its really great that this tool also downloads the chat log as txt, but with what software/player can I actually watch the VODs with the chat like on twitch?
Error occurs when stream goes offline and download doesn't finish
Describe the bug
This error always occurs when the stream being downloaded goes offline, and instead of finishing the download and merge, it actually throws this error
Error log
C:\Windows\System32>twitch-archiver -w -V -c halifax -d "D:\Apps\twitch-archiver"
Launching Twitch-Archiver in watch mode.
2024-01-19 23:30:23 [ INFO] Twitch Archiver v4.0.3 - https://github.com/Brisppy/twitch-archiver
2024-01-19 23:30:23 [ INFO] Now archiving channel 'halifax'.
2024-01-19 23:30:29 [ INFO] 56 VOD(s) in download queue.
100% - [#########################] - 2272 / 2272 - ETA: 00:00:00
100% - [#########################] - 2272 / 2272 - ETA: 00:00:00
100% - [#########################] - 2272 / 2272 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
100% - [#########################] - 2330 / 2330 - ETA: 00:00:00
2024-01-19 23:41:28 [ ERROR] Error fetching VOD status. Error: API request to https://gql.twitch.tv/gql failed. Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2024-01-19 23:41:44 [ ERROR] Exception occurred inside DownloadHandler. [' File "C:\\Users\\g2\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\twitcharchiver\\processing.py", line 274, in _start_download\n _downloader.start()\n', ' File "C:\\Users\\g2\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\twitcharchiver\\downloaders\\video.py", line 203, in start\n raise VideoDownloadError(exc) from exc\n']
2024-01-19 23:41:44 [ ERROR] Error archiving VOD {'vod_id': 2037476415, 'stream_id': 0, 'title': '🍁 @Halifax || Friday Night Shenanigans! 🍁', 'description': '', 'created_at': datetime.datetime(2024, 1, 19, 20, 8, 27), 'published_at': datetime.datetime(2024, 1, 19, 20, 8, 27), 'thumbnail_url': 'https://vod-secure.twitch.tv/_404/404_processing_320x180.png', 'duration': 23522, 'chat_archived': False, 'video_archived': False}.
Traceback (most recent call last):
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connection.py", line 461, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 1419, in getresponse
response.begin()
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 331, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 300, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 844, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\util\retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\util\util.py", line 38, in reraise
raise value.with_traceback(tb)
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connection.py", line 461, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 1419, in getresponse
response.begin()
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 331, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\http\client.py", line 300, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\api.py", line 118, in post_request
_r = self._session.post(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\downloaders\video.py", line 164, in start
self.vod.refresh_vod_metadata()
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\vod.py", line 150, in refresh_vod_metadata
self._fetch_metadata()
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\vod.py", line 124, in _fetch_metadata
_r = self._api.gql_request(
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\api.py", line 155, in gql_request
_r = self.post_request("https://gql.twitch.tv/gql", j=_q, h=_h)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\api.py", line 128, in post_request
raise RequestError(url, err) from err
twitcharchiver.exceptions.RequestError: API request to https://gql.twitch.tv/gql failed. Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\processing.py", line 274, in _start_download
_downloader.start()
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\downloaders\video.py", line 203, in start
raise VideoDownloadError(exc) from exc
twitcharchiver.exceptions.VideoDownloadError: API request to https://gql.twitch.tv/gql failed. Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Operating system
Windows 11
The file ".lock" is not created
I'm running the script on linux inside a docker container, all works but the .lock file is not created and found.
Logs:
DEBUG: Database columns match current version.
INFO: User EnderVaid ID is 172220374.
INFO: All VODs have been grabbed from Twitch.
INFO: Available VODs: [1228653244, 1227697884, 1227697623, 1226794856, 1224981498, 1223875519, 1221952999, 1220167274, 1218447210, 1217448860, 1216420614]
INFO: Channel is currently offline.
INFO: Downloaded VODs: []
INFO: VOD Queue: [1228653244, 1227697884, 1227697623, 1226794856, 1224981498, 1223875519, 1221952999, 1220167274, 1218447210, 1217448860, 1216420614]
INFO: Retrieving VOD: 1228653244
Traceback (most recent call last):
File "/home/container/twitch-vod-archiver.py", line 452, in
main()
File "/home/container/twitch-vod-archiver.py", line 401, in main
with open(Path(VOD_DIRECTORY, VOD_INFO['user_name'], '.lock.' + str(vod_id)), 'w') as lockfile:
FileNotFoundError: [Errno 2] No such file or directory: 'vod/EnderVaiD/.lock.1228653244'
container@pterodactyl~ Server marked as offline...
Specifying Custom temp folder
Describe the solution you'd like
A way to specify the path for a custom temp folder in the arguments, instead of using the system default.
Concurrent mass chat log downloads from multiple VODs
Is your feature request related to a problem? Please describe.
Need to download chat logs from many VODs. Current one-by-one downloads are very slow.
Describe the solution you'd like
Allow parallel downloads - multiple chat log downloads running simultaneously across threads.
Describe alternatives you've considered
None, only option is slow one-by-one downloads.
Describe how this feature would help YOU meet your archiving goals
Would speed up downloading many chat logs for my archives.
Describe how this feature would help OTHERS meet their archiving goals
Faster mass chat log downloads for anyone archiving lots of VODs.
Additional context
https://github.com/Daniele122898/TwitchMassChatDownloader
That guy implemented it in his program, but it's old, non-configurable and looks kinda abandoned.
IndexError on skip-live-streams
Describe the bug
Missing check on parameter -N if channel_data is returned
Supplied arguments
python3 twitch-archiver.py -d /data/twitch -c lillimontv -t 1 -N
Error log
2022-12-06 22:34:41 [ INFO] Twitch Archiver v2.2.0 - https://github.com/Brisppy/twitch-archiver
2022-12-06 22:34:42 [ INFO] OAuth token verified successfully. Expiring in 5075700
2022-12-06 22:34:43 [ INFO] Now archiving channel 'lillimontv'.
2022-12-06 22:34:44 [ INFO] Online VODs: 22
2022-12-06 22:34:44 [ INFO] Downloaded vods: 17
2022-12-06 22:34:44 [ INFO] 5 VOD(s) in download queue.
Traceback (most recent call last):
File "twitch-archiver.py", line 41, in <module>
main()
File "/root/twitch-archiver/src/__init__.py", line 142, in main
process.get_channel(args.get('channel'))
File "/root/twitch-archiver/src/processing.py", line 209, in get_channel
elif self.no_stream and stream_id == int(channel_data[0]['id']):
IndexError: list index out of range
Operating system
Debian 11
Error occurs if the streamer's name contains CJK language
Describe the bug
If the streamer's name is multibytes, error will occur
Supplied arguments
twitch-archiver --archive-only --chat -c dmf_kyochan
Error log
2024-01-16 17:38:26 [ INFO] Twitch Archiver v4.0.1 - https://github.com/Brisppy/twitch-archiver
2024-01-16 17:38:27 [ INFO] Now archiving channel '恭一郎'.
Traceback (most recent call last):
File "/usr/local/bin/twitch-archiver", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/__init__.py", line 276, in main
process.get_channel(channels)
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/processing.py", line 74, in get_channel
channel_videos: list[Vod] = channel.get_channel_videos()
File "/usr/local/lib/python3.10/dist-packages/twitcharchiver/channel.py", line 351, in get_channel_videos
for v in _r.json()[0]["data"]["user"]["videos"]["edges"]
TypeError: 'NoneType' object is not subscriptable
[OPTIONAL] Applicable files
If your issue is related to a VOD video or other file, upload it somewhere and place the link here.
Operating system
WSL2
Python 3.10.12
Additional context
It worked in version 3, but stopped working after version 4
functioning is properly if its name does not contain multibytes characters
twitch-archiver --archive-only --chat -c LIRIK
2024-01-16 17:33:03 [ INFO] Twitch Archiver v4.0.1 - https://github.com/Brisppy/twitch-archiver
2024-01-16 17:33:03 [ INFO] Now archiving channel 'LIRIK'.
2024-01-16 17:33:04 [ INFO] 6 VOD(s) in download queue.
2024-01-16 17:33:04 [ INFO] Skipping VOD as it is live and no-stream argument provided.
2024-01-16 17:33:51 [ INFO] Found 23893 chat messages.
2024-01-16 17:34:49 [ INFO] Found 53907 chat messages.
2024-01-16 17:34:50 [ INFO] Found 57602 chat messages.
100% - [#########################] - 23692 / 23692 - ETA: 00:00:00
Question about custom Exception implementation
Hi @Brisppy,
I'm looking through some of the exceptions in exceptions.py and noticed that the following format is used for custom exceptions.
class ChatExportError(Exception):
def __init__(self, error):
self.message = f'Chat export failed. Error: {error}'From looking at the code, nothing appears (please correct me if I'm wrong) to read the "message" attribute of these custom exceptions nor are these messages printed out console output when the custom exceptions are raised. The custom formatted strings in the exceptions don't appear to be used or output.
The "message" attribute for exceptions appear to be deprecated in Python 3.0.
Does there happen to be an intended use for the "message" attribute in the custom exception?
Errors on download for specific video
Describe the bug
Somehow this video produces many status-code received errors. But works fine if downloaded by yt-dlp from the same machine/IP.
Supplied arguments
python3 twitch-archiver.py -v 1647999281 -t 4 -V -D
Error log
2022-12-08 00:39:46 [ DEBUG] Arguments: {'channel': None, 'vod_id': ['1647999281'], 'client_id': None, 'client_secret': None, 'chat': False, 'video': True, 'threads': 4, 'quality': 'best', 'directory': PosixPath('/root/twitch-archiver'), 'watch': False, 'stream_only': False, 'no_stream': False, 'log_file': False, 'config_dir': PosixPath('/root/.config/twitch-archiver'), 'pushbullet_key': False, 'quiet': 0, 'debug': 10, 'show_config': False}
2022-12-08 00:39:46 [ INFO] Twitch Archiver v2.2.1 - https://github.com/Brisppy/twitch-archiver
2022-12-08 00:39:46 [ DEBUG] Loading config from file.
2022-12-08 00:39:46 [ DEBUG] Settings prior to loading config: {'client_id': '************************25qogj', 'client_secret': '************************wqgoh0', 'oauth_token': '************************stgjwt', 'pushbullet_key': ''}
2022-12-08 00:39:46 [ DEBUG] Generating config from provided arguments.
2022-12-08 00:39:46 [ DEBUG] Settings after loading config: {'client_id': '************************25qogj', 'client_secret': '************************wqgoh0', 'oauth_token': '************************stgjwt', 'pushbullet_key': ''}
2022-12-08 00:39:46 [ DEBUG] Performing Twitch authentication.
2022-12-08 00:39:46 [ DEBUG] Verifying OAuth token.
2022-12-08 00:39:47 [ INFO] OAuth token verified successfully. Expiring in 4981795
2022-12-08 00:39:47 [ INFO] Now processing VOD: 1647999281
2022-12-08 00:39:47 [ INFO] VOD offline.
2022-12-08 00:39:48 [ INFO] Grabbing video...
2022-12-08 00:39:49 [ DEBUG] Downloading segment https://d1ymi26ma8va5x.cloudfront.net/6f122e645c399a4e0051_lillimontv_40054216216_1668018595/chunked/3492.ts to /root/twitch-archiver/2022-11-09_18-30-00 - AMOGUS mit _wer_ __ FSK18 _ __merch __nitrado __klazmo __tiktok __dn - 1647999281/parts/03492.ts
2022-12-08 00:39:49 [ DEBUG] Downloading segment https://d1ymi26ma8va5x.cloudfront.net/6f122e645c399a4e0051_lillimontv_40054216216_1668018595/chunked/3493.ts to /root/twitch-archiver/2022-11-09_18-30-00 - AMOGUS mit _wer_ __ FSK18 _ __merch __nitrado __klazmo __tiktok __dn - 1647999281/parts/03493.ts
2022-12-08 00:39:49 [ DEBUG] Downloading segment https://d1ymi26ma8va5x.cloudfront.net/6f122e645c399a4e0051_lillimontv_40054216216_1668018595/chunked/3494.ts to /root/twitch-archiver/2022-11-09_18-30-00 - AMOGUS mit _wer_ __ FSK18 _ __merch __nitrado __klazmo __tiktok __dn - 1647999281/parts/03494.ts
2022-12-08 00:39:49 [ DEBUG] Downloading segment https://d1ymi26ma8va5x.cloudfront.net/6f122e645c399a4e0051_lillimontv_40054216216_1668018595/chunked/3495.ts to /root/twitch-archiver/2022-11-09_18-30-00 - AMOGUS mit _wer_ __ FSK18 _ __merch __nitrado __klazmo __tiktok __dn - 1647999281/parts/03495.ts
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ ERROR] Code other than 200 received when trying to download segment.
2022-12-08 00:39:49 [ DEBUG] Maximum retries for segment 03495 reached.
Operating system
Ubuntu 20.04
Additional context
Also tried to reduce thread count to 1, removed part files and tried again later - same error. Happens for multiple VODs by multiple different streamers, but not all the time and persistent for the same videos/chunks.
list index out of range error
Describe the bug
Error download VOD even if its available in browser - could be caused by the fact that the channel has been renamed recently.
Supplied arguments
python3 twitch-archiver.py -d /data/twitch -c lillimon -t 1 -N -D
Error log
2023-01-23 11:03:38 [ INFO] 1 VOD(s) in download queue.
2023-01-23 11:03:38 [ DEBUG] VOD queue: {40374471912: ('1715379729', True, True)}
2023-01-23 11:03:38 [ DEBUG] Processing VOD 1715379729 by lillimon
2023-01-23 11:03:38 [ DEBUG] Creating lock file for VOD.
2023-01-23 11:03:38 [ DEBUG] Database path: /home/downloader/.config/twitch-archiver/vods.db
2023-01-23 11:03:38 [ DEBUG] Connection to SQLite DB successful.
2023-01-23 11:03:38 [ DEBUG] Executing SQL statement: SELECT vod_id, video_archived, chat_archived FROM vods WHERE stream_id IS ?
2023-01-23 11:03:38 [ DEBUG] Values: {'stream_id': 40374471912}
2023-01-23 11:03:38 [ INFO] Now processing VOD: 1715379729
2023-01-23 11:03:38 [ INFO] VOD offline.
2023-01-23 11:03:39 [ INFO] Grabbing video...
2023-01-23 11:03:40 [ WARNING] Error 403 or 404 returned when downloading VOD parts - VOD was likely deleted.
2023-01-23 11:03:40 [ INFO] Grabbing chat logs...
2023-01-23 11:03:40 [ DEBUG] Grabbing chat logs from offset: 22352
2023-01-23 11:03:41 [ INFO] Found 15 messages.
2023-01-23 11:03:41 [ INFO] Merging VOD parts. This may take a while.
2023-01-23 11:03:41 [ ERROR] Error downloading VOD 1715379729.
Traceback (most recent call last):
File "/home/downloader/twitch-archiver/src/processing.py", line 394, in get_vod_connector
Utils.combine_vod_parts(vod_json, print_progress=False if self.quiet else True)
File "/home/downloader/twitch-archiver/src/utils.py", line 138, in combine_vod_parts
last_part = int(vod_parts[-1].name.strip('.ts'))
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/downloader/twitch-archiver/src/processing.py", line 401, in get_vod_connector
raise VodMergeError(e)
src.exceptions.VodMergeError: list index out of range
2023-01-23 11:03:41 [ DEBUG] No VOD information returned to channel function, downloader exited with error.
Operating system
Linux
Can't retrive CLIENT_ID and OAUTH_TOKEN:
I can't get CLIENT_ID and OAUTH_TOKEN. I also tried with other browsers but I can't, could the procedure have changed?
Time data error when trying to download a vod
Describe the bug
There was a time data error when trying to download a vod that doesnt match the format in v4.0.0.rc4
Supplied arguments
Arguments supplied to twitch-archiver which produced the error.
Error log
PS D:\Apps\twitch-archiver> twitch-archiver -w -V -c chess --config-dir "D:\Apps\twitch-archiver\Temp\run" -d "D:\Apps\twitch-archiver"
Launching Twitch-Archiver in watch mode.
2024-01-06 12:50:13 [ WARNING] Currently using a development or release candidate build. These may be unfinished or contain serious bugs. Report any issues you encounter to https://github.com/Brisppy/twitch-archiver/issues.
2024-01-06 12:50:13 [ INFO] Twitch Archiver v4.0.0.rc4 - https://github.com/Brisppy/twitch-archiver
2024-01-06 12:50:17 [ INFO] Now archiving channel 'Chess'.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Scripts\twitch-archiver.exe\__main__.py", line 7, in <module>
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\__init__.py", line 174, in main
process.get_channel(channels)
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\processing.py", line 69, in get_channel
channel_videos: list[Vod] = channel.get_channel_videos()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\channel.py", line 293, in get_channel_videos
_videos = [Vod(vod_info=v['node']) for v in _r.json()[0]['data']['user']['videos']['edges']]
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\vod.py", line 54, in __init__
self._parse_dict(vod_info)
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\site-packages\twitcharchiver\vod.py", line 105, in _parse_dict
datetime.strptime(vod_info['publishedAt'], '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=timezone.utc).timestamp()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\_strptime.py", line 555, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\g2\AppData\Local\Programs\Python\Python312\Lib\_strptime.py", line 333, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '2015-07-23T18:51:08.056376Z' does not match format '%Y-%m-%dT%H:%M:%SZ'
Operating system
Windows 11
Additional context
Add any other context about the problem here.
tcd requirement
I tried to find the tcd binary for downloading twitch chat and could only find https://github.com/acid1103/tcd but it looks like a heavy program that compiles gui elements. Is this the same binary that you use?
More metadata
Is it possible to extract more metadata of the VoD? As far as I know yt-dlp extracts additionally chapters (change of the title / change of game) by timestamp.
Also an additional parameter to the script to download the thumbnail(s) would be great so it's possible to show a gallery with thumbnails of all downloaded streams - like on archive.org.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.