arnevogel / concat Goto Github PK

On Windows 7 64-bit, I tried to download parts of at least 2 VODs that were only 1 and 2 months old. Neither of them worked, and both times I got this message:

Couldn't find quality: chunked
No available quality options found

This is odd because I've downloaded some older vods before when they were reuploaded as highlights.

Is this an issue with Twitch itself or something that can be done by concat? Is there any way to work around this other than ensuring I try to download the vod earlier?

I was trying to run concat and ran into the same problems as #36 and #16 but found out the fix.

When downloading the file, if using Safari, the extension will sometimes be switched from no extension to a .dms file, so you have to remove the extension.

Once you get the file without an extension, you have to run chmod +x ./concat_mac in terminal to associate the file as a unix executable or else terminal won't allow you to run it.

After that you should be able to run the file using ./concat_mac from terminal and continue to use the functionality as stated in the README.

For better workflow,
if user intends to query only quality info by -qualityinfo.
Should not check ffmpeg and let user proceeds

ffmepg can only process 100-200 files at a time. We should perform a divide and conquer algorithm for runs with many chunks.

Also, if ffmpeg fails, the concat proceeds to... delete all those precious downloaded chunks! Oops.

Thanks for the tool, though!

I have been using concat for years (great util!); however, just recently, our family started hitting internet data caps, so today I tried to switch from 720p60 as our default, to something lower, this is when I discovered an intermittent issue at combine. Most stream downloads work, others are quite stubborn and do not.

I tried to find one that always fails for easier reproduction of the error; usually everything is fine, or at most, a retry at a different setting will work, though this Minecraft vod I tried to grab for the kids fails consistently at all settings (493274494).

I used the -qualityinfo flag to get valid / available quality settings.

The download seems to work fine; ie, at 720p60 this 4hr stream will grab ~5GB of chunks, at 480p30 this 4hr stream will grab ~2GB of chunks, at 360p30 this 4hr stream will grab ~1GB of chunks, and at 160p30 this 4hr stream will grab ~512MB of chunks. If you look at the 'thumbnails' for each chunk, they are different and show progression through the entire 4hrs, and each piece can indeed be played directly in a media player.

However; when concat_win stitches, the output is approx 10% of that total download size, and while an mp4 file is produced, there is only a few seconds of video, and four hours of audio.

I tried downgrading to v26, then v25, same issue. I have not gone back further.

I tried with ffmpeg v421 and v401, same issue. I have not gone back further.

I tried manually combining the resulting ts files with ffmpeg (not sure if I am using correct syntax) and see errors relating to "non-monotonous DTS in output stream" and "non-existing PPS 0 referenced".

For the most part I have no issues and am happy to abandon this particular vod; already used ridiculous amounts of data in all this testing, but thought it worth reporting in case others encounter the problem and the issue is able to be accounted for / corrected.

I am also happy to test alternate manual combines if you have any suggestions at the ffmpeg syntax.

After checking the patch from #63 I still got no valid quality options for a subscriber only VOD I do have access to by viewing directly on Twitch. (Resulted in "error_code":"vod_manifest_restricted" in the usher.twitch.tv response)

Some debugging and checking for the /api/vods url on VOD playback in browser resulted in the confirmation that you can visit that url with ?client_id=yourclientid as well as ?oauth_token=yourauthtoken to get a token and sig. The first one returns info based on set cookies, which are obviously not there on the command line using concat and the latter is personal to your account after signing in, returning a non-null userid in the token.

This can be confirmed by trying to view a subscriber only VOD while logged in in browser and comparing the /api/vods and usher urls being requested with ?oath_token=undefined or your actual auth token.

Having built a version using oath_token instead of client_id actually resulted in a working subscriber-only VOD download from the command line.

TL;DR: client_id is not the holy grail to get vod downloads to work, using your personal oath_token is

Been successfully using windows version and decided to try OSX.

It segfaults on start.

$ concat_mac.dms 
Segmentation fault: 11

OSX 10.13.3

Hey there,

Would like to request a feature similar to npm's progress bar when handling downloads, see the gif below:

My reasoning behind this is the fact that a series of dots appearing isn't exactly the most helpful progress indicator, and only really serves as an indicator that the tool is still responding.

Thanks,
Elliott

I have noticed that the length that you specify with -start="" and -end="" differs with the length of the video that get outputted. Normally starting with -start="0 0 0" -end="1 0 0" should result in an one hour long video.

I think this happens when the duration of the chunks downloaded from twitch is not the same as the TARGETDURATION in the m3u8 file.

Does this work for twitch vods which are only accessable to authenticated users?

{"error":"Gone","status":410,"message":"this API has been removed."}

Concat's -start and -end flags are not working correctly for me. When I run the command ./concat_linux -vod="383949775" -start="1 7 15" -end="1 7 37", I get a video that is 35 seconds long when it should in fact be 22. I have also compared it with the actual VOD at twitch.tv/videos/383949775 and the trimming is indeed wrong. It looks like it does it from "1 7 5" to "1 7 40" instead (in the actual VOD timestamp).

I am running concat on Ubuntu 16.04 and am using the latest version.

This is a great project and I would really appreciate any input from other contributors, I will also look into the code myself.

Trying to download a VoD and I received an error 'Couldn't access usher api'. I have downloaded a few VoDs before with no issue

(Windows version)
So basically any download I try to do fails because I get an index out of range error. Weird thing with this is also that the path shown in the error message is probably the path you had your program in, but does not exist on my machine.
Happens with the newest release, but also with older ones.

panic: runtime error: index out of range goroutine 1 [running]: main.downloadPartVOD(0xc00000a0f0, 0x9, 0x923731, 0x1, 0x70acfc, 0x4, 0x70b515, 0x7, 0x70a86b, 0x1, ...) /home/arne/code/test/concat/main.go:341 +0x208b main.main() /home/arne/code/test/concat/main.go:681 +0x59b

With the latest version I get a "Memory access error". Twitch seems to have changed something, because with an older release of concat I don't get this error but instead it tries to find the VOD and the suggested quality option, but it doesn't work either.^^

I have ffmpeg installed and ffmpeg -version works etc so I know it is surely installed however I get

Combining parts

ffmpeg error
Deleting chunks
Deleting temp dir
All done!

Please output the ffmpeg err code so we can get useful information from it. Also in this event I feel like we should be prompted as to whether we want to delete the chunks or not - in this event I would much rather just have the chunks than nothing.

trying to download a vod, when finished downloading it tries to combine the parts but it throws an ffmpeg error as shown below.
./concat_ubuntu -vod=111836368 -start="0 0 0"

......................................................
Combining parts

ffmpeg error
Deleting chunks
Deleting temp dir
All done!
root@vps:~/twitch#

This only happens on "long" vods. It doesnt happen on https://www.twitch.tv/videos/112067373 .

Since a few days http://api.twitch.tv/api returns 410.Gone status, while http://usher.twitch.tv/vod is unreachable. I know you don't support this little program anymore, but can you quickly confirm that the APIs are completely gone and there's nothing to be done to fix it?

I'm trying to run concat_mac but it keeps giving me the error

Could not find ffmpeg, make sure to have ffmpeg avaliable on your system.

But I do have it, I placed ffmpeg in the same directory as concat_mac . Here's the ls listing.

bin % ls
concat_mac	ffmpeg		ffplay		ffprobe

Where is concat looking for ffmpeg?

I'm on MacOS 10.15.5

When you Download a Full VOD or a VOD time frame. If the first chunk of either the section or the Full VOD is muted by Twitch because of copyright the entire VOD stays muted. Even if the audio is back at some point in the vod the download has still no audio.

When you start a Download 1 minute after the muted part audio is there and even after at a later time the audio is muted by Twitch it is muted in the download too but the audio resumes after the muted part.

It only occurs if the VOD starts muted by Twitch. The download must have had audio before. If its muted right at the beginning the whole Vod stays muted. If it had audio before it works like in the twitch player.

I'm too familiar with command line, but how do I run this on macOS?

I have ffmpeg installed. I tried running it via terminal and get command not found.

As of today i get this response trying to download a VOD:

Contacting Twitch Server

tokenAPILink: https://api.twitch.tv/api/vods/509622116/access_token?&client_id=aokchnui2n8q38g0vezl9hq6htzy4c

Sig: <nil>, Token: <nil>

usherAPILink: http://usher.twitch.tv/vod/509622116?nauthsig=&nauth=&allow_source=true

Usher API response:

<table border="1"><tr><td><b>url</b></td><td>/vod/509622116?nauthsig=&lt;nil&gt;&amp;nauth=&lt;nil&gt;&amp;allow_source=true</td></tr>
<tr><td><b>error_code</b></td><td>nauth_sig_invalid</td></tr>
<tr><td><b>type</b></td><td>error</td></tr>
<tr><td><b>error</b></td><td>Bad auth token</td></tr></table>

map[]

Have you (or Twitch) changed the way auth works?

Encountered on Ubuntu 17.10 x86_64. When downloading a large VOD with a lot of chunks, concat's combining phase will sometimes fail with a "Too many open files/ffmpeg error" message. Technically this is a system issue, and can be worked around by increasing ulimit -n (default was 1024 and concat succeeded after setting it to 99999), however it'd be great if concat handled this case internally or if there was an option to not delete chunks after downloading so they could be concatenated manually.

When I try to do any video this happens, is there something I'm missing?

What is this *.dms and how do i use it? the Internet refers to Amiga disk images (this can't be right), but i can't either open it with unzip or mount it with mount. Can you just, you know, package it as a usual osx *.dmg? I'm too lazy to build the source every time it updates ;P.

I was able to use this program just fine for months. Now every time I attempt to use the program to download a vod, this happens.

  Program 1 
    Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 162 kb/s
    Stream #0:1[0x101]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuvj420p(pc, bt709), 1920x1080, 60 fps, 60 tbr, 90k tbn, 120 tbc
    Stream #0:2[0x102]: Data: timed_id3 (ID3  / 0x20334449)
[mp4 @ 0x18b64a0] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x18b64a0] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to '236606799.mp4':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuvj420p, 1920x1080, q=2-31, 60 fps, 60 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 162 kb/s
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help

ffmpeg error
Deleting chunks
Deleting temp dir
All done!

Ive been trying for the last few hours to download a specific VOD, and due to its size and the instability of my internet connection, I can never get it to complete. I have all these temporary files downloaded, but restarting the download doesnt seem to make use of them.

What would it take to implement resume functionality? I've never used Go before, but I would be happy to try implementing it if I was pointed in the right direction.

I found a misspelling in the program. When downloading the full VOD, the following message is printed to the console:

Dowbloading full vod

As many may not include ffmpeg in build path
or put concat in the same folder as ffmpeg executable,
i suggest provide location input.
E.g. -ffmepg "D;/ffmpeg/bin/ffmpeg.exe"

I want to change the download path
and
"file name" already exists , Can I change the file name automatically?

Bug Report

Issue: Cannot download videos.

Command: concat -vod="123456789"

Expected: begins a Twitch video download.

Actual:

user@host:~$ concat -vod="482122315"
Contacting Twitch Server
Couldn't find quality: chunked
No available quality options found

Test video-ids: ["482122315", "7695281", "378008253", "473298056"]

Test Environments:

• Docker-container (from python)
• CircleCI (docker + ci utils)
• Desktops (Pop_os!, Ubuntu)

Other info:
I saw a similar issue with pytube when YouTube changed some of their back-end stream options. It is likely this is the result of a Twitch update, considering your app has been working great for the past few months.

I wondered if the API was broken so I tried this:
curl https://api.twitch.tv/api/vods/482122315/access_token?client_id=aokchnui2n8q38g0vezl9hq6htzy4c

And the repsonse was not great:
{"error":"Gone","status":410,"message":"this API has been removed."}

Issue: Cannot download videos , i try new vods and vods that had already been downloaded and it doesn't work.
The Error:
"Couldn't find quality: chunked"

Environments: Windows 10 64 bits

Hi! I've really appreciated this tool and I saw the Wiki page for client IDs which was super useful for me. I'd like to pay it forward and expand the wiki a bit just to cover this program in more detail, and to add/update links to the new wiki pages in the readme and this program's error messages.

I cannot push my wiki to this repo because github doesn't do pull requests on wikis. You could essentially rebase this repo's wiki with the wiki in my fork. You can get my wiki files by cloning my fork's wiki, copying those files, and applying it to your own wiki. The main.go file would also need any wiki links updated to the new page titles/headers as needed.

My fork isn't perfect yet but I at least wanted to start this dialog. Let me know what you think: https://github.com/DeerTears/concat/wiki

when i tried to download from 3:11:00 it downloaded from 2:25

with some VODs, the clip downloaded doesn't match the timestamp inputted based on what the Twitch player displays at that timestamp.
ex: with VOD 558361951
with argument -start="7 16 20"

this is a frame 10 seconds into the clip downloaded with concat, so it should be 7:16:30

on the twitch player: this frame is actually at 7:16:26

This offset is not constant either, at -start="0 3 10" of the same VOD concat is about 10 seconds ahead of the Twitch player

Using the max-concurrent-downloads flag for some reason completely crashes my router.
Once the router reboots, it blocks all http requests so I can't get into the router control panel.
This is what was causing issue #57 as the usher api uses http.

after i download the exe and click open this thing pops up then disappears quickly
https://prnt.sc/ky094q
is there any other way i can install this

Would love to see a download option for audio only.
Loving this tool big time! But would love that additional option.

As commented here you need to manually rename the VOD file that is the result of the download & concat.

Adding a -filename option to use that as preferred filename after concat when provided could be nice to have. (While still keeping the Twitch VOD ID for the temp folder; so resuming downloads still works even when the filename option isn't provided for a 2nd attempt).

Q:>concat_win -vod 312517122 -start="0 25 27" -end "0 28 04"
Contacting Twitch Server
Could not access twitch token api

Did they change the API?

Hey there,

Would be nice if the repo had binaries so I could just git pull to get my updates of this.

Thanks,
Elliott

Somehow (I think recent) twitch videos don't download:

>concat_win.exe -vod 180380104
Contacting Twitch Server
Selected quality: chunked
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
panic(0x797f40, 0xc082002050)
        /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3f4
main.downloadPartVOD(0xc082002250, 0x9, 0x80b278, 0x1, 0x80dd48, 0x4, 0x80d5d0, 0x7)
        /home/arne/Dokumente/go/src/github.com/arnevogel/concat/main.go:355 +0x2816
main.main()
        /home/arne/Dokumente/go/src/github.com/arnevogel/concat/main.go:494 +0x499

>concat_win.exe -vod 179341125
Contacting Twitch Server
Selected quality: chunked
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
panic(0x797f40, 0xc082002050)
        /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3f4
main.downloadPartVOD(0xc082002250, 0x9, 0x80b278, 0x1, 0x80dd48, 0x4, 0x80d5d0, 0x7)
        /home/arne/Dokumente/go/src/github.com/arnevogel/concat/main.go:355 +0x2816
main.main()
        /home/arne/Dokumente/go/src/github.com/arnevogel/concat/main.go:494 +0x499

When starting the download with only the -start parameter the program downloads the full VOD anyway, instead of starting at the specified point.

Hi , I really like how this tool works, but after internet disconnection , will it resume the download and combine of parts or will it start over ?
thanx

Was wondering if a Mac executable can be added for the newest release. Not familiar with Go so I'm not sure if I can just compile it myself. Thanks!

So every single time I've attempted to use the program, I get the error "too many open files." And then it doesn't export my VODs. I posted the terminal log in a .txt file.
Error.txt

I'm on Linux Ubuntu 16.04. I have an i5-7500 with a GTX 1060 6gb.

Hey love your tool. Just wanted to check if you could see into the issue- No videos are downloading - by default its tryiing to download chunked quality and looks like twitch is not providing any quality options anymore. So no videos are downloading

When you try to download vod with start and end timestamps, it downloads incorrect part of the vod. The downloader also doesn't work properly on your official site.

From what I tested its always a different delay. Maybe script doesn't calculate chunks properly? I think before (when concat was still working) twitch was sending 5 sec chunks and now they send 12 second chunk.