alexaplin / nndownload Goto Github PK
View Code? Open in Web Editor NEWDownload and process links from Niconico (nicovideo.jp)
License: MIT License
Download and process links from Niconico (nicovideo.jp)
License: MIT License
http://site.nicovideo.jp/nicocas/pc/
https://live.nicovideo.jp/recent?tab=nicocas
Nico is experimenting with additional live features. As part of this, the live infrastructure seems to have moved to DMC.
Page URL: https://cas.nicovideo.jp/user/[user_id]/[live_id]
API root URL: https://api.cas.nicovideo.jp/v1/services/live/programs/[live_id]
General flow seems to be:
OPTIONS to /watching-qualities
Access-Control-Request-Method: GET
Access-Control-Request-Headers: x-connection-environment,x-frontend-id
Origin: https://cas.nicovideo.jp
GET to /watching-qualities
X-Frontend-Id: 91
X-Connection-Environment: ethernet
OPTIONS to /watching
Access-Control-Request-Method: GET
Access-Control-Request-Headers: content-type,x-connection-environment,x-frontend-id
Origin: https://cas.nicovideo.jp
POST to /watching
Content-Type: application/json
X-Frontend-Id: 91
X-Connection-Environment: ethernet
audienceToken
cookieactionTrackId
: unique identifier, e.g. d558874173_151779710194
. Must be of the form [0-9a-f]{10}_[epoch_value]isBroadcaster
: false
streamCapacity
: highest quality reported by /watching-qualitiesstreamProtocol
: https
streamQuality
: auto
or highest available[data][streamServer][url]
PUT to /watching every 20 seconds
On at least one video, the script is throwing an error when trying to get uploader and uploader_id here:
template_params["uploader"] = params["owner"]["nickname"].strip(" さん")
template_params["uploader_id"] = int(params["owner"]["id"])
It appears that the video has no "owner".
I've modified the script so that section reads:
if params["owner"]:
template_params["uploader"] = params["owner"]["nickname"].strip(" さん")
template_params["uploader_id"] = int(params["owner"]["id"])
else:
template_params["uploader"] = ""
template_params["uploader_id"] = 0
Which seems to solve the issue. Not sure if this can occur in videoDetail (why is that not mutually exclusive with video?)
Also, that strip() is potentially destructive if the nickname has さ or ん at the beginning or end of their real nickname.
size_low
and size_high
in bytes can be extracted from the thumb info API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm1097445.
Whoops!
If I wish to download only the metadata of a niconico video, I have to download the entire video and then delete it afterwards. For some videos this can be very time-consuming, so preferably there should be an argument flag that skips downloading the video.
Maybe something like --no-video
or --only-metadata
For example, these videos.
『 Antinomie 』/ ELECTROCUTICA
http://www.nicovideo.jp/watch/sm33004336
TGM3 SHIRASE(W) 3:53:85
http://www.nicovideo.jp/watch/sm33307637
Playlists, e.g. mylist/9594806.
Thumbnails are stored on tn-skr[1-4].smilevideo.jp?smile=[id]. Large thumbnails started to appear in late 2011 and can be retrieved by appending .L, e.g. http://tn-skr1.smilevideo.jp/smile?i=23862638.L.
The thumbnail and other metadata for a video can be retrieved from the API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm23862638.
.flv and .swf videos previously only available over the Flash player (e.g. https://www.nicovideo.jp/watch/nm5142117 and https://www.nicovideo.jp/watch/nm11960162) are now converted for the HTML5 player. Noticed changes:
Allow filenames to be populated with metadata, e.g. -f [%user% (%id%)] %video_id% - %title% (%date%).
Having downloaded files without the --force-high-quality
during economy hours and then firing up the script with the flag --force-high-quality
causes the file to be resumed: "Resuming previous download."
Resulting in "broken" file. The previous, low quality, video can still be played, but other data is lost.
Expected behaviour:
Upon detection of higher quality, the lower quality should be moved or replaced by the newer/better file.
Nicocas is no longer strictly experimental. The cas.nicovideo.jp subdomain seems to have been retired for live pages but is still used for the API.
Illegal path characters in the title or username are not substituted or removed and will cause an exception on write.
Restarting the script on user/mylist that has more than 50 entries, will cause the script to run into rate limit that is imposed on the video pages. The reported error is: requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://www.nicovideo.jp/watch/sm...
The options used to run the script are: --dump-metadata --download-thumbnail --download-comments --force-high-quality
Example:
I archive a user that has exactly 50 entries. After all videos are archived, I restart the script with the same parameters.
Depending on the mood of NND after approximately 3 minutes and 30 second, and between fetching the 43rd-47th video, NND will respond with a 503 error.
The video page will be available after 30-60 seconds.
Suggestion:
If the script encounters a 503 error, catch that exception and wait 60 seconds before restarting the download, is possible multiple(3) retries.
Optionally, have a flag that allows to define wait time, otherwise a global variable that the user can edit inside the script.
After I logged in withpython nndownload.py -u [username] -p [password] -o [output directory] [url]
, it just showed
Logging in...
Logged in.
and nothing comes out and seems the recorder doesn't work.
After I deleted the -o
segement, it comes error like
Traceback (most recent call last):
File "nndownload.py", line 713, in main
process_url_mo(session, url_mo)
File "nndownload.py", line 678, in process_url_mo
request_rtmp(session, url_id)
File "nndownload.py", line 176, in request_rtmp
urls = urllib.parse.unquote(nama_info.getElementsByTagName("contents")[0].firstChild.nodeValue).split(",")
IndexError: list index out of range
How to fix this?
Successive calls are required to retrieve all comments.
Community streams seem to require custom commands as opposed to official streams (#10).
I download this manga get this issue.
nndownload.py https://seiga.nicovideo.jp/comic/33772
Logging in...
Logged in.
Downloading comic 33772...
1/17
Traceback (most recent call last):
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1330, in
main()
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1318, in main
process_url_mo(session, url_mo)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1280, in process_url_mo
download_manga(session, url_id)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 564, in download_manga
download_manga_chapter(session, chapter_id)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 510, in download_manga_chapter
template_params = collect_seiga_manga_parameters(document, template_params)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 496, in collect_seiga_manga_parameters
template_params["uploader_id"] = SEIGA_USER_ID_RE.search(document.select("dd.user_name a")[0]["href"]).group(1)
IndexError: list index out of range
optparse is deprecated since Python 2.7.
From what I've seen the nama API reports all RTMP stream URLS as invalid now. Channel and community streams continue to offer RTMP, likely for older devices, but on browsers the HLS stream is what's actually served.
So after installing it from pip
, users can call it globally as a CLI tool (if they added /Scripts/
to their path).
I'm not very familiar in this area but I think it should be something covered here.
Add an option to load a user's credentials (username, password) from a file, rather than supplying them both on the commandline. This improves security on Linux, where command line arguments are visible to anyone who can view running processes; and also makes it slightly easier to use generated passwords.
NicoMovieMaker videos (nm#) may not be viewable in an external player. Videos with still images (e.g. nm11960162) will play fine with a pass through ffmpeg. Slideshows and GIFs (e.g. nm2829323) may play with improper frame timings or loops. Flash animations (e.g. nm4809023) will likely not play at all and may require other libraries or solutions.
When the video type is flv
as reported by the thumb info API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm1715919, the HTML5 player will always be served the low quality mp4
source. swf
movies are unaffected as the HTML5 player always redirects to the Flash player.
Nico has been making various changes leading up to version (く) starting June 28th. Some posting and quality restrictions have been lifted, but it's unclear whether the DMC process will change. As of now, everything seems to work as expected with the script.
http://blog.nicovideo.jp/niconews/74201.html
http://blog.nicovideo.jp/niconews/71833.html
Easy enough to get the RTMP tokens with the player API, e.g. http://watch.live.nicovideo.jp/api/getplayerstatus?v=lv305524399. Official and user broadcasts are different. For region locked streams, the region must match to request the API, but the RTMP link can be used in outside regions.
WebSockets are now used for Niconama streams.
Right now we report Niconama as invalid URLs.
Use --english to dump all metadata, not just comments.
Returns None
when a custom filepath doesn't specify a directory.
Easy enough to get the RTMP tokens with the player API, e.g. http://watch.live.nicovideo.jp/api/getplayerstatus?v=lv305524399. Official and user broadcasts are different. We could also take the streamlink/streamlink approach for live broadcasts and launch a player directly after. For region locked streams, the region must match to request the API, but the RTMP link can be used in outside regions.
We've outlived having a flat script.
Store .json objects.
Python nndownload.py -c -f -u MailAddress -p Password http://www.nicovideo.jp/watch/so22958427
Logging in...
Logged in.
Traceback (most recent call last):
File "C:\Software\nndownload\nndownload.py", line 697, in main
process_url_mo(session, url_mo)
File "C:\Software\nndownload\nndownload.py", line 654, in process_url_mo
request_video(session, url_id)
File "C:\Software\nndownload\nndownload.py", line 222, in request_video
template_params = perform_api_request(session, document)
File "C:\Software\nndownload\nndownload.py", line 585, in perform_api_request
template_params = collect_parameters(template_params, params)
File "C:\Software\nndownload\nndownload.py", line 623, in collect_parameters
template_params["uploader"] = params["uploaderInfo"]["nickname"].rstrip(" さん") if params["uploaderInfo"] else None
KeyError: 'uploaderInfo'
Whenever the script meets an exception and exits, the script exits with return code 0.
Example:
...
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
$ echo $?
0
Expected: Return code 1
Retrieve serverUrl
from the thread
object in data-api-data
, which should usually be http://nmsg.nicovideo.jp/api/. POST the default
thread ID:
<packet><thread thread="[id]" version="20061206" res_from="-1000" scores="1"/></packet>
This returns an XML document that can be converted to .ass with danmaku2ass.
websockets doesn't currently support this, but it's pending inclusion upstream. There are hacky ways to include this. I'm not entirely sure how this affects region locked streams, but I was able to generate a working stream through a proxy.
Because nicovideo is a paid member, single-threaded speed limit, but if you use multi-threading, you can achieve very fast speed.
Some videos report size_low
and size_high
as 1, so we need another way to determine quality.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.