alexaplin / nndownload Goto Github PK

Retrieve serverUrl from the thread object in data-api-data, which should usually be http://nmsg.nicovideo.jp/api/. POST the default thread ID:

<packet><thread thread="[id]" version="20061206" res_from="-1000" scores="1"/></packet>

This returns an XML document that can be converted to .ass with danmaku2ass.

Allow filenames to be populated with metadata, e.g. -f [%user% (%id%)] %video_id% - %title% (%date%).

Having downloaded files without the --force-high-quality during economy hours and then firing up the script with the flag --force-high-quality causes the file to be resumed: "Resuming previous download."

Resulting in "broken" file. The previous, low quality, video can still be played, but other data is lost.

Expected behaviour:
Upon detection of higher quality, the lower quality should be moved or replaced by the newer/better file.

After I logged in withpython nndownload.py -u [username] -p [password] -o [output directory] [url], it just showed

Logging in...
Logged in.

and nothing comes out and seems the recorder doesn't work.

After I deleted the -o segement, it comes error like

Traceback (most recent call last):
  File "nndownload.py", line 713, in main
    process_url_mo(session, url_mo)
  File "nndownload.py", line 678, in process_url_mo
    request_rtmp(session, url_id)
  File "nndownload.py", line 176, in request_rtmp
    urls = urllib.parse.unquote(nama_info.getElementsByTagName("contents")[0].firstChild.nodeValue).split(",")
IndexError: list index out of range

How to fix this?

On at least one video, the script is throwing an error when trying to get uploader and uploader_id here:

        template_params["uploader"] = params["owner"]["nickname"].strip(" さん")
        template_params["uploader_id"] = int(params["owner"]["id"])

It appears that the video has no "owner".

I've modified the script so that section reads:

        if params["owner"]:
            template_params["uploader"] = params["owner"]["nickname"].strip(" さん")
            template_params["uploader_id"] = int(params["owner"]["id"])
        else:
            template_params["uploader"] = ""
            template_params["uploader_id"] = 0

Which seems to solve the issue. Not sure if this can occur in videoDetail (why is that not mutually exclusive with video?)

Also, that strip() is potentially destructive if the nickname has さ or ん at the beginning or end of their real nickname.

Whoops!

Because nicovideo is a paid member, single-threaded speed limit, but if you use multi-threading, you can achieve very fast speed.

We've outlived having a flat script.

Community streams seem to require custom commands as opposed to official streams (#10).

Right now we report Niconama as invalid URLs.

Some videos report size_low and size_high as 1, so we need another way to determine quality.

Store .json objects.

optparse is deprecated since Python 2.7.

Whenever the script meets an exception and exits, the script exits with return code 0.
Example:

...
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
$ echo $?
0

Expected: Return code 1

WebSockets are now used for Niconama streams.

When the video type is flv as reported by the thumb info API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm1715919, the HTML5 player will always be served the low quality mp4 source. swf movies are unaffected as the HTML5 player always redirects to the Flash player.

Python nndownload.py -c -f -u MailAddress -p Password http://www.nicovideo.jp/watch/so22958427

Logging in...
Logged in.
Traceback (most recent call last):
  File "C:\Software\nndownload\nndownload.py", line 697, in main
    process_url_mo(session, url_mo)
  File "C:\Software\nndownload\nndownload.py", line 654, in process_url_mo
    request_video(session, url_id)
  File "C:\Software\nndownload\nndownload.py", line 222, in request_video
    template_params = perform_api_request(session, document)
  File "C:\Software\nndownload\nndownload.py", line 585, in perform_api_request
    template_params = collect_parameters(template_params, params)
  File "C:\Software\nndownload\nndownload.py", line 623, in collect_parameters
    template_params["uploader"] = params["uploaderInfo"]["nickname"].rstrip(" さん") if params["uploaderInfo"] else None
KeyError: 'uploaderInfo'

If I wish to download only the metadata of a niconico video, I have to download the entire video and then delete it afterwards. For some videos this can be very time-consuming, so preferably there should be an argument flag that skips downloading the video.

Maybe something like --no-video or --only-metadata

Illegal path characters in the title or username are not substituted or removed and will cause an exception on write.

.flv and .swf videos previously only available over the Flash player (e.g. https://www.nicovideo.jp/watch/nm5142117 and https://www.nicovideo.jp/watch/nm11960162) are now converted for the HTML5 player. Noticed changes:

• Converted videos are served by default and to logged out users.

I download this manga get this issue.

nndownload.py https://seiga.nicovideo.jp/comic/33772
Logging in...
Logged in.
Downloading comic 33772...
1/17
Traceback (most recent call last):
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1330, in
main()
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1318, in main
process_url_mo(session, url_mo)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 1280, in process_url_mo
download_manga(session, url_id)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 564, in download_manga
download_manga_chapter(session, chapter_id)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 510, in download_manga_chapter
template_params = collect_seiga_manga_parameters(document, template_params)
File "D:\C_Drive\Users\usr\AppData\local\Programs\Python\Python37-32\Scripts\nndownload.py", line 496, in collect_seiga_manga_parameters
template_params["uploader_id"] = SEIGA_USER_ID_RE.search(document.select("dd.user_name a")[0]["href"]).group(1)
IndexError: list index out of range

Easy enough to get the RTMP tokens with the player API, e.g. http://watch.live.nicovideo.jp/api/getplayerstatus?v=lv305524399. Official and user broadcasts are different. We could also take the streamlink/streamlink approach for live broadcasts and launch a player directly after. For region locked streams, the region must match to request the API, but the RTMP link can be used in outside regions.

Thumbnails are stored on tn-skr[1-4].smilevideo.jp?smile=[id]. Large thumbnails started to appear in late 2011 and can be retrieved by appending .L, e.g. http://tn-skr1.smilevideo.jp/smile?i=23862638.L.

The thumbnail and other metadata for a video can be retrieved from the API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm23862638.

For example, these videos.

『 Antinomie 』/ ELECTROCUTICA
http://www.nicovideo.jp/watch/sm33004336

TGM3 SHIRASE(W) 3:53:85
http://www.nicovideo.jp/watch/sm33307637

size_low and size_high in bytes can be extracted from the thumb info API, e.g. http://ext.nicovideo.jp/api/getthumbinfo/sm1097445.

websockets doesn't currently support this, but it's pending inclusion upstream. There are hacky ways to include this. I'm not entirely sure how this affects region locked streams, but I was able to generate a working stream through a proxy.

So after installing it from pip, users can call it globally as a CLI tool (if they added /Scripts/ to their path).

I'm not very familiar in this area but I think it should be something covered here.

Playlists, e.g. mylist/9594806.

Restarting the script on user/mylist that has more than 50 entries, will cause the script to run into rate limit that is imposed on the video pages. The reported error is: requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://www.nicovideo.jp/watch/sm...
The options used to run the script are: --dump-metadata --download-thumbnail --download-comments --force-high-quality

Example:
I archive a user that has exactly 50 entries. After all videos are archived, I restart the script with the same parameters.
Depending on the mood of NND after approximately 3 minutes and 30 second, and between fetching the 43rd-47th video, NND will respond with a 503 error.

The video page will be available after 30-60 seconds.

Suggestion:
If the script encounters a 503 error, catch that exception and wait 60 seconds before restarting the download, is possible multiple(3) retries.
Optionally, have a flag that allows to define wait time, otherwise a global variable that the user can edit inside the script.

Easy enough to get the RTMP tokens with the player API, e.g. http://watch.live.nicovideo.jp/api/getplayerstatus?v=lv305524399. Official and user broadcasts are different. For region locked streams, the region must match to request the API, but the RTMP link can be used in outside regions.

Add an option to load a user's credentials (username, password) from a file, rather than supplying them both on the commandline. This improves security on Linux, where command line arguments are visible to anyone who can view running processes; and also makes it slightly easier to use generated passwords.

Returns None when a custom filepath doesn't specify a directory.

From what I've seen the nama API reports all RTMP stream URLS as invalid now. Channel and community streams continue to offer RTMP, likely for older devices, but on browsers the HLS stream is what's actually served.

Nicocas is no longer strictly experimental. The cas.nicovideo.jp subdomain seems to have been retired for live pages but is still used for the API.

http://site.nicovideo.jp/nicocas/pc/
https://live.nicovideo.jp/recent?tab=nicocas

Nico is experimenting with additional live features. As part of this, the live infrastructure seems to have moved to DMC.

Page URL: https://cas.nicovideo.jp/user/[user_id]/[live_id]
API root URL: https://api.cas.nicovideo.jp/v1/services/live/programs/[live_id]

• OPTIONS /watching-qualities
  • GET /watching-qualities
    • Available qualities (superhigh, high, middle, low)
• OPTIONS /watching
  • POST /watching
    • 201 Created
    • Program metadata, commenting WebScoket server and threads, master stream URL
  • PUT /watching
    • 200 OK
    • Stream URL

General flow seems to be:

• OPTIONS to /watching-qualities

  • Access-Control-Request-Method: GET
  • Access-Control-Request-Headers: x-connection-environment,x-frontend-id
  • Origin: https://cas.nicovideo.jp

• GET to /watching-qualities

  • X-Frontend-Id: 91
  • X-Connection-Environment: ethernet

• OPTIONS to /watching

  • Access-Control-Request-Method: GET
  • Access-Control-Request-Headers: content-type,x-connection-environment,x-frontend-id
  • Origin: https://cas.nicovideo.jp

• POST to /watching

  • Content-Type: application/json
  • X-Frontend-Id: 91
  • X-Connection-Environment: ethernet
  • audienceToken cookie
  • JSON data:
    • actionTrackId: unique identifier, e.g. d558874173_151779710194. Must be of the form [0-9a-f]{10}_[epoch_value]
    • isBroadcaster: false
    • streamCapacity: highest quality reported by /watching-qualities
    • streamProtocol: https
    • streamQuality: auto or highest available
  • Master stream URL in response at [data][streamServer][url]
    • [root_dmc_url] + [playlist], e.g. https://pa055fd0b1d.dmc.nico/hlslive/ht2_nicolive/nicolive-production-pg9104945840708_6b5508405a67e8b4eb92202994cae713281e54c2d98da1f7f900e7c7c3308f6b/4/ts/playlist.m3u8?ht2_nicolive=34523456.0wwmcd_pnv8ek_9toj69v66xwy

• PUT to /watching every 20 seconds

  • Master and playlist URLs will change and must be updated

Use --english to dump all metadata, not just comments.

Nico has been making various changes leading up to version (く) starting June 28th. Some posting and quality restrictions have been lifted, but it's unclear whether the DMC process will change. As of now, everything seems to work as expected with the script.

http://blog.nicovideo.jp/niconews/74201.html
http://blog.nicovideo.jp/niconews/71833.html

NicoMovieMaker videos (nm#) may not be viewable in an external player. Videos with still images (e.g. nm11960162) will play fine with a pass through ffmpeg. Slideshows and GIFs (e.g. nm2829323) may play with improper frame timings or loops. Flash animations (e.g. nm4809023) will likely not play at all and may require other libraries or solutions.

Successive calls are required to retrieve all comments.