Problem

We are not capturing all edition labels and generating quite a few false positives. We should also capture Kodi and Plex standards.

Plex correctly points out that version and edition are different

Versions all represent the same release of an item. So, you can have multiple versions (1080p vs 480p, HEVC vs H.264, MP4 vs MKV) of The Empire Strikes Back, but they’re all for the same theatrical release of the movie.

Editions represent different releases of an item. So, the “theatrical release” vs the “Special Edition” of The Empire Strikes Back. Or “Theatrical” vs “Director’s Cut” vs “Final Cut” of Blade Runner. Editions would also be appropriate for a 2D vs 3D version of a movie.

Plex

@see https://support.plex.tv/articles/200381043-multi-version-movies/
@see https://support.plex.tv/articles/multiple-editions/

• title (year) {edition-Edition Name}
  i.e. Blade Runner (1982) {edition-Director's Cut}.mkv

Note that Plex also adds the imdb and tmdb codes in curly brackets:

• title (year) {imdb-tt0372784}.mp4
• title (year) {tmdb-272}.mp4

Jellyfin

@see https://jellyfin.org/docs/general/server/media/movies/
uses everything after " - ":

• title (year) - Edition Name
  i.e. Blade Runner (1982) - The Final Cut

Kodi

#see https://kodi.wiki/view/Video_versions
uses everything after " - ":

• title (year) - Edition Name
  i.e. Blade Runner (1982) - The Final Cut [bluray]

Solution

Change Version column name to Edition.

Change the regex and regex order:

1. Search for '.* {edition-...} .*' (plex)
2. Search for '.* (\d{4}) - ...' (kodi & Jellyfin)
3. Search for '.* Director's Cut|Theatrical Cut|Extended Cut .*' (fallback)

This is probably because trying to do this all in jq and needs optimising

In some cases, the user may want to list the release type in the filename (see Edition column: after - - Jellyfin and in {} - Plex).

This can lead to useless duplication of data.

The Edition column should strip Release type text from the field. It may be useful to set up a function that returns only the regex for release types words - this will make for a single point of code and easier maintenance.

It appears that the code is not fully posix compliant yet.

Arch links sh to the bash processing environment. This leads to issues undetected on Ubuntu:

grep: warning: stray \ before white space
grep: warning: stray \ before :
grep: warning: stray \ before -

Along with the progressbar not displaying correctly on arch.

This script should work on sh, dash as well as bash environments.

Plex allows for multiple versions of as video (alongside Edition), e.g.:

Blade Runner (1982).4k.hevc {edition-Director's Cut}.mkv

We should also search for the following keywords after date (4 digits):

• avc
  • avc
  • h 264
  • h264
  • mpeg-4 part 10
• hevc
  • hevc
  • h 265
  • h265
  • mpeg-h part 2
• vvc
  • h266
  • h.266
• mpeg-4
  • mpeg4
  • mpeg 4
• mpeg-2
  • mpeg2
  • mpeg 2
• av-1
  • av1
  • av-1
• 10-bit
  • 10bit
  • 10 bit
• dc
• vc-1
• hdr
• hdr10
• hdr10+
• vp9
• divx
• mjpeg
• 3d

references:

• https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding
• https://en.wikipedia.org/wiki/List_of_codecs#Video_compression_formats
• https://support.plex.tv/articles/200381043-multi-version-movies/
• https://target-video.com/best-video-codecs-to-use-and-how-to-choose-the-right-one/
• https://en.wikipedia.org/wiki/VC-1
• https://en.wikipedia.org/wiki/MPEG-2
• https://en.wikipedia.org/wiki/List_of_codecs
• https://videocompressionguru.medium.com/mpeg-4-vs-avc-h-264-vs-mp4-what-is-the-difference-f3c166c1cf56

It would be very convenient to have CLI args for all of the .env args.

This would allow one-off changing of parameters for a single run, and as a side effect make the scripts self documenting.

The script was created on Ubuntu (bash shell).

It is untested on POSIX systems and other shells. Testing needs to be done on other systems/shells and possibly syntax updated to ensure the maximum portability.

Some very good comments in https://unix.stackexchange.com/questions/555099/what-is-the-most-portable-shell-and-relevant-best-practices-to-follow

mediainfo and ffprobe are currently unable to detect these. This means that if they are not in the filename, then they are never found.

As soon as a way of detecting these is found, it should be implemented

Currently, the script assumes a single Audio or Video stream, and looks at only the first.

After optimisation - fetching the full ffprobe result once and using jq to extract the required value from the stream/s, we now get the problem of potential multiple values (separated by newline). e.g.:

AVC
MJPEG
MJPEG

These need to be represented in a sensible manner, in a single cell in the CSV/spreadsheet. and also pair with matching columns, such as Resolution|Video

The individual settings are too fine-grained. For readability and simpler configuration, these should be combined into a single .env variable:

detect_if_not_in_filename

Proposal

Allow the user to specify the media scanning package or allow automatic selection in the .env file. If the config is not set, then fallback to automatic detection.

If the package specified does not exist, then fail gracefully.

If automatic, then prefer ffprobe, and fallback to mediainfo if ffprobe not present. If neither packages are available, exit gracefully.

Rationale:

mediainfo can take a long time to return results, compared to ffprobe. Sample results and times on a troublesome file:

$ time ffprobe -v quiet -print_format json  -show_format -show_streams sample_video.mkv
...
real	0m0.295s
user	0m0.040s
sys	0m0.075s

$  time mediainfo  --Output=JSON sample_video.mkv
...
real	0m23.114s
user	0m0.646s
sys	0m1.221s

The file was local (external hard disk). Repeated tests on the same file showed similar timings. The time differences (0.295s vs 23.114s) are huge!

Considerations

• There may be regression on the colour sampling (this seems difficult in ffprobe)
• It may be possible to reduce the mediainfo scan time by specifying only the information we need
• Are there solutions other than ffprobe or mediainfo?
• The fetching of the JSON data will need to be abstracted into functions, which will add a "little" extra processing time. But looking at the difference between ffprobe and mediainfo, who cares?
• There appears to be no time difference between JSON and text format outputs, so this is not a factor.
• There are reports of inconsistencies in the data between ffprobe and mediainfo, which is the most reliable?
• It might be useful to allow an override of the cofig in the CLI, i.e ./archive_list.sh --scanner=ffprobe /path/to/dir/

The entrypoint script archive_list.sh has been replaced by video_list_csv.sh.

At the 3 month date after the file was marked as deprecated, archive_list.sh should be deleted (end of April 2024).

We need to handle:

• Single file extras (suffixed with " - ...")
• Directories containing specific extras
• extras for TV Shows
• extras for TV seasons

In addition what would the column structure look like?
the user should be given the configuration option to group extras under movie/series/season?

In some cases, the Video and Audio columns can become quite cluttered with multiple streams. Not all users was the full details, and only want to see the default stream.

A user should be able to define what level of information they want, with a simple config item in the .env.

Currently we are getting bad Title for trailers, etc:

trailer

These should be:

<dir_name> - trailer

We are currently searching for Year in the filename. While this is fine, we should also look in the containing directory, because this seems to be the standard...

├── TV Show Name (date)
│   ├── Season 1
│   │   ├── TV.Show.Name.S01E01.foo.bar.mkv
│   │   ├── ...
│   ├── Season 2
│   │   ├──...
│   ├── ...

Look for date in the filename. If not exists, then search the directory name 2-levels up.

new option to be global in the .env, called force_detect.

It should be turned off by default.

The codebase is becoming too long. For readability and maintenance, each piece of column code shouold be placed in its own file as a function, e.g.:

functions/title.sh

This will greatly improve the UX.

However, the progress bar will output to stdout, so the destination of the CSV will need to be input as a parameter rather than as a CLI file pipe.

ffprobe does not support detection of HDR10 or HDR10+ yet.

Although it is the default, preferred scanner due to being much faster than mediainfo, this is a major issue, since DV, HDR10 & HDR10+ are very much mainstream now.

Plex, Emby, Jellyfin and Kodi all have their own forks to implement this. It may be worth investigating integrating one of their packages somehow.

Or add a new column: Language

Sample JSON snippet where this occurred with force_detect=1 and detect_if_not_in_filename=1:

{
      "index": 0,
      "codec_name": "hevc",
      "codec_long_name": "H.265 / HEVC (High Efficiency Video Coding)",
      "profile": "Main 10",
      "codec_type": "video",
      "codec_tag_string": "[0][0][0][0]",
      "codec_tag": "0x0000",
      "width": 3840,
      "height": 2160,
      "coded_width": 3840,
      "coded_height": 2160,
      "closed_captions": 0,
      "film_grain": 0,
      "has_b_frames": 1,
      "sample_aspect_ratio": "1:1",
      "display_aspect_ratio": "16:9",
      "pix_fmt": "yuv420p10le",
      "level": 153,
      "color_range": "tv",
      "color_space": "bt2020nc",
      "color_transfer": "smpte2084",
      "color_primaries": "bt2020",
      "chroma_location": "topleft",
      "refs": 1,
      "r_frame_rate": "24000/1001",
      "avg_frame_rate": "24000/1001",
      "time_base": "1/1000",
      "start_pts": 0,
      "start_time": "0.000000",
      "extradata_size": 798,
      "disposition": {
        "default": 0,
        "dub": 0,
        "original": 0,
        "comment": 0,
        "lyrics": 0,
        "karaoke": 0,
        "forced": 0,
        "hearing_impaired": 0,
        "visual_impaired": 0,
        "clean_effects": 0,
        "attached_pic": 0,
        "timed_thumbnails": 0,
        "non_diegetic": 0,
        "captions": 0,
        "descriptions": 0,
        "metadata": 0,
        "dependent": 0,
        "still_image": 0
      },
      "tags": {
        "language": "eng",
        "BPS-eng": "64464572",
        "DURATION-eng": "01:34:05.640000000",
        "NUMBER_OF_FRAMES-eng": "135360",
        "NUMBER_OF_BYTES-eng": "45492971164",
        "SOURCE_ID-eng": "001011",
        "_STATISTICS_WRITING_APP-eng": "MakeMKV v1.17.6 linux(x64-release)",
        "_STATISTICS_WRITING_DATE_UTC-eng": "2024-03-31 01:08:10",
        "_STATISTICS_TAGS-eng": "BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES SOURCE_ID"
      },
      "side_data_list": [
        {
          "side_data_type": "DOVI configuration record",
          "dv_version_major": 1,
          "dv_version_minor": 0,
          "dv_profile": 7,
          "dv_level": 6,
          "rpu_present_flag": 1,
          "el_present_flag": 1,
          "bl_present_flag": 1,
          "dv_bl_signal_compatibility_id": 6
        }
      ]
    }

Having an entry list script name that is different to the repository name is a little confusing.

Tasks:

• Move archive_list.sh into /includes/
• Create a wrapper script in the root directory: video_list_csv.sh
• Clone video_list_csv.sh into archive_list.sh in the root directory, for backwards compatibility.
  • This will be deprecated at a later date