ymagis / clairmeta Goto Github PK

@orzel : When probing a very long image sequence, we don't need individual file names in the report.

Improve check robustness :

• Dedicated Atmos flag check, instead of fails of subsequent object count check
• Fail gracefully if the required keys related to objects count are not present

This is for Interop/Cinecanvas subtitles.

I have a subtitle file with the following:

<?xml version="1.0" encoding="UTF-8"?>
<DCSubtitle Version="1.0">
(...)
<Subtitle SpotNumber="976" TimeIn="00:00:08:115" TimeOut="00:00:14:083" FadeUpTime="00:00:02:000" FadeDownTime="00:00:01:000">

Clairmeta fails with:

2019-12-02 05:56:15,733 - Clairmeta - INFO - Error(s) :
	check_subtitle_cpl_st_timing - Check unknown error
Traceback (most recent call last):
  File "/home/mhm/ClairMeta/clairmeta/dcp_check_base.py", line 118, in run_check
    check_res = func(*args)
  File "/home/mhm/ClairMeta/clairmeta/dcp_check_subtitle.py", line 376, in check_subtitle_cpl_st_timing
    if f_s and f_s > dur:
TypeError: '>' not supported between instances of 'str' and 'int'

TI's Cinecanvas spec v1.0 says for both the FadeUpTime (and similairly for FadeDownTime) of the Subtitle element:

The time that it takes to bring a subtitle fully on screen. A subtitle begins to appear at the time specified in the TimeIn attribute, and fades in over the time specified in this attribute. FadeUpTime is in the format TTT, where TTT = ticks. A “tick” is defined as 4 msec and has a range of 0 to 249. This definition of tick was chosen because it will allow frame accurate timing at multiple frame rates, without specifying the display frame rate in the subtitle file.

Default FadeUpTime = 20 (this is 80 msec)

CineCanvas spec v1.1 adds the following text:

For fade times greater than one second, the full time format of HH:MM:SS:TTT or HH:MM:SS.sss must be used. See the attribute description for TimeIn or TimeOut for the definition of these formats. FadeUpTime will be limited to a maximum of 8 seconds. Any time greater than this will be replaced with the max time of 8 seconds.

And changes the last sentance to (emphasis mine):

This definition of time was chosen because it will allow frame accurate timing at multiple frame rates, without specifying the display frame rate in the subtitle file.

Strictly speaking my subtitle file above does not validate (because it specifies <DCSubtitle Version="1.0">), but maybe handle this case more gracefully?

According to ST429-2:2013

9.11 Hash Element
The Hash element shall be present in an asset when the KeyId element is present (i.e., when the referenced Track File is encrypted).

I don't there is a check for this at the moment?

This is obviously only strictly required for SMPTE packages on encrypted track files but I think it would be really useful to always perform this check, but maybe produce a warning where not strictly mandated by the standards

I have a SMPTE DCP with subtitles:

http://dcpomatic.com/Clairmeta_FTR-1_F_MOS_2K_20190106_SMPTE_OV.zip

ClairMeta says "Namespace unknown : dcst"

The XML (inside the subtitle MXF) starts

<dcst:SubtitleReel xmlns:dcst="http://www.smpte-ra.org/schemas/428-7/2010/DCST" xmlns:xs="http://www.w3.org/2001/XMLSchema">

I think ClairMeta is incorrectly looking for "dcst" as one of its accepted namespaces (in settings.py) instead of looking for http://www.smpte-ra.org/schemas/428-7/2010/DCST

If I'm right about this, I don't know how you fix it as AFAICS asdcp-info only returns the namespace name, not its value. check_subtitle_cpl_xml() gets 'NamespaceName' from the probe.

Let me know if I can provide any more information.

Thanks!

MainClosedCaption element from http://www.digicine.com/PROTO-ASDCP-CC-CPL-20070926 are not in the cpl_list from the DCP metadata

We currently still use magic based file type check in the sequence validation tool (eg. DSM, DCDM). We should remove those check as they are not reliable and can produce different results across system and versions of the file utility.

A more useful test would be to actually decode the image with Pillow / OpenImageIO / ... and check it's type, we then can rely on the fact that all images should be of the same size to assume every others ones have the same type.

Update the default check profile to consider check_assets_cpl_filename_uuid as a warning.

I am incorporating clairmeta into a python script by using your instructions:

dcp = DCP(absolute_dcp_path)
        # Parse DCP
        dcp.parse()
        # Check DCP
        dcp.check(hash_callback=console_progress_bar)

I would like to know what arguments I could use to get the JSON or XML outputs?

Thanks,

Kieran O'Leary
Irish Film Institute.

When checking file sequences, we have a check for detecting file size differences in sequences (should not happen for non-compressed images).

However, due to images originating from different softwares, there can be small differences in the metadata header, that's why we have a tolerance. Currently at 0.01% (https://github.com/Ymagis/ClairMeta/blob/develop/clairmeta/sequence_check.py#L97), this is too small, we have to increase it (eg. 2.5%) and make this a custom setting (https://github.com/Ymagis/ClairMeta/blob/develop/clairmeta/settings.py#L173).

Please relax the check on encryption coherence across reels with regards to encryption.

The below references do not mention encryption and indeed, I have several sample compositions containing a mix of encrypted and unencrypted trackfiles that playout without problems.

• ST 429-2:2013 section 8.7:
  Essence tracks in a Composition shall have homogenous encoding parameter values throughout the Composition. Picture essence shall have constant frame rate and pixel array size. Sound essence shall have constant sample rate, language, channel count, and channel assignment parameters.

• mpeg_ii_pack_constraints_2v5.doc section 7.2:
  The edit rates of all Assets must be equal and constant for the duration of the Composition.

• mpeg_ii_pack_constraints_2v5.doc section 7.3
  The following picture-related parameters shall not change for the duration of the Composition Playlist:
  -- Raster size
  -- Screen Aspect Ratio
  -- Frame Rate

• mpeg_ii_pack_constraints_2v5.doc section 7.4
  All sound track files referenced by a given composition must share identical:
  -- baseband sample rate
  -- baseband sample bit depth
  -- language
  -- channel format (presentation format)

• mpeg_ii_pack_constraints_2v5.doc section 7.5
  All subtitle tracks referenced by a given Composition Playlist must share identical language.

Hi,
Are there any instructions for adding the dependencies on Windows? I'm thinking of asdcp and libmaic in particular. You can use homebrew or apt-get on non windows machines, but on Windows (we do not have access to Windows Subsystem for Linux), this becomes a bit more awkward.

When parsing a file sequence, the initialization for the first file seems to be off, the default StartIndex and EndIndex value should be the current file index and not 0.

See : https://github.com/Ymagis/ClairMeta/blob/develop/clairmeta/utils/probe.py#L410

@DerouineauNicolas

I can't find any reference to that a filename extension of ".xml" is required. Indeed I have several Interop DCP:s with CineCanvas subtitles with a ".XML" extension (upper case) that play without problems.

Hi,

On windows 7, I get an error when clairmeta is imported:
2018-11-19 12:32:56,288 - Clairmeta - ERROR - Could not intialize log file : [Error 3] The system cannot find the path specified: 'C:\\Users\\PCCC/Library/Logs'
This isn't a fatal error, clairmeta continues to work, but it probably should be changed.

My guess is that it originates from this line:

• create ~/Library
• ask the user to create ~/Library
• check if not unix and if not, use some other default logging location on Windows

When building ubuntu trusty debian package, the process halt when attempting to install pipenv. This seems related to the fact that pipenv no longer supports Python 3.4.

pip3 install pipenv
Package 'pip' requires a different Python: 3.4.3 not in '>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'

On some DCP, DBOX flag is set to False :

=> "CPL dbox": false,
=> "CPL D-BOX": false,
=> "CPL MotionSimulator": "DBOX"
=> "NamingConvention.AudioType.MotionSimulator" = True

CPL_DBOX.xml.txt
(CPL xml -> xml.txt because "github rules")

On Ubuntu 16.04, the mime-type reported for .xml files is "application/xml". So all DCPs fail validation because dcp_check_utils.py only accepts "text/xml". I guess Mac OS X reports a mime-type of "text/xml"?

The problem can be corrected by accepting both mime-types:

$ diff -u /usr/lib/python3.5/dist-packages/clairmeta/dcp_check_utils.py.orig /usr/lib/python3.5/dist-packages/clairmeta/dcp_check_utils.py
--- /usr/lib/python3.5/dist-packages/clairmeta/dcp_check_utils.py.orig	2018-03-19 17:59:01.000000000 +0100
+++ /usr/lib/python3.5/dist-packages/clairmeta/dcp_check_utils.py	2018-03-28 21:11:02.000000000 +0200
@@ -22,7 +22,7 @@
 def check_xml(xml_path, xml_ns, schema_type, schema_dcp):
     # Correct file type (magic number)
     xml_magic = magic.from_file(xml_path, mime=True)
-    if xml_magic != "text/xml":
+    if xml_magic != "text/xml" and xml_magic != "application/xml":
         raise CheckException(
             "File type unknown, expected XML Document but got {}".format(
                 xml_magic))

But maybe the mime-type should not be checked at all, only the content? What mime-type would be reported on various Windows versions? It seems a fragile check, and actually irrelevant. DCP players only care about the content, as far as I know.

I'm using Windows 7 and python 2.7. I found a DCP that only fails on windows. It seems that it might be that mediainfo or other tools are adding new line characters and perhaps other issues that could be creating false positives with various if/else statements..
The only issue that this DCP has on OSX is the error regarding the non compliant ContentTitleText:
I've changed some filepaths and DCP names but otherwise this is the log output:

C:\Users\PCCCC>python -m clairmeta.cli check -type dcp "generic-dcp-name"
2018-11-19 12:48:13,345 - Clairmeta - ERROR - Could not intialize log file : [Er
ror 3] The system cannot find the path specified: 'C:\\Users\\PCCCC/Library/L
ogs'
2018-11-19 12:48:13,480 - Clairmeta - INFO - Probing DCP : generic-dcp-name
2018-11-19 12:48:13,782 - Clairmeta - INFO - Total time : 0.30 seconds
2018-11-19 12:48:13,923 - Clairmeta - INFO - Checking DCP : generic-dcp-name
2018-11-19 12:49:39,211 - Clairmeta - INFO - DCP : generic-dcp-name
2018-11-19 12:49:39,211 - Clairmeta - INFO - Size : 7.63 GiB
2018-11-19 12:49:39,211 - Clairmeta - INFO - Warning(s) :
        check_picture_cpl_avg_bitrate - 36949a61-4d79-4961-a2c0-d00eea868408_cpl
.xml (Asset 3e2ea821-73a1-484b-8fee-aa31acffe482_j2c.mxf) : Exceed DCI safe aver
 Mb/sitrate (245.0 Mb/s) : 95.67Mb/s
        check_assets_cpl_labels_schema - 36949a61-4d79-4961-a2c0-d00eea868408_cp
l.xml (Asset dc7b26ee-735b-41f4-be12-dace53c4a08b_pcm.mxf) : MXF Label incoheren
 but expected MXFInterop
        check_assets_cpl_labels_schema - 36949a61-4d79-4961-a2c0-d00eea868408_cp
l.xml (Asset 3e2ea821-73a1-484b-8fee-aa31acffe482_j2c.mxf) : MXF Label incoheren
 but expected MXFInterop
        check_dcnc_compliance - ContentTitle must have 12 parts, 9 found
2018-11-19 12:49:39,211 - Clairmeta - INFO - Error(s) :
        check_picture_cpl_max_bitrate - 36949a61-4d79-4961-a2c0-d00eea868408_cpl
.xml (Asset 3e2ea821-73a1-484b-8fee-aa31acffe482_j2c.mxf) : Exceed DCI maximum b
 Mb/se (250.05 Mb/s) : 244.94Mb/s
        check_assets_cpl_labels - 36949a61-4d79-4961-a2c0-d00eea868408_cpl.xml (
Asset dc7b26ee-735b-41f4-be12-dace53c4a08b_pcm.mxf) : MXF Label invalid : MXFInt
erop
        check_assets_cpl_metadata - 36949a61-4d79-4961-a2c0-d00eea868408_cpl.xml
 (Asset dc7b26ee-735b-41f4-be12-dace53c4a08b_pcm.mxf) : Encrypted metadata misma
tch, CPL claims False but MXF No
        check_assets_cpl_labels - 36949a61-4d79-4961-a2c0-d00eea868408_cpl.xml (
Asset 3e2ea821-73a1-484b-8fee-aa31acffe482_j2c.mxf) : MXF Label invalid : MXFInt
erop
        check_assets_cpl_metadata - 36949a61-4d79-4961-a2c0-d00eea868408_cpl.xml
 (Asset 3e2ea821-73a1-484b-8fee-aa31acffe482_j2c.mxf) : Encrypted metadata misma
tch, CPL claims False but MXF No
2018-11-19 12:49:39,213 - Clairmeta - INFO - Total check : 53
2018-11-19 12:49:39,213 - Clairmeta - INFO - Total time : 85.29 sec
2018-11-19 12:49:39,213 - Clairmeta - INFO - Validation : Fail

I believe that an odd number of audio channels are never valid. This is because physical AES pairs are used in almost all media block implementations. I have personal experience with Dolby Cat862/DSS200 fails at playback with odd number of channels.

Also most people find it unsafe with an even number of channels <6, even if I have no evidence of it NOT working. Especially I have successfully played many played many two-channel DCP:s. The usual thing done is to pad any unused channels (no. 1-6) with silence.

Reference 1:
http://isdcf.com/papers/ISDCF-Doc4-Audio-channel-recommendations.pdf

Note 1. Not all channels need to be present in a given DCP. For instance, only the first 8 channels should be used when delivering 5.1 + HI/VI content. In all cases, an even number of channels shall be used.

Reference 2:

To: Main ISDCF <[email protected]>
Date: Mon, 27 Jan 2014 20:18:29 +0000
Message-ID: <CF0BFC61.21F4F%[email protected]>
Subject: Re: [ISDCF] Channel count in Interop compositions

In the past, Universal has made Interop DCPs of classic titles with ch. 1 & 2 only (L/R), and with ch. 1, 2, 3 & 4 only (L/R/C/LFE). We've not experienced any playback problems in the field. However, having thought about it more and just to be safe, we now include six channels of audio on all DCPs, putting null audio on "empty" channels. And, yes, audio needs to be in AES pairs (no odd number of channels).

On init_ we have:

for d in deps:
if not check_command(d):
get_log().warning("Missing dependency : {}".format(d))

And we cannot disable this part.
The only solution is to use redirect_stdout on include clairmeta lib

Idea: using getenv to disable all output log ?

For SMPTE subtitles the uuid in the CPL refers to the mxf container, not to the uuid in the wrapped subtitle xml. This check fails for unencrypted SMPTE subtitles because it compares the MainSubtitle Id element in the CPL against the XML. This is correct for Interop (which is not mxf-wrapped) but not for SMPTE.

Please instead check against the subtitle mxf container uuid.

ST429-2:2009:

8.4.4 Maximum Rate of Occurrence for On-Screen Timed Text
Up to three (3) subtitle instances may be visible on screen at any time. The visibility period of an instance shall include fade-in and fade-out times. A subtitle instance shall contain no more than six (6) Text elements or three (3) Image elements.

ST429-2:2013:

8.4.4 Maximum Rate of Occurrence for On-Screen Timed Text
Up to two (2) subtitle instances may be visible on screen at any time. The visibility period of an instance shall include fade-in and fade-out times. A subtitle instance shall contain no more than six (6) Text elements or three (3) Image elements.

Does clairmeta currently check the above constraints?

Last version succeeded but with the following warning :
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

Now it fails with :
Cannot uninstall 'six'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I guess we should build the package in a clean virtual env.

I have a subtitle XML with the following (pretty-printed)

<?xml version="1.0" encoding="UTF-8"?>
<dcst:SubtitleReel xmlns:dcst="http://www.smpte-ra.org/schemas/428-7/2010/DCST" xmlns:xs="http://www.w3.org/2001/XMLSchema">
(...)
  <dcst:LoadFont ID="">urn:uuid:47471663-927b-4008-957a-faf6fd14be32</dcst:LoadFont>
  <dcst:SubtitleList>
    <dcst:Font AspectAdjust="1.0" Color="FFFFFFFF" Effect="border" EffectColor="FF000000" ID="" Italic="no" Script="normal" Size="42" Underline="no" Weight="normal">
(...)

This gives the following error on a Barco ICMP player:

Dec  9 12:53:12 icmp-0e26fd user.err GIP: XML Parse: ERROR on line 2 , column 586
Dec  9 12:53:12 icmp-0e26fd user.err GIP: XML Parse: Error string = parsing aborted
Dec  9 12:53:12 icmp-0e26fd user.err GIP: XML Parse: Error Info string = Bad LoadFont Tag
Dec  9 12:53:12 icmp-0e26fd user.err GIP: XML Parse: Parser returned with error.
Dec  9 12:53:12 icmp-0e26fd user.err GIP: The error occured in: /tmp/subtitles/published/980d2baf-4c93-4c44-ab78-bfcd8fd3c6b7/980d2baf-4c93-4c44-ab78-bfcd8fd3c6b7.xml
Dec  9 12:53:12 icmp-0e26fd user.err GIP: SubtitleError--Subtitle-XML Parsing Error
Dec  9 12:53:12 icmp-0e26fd user.info GIP: Flush Overlay
Dec  9 12:53:12 icmp-0e26fd user.err SMS: Player- Player subtitles: activate gip subtitles failed: 0
Dec  9 12:53:12 icmp-0e26fd user.err SMS: Player- Player error: player - subtitles validation error - cannot play
Dec  9 12:53:12 icmp-0e26fd user.info SMS: Report- |SHOW-SELECT| [BomberHarris_SHR-1-30_F-143_EN-SV_20_2K_20191208_SMPTE_OV]
Dec  9 12:53:12 icmp-0e26fd user.debug SMS: Player- starting streams at frame id: 0 - streams: AV
Dec  9 12:53:12 icmp-0e26fd user.debug SM: IMB Controller- Sync Payload Updated (frameId: 0, status: 0, delay: 0, offset: 0)
Dec  9 12:53:12 icmp-0e26fd user.debug SMS: Player- Player subtitles: updating gip timeline: 00:00:00:000
Dec  9 12:53:13 icmp-0e26fd user.info GIP: Overlay_Flush(): free buffers = 3
Dec  9 12:53:13 icmp-0e26fd user.info GIP: Overlay_Flush() done

Also attaching the subtitle XML in full:

980d2baf-4c93-4c44-ab78-bfcd8fd3c6b7.zip

It would be nice if clairmeta would check empty ID attributes for LoadFont and Font. Should probably also be done for DCSubtitle v1.0 and v1.1 as well as 428-7:2014.

For each assets, add the asset type (Picture, Sound, Auxdata) into the asset dict
( the asset type is only the key of the asset dict )

• Think about to avoid to use asset_type as a dict key: In the future, if we have multi-AuxData, all AuxData will be destroyed by the last AuxData )

Hello,

I has builded a package on debian buster, the package has build successful.

travis-ci

But the deploy process fail, with a message:

Bintray response: 400 Bad Request. Unable to upload files: Could not extract metadata for artifact 'libpf-java_5.5.0-4_all.deb', content might be malformed.

I has builded the same packages to (wheezy and jessie without problem)

Bintray

Best regards

Similar to #50

When probing a DCP, we currently have a field size that returns the actual size of the directory. We could also provide the expected size from adding up all assets listed in the AssetMap.

@benjamingymg

We are not sure what this error means, how serious it is, and what the line 17 refers to. We can share the XML privately if needed.

2019-07-11 14:51:36,700 - Clairmeta - INFO - Probing DCP : /Volumes/test/e0c17f87-7386-40b7-916e-848b744bfb5c/objects/SONGOFGRANITE_FTR_F_EN-EN_51_2K_20170301_OL_IOP_OV
2019-07-11 14:55:37,132 - Clairmeta - INFO - Total time : 240.43 seconds
2019-07-11 14:55:39,102 - Clairmeta - INFO - Checking DCP : /Volumes/test/e0c17f87-7386-40b7-916e-848b744bfb5c/objects/SONGOFGRANITE_FTR_F_EN-EN_51_2K_20170301_OL_IOP_OV
2019-07-11 15:19:42,585 - Clairmeta - INFO - DCP : /Volumes/test/e0c17f87-7386-40b7-916e-848b744bfb5c/objects/SONGOFGRANITE_FTR_F_EN-EN_51_2K_20170301_OL_IOP_OV
2019-07-11 15:19:42,585 - Clairmeta - INFO - Size : 72.07 GiB
2019-07-11 15:19:42,585 - Clairmeta - INFO - Warning(s) :
	check_dcnc_compliance - ContentTitle must have 12 parts, 10 found
2019-07-11 15:19:42,585 - Clairmeta - INFO - Error(s) :
	check_pkl_xml - 2f319f96-c80d-48f8-931c-d79a5eea6eac_pkl.xml : Schema validation error : Element '{http://www.w3.org/2001/XMLSchema}import': Failed to parse the XML resource 'http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd'., line 17
	check_cpl_xml - 662311b2-8ecc-43c3-95a1-3f831f96515c_cpl.xml : Schema validation error : Element '{http://www.w3.org/2001/XMLSchema}import': Failed to parse the XML resource 'http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd'., line 17
	check_am_xml - ASSETMAP : Schema validation error : Element '{http://www.w3.org/2001/XMLSchema}import': Failed to parse the XML resource 'http://www.w3.org/TR/2002/REC-xmldsig-core-20020212/xmldsig-core-schema.xsd'., line 17
2019-07-11 15:19:42,585 - Clairmeta - INFO - Total check : 59
2019-07-11 15:19:42,585 - Clairmeta - INFO - Total time : 1443.48 sec
2019-07-11 15:19:42,586 - Clairmeta - INFO - Validation : Fail

It looks like the DCDM check doesn't do anything if no errors are found. The code returns True if no issues are found, but when running from the command line, it would be helpful if some sort of Success judgement was displayed.

Have a uniform way of specifying the relevant standards for each check performed by ClairMeta. This will most likely be present in the checks docstring.

When assets referenced in the CompositionPlayList are missing, display the Essence type (Image, Sound, Aux, ...) as part of the error message.

These checks are concerned :

• check_assets_cpl_missing_from_vf
• check_assets_cpl_missing_from_multi_cpl
• check_link_ov_asset

Please see the following references:

ISO 15444-1:2006 Amendment 1

A.10.1 Codestream restrictions for digital cinema

(...)

	2K digital cinema profile	4K digital cinema profile
Application specific restrictions
Max compressed bytes for any image frame (aggregate of all 3 color components)	1302083 bytes for 24 fps 651041 bytes for 48 fps	1302083 bytes (for 24 fps)
Max compressed bytes for any single color component of an image frame	1041666 bytes for 24 fps 520833 bytes for 48 fps	1041666 bytes for 2K portion of each component (for 24 fps)

DCSS v1.3 ( https://www.dcimovies.com/specification/DCI_DCSS_Ver1-3_2018-0627.pdf )

4.3.3. Codestream Specification
All codestreams shall fully conform with [ISO 15444-1:2006 Amendment 1], as more fully constrained as follows:

(...)

• For a frame rate of 24 FPS, a 2K distribution shall have a maximum of 1,302,083 bytes per frame (aggregate of all three color components including headers). Additionally, it shall have a maximum of 1,041,666 bytes per color component per frame including all relevant tile-part headers
• For a frame rate of 48 FPS, a 2K distribution shall have a maximum of 651,041 bytes per frame (aggregate of all three color components including headers). Additionally, it shall have a maximum of 520,833 bytes per color component per frame including all relevant tile-part headers.
• A 4K distribution shall have a maximum of 1,302,083 bytes per frame (aggregate of all three color components including headers). Additionally, the 2K portion of each frame shall satisfy the 24 FPS 2K distribution requirements as stated above.

Note: For information purposes only, this yields a maximum of 250 Mbits/sec total and a maximum of 200 Mbits/sec for the 2K portion of each color component.

CTP v1.2 ( https://www.dcimovies.com/compliance_test_plan/DCI_CTP_v12.pdf )

4.5.2. Image Compression Standard & Encoding Parameters

(...)

• Verify that every frame is correctly JPEG 2000 encoded as described in [ISO-15444-1]. Verify that the proper JPEG 2000 encoding parameters as specified in [ISO-15444-1-AMD-1] were used. The Codestream Specifications for 2K and 4K distributions are listed in [DCI-DCSS-1-2], section [4.3]. This can be achieved by using JPEG 2000 decoding software. (...) If any frame fails to correctly decode or does not conform to the appropriate Codestream Specifications, this is cause to fail the test.

(...)

C.5.1. j2c-scan Source Code Listing

SMPTE ST 429-2:2013

10.2.2 Compression
Picture essence shall be compressed using JPEG 2000 [ISO/IEC 15444-1] as constrained by [ISO/IEC 15444-1:2004/Amd 1:2006].

SMPTE ST 429-4:2006

5 JPEG 2000 Codestream Restrictions
The JPEG 2000 codestreams shall be restricted according to ISO-IEC 15444-1 Amd 1. This amendment defines two profiles for D-Cinema usage referred to in this standard as 2K and 4K.

P-HFR -- Prototype DCP for High Frame Rate Projection ( https://isdcf.com/papers/ISDCF-HighFrameRate-DCP.pdf )

2.1 JPEG2000 Codestream Formulation
JPEG 2000 Codestreams used in P-HFR files shall conform to ISO/IEC 15444-1:2004, “JPEG 2000 Image Coding System: Core Coding System” and ISO/IEC 15444-1:2004/Amd 1:2006,
Profiles for Digital Cinema Applications” except that the constraints on bit rate shall be expanded as follows:

• The data rate of the total codestream shall not exceed 500Mb/s
• The combined data rates of the Cx and Cz channels shall not exceed 250Mb/s
• The data rate of the Y channel alone shall not exceed 500Mb/s

(...)

2.3 Picture Track File Formulation
A stereoscopic Picture Track File in a P-HFR composition shall conform to SMPTE ST 429-13:2009 with the following exceptions:

• The file shall contain only 2k images.
• In the MXF FileDescriptor class, the SampleRate property shall be set to indicate the total frame rate of the file. For stereoscopic content this value shall be twice the EditRate (i.e., one of 96/1, 100/1 or 120/1).

(...)

2.4 Composition Formulation
(...) All files in a P-HFR composition shall have equal edit rate, one of 48/1, 50/1 or 60/1.

For Interop:
MainSubtitle Duration must be equal to MainPicture Duration.

For SMPTE:
MainSubtitle duration is allowed to be less than or equal to MainPicture Duration. See references below

• ST 429-2:2013 section 9.4:
  The Duration element shall be present within every Asset element that refers to an external track file. The value of all Duration elements in a reel, with the exception of timed text elements, shall be equal. The Duration of the Reel shall be determined by the MainPicture element, per the provisions of [SMPTE ST 429-7], or the MainStereoscopicPicture element, whichever is present.

• ST 429-2:2013 section 9.6:
  Each Reel element in a Composition Playlist document shall contain one (1) MainPicture element [SMPTE ST 429-7] or one (1) MainStereoscopicPicture element [SMPTE ST 429-10]. This element shall refer to a Picture Track File as defined by [SMPTE ST 429-3]. If the element name is MainStereoscopicPicture, the referenced Track File shall also conform to [SMPTE ST 429-10].

• ST 429-7:2006 section 5:
  At the start of a given Reel, playback of all Assets contained within the Reel shall start simultaneously at the Entry Point given for each respective Track File. The duration of a Reel shall be equal to the duration of the Reel’s MainPicture Asset, or the Asset with the shortest duration if the MainPicture Asset is not present.
  (Figure 3 clearly shows a MainSubtitle Asset with shorter Duration than the two other assets)

• mpeg_ii_pack_constraints_2v5.doc section 7.5:
  All subtitle tracks referenced by a given Composition Playlist must share identical language.The subtitle file shall not contain any subtitles outside the playable portion determined by the EntryPoint and Duration of the subtitle asset.

• mpeg_ii_pack_constraints_2v5.doc section 7.8:
  All Assets referenced by a single Reel shall have the same Duration

I have an CPL with the following element:

<AnnotationText>Magnus är sexist (burnt in subtitles)</AnnotationText>

Clairmeta fails with the following:

	check_cpl_contenttitle_annotationtext_match - Check unknown error
Traceback (most recent call last):
  File "clairmeta/dcp_check_base.py", line 118, in run_check
    check_res = func(*args)
  File "clairmeta/dcp_check_cpl.py", line 87, in check_cpl_contenttitle_annotationtext_match
    "mismatch : {} / {}".format(ct, at))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 7: ordinal not in range(128)

	check_cpl_contenttitle_pklannotationtext_match - Check unknown error
Traceback (most recent call last):
  File "clairmeta/dcp_check_base.py", line 118, in run_check
    check_res = func(*args)
  File "clairmeta/dcp_check_cpl.py", line 111, in check_cpl_contenttitle_pklannotationtext_match
    "AnnotationText mismatch : {} / {}".format(ct, at))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 7: ordinal not in range(128)

Even though the XML schema specified in the two later versions of ST428-7 makes the LoadFont element optional by specifying:

<xs:element name="LoadFont" minOccurs="0" maxOccurs="unbounded">

This is further constrained by ST 429-2:2013:

8.4.1 Fonts for Timed Text
When Text elements are present in the Timed Text essence, one (1) LoadFont element shall be present. Timed Text essence shall not contain more than one (1) LoadFont element.

And also mentioned (more loosely) in ST 428-7:2014:

5.11 LoadFont Element [instance optional]
(...)
When one or more Text elements are present, at least one LoadFont element shall also be present.

That is, exactly one LoadFont element must be present whenever Text elements are used later in the file. The only time that the LoadFont element can be omitted is when no Text elements are present in the file (i.e. Image elements referencing PNG files are used instead).

Does clairmeta currently perform this test?

The XSD for SMPTE subs says

<xs:element name="Language" type="xs:language" minOccurs="0" default="en"/>

so I don't think ClairMeta should give an error if no tag is present in some SMPTE subtitle XML.

Thanks!

On isdcf.py :

fields_list = str.split('_')
if len(fields_list) != 12:
    error_list.append("ContentTitle must have 12 parts, {} found".format(
        len(fields_list)))

Some DCPs are wrong NamingConvention Title...
Like:

• Preshow_TLR-Stuff_F_EN-XX_INT_51_4K_20170227_XXX_IOP_OV
• XXX-HFR-2D-060_SHR_C_INT-XX_INT_51_2K_YMA_20130121_XXX_OV
• RandomMovie_FTR-Live_F-177_EN-XX_51_2K_PAT_20190129_XXX_IOP_OV

The parsing give an error with this kind of title.

Solution:
split each "_" and look at the good elements using rules.
If "ContentKind" is in wrong place, the regex can match and find elements

• adding a flag "NamingConventionCompliance" : False (or True if all is ok)

Uploading the generated .deb don't work for bionic, travis report :

Unable to upload files: Could not extract metadata for artifact "pool/main/bionic/asdcplib/asdcplib_2.10.31_amd64.deb" content might be 'malformed.

Support for ISDCF recommandation, eg. Wild Track Audio format for SMPTE DCP B2.1.

These should be warnings.

https://isdcf.com/papers/ISDCF-Doc4-Audio-channel-recommendations.pdf
https://isdcf.com/t/SMPTE-DCP-Tests/

Hi,
the bintray repo (https://dl.bintray.com/ymagis/Clairmeta/dists/) has a distro name typo, Ubuntu artful (17.10) not Ubuntu rtful.

Thanks for this awesome project/tool.

Is there an reference for this constraint on Interop DCP:s?
https://github.com/Ymagis/ClairMeta/blob/develop/clairmeta/dcp_check.py#L157-L172

I can't find one..

The check_dcnc_field_claim_subtitle() test does not currently take this into account.

Please see: http://isdcf.com/dcnc/home/appendix-1-language-codes-intro.html

Subtitles are either generated in the theatre by the Digital Cinema projector, or are burned into the image by the studio. If the Language Code is UPPER CASE, the sub-titles are generated in real time by the projection system. If the Language Code is lower case, the subtitles are already "burned" into the image.

For the Probe functionality, we could systematically include the result of the CPLs ContentTitleText naming convention parsing, leaving field with default value if not found.

Currently it is not present if an error / warning is detected.

It would be nice if the shell exit code was not '0' when the validation result is 'Fail'.

$ python3 -m clairmeta.cli check -type dcp ./
&& echo "ALL OK! (\$? == 0)"
2018-03-28 22:35:31,542 - Clairmeta - INFO - Probing DCP : .
...
2018-03-28 22:35:32,133 - Clairmeta - INFO - Error(s) :
	check_document_signature - pkl_347c0542-0b21-4953-83f2-c7903716409b.xml : XML Digest mismatch, signature can't be checked
	check_assets_am_size - ASSETMAP.xml (Asset pkl_347c0542-0b21-4953-83f2-c7903716409b.xml) : Invalid size value, expected 8092 but got 8091
2018-03-28 22:35:32,133 - Clairmeta - INFO - Total check : 77
2018-03-28 22:35:32,133 - Clairmeta - INFO - Total time : 0.33 sec
2018-03-28 22:35:32,133 - Clairmeta - INFO - Validation : Fail

False
ALL OK! ($? == 0)

Study the possibility of deep inspection of individuals J2C frames embedded in the MXF container. This might include :

• Decoding frame without error
• Structure and compliance of the J2C codestreams with appropriate norm
• Bitrate graph generation (don't need deep inspection for that though)

This only apply to non-encrypted DCPs or ones where a KDM is available and should be made optional due to the execution time overhead.

Main issue is to find a non commercial jp2k decoder that would be compliant with cinema standard and fast enough.

We have a sequence where one corrupted tif resulted in DCP-O-Matic not being able to make a DCP. After doing an audit in exiftool, we found a corrupt frame. ClairMeta approved this sequence as the filesize, mimetype and sequence continuity were all ok. I suggest some further probing, for example:
Width/Height, Bit Depth , some sort of check that a valid tiff header is present. Whatever is the least intensive for the greatest chance of detecting corruption.
Here's the exiftool report on the corrupt frame:

exiftool E:\DCDM\video\Reel_3\DCDM_Tiff16bit-XYZ_DCI2K-Scope_Reel03_075580.tif
ExifTool Version Number         : 11.19
File Name                       : _DCDM_Tiff16bit-XYZ_DCI2K-Scope_Reel03_075580.tif
Directory                       : E:/DCDM/video/Reel_3
File Size                       : 10 MB
File Modification Date/Time     : 2019:02:08 14:41:08+00:00
File Access Date/Time           : 2019:02:08 14:41:08+00:00
File Creation Date/Time         : 2019:02:08 14:41:08+00:00
File Permissions                : rw-rw-rw-
File Type                       : TIFF
File Type Extension             : tif
MIME Type                       : image/tiff
Exif Byte Order                 : Little-endian (Intel, II)

and here's the previous frame which is fine:

ExifTool Version Number         : 11.19
File Name                       : DCDM_Tiff16bit-XYZ_DCI2K-Scope_Reel03_075579.tif
Directory                       : New folder
File Size                       : 10 MB
File Modification Date/Time     : 2018:05:11 02:23:31+01:00
File Access Date/Time           : 2020:03:10 09:44:41+00:00
File Creation Date/Time         : 2020:03:10 09:44:41+00:00
File Permissions                : rw-rw-rw-
File Type                       : TIFF
File Type Extension             : tif
MIME Type                       : image/tiff
Exif Byte Order                 : Little-endian (Intel, II)
Image Width                     : 2048
Image Height                    : 858
Bits Per Sample                 : 16 16 16
Compression                     : Uncompressed
Photometric Interpretation      : RGB
Strip Offsets                   : 8
Samples Per Pixel               : 3
Rows Per Strip                  : 858
Strip Byte Counts               : 10543104
Planar Configuration            : Chunky
Image Size                      : 2048x858
Megapixels                      : 1.8

Best,

Kieran.

On Mac, after installing:

easy_install pip
pip install clairmeta

The libmagic library is missing, easy_install libmagic or pip install libmagic doesn't help.

http://eidr.org/documents/EIDR_in_Digital_Cinema.pdf