1. PSNR for lossy compression with various compression settings
2. lossless: compare round-trip compress-decompress image with original image, to verify lossyless

Basic commands and references on how to run jyplyzer against the example files should also include how users read the output.

Dev Effort

0.5D

Description

When opening the Jpylyzer Website over HTTPS, CSS is blocked by modern browsers to avoid loading it over an unsecure connection. Images included via HTTP are loaded but show a warning in the browser's console.

To avoid this you could:

1. link all resources via their HTTPS links which works when opening the page over HTTP and HTTPS
2. use relative links

See comments here:

https://github.com/openplanets/jpylyzer/tree/master/doc

An alternative would be to discontinue the PDF version, and make a completely self-contained HTML version (CSS, images embedded inside HTML) available for download / offline viewing. This is no problem for Pandoc.

In usage instructions add desc of --nullxml and --nopretty options. Do this once binary builds are ready.

Mandatory Requirement

Detection of LUT-based ICC profiles doesn't work (mistyped AToB0Tag).

Debain does not like this. The libraries and binary probably need separating.

When building the debian package.
More info on this @ http://lintian.debian.org/tags/unstripped-binary-or-object.html

While analysing one corrupted JPX image, jpylyzer (1.1.3/1.1.4) appeared to enter some indefinite loop.

Jpylyzer crashes with a Unicode Decode Error if the name or path of an analysed file contains non-ASCII characters, e.g.:

jpylyzer NL-Albert-Mülli_1-2_new_Page_2_Image_0001.jpf

The problem also occurs if the (full) path to the file contains such characters (since jpylyzer adds the full path to its output!).

Will be fixed in version 1.6.1!

Update

The fix in 1.6.1 now causes an attribute error when running jpylyzer under Python 3.2 because the decode attribute does not exist for str objects in that version! Will be fixed in 1.6.2 (1.6.1 Win binaries and Deb packages are unaffected by this).

If a tile-part contains any optional marker segments (COD, COC, QCD, etc.) it will report the following error:

- <contiguousCodestreamBox>
- <tileParts>
- <tilePart>
  <foundSODMarker>False</foundSODMarker> 
  </tilePart>
  </tileParts>
  </contiguousCodestreamBox>

This is because jpylyzer erroneously expects the SOD (start-of-data) marker at a fixed byte position after the SOT.

As a result, images that contain any of these markers will fail validation.

This applies, for example, to JP2s that were created using OpenJPEG (1.5) using the default settings, even though such images are really valid.

For Win, Debian links paths remain identical apart from the version number. So making the version number a site variable would simplify updating to new versions.

Also: display current version number on front page!

Something like:

--nullxml : extract null-terminated XML content from XML and UUID boxes (doesn't affect validation)

Update http://openplanets.github.io/jpylyzer/usage.html as well.

Apparently it shouldn't! This is a mandatory requirement

While trying to create a Debian package for jpylyzer on a local machine I ran into a number of issues:

https://github.com/openplanets/jpylyzer/wiki/Debian-packaging

Some of these are largely a matter of better developer documentation (e.g. the BUILD_HOWTO_LINUX). Lintian errors need a more in-depth look.

Some software products use UUID boxes for storing XMP metadata. See the XMP spec:

http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/cs6/XMPSpecificationPart3.pdf

Page 16 of that document does mention that XMP packets are stored in a UUID box for JPEG 2000 (confusingly they refer to Part 12 -ISO Base Media Format- of the standard there, which is a container format for video. This is something else than JP2/ Part 1 (even though it uses a similar box structure), but both Exiftool and Kakadu have adopted this for JP2 (and possibly other products as well).

Currently jpylyzer isn't able to read XMP metadata in a UUID box. Support for this will be added in a future version of jpylyzer.

Requested by Clemens Neudecker as user option. Will look at this for upcoming version (e.g. -o switch)

The valid example files in the package are 20Mb+. These could do with being different or a lot smaller.

This is somewhat related to issue #23. It would be nice if there was a DTD for jpylyzer produce XML instances. Thanks

Jpylyzer will enter an infinite loop for images that contain only one single tile-part (caused by the fact that the value of psot is 0 in that case, and jpylyzer doesn't handle this well).

This applies e.g. to some GeoJP2 images created with the ECW JPEG 2000 SDK. Incidentally JHOVE 1.6 will report an ill-formed codestream for these images, so most likely it also doesn't handle the zero psot values correctly.

Will be fixed in jpylyzer 1.5.3 ...

XML contains null byte in case of JP2s with UUID Info Box, which means that output is not well-formed and cannot be opened with XML editor/viewer.

Happens at least with the Win32 binary under Windows 7 (strangely if I remember well problem didn't happen under XP).

A bit embarrassing, as the new sample images all contain this (otherwise not very common) box.

Solution: try to strip away last character (which should be null byte according to standard, but I assumed that elementtree would take care of this automatically!)

Under various circumstances (e.g. corrupted files) XML output may contain illegal control characters (e.g. null bytes). For properties that are directly converted from bytes to text there is already a mechanism that will suppress null characters, but they may stil occur in UTF-8 and Latin-encoded text.

This may affect output related to codestream comments (Latin), and URL boxes within UUID Info boxes (UTF-8).

XML boxes might be affected as well, although the built-in well formedness check in the XML box validator function will probably prevent any problems here.

Possible solutions:

1. Use containsControlCharacters directly in URL and Codestream validator functions (i.e. before decoding to UTF-8/Latin)
2. OR replace control characters with something else

While at it, containsControlCharacters should catch all characters in the 0-31 byte range (none of which are allowed in XML and none of which can be replaced with entity references)

Some malformed (deliberately bit-flipped) JP2s appear to make jpylyzer hang. e.g. cc-16-kdu.jp2-killed-10918 runs for a long time and then stops, outputting simply 'Killed'.

anj@anj-VirtualBox:~/bitwiser$ hexdump -C cc-16-kdu-hangs/cc-16-kdu.jp2-killed-10918 > hd1
anj@anj-VirtualBox:~/bitwiser$ hexdump -C src/test/resources/cc-16-kdu.jp2 > hd2
anj@anj-VirtualBox:~/bitwiser$ diff hd*
131c131
< 00000830  00 6a 70 32 63 ff 4f ff  51 00 32 00 00 80 00 00  |.jp2c.O.Q.2.....|

---
> 00000830  00 6a 70 32 63 ff 4f ff  51 00 32 00 00 00 00 00  |.jp2c.O.Q.2.....|

More interestingly, cc-16-kdu.jp2-killed-22181 appears to cause an infinite loop:

....
User warning: ignoring unknown box
User warning: ignoring unknown box
User warning: ignoring unknown box
....

There may be further interesting modes of failure hidden in the other examples I have. I'll see about attaching them... Seems I cannot - so I uploaded the zip of fails here: https://dl.dropbox.com/u/135740/cc-16-kdu-hangs.zip

I have three jp2 images which are named:

"x-fmt-392 (JPEG2000) [1].jp2"
"x-fmt-392 (JPEG2000) [2].jp2"
"x-fmt-392 (JPEG2000) [3].jp2"

When running any of the commands:

jpylyzer 'x-fmt-392 (JPEG2000) [1].jp2'
jpylyzer "x-fmt-392 (JPEG2000) [1].jp2"
jpylyzer x-fmt-392\ (JPEG2000)\ [1].jp2

the output is:
"User warning: no images to check!"

Similarly, running:

jpylyzer *.jp2

produces:
jpylyzer: error: unrecognized arguments: x-fmt-392 (JPEG2000) [2].jp2 x-fmt-392 (JPEG2000) [3].jp2

Running either of

jpylyzer '.jp2'
jpylyzer ".jp2"

results in the correct running of the program. Likewise if I rename one of the files "test.jp2", I am able to run the program on that file on its own.

The thought was to prompt for the users JP2 profile on first setup, not sure if this is needed. If not then the template file should be removed.

Section 7.9 of the User Manual documents the Tile-part level properties and tests, but a description of test foundSODMarker is missing.

This test checks if the last marker in every tile-part is a start of data marker.

Will be added in next revision of Manual.

copyright contains dh_make todo boilerplate
Either needs to be remove or the section filled in acurately

this is related to the config #8

JP2 filespec states that (non-binary) codestream comments use iso-8859-15 / Latin encoding. However jpylyzer doesn't check for the presence of code points that are undefined in that encoding. There should be a test for this.

Introduced in 1.11.1.

Dev Effort

TBC

Description

Currently jpylyzer doesn’t support the full set of marker segments that can occur in a codestream. Missing ones should ideally be included at some point (although some of them are pretty rare).

A good first step would be to create empty validator functions for all missing marker segments, which is quick and easy to implement. This would allow jpylyzer to report at least the presence of these markers in a file (without providing any details about its fields). This is already very useful information by itself, as some decoders appear to choke on some of the more obscure marker segments, so knowing they're there can already be helpful.

(Submitted by email by Thomas Ledoux of BnF while using Jython)

1/ when called from java, the scriptName as no value (probably because sys.argv[0] is not valuated), it would be nice to add a safeguard
if len(scriptName) == 0:
scriptName = 'jpylyzer'
so that the output fills the

These need removing, or content added

When building the package the changelog should be downloaded from http://p2-registry.ecs.soton.ac.uk/opf/gh2ch/

URL is encoded as a null-terminated UTF-8 string. Add check for presence of null terminator.

The source document of the User Manual is currently an MS Word document, which is a bit awkward to edit/maintain. Apart from that it creates a dependency on proprietary software (LibreOffice / OpenOffice will mess up the layout). Some of the figures were originally created in MS Powerpoint, has similar problems.

An alternative would be to migrate the User Manual to Markdown Extra (which includes table support), and to provide the figures as SVG. This would lower the barrier to contributing to the User Manual, and it would simplify things as well. In combination with a tool like Pandoc it would also enable us to generate versions of the User Manual in pretty much any desired delivery format (HTML, PDF, EPUB, etc.).

How to do this?

Possible workflow:

1. Convert Word doc to HTML

2. Clean up HTML if needed (e.g. HTMLTidy)

3. Convert HTML to Markdown. In Pandoc:

   pandoc -f html -t markdown_phpextra umanual.html > umanual.md

4. Manually clean up result

When trying to build the packages for jpylyzer1.10.4 it uses version number 1.10.1 instead. No idea where this is coming from!

Mathieu Malaterre suggested to include a bit more info on any "unknown" boxes encountered by jpylyzer, such as the associated string. Will look at this for jpylyzer 1.6!

Thanks for providing setup.py now ! It would be nice if you could also import the remaining diff from the actual debian package:

http://patch-tracker.debian.org/patch/series/view/jpylyzer/1.5.0-2/addsetup.patch

Thanks

It would be nice to have a proper jpylyzer home page, which would be pretty easy to set up with Github pages.

Also submitted by Thomas Ledoux (e/mail)±

2/ it's surprising that the XML output doesn't have any namespace defined (that would be handy to validate the output and to mangle it with XSLT...)
The modification seems straightforward
replace
root=ET.Element('jpylyzer')
by something like
root=ET.Element('jpylyzer', {'xmlns': 'http://openplanetsfoundation.org/ns/jpylyzer'})
I suppose the main problem is to decide which URI to choose (openplanets or kb ???)

While trying to build Debian packages for jpylyzer 1.5 on AWS the following problems occur (both for AMD 64 and i386) (copied from 'state.txt'):

Switched to branch 'master'
From git://github.com/openplanets/jpylyzer
   8d15734..cc1d25d  master     -> origin/master
 * [new tag]         1.5.0      -> 1.5.0
From git://github.com/openplanets/jpylyzer
 * [new tag]         1.1.5      -> 1.1.5
 * [new tag]         1.1.6      -> 1.1.6
 * [new tag]         1.2.0      -> 1.2.0
 * [new tag]         1.3.0      -> 1.3.0
 * [new tag]         1.4.0      -> 1.4.0
Updating 8d15734..cc1d25d
Fast-forward
 boxvalidator.py        |  358 ++-
 buildWin32.bat         |    1 +
 byteconv.py            |   24 +
 config.py              |    1 +
 debian/jpylyzer.pod    |    4 +
 etpatch.py             |    6 +-
 jpylyzer.py            |   68 +-
 jpylyzerUserManual.pdf |11787 +++++++++++++++++++++++++-----------------------
 8 files changed, 6455 insertions(+), 5794 deletions(-)
 create mode 100644 config.py
Note: checking out '1.5.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at cc1d25d... By default only report results of individual tests that failed, added --verbose switch which will force reporting of everything (second try)
--2012-04-10 08:42:40--  http://p2-registry.ecs.soton.ac.uk/opf/gh2ch/?repo=https://github.com/openplanets/jpylyzer&software=jpylyzer&distribution=stable&urgency=low
Resolving p2-registry.ecs.soton.ac.uk... 152.78.189.23
Connecting to p2-registry.ecs.soton.ac.uk|152.78.189.23|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: get_changelog.php?repo=https://github.com/openplanets/jpylyzer&software=jpylyzer&distribution=stable&urgency=low [following]
--2012-04-10 08:42:40--  http://p2-registry.ecs.soton.ac.uk/opf/gh2ch/get_changelog.php?repo=https://github.com/openplanets/jpylyzer&software=jpylyzer&distribution=stable&urgency=low
Reusing existing connection to p2-registry.ecs.soton.ac.uk:80.
HTTP request sent, awaiting response... 200 OK
Length: 7829 (7.6K) [text/plain]
Saving to: `debian/changelog'

     0K .......                                               100%  151M=0s

2012-04-10 08:42:45 (151 MB/s) - `debian/changelog' saved [7829/7829]

Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at /usr/share/perl5/Dpkg/BuildFlags.pm line 109.
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): 
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor): -Wl,-Bsymbolic-functions
dpkg-buildpackage: source package jpylyzer
dpkg-buildpackage: source version 1.5.0
dpkg-buildpackage: source changed by Johan van der Knijff <[email protected]>
 dpkg-source --before-build jpylyzer-1.5.0
dpkg-buildpackage: host architecture i386
 debian/rules clean
dh_testdir
dh_testroot
rm -f build-stamp
rm -f jpylyzer.1
make clean
make[1]: Entering directory `/home/ubuntu/jpylyzer-1.5.0'
rm -fR build
rm -fR dist
rm -f jpylyzer.spec
rm -f logdict2.7.2.final.0-1.log
rm -f logdict2.7.2.final.0-2.log
rm -f logdict2.7.2.final.0-3.log
rm -f warnjpylyzer.txt
make[1]: Leaving directory `/home/ubuntu/jpylyzer-1.5.0'
dh_clean
    rm -f debian/jpylyzer.substvars
    rm -f debian/jpylyzer.*.debhelper
    rm -rf debian/jpylyzer/
    rm -f debian/*.debhelper.log
    rm -f debian/files
    find .  \( \( -type f -a \
            \( -name '#*#' -o -name '.*~' -o -name '*~' -o -name DEADJOE \
         -o -name '*.orig' -o -name '*.rej' -o -name '*.bak' \
         -o -name '.*.orig' -o -name .*.rej -o -name '.SUMS' \
         -o -name TAGS -o \( -path '*/.deps/*' -a -name '*.P' \) \
        \) -exec rm -f {} \; \) -o \
        \( -type d -a -name autom4te.cache -prune -exec rm -rf {} \; \) \)
 dpkg-source -b jpylyzer-1.5.0
dpkg-source: warning: no source format specified in debian/source/format, see dpkg-source(1)
dpkg-source: info: using source format `1.0'
dpkg-source: info: building jpylyzer in jpylyzer_1.5.0.tar.gz
dpkg-source: info: building jpylyzer in jpylyzer_1.5.0.dsc
 debian/rules build
dh_testdir
dh_testdir
touch build-stamp
make build
make[1]: Entering directory `/home/ubuntu/jpylyzer-1.5.0'
pymakespec --onefile jpylyzer.py
make[1]: pymakespec: Command not found
make[1]: *** [build] Error 127
make[1]: Leaving directory `/home/ubuntu/jpylyzer-1.5.0'
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
cp: omitting directory `jpylyzer'
rm: cannot remove `jpylyzer': Is a directory
DONE

With Github's decision to drop support for separate file uploads we have no location for storing new versions of the Windows binaries and the documentation PDF.

Solution: include both in repo itself (these files are pretty small anyway).

Once this is done the links on the OPF jpylyzer page need to be updated as well.

If extracted XML contains end-of-line characters, they are removed in jpylyzer's output. This happens here (byteconv.py):

def removeControlCharacters(string):
    # Remove control characters from string
    # Source: http://stackoverflow.com/a/19016117/1209004
    return "".join(ch for ch in string if unicodedata.category(ch)[0]!="C")

I tried to fix this by changing the Unicode category code from C to Cc. However, in that case null characters aren't filtered out, even though they are part of the Cc category. See:

http://www.fileformat.info/info/unicode/category/Cc/list.htm

Might be a bug in unicodedata. For now I'll just leave it as it is.

missing-dependency-on-libc needed by usr/local/jpylyzer/_codecs_cn.so needed by 22 files

Jpylyzer loads the whole image into memory, which causes an exception if system memory is exceeded. No easy solution for this, as changing this behaviour would require fundamentally different way of parsing.

To do:

• Add exception handler so memory errors are handled gracefully
• Perhaps add output element that reports jpylyzer success/failure at analysing file

Following:

openpreserve/jpylyzer-test-files#1

Obviously full validation of ICC profiles is out of jpylyzer's scope, but maybe add test for cases like this where ICC profile is obviously damaged?

While running jpylyzer with --nullxml OPTION on this file:

http://sdowww.lmsal.com/sdomedia/hv_jp2kwrite/v0.8/jp2/AIA/2014/02/01/304/2014_02_01__00_11_07_13__SDO_AIA_AIA_304.jp2

Doing this with Python 2.7, some of the resulting XML elements are empty, even though in reality they do contain text (e.g. look at <HV_COMMENT> at the bottom). Using Python 3.3 it works correctly. So something goes wrong while parsing the XML. Could be a bug in ElementTree.

This also affects the Windows executables / Debian packages, since they are built using Python 2.7.