cjxl fails on "MotionPhoto" jpegs about libjxl HOT 1 CLOSED

More generally it would be good if we have a better way to treat "semantically meaningful" tail data when transcoding JPEG files. Besides these "MotionPhoto" cases, there is also UltraHDR (which has a gain map in the tail data) and MPF in general (Multi Picture Format) like some examples I've seen where the main image has fake bokeh applied to the background but the original image with a sharp background is still there in the tail data; also just a concatenation of JPEG files (which is something e.g. ffmpeg can ingest) could be seen as an example of this.

Currently tail data is stored in the jbrd and just treated like any other "unknown" data, like unknown APP markers or padding bits: it simply gets Brotli-compressed and in that way it becomes "hidden" in the JXL file, in the sense that the only way to retrieve such info is to apply the JPEG bitstream reconstruction procedure first.

It would be cleaner if we would define some kind of recursive structure in the file format where there is an optional box, name TBD but for the purpose of discussion we could call it tail, which contains the "tail data" of the main image. Then if the tail data is a movie, it could simply be copied into such a tail box, but if the tail data is another JPEG bitstream, then JXL's JPEG bitstream recompression could be applied recursively to that tail JPEG, and the tail box of the main image would then contain another JXL file. This could be nested arbitrarily deep, e.g. if you would concatenate three JPEGs then the third one would be in the tail box of the JXL file corresponding to the second one, which would be in the tail box of the first one.

Maybe instead of tail it could be called next or something. Such a structure would not allow seeking (it's a linked list, not a random access data structure), but for sequential access it would be fine. It would also make it straightforward to strip the tail data while still allowing reconstruction of the main image.

This method could also be used to extend the JXL file format to allow representing sequences of images that are coded completely independently (i.e. not as frames within a single JXL codestream), which can be useful since multi-frame single-codestream JXL has the limitation that all frames need to have the same dimensions and colorspace.

I am turning this issue into a discussion since this is not just a libjxl issue but changing anything here would require extending the spec (ISO/IEC 18181-2 in this case).

from libjxl.