Incorrect de-serialization leading to stream corruption in Big Endian systems about compress HOT 9 CLOSED

The root cause is that the LZF compression in BE systems carried out by UnsafeChunkEncoderBE class incorrectly computes the matching patterns in the stream, using word and double word sequences (methods _findMatchLength and _findLongMatchLength in class com/ning/compress/lzf/impl/UnsafeChunkEncoderBE.java). Specifically, the way the trailing null bytes are computed using the method _leadingBytes.

Endian difference does not exist for a word or a double word value residing in a register, so computing the trailing 0s for such a number is always obtained through Long.numberOfTrailingZeros, irrespective of the platform endian-ness. [ please refer to http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html#numberOfTrailingZeros%28long%29 to see that the API is endian neutral].

Please let me know if you need additional more information on this (including a draft proposal for fix.)

Thanks,
Gireesh.

from compress.

Ouch. Yes, this is a serious problem and has to be fixed.

from compress.

Quick note: no progress yet since I was out on vacation for past 3 weeks. Hoping to address this soon.

from compress.

Any update on this? were you able to make progress?
Please review: https://bugzilla.redhat.com/show_bug.cgi?id=1115264
The same issue reported in Fedora project, has been resolved, the patch is available in this link, which resolves the issue.

Please let me know if I can be of any assistance further.

from compress.

Unfortunately I have not had time to look into this due to other crises (not related to this project). But I hope to get back to it as soon as possible, since this is a critical problem. Thank you for the link, that should be helpful.

from compress.

@gireeshpunathil Sorry for the long delay; I finally got back and patched things as suggested. I also simplified handling a bit, without (I hope!) changing behavior itself. However, I don't have direct access to a big-endian system right now (will try to find one).
I did release 1.0.3, so if you can easily double-check tests, that'd be great. If not, I'll eventually find a way to verify it.

from compress.

Thanks, I will do the testing and will let you know the result.

from compress.

@cowtowncoder, I tested with your fix, and confirm that it fixes the reported problem. Thank you very much!

from compress.

@gireeshpunathil thank you for doing all the detective work here!

from compress.