UnsafeChunkDecoder in ning-compress-0.9.0 leads to SIGSEV in gc about compress HOT 14 CLOSED

If you could email me the test file, that would be great, since this probably depends on data. I have tested this with a few files, including jvm-compressor-benchmark files, but obviously not yet with enough exemplars. :)

from compress.

Ok I see the files, thanks!

from compress.

Quick question: I can't actually test the issue with the code and files -- but I was wondering if this is sporadic, as there's number of repetitions? If so, how often do you see failure?

from compress.

Btw, it is also possible to pass UnsafeChunkDecoder as the ChunkDecoder to LZFInputStream, to force use of particular one. But even with that tests pass for me on Mac... maybe I need to find a Ubuntu box.

from compress.

I pushed in elasticsearch an upgrade to 0.9 as well, and a user reported JVM crashing when running elasticsearch tests (I did not see it...). Here is more info: elastic/elasticsearch@4bbf298#commitcomment-714854.

from compress.

I think the issue is caused by a buffer overflow in the copyLong method. When I put the following assert here 3 tests start to fail.

assert(destIndex + 8 <= dest.length);

from compress.

That sounds plausible: there are checks in 'copyUpTo32' to avoid this problem, but not in 'copyLong'. I wish I could easily reproduce this on my system, but since I can add asserts perhaps I can figure out an efficient way to avoid this corner case.
It is interesting that this does not seem to affect all platforms; issues with 'copyUpTo32' were very obvious. Perhaps it also depends on memory allocation details.
Thank you for pointing this out -- it gives one lead, should be able to figure out possible patch.

from compress.

I think you should be able to reproduce this on your system. Just add the assert and run ning/compress tests. testIncrementalWithFullReads(com.ning.compress.lzf.TestLZFInputStream), testIncrementalWithMinimalReads(com.ning.compress.lzf.TestLZFInputStream) and testUnsafeCodec(com.ning.compress.lzf.TestLZFRoundTrip) fail every time I run tests on Snow Leopard (64 bit java 1.6.0_29) and Ubuntu (32 bit java 1.6.0_23).

from compress.

Ok: I added boundary check which should prevent write past end of allowed buffer. I would greatly appreciate if anyone with "succesfully crashing" platform could see if this takes care of the issue.

from compress.

@imotov: yes, I was able to verify that. What I mean that I can verify that this aspect gets properly handled, and hopefully that is enough. I just can not yet verify that this is the only issue, i.e. before declaring this solved I hope it can be independently verified.

from compress.

It fixed the issue for elasticsearch. Thanks!

from compress.

Ok good! I will do 0.9.1 build then. I hope Wolfgang can also verify issue on his end, I hope this can be closed.

from compress.

I pulled 0.9.1 and all tests pass. Congrats! Thanks for everyone for fixing this so fast.

from compress.

Great! Big thank you to everyone who helped.

from compress.