DirectBuffer grows larger than available space. about 4mc HOT 2 CLOSED

In 4mc the LZ4 compression/decompression is bounded up to 4MB (4 * 1024 * 1024), which is an hard fixed limit.
When creating a compressor or decompressor the direct buffer is allocated twice (once for compressed data buffer, and for output uncompressed data buffer).
So each single decompressor istantiated will need no more than 2x 4MB direct buffers.

If you have some simple test case or directions to try and reproduce your issue, I can try.

from 4mc.

This should be anyway fixed now by the direct buffer caching introduced, working good both on cluster distributed jobs and on HDFS clients using 4mc codecs.

from 4mc.