Arithmetic operation resulted in an overflow. about easycompressor HOT 6 CLOSED

Relevant issues:
MiloszKrajewski/K4os.Compression.LZ4#42
MiloszKrajewski/K4os.Compression.LZ4#43

from easycompressor.

@sgf @mjebrahimi

Basically this is a problem with the LZ4 itself and has nothing to do with EasyCompressor and I do not know the reason for this.

I disagree. There is no problem with LZ4 (in this respect), the problem is in your library.

I found that sometimes LZ4 throws an exception when decompressing large files.
In this case, depending on the file size, one of these errors may occur:

• OverflowException: Arithmetic operation resulted in an overflow
• ArgumentOutOfRangeException: Specified argument was out of the range of valid values

No, it does not. It throws in EasyCompressor.LZ4\LZ4Compressor.cs. To be very specific, in:

protected override byte[] BaseDecompress(byte[] compressedBytes)
{
    var target = new byte[compressedBytes.Length * 255].AsSpan();
    int decoded = LZ4Codec.Decode(compressedBytes, target);
    return target.Slice(0, decoded).ToArray();
}

you allocate compressed length * 255 bytes. This mean that ~8.1 MB (8421504 B to be exact) file is enough to allocate over 2GB of memory. Everything over it will cause excaption: ArgumentOutOfRangeException or OverflowException (depending on compiler settings, what fails first: multiplication arithmetic overflow or memory allocation).

...and the StackTrace of original report confirms that:

StackTrace:
at EasyCompressor.LZ4Compressor.BaseDecompress(Byte[] compressedBytes) in D:\a\EasyCompressor\EasyCompressor\src\EasyCompressor.LZ4\LZ4Compressor.cs:line 42
at EasyCompressor.BaseCompressor.Decompress(Byte[] compressedBytes) in D:\a\EasyCompressor\EasyCompressor\src\EasyCompressor\Compressor\BaseCompressor.cs:line 80
at ConsoleAppCompressTest.Program.Main(String[] args) in G:\Src\my\M3\Mir3Apps\ConsoleAppCompressTest\Program.cs:line 15

(the problem is in EasyCompressor.LZ4Compressor.BaseDecompress

Just try this snippet:

int x = 8421505 * 255;

from easycompressor.

So is it right to always use the stream method since the original source size is usually unknown at decompression time?

Not really. There are several ways to make size known. Like storing it? While compressing you can allocate 4 byes more and actually store length in 4 first bytes (or 4 last bytes if you wish).

Or you can use LZ4Pickler which was designed to do exactly that?

from easycompressor.

I found that sometimes LZ4 throws an exception when decompressing large files.
In this case, depending on the file size, one of these errors may occur:

• OverflowException: Arithmetic operation resulted in an overflow
• ArgumentOutOfRangeException: Specified argument was out of the range of valid values

Basically this is a problem with the LZ4 itself and has nothing to do with EasyCompressor and I do not know the reason for this.
I have opened this issue to follow this case.

The MiloszKrajewski/lz4net repo is no longer maintained and has been replaced by MiloszKrajewski/K4os.Compression.LZ4 and This LZ4Compressor just is a wrapper on it and nothing else.

from easycompressor.

@MiloszKrajewski OK, you're right.
So is it right to always use the stream method since the original source size is usually unknown at decompression time?

from easycompressor.

Great point.
So I refactored the code to store the actual size length (4 bytes) at first.
Thanks, @MiloszKrajewski.

from easycompressor.