Google Cloud Storage | CRC32c of large files. about java-storage HOT 3 CLOSED

I think, the way you upload files is the right way.
Because instances of Blob and BlobInfo classes are immutable objects, you cannot modify Crc32 value of an existing object. So, to enable data validation you need to provide CRC32hash before you open a WriteChannel. The hash calculated after WriteChannel is opened can't be used for validation.

To speed up crc32 hash calculating for your local files I would recommend you to increase the buffer size. 1k seems to be very small size. On my local machine resizing from 1k to 128k reduced hash calculation time from 50 seconds to 4. With the buffer of 1M size calculation took 3 seconds. Experimenting with the size will allow you to figure out the most optimal value.

I hope my answer will help.
Thanks

from java-storage.

I also noticed a bug in your code of CRC calculation:

Instead of:
hasher = hasher.putBytes(ByteBuffer.wrap(buffer, 0, limit-1));

should be:
hasher = hasher.putBytes(ByteBuffer.wrap(buffer, 0, limit));

Otherwise, the last byte in each chunk will be ignored.

from java-storage.

@Tiozi I hope the question is answered, if you need more details please provide more info.

from java-storage.